2024-09-05T16:25:39.677 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-05T16:25:39.694 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-05T16:25:39.778 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/skanta-2024-09-05_12:54:40-rados-wip-bharath4-testing-2024-09-02-0957-distro-default-smithi/7890779 branch: wip-bharath4-testing-2024-09-02-0957 description: rados/singleton/{all/ec-lost-unfound mon_election/classic msgr-failures/few msgr/async-v2only objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-hybrid} rados supported-random-distro$/{centos_latest}} email: skanta@redhat.com first_in_suite: false job_id: '7890779' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: skanta-2024-09-05_12:54:40-rados-wip-bharath4-testing-2024-09-02-0957-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-bharath4-testing-2024-09-02-0957 ceph: conf: global: mon client directed command retry: 5 mon election default strategy: 1 ms bind msgr1: false ms bind msgr2: true ms inject socket failures: 5000 ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon scrub interval: 300 osd: bluefs allocator: hybrid bluestore allocator: hybrid bluestore block size: 96636764160 bluestore fsck on mount: true bluestore write v2 random: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - \(MON_DOWN\) sha1: ac9df2d66a4c8e4e8634a3fe39168f4f4683bc03 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: ac9df2d66a4c8e4e8634a3fe39168f4f4683bc03 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-bharath4-testing-2024-09-02-0957 sha1: ac9df2d66a4c8e4e8634a3fe39168f4f4683bc03 owner: scheduled_skanta@teuthology priority: 99 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mon.b - mon.c - mgr.x - osd.0 - osd.1 - osd.2 - osd.3 seed: 5171 sha1: ac9df2d66a4c8e4e8634a3fe39168f4f4683bc03 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-bharath4-testing-2024-09-02-0957 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_ac9df2d66a4c8e4e8634a3fe39168f4f4683bc03/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: ac9df2d66a4c8e4e8634a3fe39168f4f4683bc03 targets: smithi087.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEBEONroSHW8yf0u1X9DR7rduLLOszGC4pmixy6wxEg7QaYnz4xpcLgxvC7B4TbUV5WQw5IYbxwGn5Dsq0/QvF8= tasks: - install: null - ceph: create_rbd_pool: false log-ignorelist: - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(OBJECT_ - \(SLOW_OPS\) - slow request - \(POOL_APP_NOT_ENABLED\) pre-mgr-commands: - sudo ceph config set mgr mgr_pool false --force - ec_lost_unfound: null teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-05_12:54:40 tube: smithi user: skanta verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-05T16:25:39.778 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_ac9df2d66a4c8e4e8634a3fe39168f4f4683bc03/qa; will attempt to use it 2024-09-05T16:25:39.780 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_ac9df2d66a4c8e4e8634a3fe39168f4f4683bc03/qa/tasks 2024-09-05T16:25:39.780 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-05T16:25:39.782 INFO:teuthology.task.internal:Checking packages... 2024-09-05T16:25:39.836 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'ac9df2d66a4c8e4e8634a3fe39168f4f4683bc03' 2024-09-05T16:25:39.837 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-05T16:25:39.837 INFO:teuthology.packaging:ref: None 2024-09-05T16:25:39.837 INFO:teuthology.packaging:tag: None 2024-09-05T16:25:39.837 INFO:teuthology.packaging:branch: wip-bharath4-testing-2024-09-02-0957 2024-09-05T16:25:39.837 INFO:teuthology.packaging:sha1: ac9df2d66a4c8e4e8634a3fe39168f4f4683bc03 2024-09-05T16:25:39.837 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-bharath4-testing-2024-09-02-0957 2024-09-05T16:25:39.981 INFO:teuthology.task.internal:Found packages for ceph version 19.3.0-4650.gac9df2d6 2024-09-05T16:25:39.983 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-05T16:25:39.989 INFO:teuthology.task.internal:no buildpackages task found 2024-09-05T16:25:39.989 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-05T16:25:40.119 INFO:teuthology.task.internal:Saving configuration 2024-09-05T16:25:40.137 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-05T16:25:40.169 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-05T16:25:40.189 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi087.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/skanta-2024-09-05_12:54:40-rados-wip-bharath4-testing-2024-09-02-0957-distro-default-smithi/7890779', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-05 16:20:45.164678', 'locked_by': 'scheduled_skanta@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEBEONroSHW8yf0u1X9DR7rduLLOszGC4pmixy6wxEg7QaYnz4xpcLgxvC7B4TbUV5WQw5IYbxwGn5Dsq0/QvF8='} 2024-09-05T16:25:40.189 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-05T16:25:40.249 INFO:teuthology.task.internal:roles: ubuntu@smithi087.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2024-09-05T16:25:40.249 INFO:teuthology.run_tasks:Running task console_log... 2024-09-05T16:25:40.325 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f5de13ffbe0>, signals=[15]) 2024-09-05T16:25:40.325 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-05T16:25:40.349 INFO:teuthology.task.internal:Opening connections... 2024-09-05T16:25:40.349 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi087.front.sepia.ceph.com 2024-09-05T16:25:40.350 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi087.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-05T16:25:40.435 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-05T16:25:40.442 DEBUG:teuthology.orchestra.run.smithi087:> uname -m 2024-09-05T16:25:40.514 INFO:teuthology.orchestra.run.smithi087.stdout:x86_64 2024-09-05T16:25:40.528 DEBUG:teuthology.orchestra.run.smithi087:> cat /etc/os-release 2024-09-05T16:25:40.572 INFO:teuthology.orchestra.run.smithi087.stdout:NAME="CentOS Stream" 2024-09-05T16:25:40.572 INFO:teuthology.orchestra.run.smithi087.stdout:VERSION="9" 2024-09-05T16:25:40.572 INFO:teuthology.orchestra.run.smithi087.stdout:ID="centos" 2024-09-05T16:25:40.572 INFO:teuthology.orchestra.run.smithi087.stdout:ID_LIKE="rhel fedora" 2024-09-05T16:25:40.572 INFO:teuthology.orchestra.run.smithi087.stdout:VERSION_ID="9" 2024-09-05T16:25:40.572 INFO:teuthology.orchestra.run.smithi087.stdout:PLATFORM_ID="platform:el9" 2024-09-05T16:25:40.572 INFO:teuthology.orchestra.run.smithi087.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-05T16:25:40.573 INFO:teuthology.orchestra.run.smithi087.stdout:ANSI_COLOR="0;31" 2024-09-05T16:25:40.573 INFO:teuthology.orchestra.run.smithi087.stdout:LOGO="fedora-logo-icon" 2024-09-05T16:25:40.573 INFO:teuthology.orchestra.run.smithi087.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-05T16:25:40.573 INFO:teuthology.orchestra.run.smithi087.stdout:HOME_URL="https://centos.org/" 2024-09-05T16:25:40.573 INFO:teuthology.orchestra.run.smithi087.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-05T16:25:40.573 INFO:teuthology.orchestra.run.smithi087.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-05T16:25:40.573 INFO:teuthology.orchestra.run.smithi087.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-05T16:25:40.574 INFO:teuthology.lock.ops:Updating smithi087.front.sepia.ceph.com on lock server 2024-09-05T16:25:40.595 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-05T16:25:40.605 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-05T16:25:40.691 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-05T16:25:40.691 DEBUG:teuthology.orchestra.run.smithi087:> test '!' -e /home/ubuntu/cephtest 2024-09-05T16:25:40.709 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-05T16:25:40.774 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-05T16:25:40.774 DEBUG:teuthology.orchestra.run.smithi087:> test -z $(ls -A /var/lib/ceph) 2024-09-05T16:25:40.802 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-05T16:25:40.878 INFO:teuthology.run_tasks:Running task kernel... 2024-09-05T16:25:40.967 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-05T16:25:40.967 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-05T16:25:40.967 DEBUG:teuthology.orchestra.run.smithi087:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-05T16:25:40.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-05T16:25:40.985 DEBUG:teuthology.orchestra.run.smithi087:> uname -r 2024-09-05T16:25:41.044 INFO:teuthology.orchestra.run.smithi087.stdout:5.14.0-503.el9.x86_64 2024-09-05T16:25:41.045 INFO:teuthology.task.kernel:Running kernel on smithi087: 5.14.0-503.el9.x86_64 2024-09-05T16:25:41.045 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum install -y kernel 2024-09-05T16:25:48.037 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS Stream 9 - BaseOS 1.6 MB/s | 8.2 MB 00:05 2024-09-05T16:26:02.170 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS Stream 9 - AppStream 1.6 MB/s | 20 MB 00:13 2024-09-05T16:26:07.329 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS Stream 9 - CRB 4.7 MB/s | 6.5 MB 00:01 2024-09-05T16:26:09.779 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS Stream 9 - Extras packages 17 kB/s | 19 kB 00:01 2024-09-05T16:26:10.541 INFO:teuthology.orchestra.run.smithi087.stdout:Extra Packages for Enterprise Linux 36 MB/s | 23 MB 00:00 2024-09-05T16:26:15.905 INFO:teuthology.orchestra.run.smithi087.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-09-05T16:26:17.854 INFO:teuthology.orchestra.run.smithi087.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-05T16:26:17.854 INFO:teuthology.orchestra.run.smithi087.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-05T16:26:17.945 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-09-05T16:26:17.946 INFO:teuthology.orchestra.run.smithi087.stdout:Nothing to do. 2024-09-05T16:26:17.947 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-09-05T16:26:18.057 DEBUG:teuthology.orchestra.run.smithi087:> echo no | sudo yum reinstall kernel || true 2024-09-05T16:26:18.593 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:00:03 ago on Thu 05 Sep 2024 04:26:15 PM UTC. 2024-09-05T16:26:18.724 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-09-05T16:26:18.732 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-05T16:26:18.732 INFO:teuthology.orchestra.run.smithi087.stdout: Package Architecture Version Repository Size 2024-09-05T16:26:18.732 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-05T16:26:18.732 INFO:teuthology.orchestra.run.smithi087.stdout:Reinstalling: 2024-09-05T16:26:18.732 INFO:teuthology.orchestra.run.smithi087.stdout: kernel x86_64 5.14.0-503.el9 baseos 1.9 M 2024-09-05T16:26:18.732 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:26:18.732 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-09-05T16:26:18.733 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-05T16:26:18.733 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:26:18.733 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 1.9 M 2024-09-05T16:26:18.733 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 0 2024-09-05T16:26:18.733 INFO:teuthology.orchestra.run.smithi087.stderr:Operation aborted. 2024-09-05T16:26:18.789 INFO:teuthology.orchestra.run.smithi087.stdout:Is this ok [y/N]: 2024-09-05T16:26:18.790 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum reinstall -y kernel || true 2024-09-05T16:26:19.323 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:00:04 ago on Thu 05 Sep 2024 04:26:15 PM UTC. 2024-09-05T16:26:19.447 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-09-05T16:26:19.448 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-05T16:26:19.448 INFO:teuthology.orchestra.run.smithi087.stdout: Package Architecture Version Repository Size 2024-09-05T16:26:19.448 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-05T16:26:19.448 INFO:teuthology.orchestra.run.smithi087.stdout:Reinstalling: 2024-09-05T16:26:19.448 INFO:teuthology.orchestra.run.smithi087.stdout: kernel x86_64 5.14.0-503.el9 baseos 1.9 M 2024-09-05T16:26:19.448 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:26:19.449 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-09-05T16:26:19.449 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-05T16:26:19.449 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:26:19.449 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 1.9 M 2024-09-05T16:26:19.449 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 0 2024-09-05T16:26:19.449 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-09-05T16:26:20.549 INFO:teuthology.orchestra.run.smithi087.stdout:kernel-5.14.0-503.el9.x86_64.rpm 1.8 MB/s | 1.9 MB 00:01 2024-09-05T16:26:20.550 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-09-05T16:26:20.550 INFO:teuthology.orchestra.run.smithi087.stdout:Total 1.7 MB/s | 1.9 MB 00:01 2024-09-05T16:26:20.581 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-09-05T16:26:20.815 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-09-05T16:26:20.816 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-09-05T16:26:20.846 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-09-05T16:26:20.846 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-09-05T16:26:21.009 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-09-05T16:26:21.143 INFO:teuthology.orchestra.run.smithi087.stdout: Reinstalling : kernel-5.14.0-503.el9.x86_64 1/2 2024-09-05T16:26:21.143 INFO:teuthology.orchestra.run.smithi087.stdout: Cleanup : kernel-5.14.0-503.el9.x86_64 2/2 2024-09-05T16:26:21.811 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: kernel-5.14.0-503.el9.x86_64 2/2 2024-09-05T16:26:21.812 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : kernel-5.14.0-503.el9.x86_64 1/2 2024-09-05T16:26:22.510 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : kernel-5.14.0-503.el9.x86_64 2/2 2024-09-05T16:26:22.510 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:26:22.510 INFO:teuthology.orchestra.run.smithi087.stdout:Reinstalled: 2024-09-05T16:26:22.510 INFO:teuthology.orchestra.run.smithi087.stdout: kernel-5.14.0-503.el9.x86_64 2024-09-05T16:26:22.510 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:26:22.510 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-09-05T16:26:22.687 DEBUG:teuthology.orchestra.run.smithi087:> rpm -q kernel | sort -rV | head -n 1 2024-09-05T16:26:22.746 INFO:teuthology.orchestra.run.smithi087.stdout:kernel-5.14.0-503.el9.x86_64 2024-09-05T16:26:22.747 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-503.el9.x86_64 2024-09-05T16:26:22.747 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-09-05T16:26:22.747 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-05T16:26:22.748 DEBUG:teuthology.orchestra.run.smithi087:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-05T16:26:22.781 INFO:teuthology.orchestra.run.smithi087.stdout:ttyS1 2024-09-05T16:26:22.807 DEBUG:teuthology.parallel:result is None 2024-09-05T16:26:22.807 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-05T16:26:22.818 INFO:teuthology.task.internal:Creating test directory... 2024-09-05T16:26:22.819 DEBUG:teuthology.orchestra.run.smithi087:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-05T16:26:22.861 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-05T16:26:22.868 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-05T16:26:22.961 INFO:teuthology.task.internal:Creating archive directory... 2024-09-05T16:26:22.961 DEBUG:teuthology.orchestra.run.smithi087:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-05T16:26:22.987 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-05T16:26:23.062 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-05T16:26:23.065 DEBUG:teuthology.orchestra.run.smithi087:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-05T16:26:23.112 INFO:teuthology.orchestra.run.smithi087.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-05T16:26:23.122 INFO:teuthology.orchestra.run.smithi087.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-05T16:26:23.124 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-05T16:26:23.164 INFO:teuthology.task.internal:Configuring sudo... 2024-09-05T16:26:23.164 DEBUG:teuthology.orchestra.run.smithi087:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-05T16:26:23.190 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-05T16:26:23.270 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-05T16:26:23.271 DEBUG:teuthology.orchestra.run.smithi087:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-05T16:26:23.286 DEBUG:teuthology.orchestra.run.smithi087:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-05T16:26:23.370 DEBUG:teuthology.orchestra.run.smithi087:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-05T16:26:23.417 DEBUG:teuthology.orchestra.run.smithi087:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-05T16:26:23.500 DEBUG:teuthology.orchestra.run.smithi087:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-05T16:26:23.539 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-09-05T16:26:23.540 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-05T16:26:23.604 DEBUG:teuthology.orchestra.run.smithi087:> sudo service rsyslog restart 2024-09-05T16:26:23.677 INFO:teuthology.orchestra.run.smithi087.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-05T16:26:24.193 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-05T16:26:24.213 INFO:teuthology.task.internal:Starting timer... 2024-09-05T16:26:24.213 INFO:teuthology.run_tasks:Running task pcp... 2024-09-05T16:26:24.315 INFO:teuthology.run_tasks:Running task selinux... 2024-09-05T16:26:24.378 DEBUG:teuthology.orchestra.run.smithi087:> sudo service auditd rotate 2024-09-05T16:26:24.416 INFO:teuthology.orchestra.run.smithi087.stdout:Rotating logs: 2024-09-05T16:26:24.418 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-05T16:26:24.418 DEBUG:teuthology.orchestra.run.smithi087:> /usr/sbin/getenforce 2024-09-05T16:26:24.490 INFO:teuthology.orchestra.run.smithi087.stdout:Permissive 2024-09-05T16:26:24.491 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi087.front.sepia.ceph.com': 'permissive'} 2024-09-05T16:26:24.491 DEBUG:teuthology.orchestra.run.smithi087:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-05T16:26:24.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-05T16:26:24.557 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-05T16:26:24.557 DEBUG:teuthology.orchestra.run.smithi087:> sudo /usr/sbin/setenforce permissive 2024-09-05T16:26:24.626 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-05T16:26:24.674 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-09-05T16:26:24.690 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-09-05T16:26:26.127 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-05T16:26:26.588 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-05T16:26:26.593 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi087.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-05T16:30:31.547 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi087.front.sepia.ceph.com')] 2024-09-05T16:30:31.571 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi087.front.sepia.ceph.com' 2024-09-05T16:30:31.573 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi087.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-05T16:30:31.649 DEBUG:teuthology.orchestra.run.smithi087:> true 2024-09-05T16:30:31.723 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi087.front.sepia.ceph.com' 2024-09-05T16:30:31.724 INFO:teuthology.run_tasks:Running task clock... 2024-09-05T16:30:31.737 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-05T16:30:31.737 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-05T16:30:31.738 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-05T16:30:31.793 INFO:teuthology.orchestra.run.smithi087.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-05T16:30:31.807 INFO:teuthology.orchestra.run.smithi087.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-05T16:30:31.831 INFO:teuthology.orchestra.run.smithi087.stderr:sudo: ntpd: command not found 2024-09-05T16:30:31.842 INFO:teuthology.orchestra.run.smithi087.stdout:506 Cannot talk to daemon 2024-09-05T16:30:31.856 INFO:teuthology.orchestra.run.smithi087.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-05T16:30:31.870 INFO:teuthology.orchestra.run.smithi087.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-05T16:30:31.934 INFO:teuthology.orchestra.run.smithi087.stderr:bash: line 1: ntpq: command not found 2024-09-05T16:30:31.937 INFO:teuthology.orchestra.run.smithi087.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-05T16:30:31.937 INFO:teuthology.orchestra.run.smithi087.stdout:=============================================================================== 2024-09-05T16:30:31.937 INFO:teuthology.orchestra.run.smithi087.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-05T16:30:31.937 INFO:teuthology.orchestra.run.smithi087.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-05T16:30:31.937 INFO:teuthology.orchestra.run.smithi087.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-05T16:30:31.937 INFO:teuthology.orchestra.run.smithi087.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-05T16:30:31.938 INFO:teuthology.run_tasks:Running task install... 2024-09-05T16:30:31.946 DEBUG:teuthology.task.install:project ceph 2024-09-05T16:30:31.947 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'ac9df2d66a4c8e4e8634a3fe39168f4f4683bc03'}} 2024-09-05T16:30:31.947 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'ac9df2d66a4c8e4e8634a3fe39168f4f4683bc03'} 2024-09-05T16:30:31.947 INFO:teuthology.task.install:Using flavor: default 2024-09-05T16:30:31.952 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-05T16:30:31.952 INFO:teuthology.task.install:extra packages: [] 2024-09-05T16:30:31.953 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'ac9df2d66a4c8e4e8634a3fe39168f4f4683bc03', 'tag': None, 'wait_for_package': False} 2024-09-05T16:30:31.953 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=ac9df2d66a4c8e4e8634a3fe39168f4f4683bc03 2024-09-05T16:30:32.101 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-bharath4-testing-2024-09-02-0957/ac9df2d66a4c8e4e8634a3fe39168f4f4683bc03/centos/9/flavors/default/ 2024-09-05T16:30:32.101 INFO:teuthology.task.install.rpm:Package version is 19.3.0-4650.gac9df2d6 2024-09-05T16:30:32.211 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-bharath4-testing-2024-09-02-0957/ac9df2d66a4c8e4e8634a3fe39168f4f4683bc03/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-bharath4-testing-2024-09-02-0957/ac9df2d66a4c8e4e8634a3fe39168f4f4683bc03/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-bharath4-testing-2024-09-02-0957/ac9df2d66a4c8e4e8634a3fe39168f4f4683bc03/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-05T16:30:32.211 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-09-05T16:30:32.212 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-05T16:30:32.238 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-05T16:30:32.238 DEBUG:teuthology.orchestra.run.smithi087:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/ac9df2d66a4c8e4e8634a3fe39168f4f4683bc03/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-05T16:30:32.304 DEBUG:teuthology.orchestra.run.smithi087:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-05T16:30:32.379 DEBUG:teuthology.orchestra.run.smithi087:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-05T16:30:32.443 INFO:teuthology.orchestra.run.smithi087.stdout:check_obsoletes = 1 2024-09-05T16:30:32.445 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum clean all 2024-09-05T16:30:32.669 INFO:teuthology.orchestra.run.smithi087.stdout:45 files removed 2024-09-05T16:30:32.695 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-radosgw 2024-09-05T16:30:33.241 INFO:teuthology.orchestra.run.smithi087.stdout:ceph packages for x86_64 306 kB/s | 84 kB 00:00 2024-09-05T16:30:33.536 INFO:teuthology.orchestra.run.smithi087.stdout:ceph noarch packages 97 kB/s | 24 kB 00:00 2024-09-05T16:30:33.826 INFO:teuthology.orchestra.run.smithi087.stdout:ceph source packages 7.5 kB/s | 1.9 kB 00:00 2024-09-05T16:30:34.450 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS Stream 9 - BaseOS 14 MB/s | 8.2 MB 00:00 2024-09-05T16:30:37.868 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS Stream 9 - AppStream 8.8 MB/s | 20 MB 00:02 2024-09-05T16:30:44.336 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS Stream 9 - CRB 2.4 MB/s | 6.5 MB 00:02 2024-09-05T16:30:45.745 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS Stream 9 - Extras packages 91 kB/s | 19 kB 00:00 2024-09-05T16:30:46.515 INFO:teuthology.orchestra.run.smithi087.stdout:Extra Packages for Enterprise Linux 35 MB/s | 23 MB 00:00 2024-09-05T16:30:51.929 INFO:teuthology.orchestra.run.smithi087.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-09-05T16:30:54.147 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-09-05T16:30:54.149 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-05T16:30:54.149 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repository Size 2024-09-05T16:30:54.149 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-05T16:30:54.149 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-09-05T16:30:54.149 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-radosgw x86_64 2:19.3.0-4650.gac9df2d6.el9 ceph 11 M 2024-09-05T16:30:54.149 INFO:teuthology.orchestra.run.smithi087.stdout:Upgrading: 2024-09-05T16:30:54.150 INFO:teuthology.orchestra.run.smithi087.stdout: librados2 x86_64 2:19.3.0-4650.gac9df2d6.el9 ceph 3.5 M 2024-09-05T16:30:54.150 INFO:teuthology.orchestra.run.smithi087.stdout: librbd1 x86_64 2:19.3.0-4650.gac9df2d6.el9 ceph 3.2 M 2024-09-05T16:30:54.150 INFO:teuthology.orchestra.run.smithi087.stdout:Installing dependencies: 2024-09-05T16:30:54.150 INFO:teuthology.orchestra.run.smithi087.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-05T16:30:54.150 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-base x86_64 2:19.3.0-4650.gac9df2d6.el9 ceph 5.5 M 2024-09-05T16:30:54.150 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-common x86_64 2:19.3.0-4650.gac9df2d6.el9 ceph 22 M 2024-09-05T16:30:54.150 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-selinux x86_64 2:19.3.0-4650.gac9df2d6.el9 ceph 25 k 2024-09-05T16:30:54.150 INFO:teuthology.orchestra.run.smithi087.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-09-05T16:30:54.150 INFO:teuthology.orchestra.run.smithi087.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-05T16:30:54.150 INFO:teuthology.orchestra.run.smithi087.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-05T16:30:54.150 INFO:teuthology.orchestra.run.smithi087.stdout: libcephfs2 x86_64 2:19.3.0-4650.gac9df2d6.el9 ceph 755 k 2024-09-05T16:30:54.150 INFO:teuthology.orchestra.run.smithi087.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-05T16:30:54.150 INFO:teuthology.orchestra.run.smithi087.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-05T16:30:54.151 INFO:teuthology.orchestra.run.smithi087.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-05T16:30:54.151 INFO:teuthology.orchestra.run.smithi087.stdout: libradosstriper1 x86_64 2:19.3.0-4650.gac9df2d6.el9 ceph 509 k 2024-09-05T16:30:54.151 INFO:teuthology.orchestra.run.smithi087.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-05T16:30:54.151 INFO:teuthology.orchestra.run.smithi087.stdout: librgw2 x86_64 2:19.3.0-4650.gac9df2d6.el9 ceph 5.5 M 2024-09-05T16:30:54.151 INFO:teuthology.orchestra.run.smithi087.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-05T16:30:54.151 INFO:teuthology.orchestra.run.smithi087.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-05T16:30:54.151 INFO:teuthology.orchestra.run.smithi087.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-05T16:30:54.151 INFO:teuthology.orchestra.run.smithi087.stdout: python3-ceph-argparse x86_64 2:19.3.0-4650.gac9df2d6.el9 ceph 45 k 2024-09-05T16:30:54.151 INFO:teuthology.orchestra.run.smithi087.stdout: python3-ceph-common x86_64 2:19.3.0-4650.gac9df2d6.el9 ceph 143 k 2024-09-05T16:30:54.151 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cephfs x86_64 2:19.3.0-4650.gac9df2d6.el9 ceph 161 k 2024-09-05T16:30:54.151 INFO:teuthology.orchestra.run.smithi087.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-05T16:30:54.151 INFO:teuthology.orchestra.run.smithi087.stdout: python3-rados x86_64 2:19.3.0-4650.gac9df2d6.el9 ceph 320 k 2024-09-05T16:30:54.151 INFO:teuthology.orchestra.run.smithi087.stdout: python3-rbd x86_64 2:19.3.0-4650.gac9df2d6.el9 ceph 306 k 2024-09-05T16:30:54.152 INFO:teuthology.orchestra.run.smithi087.stdout: python3-rgw x86_64 2:19.3.0-4650.gac9df2d6.el9 ceph 99 k 2024-09-05T16:30:54.152 INFO:teuthology.orchestra.run.smithi087.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-05T16:30:54.152 INFO:teuthology.orchestra.run.smithi087.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-05T16:30:54.152 INFO:teuthology.orchestra.run.smithi087.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-05T16:30:54.152 INFO:teuthology.orchestra.run.smithi087.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-05T16:30:54.152 INFO:teuthology.orchestra.run.smithi087.stdout:Installing weak dependencies: 2024-09-05T16:30:54.152 INFO:teuthology.orchestra.run.smithi087.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-05T16:30:54.152 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:30:54.152 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-09-05T16:30:54.152 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-05T16:30:54.152 INFO:teuthology.orchestra.run.smithi087.stdout:Install 30 Packages 2024-09-05T16:30:54.153 INFO:teuthology.orchestra.run.smithi087.stdout:Upgrade 2 Packages 2024-09-05T16:30:54.153 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:30:54.153 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 62 M 2024-09-05T16:30:54.153 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-09-05T16:30:54.747 INFO:teuthology.orchestra.run.smithi087.stdout:(1/32): ceph-base-19.3.0-4650.gac9df2d6.el9.x86 11 MB/s | 5.5 MB 00:00 2024-09-05T16:30:54.797 INFO:teuthology.orchestra.run.smithi087.stdout:(2/32): ceph-selinux-19.3.0-4650.gac9df2d6.el9. 504 kB/s | 25 kB 00:00 2024-09-05T16:30:54.914 INFO:teuthology.orchestra.run.smithi087.stdout:(3/32): libcephfs2-19.3.0-4650.gac9df2d6.el9.x8 6.3 MB/s | 755 kB 00:00 2024-09-05T16:30:54.981 INFO:teuthology.orchestra.run.smithi087.stdout:(4/32): libradosstriper1-19.3.0-4650.gac9df2d6. 7.5 MB/s | 509 kB 00:00 2024-09-05T16:30:55.090 INFO:teuthology.orchestra.run.smithi087.stdout:(5/32): ceph-radosgw-19.3.0-4650.gac9df2d6.el9. 13 MB/s | 11 MB 00:00 2024-09-05T16:30:55.148 INFO:teuthology.orchestra.run.smithi087.stdout:(6/32): python3-ceph-argparse-19.3.0-4650.gac9d 776 kB/s | 45 kB 00:00 2024-09-05T16:30:55.207 INFO:teuthology.orchestra.run.smithi087.stdout:(7/32): python3-ceph-common-19.3.0-4650.gac9df2 2.4 MB/s | 143 kB 00:00 2024-09-05T16:30:55.274 INFO:teuthology.orchestra.run.smithi087.stdout:(8/32): python3-cephfs-19.3.0-4650.gac9df2d6.el 2.4 MB/s | 161 kB 00:00 2024-09-05T16:30:55.357 INFO:teuthology.orchestra.run.smithi087.stdout:(9/32): librgw2-19.3.0-4650.gac9df2d6.el9.x86_6 15 MB/s | 5.5 MB 00:00 2024-09-05T16:30:55.383 INFO:teuthology.orchestra.run.smithi087.stdout:(10/32): python3-rados-19.3.0-4650.gac9df2d6.el 2.9 MB/s | 320 kB 00:00 2024-09-05T16:30:55.408 INFO:teuthology.orchestra.run.smithi087.stdout:(11/32): python3-rbd-19.3.0-4650.gac9df2d6.el9. 6.0 MB/s | 306 kB 00:00 2024-09-05T16:30:55.433 INFO:teuthology.orchestra.run.smithi087.stdout:(12/32): python3-rgw-19.3.0-4650.gac9df2d6.el9. 1.9 MB/s | 99 kB 00:00 2024-09-05T16:30:55.634 INFO:teuthology.orchestra.run.smithi087.stdout:(13/32): ceph-common-19.3.0-4650.gac9df2d6.el9. 15 MB/s | 22 MB 00:01 2024-09-05T16:30:55.725 INFO:teuthology.orchestra.run.smithi087.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 496 kB/s | 45 kB 00:00 2024-09-05T16:30:55.751 INFO:teuthology.orchestra.run.smithi087.stdout:(15/32): boost-program-options-1.75.0-8.el9.x86 312 kB/s | 107 kB 00:00 2024-09-05T16:30:55.776 INFO:teuthology.orchestra.run.smithi087.stdout:(16/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 468 kB/s | 160 kB 00:00 2024-09-05T16:30:55.818 INFO:teuthology.orchestra.run.smithi087.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 1.0 MB/s | 42 kB 00:00 2024-09-05T16:30:55.851 INFO:teuthology.orchestra.run.smithi087.stdout:(18/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.8 MB/s | 292 kB 00:00 2024-09-05T16:30:55.877 INFO:teuthology.orchestra.run.smithi087.stdout:(19/32): librdkafka-1.6.1-102.el9.x86_64.rpm 4.3 MB/s | 662 kB 00:00 2024-09-05T16:30:55.902 INFO:teuthology.orchestra.run.smithi087.stdout:(20/32): qatlib-24.02.0-1.el9.x86_64.rpm 2.6 MB/s | 221 kB 00:00 2024-09-05T16:30:55.927 INFO:teuthology.orchestra.run.smithi087.stdout:(21/32): qatlib-service-24.02.0-1.el9.x86_64.rp 477 kB/s | 36 kB 00:00 2024-09-05T16:30:55.952 INFO:teuthology.orchestra.run.smithi087.stdout:(22/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 629 kB/s | 47 kB 00:00 2024-09-05T16:30:56.011 INFO:teuthology.orchestra.run.smithi087.stdout:(23/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 3.6 MB/s | 309 kB 00:00 2024-09-05T16:30:56.044 INFO:teuthology.orchestra.run.smithi087.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 1.5 MB/s | 49 kB 00:00 2024-09-05T16:30:56.070 INFO:teuthology.orchestra.run.smithi087.stdout:(25/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 222 kB/s | 26 kB 00:00 2024-09-05T16:30:56.095 INFO:teuthology.orchestra.run.smithi087.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-09-05T16:30:56.128 INFO:teuthology.orchestra.run.smithi087.stdout:(27/32): re2-20211101-20.el9.x86_64.rpm 5.7 MB/s | 191 kB 00:00 2024-09-05T16:30:56.195 INFO:teuthology.orchestra.run.smithi087.stdout:(28/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 6.5 MB/s | 839 kB 00:00 2024-09-05T16:30:56.237 INFO:teuthology.orchestra.run.smithi087.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-09-05T16:30:56.313 INFO:teuthology.orchestra.run.smithi087.stdout:(30/32): libarrow-9.0.0-11.el9.x86_64.rpm 11 MB/s | 4.4 MB 00:00 2024-09-05T16:30:56.463 INFO:teuthology.orchestra.run.smithi087.stdout:(31/32): librados2-19.3.0-4650.gac9df2d6.el9.x8 13 MB/s | 3.5 MB 00:00 2024-09-05T16:30:56.513 INFO:teuthology.orchestra.run.smithi087.stdout:(32/32): librbd1-19.3.0-4650.gac9df2d6.el9.x86_ 12 MB/s | 3.2 MB 00:00 2024-09-05T16:30:56.519 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-09-05T16:30:56.519 INFO:teuthology.orchestra.run.smithi087.stdout:Total 26 MB/s | 62 MB 00:02 2024-09-05T16:30:56.623 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-09-05T16:30:56.658 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-09-05T16:30:56.659 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-09-05T16:30:57.133 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-09-05T16:30:57.133 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-09-05T16:30:58.591 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-09-05T16:30:58.672 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-09-05T16:30:58.709 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-09-05T16:30:58.735 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-09-05T16:30:58.757 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-05T16:30:58.782 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-05T16:30:58.976 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-05T16:30:59.028 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-05T16:30:59.386 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-09-05T16:30:59.536 INFO:teuthology.orchestra.run.smithi087.stdout: Upgrading : librados2-2:19.3.0-4650.gac9df2d6.el9.x86_64 7/34 2024-09-05T16:30:59.628 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: librados2-2:19.3.0-4650.gac9df2d6.el9.x86_64 7/34 2024-09-05T16:30:59.718 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-rados-2:19.3.0-4650.gac9df2d6.el9.x86_64 8/34 2024-09-05T16:30:59.754 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libcephfs2-2:19.3.0-4650.gac9df2d6.el9.x86_64 9/34 2024-09-05T16:30:59.817 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: libcephfs2-2:19.3.0-4650.gac9df2d6.el9.x86_64 9/34 2024-09-05T16:30:59.854 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-09-05T16:30:59.898 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-ceph-argparse-2:19.3.0-4650.gac9df2d6.el9. 11/34 2024-09-05T16:30:59.968 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-cephfs-2:19.3.0-4650.gac9df2d6.el9.x86_64 12/34 2024-09-05T16:30:59.990 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libradosstriper1-2:19.3.0-4650.gac9df2d6.el9.x86_6 13/34 2024-09-05T16:31:00.054 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: libradosstriper1-2:19.3.0-4650.gac9df2d6.el9.x86_6 13/34 2024-09-05T16:31:00.079 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-09-05T16:31:00.190 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/34 2024-09-05T16:31:00.222 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-09-05T16:31:00.254 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-09-05T16:31:00.279 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-09-05T16:31:00.308 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-09-05T16:31:00.363 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-09-05T16:31:00.679 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-09-05T16:31:00.727 INFO:teuthology.orchestra.run.smithi087.stdout: Upgrading : librbd1-2:19.3.0-4650.gac9df2d6.el9.x86_64 22/34 2024-09-05T16:31:00.801 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: librbd1-2:19.3.0-4650.gac9df2d6.el9.x86_64 22/34 2024-09-05T16:31:00.832 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-rbd-2:19.3.0-4650.gac9df2d6.el9.x86_64 23/34 2024-09-05T16:31:00.868 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-09-05T16:31:01.357 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-09-05T16:31:01.379 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : librgw2-2:19.3.0-4650.gac9df2d6.el9.x86_64 26/34 2024-09-05T16:31:01.451 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: librgw2-2:19.3.0-4650.gac9df2d6.el9.x86_64 26/34 2024-09-05T16:31:01.520 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-rgw-2:19.3.0-4650.gac9df2d6.el9.x86_64 27/34 2024-09-05T16:31:01.548 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-ceph-common-2:19.3.0-4650.gac9df2d6.el9.x8 28/34 2024-09-05T16:31:03.876 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-common-2:19.3.0-4650.gac9df2d6.el9.x86_64 29/34 2024-09-05T16:31:03.911 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-common-2:19.3.0-4650.gac9df2d6.el9.x86_64 29/34 2024-09-05T16:31:04.463 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-common-2:19.3.0-4650.gac9df2d6.el9.x86_64 29/34 2024-09-05T16:31:04.506 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-base-2:19.3.0-4650.gac9df2d6.el9.x86_64 30/34 2024-09-05T16:31:04.603 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-base-2:19.3.0-4650.gac9df2d6.el9.x86_64 30/34 2024-09-05T16:31:04.603 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-05T16:31:04.604 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-05T16:31:04.604 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:31:04.632 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-selinux-2:19.3.0-4650.gac9df2d6.el9.x86_64 31/34 2024-09-05T16:31:14.524 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-selinux-2:19.3.0-4650.gac9df2d6.el9.x86_64 31/34 2024-09-05T16:31:14.524 INFO:teuthology.orchestra.run.smithi087.stdout:skipping the directory /sys 2024-09-05T16:31:14.524 INFO:teuthology.orchestra.run.smithi087.stdout:skipping the directory /proc 2024-09-05T16:31:14.524 INFO:teuthology.orchestra.run.smithi087.stdout:skipping the directory /mnt 2024-09-05T16:31:14.524 INFO:teuthology.orchestra.run.smithi087.stdout:skipping the directory /var/tmp 2024-09-05T16:31:14.525 INFO:teuthology.orchestra.run.smithi087.stdout:skipping the directory /home 2024-09-05T16:31:14.525 INFO:teuthology.orchestra.run.smithi087.stdout:skipping the directory /root 2024-09-05T16:31:14.525 INFO:teuthology.orchestra.run.smithi087.stdout:skipping the directory /tmp 2024-09-05T16:31:14.525 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:31:15.547 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-radosgw-2:19.3.0-4650.gac9df2d6.el9.x86_64 32/34 2024-09-05T16:31:15.573 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-4650.gac9df2d6.el9.x86_64 32/34 2024-09-05T16:31:15.573 INFO:teuthology.orchestra.run.smithi087.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-05T16:31:15.573 INFO:teuthology.orchestra.run.smithi087.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-05T16:31:15.573 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-05T16:31:15.573 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-05T16:31:15.573 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:31:15.575 INFO:teuthology.orchestra.run.smithi087.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-05T16:31:15.667 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-05T16:31:15.667 INFO:teuthology.orchestra.run.smithi087.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-05T16:31:16.843 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-05T16:31:16.843 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-base-2:19.3.0-4650.gac9df2d6.el9.x86_64 1/34 2024-09-05T16:31:16.843 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-common-2:19.3.0-4650.gac9df2d6.el9.x86_64 2/34 2024-09-05T16:31:16.843 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-radosgw-2:19.3.0-4650.gac9df2d6.el9.x86_64 3/34 2024-09-05T16:31:16.843 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-selinux-2:19.3.0-4650.gac9df2d6.el9.x86_64 4/34 2024-09-05T16:31:16.843 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libcephfs2-2:19.3.0-4650.gac9df2d6.el9.x86_64 5/34 2024-09-05T16:31:16.843 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libradosstriper1-2:19.3.0-4650.gac9df2d6.el9.x86_6 6/34 2024-09-05T16:31:16.843 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : librgw2-2:19.3.0-4650.gac9df2d6.el9.x86_64 7/34 2024-09-05T16:31:16.843 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-ceph-argparse-2:19.3.0-4650.gac9df2d6.el9. 8/34 2024-09-05T16:31:16.844 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-ceph-common-2:19.3.0-4650.gac9df2d6.el9.x8 9/34 2024-09-05T16:31:16.844 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-cephfs-2:19.3.0-4650.gac9df2d6.el9.x86_64 10/34 2024-09-05T16:31:16.844 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-rados-2:19.3.0-4650.gac9df2d6.el9.x86_64 11/34 2024-09-05T16:31:16.844 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-rbd-2:19.3.0-4650.gac9df2d6.el9.x86_64 12/34 2024-09-05T16:31:16.844 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-rgw-2:19.3.0-4650.gac9df2d6.el9.x86_64 13/34 2024-09-05T16:31:16.844 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-09-05T16:31:16.844 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-09-05T16:31:16.844 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-09-05T16:31:16.844 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-09-05T16:31:16.844 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-09-05T16:31:16.844 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-09-05T16:31:16.844 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-09-05T16:31:16.844 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-09-05T16:31:16.844 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-09-05T16:31:16.845 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-09-05T16:31:16.845 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-09-05T16:31:16.845 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-09-05T16:31:16.845 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-09-05T16:31:16.845 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-09-05T16:31:16.845 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-09-05T16:31:16.846 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : re2-1:20211101-20.el9.x86_64 29/34 2024-09-05T16:31:16.846 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-09-05T16:31:16.846 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : librados2-2:19.3.0-4650.gac9df2d6.el9.x86_64 31/34 2024-09-05T16:31:16.846 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-09-05T16:31:16.846 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : librbd1-2:19.3.0-4650.gac9df2d6.el9.x86_64 33/34 2024-09-05T16:31:17.678 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-09-05T16:31:17.678 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:31:17.678 INFO:teuthology.orchestra.run.smithi087.stdout:Upgraded: 2024-09-05T16:31:17.678 INFO:teuthology.orchestra.run.smithi087.stdout: librados2-2:19.3.0-4650.gac9df2d6.el9.x86_64 2024-09-05T16:31:17.678 INFO:teuthology.orchestra.run.smithi087.stdout: librbd1-2:19.3.0-4650.gac9df2d6.el9.x86_64 2024-09-05T16:31:17.678 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-09-05T16:31:17.678 INFO:teuthology.orchestra.run.smithi087.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-05T16:31:17.678 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-base-2:19.3.0-4650.gac9df2d6.el9.x86_64 2024-09-05T16:31:17.678 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-common-2:19.3.0-4650.gac9df2d6.el9.x86_64 2024-09-05T16:31:17.679 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-radosgw-2:19.3.0-4650.gac9df2d6.el9.x86_64 2024-09-05T16:31:17.679 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-selinux-2:19.3.0-4650.gac9df2d6.el9.x86_64 2024-09-05T16:31:17.679 INFO:teuthology.orchestra.run.smithi087.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-09-05T16:31:17.679 INFO:teuthology.orchestra.run.smithi087.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-05T16:31:17.679 INFO:teuthology.orchestra.run.smithi087.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-05T16:31:17.679 INFO:teuthology.orchestra.run.smithi087.stdout: libcephfs2-2:19.3.0-4650.gac9df2d6.el9.x86_64 2024-09-05T16:31:17.679 INFO:teuthology.orchestra.run.smithi087.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-05T16:31:17.679 INFO:teuthology.orchestra.run.smithi087.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-05T16:31:17.679 INFO:teuthology.orchestra.run.smithi087.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-05T16:31:17.679 INFO:teuthology.orchestra.run.smithi087.stdout: libradosstriper1-2:19.3.0-4650.gac9df2d6.el9.x86_64 2024-09-05T16:31:17.679 INFO:teuthology.orchestra.run.smithi087.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-05T16:31:17.679 INFO:teuthology.orchestra.run.smithi087.stdout: librgw2-2:19.3.0-4650.gac9df2d6.el9.x86_64 2024-09-05T16:31:17.680 INFO:teuthology.orchestra.run.smithi087.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-05T16:31:17.680 INFO:teuthology.orchestra.run.smithi087.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-05T16:31:17.680 INFO:teuthology.orchestra.run.smithi087.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-05T16:31:17.680 INFO:teuthology.orchestra.run.smithi087.stdout: python3-ceph-argparse-2:19.3.0-4650.gac9df2d6.el9.x86_64 2024-09-05T16:31:17.680 INFO:teuthology.orchestra.run.smithi087.stdout: python3-ceph-common-2:19.3.0-4650.gac9df2d6.el9.x86_64 2024-09-05T16:31:17.680 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cephfs-2:19.3.0-4650.gac9df2d6.el9.x86_64 2024-09-05T16:31:17.680 INFO:teuthology.orchestra.run.smithi087.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-05T16:31:17.680 INFO:teuthology.orchestra.run.smithi087.stdout: python3-rados-2:19.3.0-4650.gac9df2d6.el9.x86_64 2024-09-05T16:31:17.680 INFO:teuthology.orchestra.run.smithi087.stdout: python3-rbd-2:19.3.0-4650.gac9df2d6.el9.x86_64 2024-09-05T16:31:17.680 INFO:teuthology.orchestra.run.smithi087.stdout: python3-rgw-2:19.3.0-4650.gac9df2d6.el9.x86_64 2024-09-05T16:31:17.680 INFO:teuthology.orchestra.run.smithi087.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-05T16:31:17.680 INFO:teuthology.orchestra.run.smithi087.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-05T16:31:17.680 INFO:teuthology.orchestra.run.smithi087.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-05T16:31:17.681 INFO:teuthology.orchestra.run.smithi087.stdout: re2-1:20211101-20.el9.x86_64 2024-09-05T16:31:17.681 INFO:teuthology.orchestra.run.smithi087.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-05T16:31:17.681 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:31:17.681 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-09-05T16:31:18.010 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-test 2024-09-05T16:31:18.562 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:00:27 ago on Thu 05 Sep 2024 04:30:51 PM UTC. 2024-09-05T16:31:18.701 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-09-05T16:31:18.702 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-05T16:31:18.702 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repository Size 2024-09-05T16:31:18.703 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-05T16:31:18.703 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-09-05T16:31:18.703 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-test x86_64 2:19.3.0-4650.gac9df2d6.el9 ceph 50 M 2024-09-05T16:31:18.703 INFO:teuthology.orchestra.run.smithi087.stdout:Installing dependencies: 2024-09-05T16:31:18.703 INFO:teuthology.orchestra.run.smithi087.stdout: libcephsqlite x86_64 2:19.3.0-4650.gac9df2d6.el9 ceph 161 k 2024-09-05T16:31:18.703 INFO:teuthology.orchestra.run.smithi087.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-05T16:31:18.703 INFO:teuthology.orchestra.run.smithi087.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-05T16:31:18.703 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:31:18.703 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-09-05T16:31:18.703 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-05T16:31:18.703 INFO:teuthology.orchestra.run.smithi087.stdout:Install 4 Packages 2024-09-05T16:31:18.703 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:31:18.704 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 51 M 2024-09-05T16:31:18.704 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 216 M 2024-09-05T16:31:18.704 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-09-05T16:31:18.904 INFO:teuthology.orchestra.run.smithi087.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.9 MB/s | 304 kB 00:00 2024-09-05T16:31:18.946 INFO:teuthology.orchestra.run.smithi087.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 64 kB 00:00 2024-09-05T16:31:18.988 INFO:teuthology.orchestra.run.smithi087.stdout:(3/4): libcephsqlite-19.3.0-4650.gac9df2d6.el9. 679 kB/s | 161 kB 00:00 2024-09-05T16:31:20.288 INFO:teuthology.orchestra.run.smithi087.stdout:(4/4): ceph-test-19.3.0-4650.gac9df2d6.el9.x86_ 33 MB/s | 50 MB 00:01 2024-09-05T16:31:20.290 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-09-05T16:31:20.290 INFO:teuthology.orchestra.run.smithi087.stdout:Total 32 MB/s | 51 MB 00:01 2024-09-05T16:31:20.317 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-09-05T16:31:20.337 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-09-05T16:31:20.337 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-09-05T16:31:20.717 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-09-05T16:31:20.717 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-09-05T16:31:21.241 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-09-05T16:31:21.313 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-05T16:31:21.354 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-05T16:31:21.381 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libcephsqlite-2:19.3.0-4650.gac9df2d6.el9.x86_64 3/4 2024-09-05T16:31:26.042 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: libcephsqlite-2:19.3.0-4650.gac9df2d6.el9.x86_64 3/4 2024-09-05T16:31:26.147 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-test-2:19.3.0-4650.gac9df2d6.el9.x86_64 4/4 2024-09-05T16:31:28.533 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-test-2:19.3.0-4650.gac9df2d6.el9.x86_64 4/4 2024-09-05T16:31:28.533 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-test-2:19.3.0-4650.gac9df2d6.el9.x86_64 1/4 2024-09-05T16:31:28.533 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libcephsqlite-2:19.3.0-4650.gac9df2d6.el9.x86_64 2/4 2024-09-05T16:31:28.533 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-05T16:31:28.837 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-05T16:31:28.837 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:31:28.837 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-09-05T16:31:28.837 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-test-2:19.3.0-4650.gac9df2d6.el9.x86_64 2024-09-05T16:31:28.837 INFO:teuthology.orchestra.run.smithi087.stdout: libcephsqlite-2:19.3.0-4650.gac9df2d6.el9.x86_64 2024-09-05T16:31:28.837 INFO:teuthology.orchestra.run.smithi087.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-05T16:31:28.837 INFO:teuthology.orchestra.run.smithi087.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-05T16:31:28.837 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:31:28.838 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-09-05T16:31:29.110 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph 2024-09-05T16:31:29.671 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:00:38 ago on Thu 05 Sep 2024 04:30:51 PM UTC. 2024-09-05T16:31:29.803 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-09-05T16:31:29.806 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-05T16:31:29.806 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repository Size 2024-09-05T16:31:29.806 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-05T16:31:29.806 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-09-05T16:31:29.806 INFO:teuthology.orchestra.run.smithi087.stdout: ceph x86_64 2:19.3.0-4650.gac9df2d6.el9 ceph 6.6 k 2024-09-05T16:31:29.806 INFO:teuthology.orchestra.run.smithi087.stdout:Installing dependencies: 2024-09-05T16:31:29.806 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mds x86_64 2:19.3.0-4650.gac9df2d6.el9 ceph 2.4 M 2024-09-05T16:31:29.806 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr x86_64 2:19.3.0-4650.gac9df2d6.el9 ceph 1.5 M 2024-09-05T16:31:29.807 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-modules-core noarch 2:19.3.0-4650.gac9df2d6.el9 ceph-noarch 291 k 2024-09-05T16:31:29.807 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mon x86_64 2:19.3.0-4650.gac9df2d6.el9 ceph 4.8 M 2024-09-05T16:31:29.807 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-osd x86_64 2:19.3.0-4650.gac9df2d6.el9 ceph 17 M 2024-09-05T16:31:29.807 INFO:teuthology.orchestra.run.smithi087.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-05T16:31:29.807 INFO:teuthology.orchestra.run.smithi087.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-05T16:31:29.807 INFO:teuthology.orchestra.run.smithi087.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-05T16:31:29.807 INFO:teuthology.orchestra.run.smithi087.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-05T16:31:29.807 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-05T16:31:29.807 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-05T16:31:29.807 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-05T16:31:29.807 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-05T16:31:29.807 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-05T16:31:29.807 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-05T16:31:29.808 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jaraco-collections 2024-09-05T16:31:29.808 INFO:teuthology.orchestra.run.smithi087.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-05T16:31:29.808 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-05T16:31:29.808 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-05T16:31:29.808 INFO:teuthology.orchestra.run.smithi087.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-05T16:31:29.808 INFO:teuthology.orchestra.run.smithi087.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-05T16:31:29.808 INFO:teuthology.orchestra.run.smithi087.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-05T16:31:29.808 INFO:teuthology.orchestra.run.smithi087.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-05T16:31:29.808 INFO:teuthology.orchestra.run.smithi087.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-05T16:31:29.808 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-05T16:31:29.808 INFO:teuthology.orchestra.run.smithi087.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-05T16:31:29.808 INFO:teuthology.orchestra.run.smithi087.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-05T16:31:29.808 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-05T16:31:29.809 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-05T16:31:29.809 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-05T16:31:29.809 INFO:teuthology.orchestra.run.smithi087.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-05T16:31:29.809 INFO:teuthology.orchestra.run.smithi087.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-05T16:31:29.809 INFO:teuthology.orchestra.run.smithi087.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-05T16:31:29.809 INFO:teuthology.orchestra.run.smithi087.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-05T16:31:29.809 INFO:teuthology.orchestra.run.smithi087.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-05T16:31:29.809 INFO:teuthology.orchestra.run.smithi087.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-05T16:31:29.809 INFO:teuthology.orchestra.run.smithi087.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-05T16:31:29.809 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:31:29.809 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-09-05T16:31:29.809 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-05T16:31:29.809 INFO:teuthology.orchestra.run.smithi087.stdout:Install 37 Packages 2024-09-05T16:31:29.809 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:31:29.810 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 31 M 2024-09-05T16:31:29.810 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 109 M 2024-09-05T16:31:29.810 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-09-05T16:31:30.741 INFO:teuthology.orchestra.run.smithi087.stdout:(1/37): ceph-19.3.0-4650.gac9df2d6.el9.x86_64.r 55 kB/s | 6.6 kB 00:00 2024-09-05T16:31:30.925 INFO:teuthology.orchestra.run.smithi087.stdout:(2/37): ceph-mgr-19.3.0-4650.gac9df2d6.el9.x86_ 5.0 MB/s | 1.5 MB 00:00 2024-09-05T16:31:30.967 INFO:teuthology.orchestra.run.smithi087.stdout:(3/37): ceph-mds-19.3.0-4650.gac9df2d6.el9.x86_ 6.9 MB/s | 2.4 MB 00:00 2024-09-05T16:31:31.092 INFO:teuthology.orchestra.run.smithi087.stdout:(4/37): ceph-mgr-modules-core-19.3.0-4650.gac9d 2.3 MB/s | 291 kB 00:00 2024-09-05T16:31:31.159 INFO:teuthology.orchestra.run.smithi087.stdout:(5/37): ceph-mon-19.3.0-4650.gac9df2d6.el9.x86_ 11 MB/s | 4.8 MB 00:00 2024-09-05T16:31:31.543 INFO:teuthology.orchestra.run.smithi087.stdout:(6/37): ceph-osd-19.3.0-4650.gac9df2d6.el9.x86_ 28 MB/s | 17 MB 00:00 2024-09-05T16:31:32.118 INFO:teuthology.orchestra.run.smithi087.stdout:(7/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 247 kB/s | 253 kB 00:01 2024-09-05T16:31:32.285 INFO:teuthology.orchestra.run.smithi087.stdout:(8/37): python3-ply-3.11-14.el9.noarch.rpm 145 kB/s | 106 kB 00:00 2024-09-05T16:31:32.852 INFO:teuthology.orchestra.run.smithi087.stdout:(9/37): python3-pycparser-2.20-6.el9.noarch.rpm 184 kB/s | 135 kB 00:00 2024-09-05T16:31:33.060 INFO:teuthology.orchestra.run.smithi087.stdout:(10/37): python3-requests-2.25.1-8.el9.noarch.r 161 kB/s | 125 kB 00:00 2024-09-05T16:31:33.102 INFO:teuthology.orchestra.run.smithi087.stdout:(11/37): python3-cryptography-36.0.1-4.el9.x86_ 646 kB/s | 1.2 MB 00:01 2024-09-05T16:31:33.203 INFO:teuthology.orchestra.run.smithi087.stdout:(12/37): lua-5.4.4-4.el9.x86_64.rpm 1.3 MB/s | 188 kB 00:00 2024-09-05T16:31:33.227 INFO:teuthology.orchestra.run.smithi087.stdout:(13/37): python3-mako-1.1.4-6.el9.noarch.rpm 1.3 MB/s | 172 kB 00:00 2024-09-05T16:31:33.253 INFO:teuthology.orchestra.run.smithi087.stdout:(14/37): python3-markupsafe-1.1.1-12.el9.x86_64 693 kB/s | 35 kB 00:00 2024-09-05T16:31:33.294 INFO:teuthology.orchestra.run.smithi087.stdout:(15/37): python3-packaging-20.9-5.el9.noarch.rp 1.1 MB/s | 77 kB 00:00 2024-09-05T16:31:33.319 INFO:teuthology.orchestra.run.smithi087.stdout:(16/37): python3-pytz-2021.1-5.el9.noarch.rpm 775 kB/s | 51 kB 00:00 2024-09-05T16:31:33.344 INFO:teuthology.orchestra.run.smithi087.stdout:(17/37): python3-toml-0.10.2-6.el9.noarch.rpm 833 kB/s | 42 kB 00:00 2024-09-05T16:31:33.394 INFO:teuthology.orchestra.run.smithi087.stdout:(18/37): luarocks-3.9.2-5.el9.noarch.rpm 3.0 MB/s | 151 kB 00:00 2024-09-05T16:31:33.419 INFO:teuthology.orchestra.run.smithi087.stdout:(19/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 1.7 MB/s | 43 kB 00:00 2024-09-05T16:31:33.445 INFO:teuthology.orchestra.run.smithi087.stdout:(20/37): lua-devel-5.4.4-4.el9.x86_64.rpm 177 kB/s | 22 kB 00:00 2024-09-05T16:31:33.470 INFO:teuthology.orchestra.run.smithi087.stdout:(21/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 3.4 MB/s | 172 kB 00:00 2024-09-05T16:31:33.504 INFO:teuthology.orchestra.run.smithi087.stdout:(22/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 414 kB/s | 11 kB 00:00 2024-09-05T16:31:33.529 INFO:teuthology.orchestra.run.smithi087.stdout:(23/37): python3-jaraco-classes-3.2.1-5.el9.noa 761 kB/s | 18 kB 00:00 2024-09-05T16:31:33.563 INFO:teuthology.orchestra.run.smithi087.stdout:(24/37): python3-jaraco-collections-3.0.0-8.el9 700 kB/s | 23 kB 00:00 2024-09-05T16:31:33.588 INFO:teuthology.orchestra.run.smithi087.stdout:(25/37): python3-jaraco-functools-3.5.0-2.el9.n 787 kB/s | 19 kB 00:00 2024-09-05T16:31:33.613 INFO:teuthology.orchestra.run.smithi087.stdout:(26/37): python3-jaraco-text-3.2.0-6.el9.noarch 790 kB/s | 20 kB 00:00 2024-09-05T16:31:33.638 INFO:teuthology.orchestra.run.smithi087.stdout:(27/37): python3-logutils-0.3.5-21.el9.noarch.r 1.8 MB/s | 46 kB 00:00 2024-09-05T16:31:33.663 INFO:teuthology.orchestra.run.smithi087.stdout:(28/37): python3-cherrypy-18.6.1-2.el9.noarch.r 1.6 MB/s | 358 kB 00:00 2024-09-05T16:31:33.689 INFO:teuthology.orchestra.run.smithi087.stdout:(29/37): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-09-05T16:31:33.714 INFO:teuthology.orchestra.run.smithi087.stdout:(30/37): python3-urllib3-1.26.5-6.el9.noarch.rp 249 kB/s | 215 kB 00:00 2024-09-05T16:31:33.739 INFO:teuthology.orchestra.run.smithi087.stdout:(31/37): python3-portend-3.1.0-2.el9.noarch.rpm 336 kB/s | 16 kB 00:00 2024-09-05T16:31:33.775 INFO:teuthology.orchestra.run.smithi087.stdout:(32/37): python3-tempora-5.0.0-2.el9.noarch.rpm 1.3 MB/s | 36 kB 00:00 2024-09-05T16:31:33.800 INFO:teuthology.orchestra.run.smithi087.stdout:(33/37): python3-pecan-1.4.2-3.el9.noarch.rpm 2.0 MB/s | 272 kB 00:00 2024-09-05T16:31:33.825 INFO:teuthology.orchestra.run.smithi087.stdout:(34/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 809 kB/s | 90 kB 00:00 2024-09-05T16:31:33.842 INFO:teuthology.orchestra.run.smithi087.stdout:(35/37): python3-zc-lockfile-2.0-10.el9.noarch. 1.2 MB/s | 20 kB 00:00 2024-09-05T16:31:33.867 INFO:teuthology.orchestra.run.smithi087.stdout:(36/37): python3-webob-1.8.8-2.el9.noarch.rpm 2.5 MB/s | 230 kB 00:00 2024-09-05T16:31:33.892 INFO:teuthology.orchestra.run.smithi087.stdout:(37/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.5 MB/s | 427 kB 00:00 2024-09-05T16:31:33.897 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-09-05T16:31:33.897 INFO:teuthology.orchestra.run.smithi087.stdout:Total 7.5 MB/s | 31 MB 00:04 2024-09-05T16:31:34.052 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-09-05T16:31:34.092 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-09-05T16:31:34.092 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-09-05T16:31:34.391 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-09-05T16:31:34.391 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-09-05T16:31:35.737 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-09-05T16:31:35.804 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-05T16:31:35.837 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-05T16:31:35.865 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-05T16:31:35.929 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-05T16:31:35.980 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-05T16:31:36.013 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-05T16:31:36.106 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-05T16:31:36.183 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-05T16:31:36.224 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-05T16:31:36.259 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-05T16:31:36.289 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-05T16:31:36.323 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-05T16:31:36.356 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-05T16:31:36.397 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-05T16:31:36.440 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-05T16:31:36.474 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-05T16:31:36.523 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-05T16:31:36.555 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-05T16:31:36.628 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-05T16:31:36.769 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-05T16:31:36.847 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-05T16:31:36.900 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-05T16:31:36.938 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-05T16:31:36.986 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-05T16:31:37.050 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-05T16:31:37.241 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-05T16:31:37.291 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-05T16:31:37.370 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-05T16:31:37.458 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-05T16:31:37.499 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-05T16:31:37.629 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-05T16:31:37.787 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-4650.gac9df2d6.el9. 32/37 2024-09-05T16:31:37.809 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-mgr-2:19.3.0-4650.gac9df2d6.el9.x86_64 33/37 2024-09-05T16:31:37.834 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-mgr-2:19.3.0-4650.gac9df2d6.el9.x86_64 33/37 2024-09-05T16:31:37.834 INFO:teuthology.orchestra.run.smithi087.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-05T16:31:37.834 INFO:teuthology.orchestra.run.smithi087.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-05T16:31:37.835 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-05T16:31:37.835 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-05T16:31:37.835 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:31:39.293 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-osd-2:19.3.0-4650.gac9df2d6.el9.x86_64 34/37 2024-09-05T16:31:39.321 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-osd-2:19.3.0-4650.gac9df2d6.el9.x86_64 34/37 2024-09-05T16:31:39.321 INFO:teuthology.orchestra.run.smithi087.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-05T16:31:39.321 INFO:teuthology.orchestra.run.smithi087.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-05T16:31:39.321 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-05T16:31:39.321 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-05T16:31:39.321 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:31:39.769 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-mon-2:19.3.0-4650.gac9df2d6.el9.x86_64 35/37 2024-09-05T16:31:39.793 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-mon-2:19.3.0-4650.gac9df2d6.el9.x86_64 35/37 2024-09-05T16:31:39.793 INFO:teuthology.orchestra.run.smithi087.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-05T16:31:39.794 INFO:teuthology.orchestra.run.smithi087.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-05T16:31:39.794 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-05T16:31:39.794 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-05T16:31:39.794 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:31:40.020 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-mds-2:19.3.0-4650.gac9df2d6.el9.x86_64 36/37 2024-09-05T16:31:40.044 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-mds-2:19.3.0-4650.gac9df2d6.el9.x86_64 36/37 2024-09-05T16:31:40.044 INFO:teuthology.orchestra.run.smithi087.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-05T16:31:40.044 INFO:teuthology.orchestra.run.smithi087.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-05T16:31:40.044 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-05T16:31:40.044 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-05T16:31:40.044 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:31:40.107 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-2:19.3.0-4650.gac9df2d6.el9.x86_64 37/37 2024-09-05T16:31:41.893 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-2:19.3.0-4650.gac9df2d6.el9.x86_64 37/37 2024-09-05T16:31:41.893 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-2:19.3.0-4650.gac9df2d6.el9.x86_64 1/37 2024-09-05T16:31:41.893 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-mds-2:19.3.0-4650.gac9df2d6.el9.x86_64 2/37 2024-09-05T16:31:41.893 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-mgr-2:19.3.0-4650.gac9df2d6.el9.x86_64 3/37 2024-09-05T16:31:41.893 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-mon-2:19.3.0-4650.gac9df2d6.el9.x86_64 4/37 2024-09-05T16:31:41.893 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-osd-2:19.3.0-4650.gac9df2d6.el9.x86_64 5/37 2024-09-05T16:31:41.893 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-4650.gac9df2d6.el9. 6/37 2024-09-05T16:31:41.893 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-05T16:31:41.893 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-05T16:31:41.893 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-05T16:31:41.894 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-05T16:31:41.894 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-05T16:31:41.894 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-05T16:31:41.894 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-05T16:31:41.894 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-05T16:31:41.894 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-05T16:31:41.894 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-05T16:31:41.894 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-05T16:31:41.894 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-05T16:31:41.894 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-05T16:31:41.894 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-05T16:31:41.894 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-05T16:31:41.894 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-05T16:31:41.895 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-05T16:31:41.895 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-05T16:31:41.895 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-05T16:31:41.896 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-05T16:31:41.896 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-05T16:31:41.896 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-05T16:31:41.896 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-05T16:31:41.896 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-05T16:31:41.896 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-05T16:31:41.897 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-05T16:31:41.897 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-05T16:31:41.897 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-05T16:31:41.897 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-05T16:31:41.897 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-05T16:31:43.054 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-05T16:31:43.054 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:31:43.055 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-09-05T16:31:43.055 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-2:19.3.0-4650.gac9df2d6.el9.x86_64 2024-09-05T16:31:43.055 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mds-2:19.3.0-4650.gac9df2d6.el9.x86_64 2024-09-05T16:31:43.055 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-2:19.3.0-4650.gac9df2d6.el9.x86_64 2024-09-05T16:31:43.055 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-modules-core-2:19.3.0-4650.gac9df2d6.el9.noarch 2024-09-05T16:31:43.056 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mon-2:19.3.0-4650.gac9df2d6.el9.x86_64 2024-09-05T16:31:43.056 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-osd-2:19.3.0-4650.gac9df2d6.el9.x86_64 2024-09-05T16:31:43.056 INFO:teuthology.orchestra.run.smithi087.stdout: lua-5.4.4-4.el9.x86_64 2024-09-05T16:31:43.056 INFO:teuthology.orchestra.run.smithi087.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-05T16:31:43.056 INFO:teuthology.orchestra.run.smithi087.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-05T16:31:43.057 INFO:teuthology.orchestra.run.smithi087.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-05T16:31:43.057 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-05T16:31:43.057 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-05T16:31:43.057 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-05T16:31:43.058 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-05T16:31:43.058 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-05T16:31:43.058 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-05T16:31:43.058 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-05T16:31:43.058 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-05T16:31:43.058 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-05T16:31:43.059 INFO:teuthology.orchestra.run.smithi087.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-05T16:31:43.059 INFO:teuthology.orchestra.run.smithi087.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-05T16:31:43.059 INFO:teuthology.orchestra.run.smithi087.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-05T16:31:43.059 INFO:teuthology.orchestra.run.smithi087.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-05T16:31:43.059 INFO:teuthology.orchestra.run.smithi087.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-05T16:31:43.060 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-05T16:31:43.060 INFO:teuthology.orchestra.run.smithi087.stdout: python3-ply-3.11-14.el9.noarch 2024-09-05T16:31:43.060 INFO:teuthology.orchestra.run.smithi087.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-05T16:31:43.060 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-05T16:31:43.060 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-05T16:31:43.061 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-05T16:31:43.061 INFO:teuthology.orchestra.run.smithi087.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-05T16:31:43.061 INFO:teuthology.orchestra.run.smithi087.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-05T16:31:43.061 INFO:teuthology.orchestra.run.smithi087.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-05T16:31:43.061 INFO:teuthology.orchestra.run.smithi087.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-05T16:31:43.062 INFO:teuthology.orchestra.run.smithi087.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-05T16:31:43.062 INFO:teuthology.orchestra.run.smithi087.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-05T16:31:43.062 INFO:teuthology.orchestra.run.smithi087.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-05T16:31:43.062 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:31:43.062 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-09-05T16:31:43.418 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-base 2024-09-05T16:31:43.982 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:00:52 ago on Thu 05 Sep 2024 04:30:51 PM UTC. 2024-09-05T16:31:44.070 INFO:teuthology.orchestra.run.smithi087.stdout:Package ceph-base-2:19.3.0-4650.gac9df2d6.el9.x86_64 is already installed. 2024-09-05T16:31:44.111 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-09-05T16:31:44.112 INFO:teuthology.orchestra.run.smithi087.stdout:Nothing to do. 2024-09-05T16:31:44.112 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-09-05T16:31:44.179 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install cephadm 2024-09-05T16:31:44.721 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:00:53 ago on Thu 05 Sep 2024 04:30:51 PM UTC. 2024-09-05T16:31:44.851 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-09-05T16:31:44.852 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-05T16:31:44.852 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repository Size 2024-09-05T16:31:44.852 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-05T16:31:44.852 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-09-05T16:31:44.852 INFO:teuthology.orchestra.run.smithi087.stdout: cephadm noarch 2:19.3.0-4650.gac9df2d6.el9 ceph-noarch 783 k 2024-09-05T16:31:44.853 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:31:44.853 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-09-05T16:31:44.853 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-05T16:31:44.853 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2024-09-05T16:31:44.853 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:31:44.853 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 783 k 2024-09-05T16:31:44.853 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 789 k 2024-09-05T16:31:44.853 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-09-05T16:31:45.136 INFO:teuthology.orchestra.run.smithi087.stdout:cephadm-19.3.0-4650.gac9df2d6.el9.noarch.rpm 2.7 MB/s | 783 kB 00:00 2024-09-05T16:31:45.136 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-09-05T16:31:45.136 INFO:teuthology.orchestra.run.smithi087.stdout:Total 2.7 MB/s | 783 kB 00:00 2024-09-05T16:31:45.136 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-09-05T16:31:45.141 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-09-05T16:31:45.141 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-09-05T16:31:45.147 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-09-05T16:31:45.147 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-09-05T16:31:45.222 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-09-05T16:31:45.696 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: cephadm-2:19.3.0-4650.gac9df2d6.el9.noarch 1/1 2024-09-05T16:31:45.786 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : cephadm-2:19.3.0-4650.gac9df2d6.el9.noarch 1/1 2024-09-05T16:31:46.119 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: cephadm-2:19.3.0-4650.gac9df2d6.el9.noarch 1/1 2024-09-05T16:31:46.336 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : cephadm-2:19.3.0-4650.gac9df2d6.el9.noarch 1/1 2024-09-05T16:31:46.337 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:31:46.337 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-09-05T16:31:46.337 INFO:teuthology.orchestra.run.smithi087.stdout: cephadm-2:19.3.0-4650.gac9df2d6.el9.noarch 2024-09-05T16:31:46.337 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:31:46.337 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-09-05T16:31:46.546 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-immutable-object-cache 2024-09-05T16:31:47.108 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:00:56 ago on Thu 05 Sep 2024 04:30:51 PM UTC. 2024-09-05T16:31:47.240 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-09-05T16:31:47.241 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-05T16:31:47.241 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repo Size 2024-09-05T16:31:47.241 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-05T16:31:47.241 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-09-05T16:31:47.241 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-4650.gac9df2d6.el9 ceph 145 k 2024-09-05T16:31:47.241 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:31:47.241 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-09-05T16:31:47.241 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-05T16:31:47.242 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2024-09-05T16:31:47.242 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:31:47.242 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 145 k 2024-09-05T16:31:47.242 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 432 k 2024-09-05T16:31:47.242 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-09-05T16:31:47.465 INFO:teuthology.orchestra.run.smithi087.stdout:ceph-immutable-object-cache-19.3.0-4650.gac9df2 652 kB/s | 145 kB 00:00 2024-09-05T16:31:47.466 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-09-05T16:31:47.466 INFO:teuthology.orchestra.run.smithi087.stdout:Total 647 kB/s | 145 kB 00:00 2024-09-05T16:31:47.466 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-09-05T16:31:47.473 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-09-05T16:31:47.473 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-09-05T16:31:47.529 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-09-05T16:31:47.530 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-09-05T16:31:47.684 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-09-05T16:31:47.757 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-4650.gac9df2d6. 1/1 2024-09-05T16:31:47.783 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-4650.gac9df2d6. 1/1 2024-09-05T16:31:47.783 INFO:teuthology.orchestra.run.smithi087.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-05T16:31:47.783 INFO:teuthology.orchestra.run.smithi087.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-05T16:31:47.783 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:31:48.694 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-4650.gac9df2d6. 1/1 2024-09-05T16:31:48.694 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:31:48.694 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-09-05T16:31:48.694 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-immutable-object-cache-2:19.3.0-4650.gac9df2d6.el9.x86_64 2024-09-05T16:31:48.694 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:31:48.694 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-09-05T16:31:48.926 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-mgr 2024-09-05T16:31:49.487 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:00:58 ago on Thu 05 Sep 2024 04:30:51 PM UTC. 2024-09-05T16:31:49.580 INFO:teuthology.orchestra.run.smithi087.stdout:Package ceph-mgr-2:19.3.0-4650.gac9df2d6.el9.x86_64 is already installed. 2024-09-05T16:31:49.623 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-09-05T16:31:49.624 INFO:teuthology.orchestra.run.smithi087.stdout:Nothing to do. 2024-09-05T16:31:49.624 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-09-05T16:31:49.695 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-mgr-dashboard 2024-09-05T16:31:50.239 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:00:59 ago on Thu 05 Sep 2024 04:30:51 PM UTC. 2024-09-05T16:31:50.372 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-09-05T16:31:50.373 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-05T16:31:50.373 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repository Size 2024-09-05T16:31:50.373 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-05T16:31:50.373 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-09-05T16:31:50.373 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-dashboard noarch 2:19.3.0-4650.gac9df2d6.el9 ceph-noarch 14 M 2024-09-05T16:31:50.373 INFO:teuthology.orchestra.run.smithi087.stdout:Installing dependencies: 2024-09-05T16:31:50.373 INFO:teuthology.orchestra.run.smithi087.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-05T16:31:50.373 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-grafana-dashboards noarch 2:19.3.0-4650.gac9df2d6.el9 ceph-noarch 34 k 2024-09-05T16:31:50.373 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-prometheus-alerts noarch 2:19.3.0-4650.gac9df2d6.el9 ceph-noarch 16 k 2024-09-05T16:31:50.373 INFO:teuthology.orchestra.run.smithi087.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-05T16:31:50.374 INFO:teuthology.orchestra.run.smithi087.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-05T16:31:50.374 INFO:teuthology.orchestra.run.smithi087.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-05T16:31:50.374 INFO:teuthology.orchestra.run.smithi087.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-05T16:31:50.374 INFO:teuthology.orchestra.run.smithi087.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-05T16:31:50.374 INFO:teuthology.orchestra.run.smithi087.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-05T16:31:50.374 INFO:teuthology.orchestra.run.smithi087.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-05T16:31:50.374 INFO:teuthology.orchestra.run.smithi087.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-05T16:31:50.374 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:31:50.374 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-09-05T16:31:50.374 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-05T16:31:50.374 INFO:teuthology.orchestra.run.smithi087.stdout:Install 12 Packages 2024-09-05T16:31:50.374 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:31:50.375 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 19 M 2024-09-05T16:31:50.375 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 158 M 2024-09-05T16:31:50.375 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-09-05T16:31:50.876 INFO:teuthology.orchestra.run.smithi087.stdout:(1/12): ceph-grafana-dashboards-19.3.0-4650.gac 238 kB/s | 34 kB 00:00 2024-09-05T16:31:50.901 INFO:teuthology.orchestra.run.smithi087.stdout:(2/12): ceph-prometheus-alerts-19.3.0-4650.gac9 97 kB/s | 16 kB 00:00 2024-09-05T16:31:51.118 INFO:teuthology.orchestra.run.smithi087.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.2 MB/s | 269 kB 00:00 2024-09-05T16:31:51.185 INFO:teuthology.orchestra.run.smithi087.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 3.3 MB/s | 1.0 MB 00:00 2024-09-05T16:31:51.243 INFO:teuthology.orchestra.run.smithi087.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 9.2 MB/s | 548 kB 00:00 2024-09-05T16:31:51.269 INFO:teuthology.orchestra.run.smithi087.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 779 kB/s | 19 kB 00:00 2024-09-05T16:31:51.419 INFO:teuthology.orchestra.run.smithi087.stdout:(7/12): ceph-mgr-dashboard-19.3.0-4650.gac9df2d 21 MB/s | 14 MB 00:00 2024-09-05T16:31:51.478 INFO:teuthology.orchestra.run.smithi087.stdout:(8/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 9.8 MB/s | 2.0 MB 00:00 2024-09-05T16:31:51.503 INFO:teuthology.orchestra.run.smithi087.stdout:(9/12): protobuf-compiler-3.14.0-13.el9.x86_64. 2.2 MB/s | 863 kB 00:00 2024-09-05T16:31:51.528 INFO:teuthology.orchestra.run.smithi087.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 615 kB/s | 31 kB 00:00 2024-09-05T16:31:51.562 INFO:teuthology.orchestra.run.smithi087.stdout:(11/12): python3-grpcio-tools-1.46.7-10.el9.x86 1.0 MB/s | 144 kB 00:00 2024-09-05T16:31:51.595 INFO:teuthology.orchestra.run.smithi087.stdout:(12/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.0 MB/s | 188 kB 00:00 2024-09-05T16:31:51.598 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-09-05T16:31:51.599 INFO:teuthology.orchestra.run.smithi087.stdout:Total 16 MB/s | 19 MB 00:01 2024-09-05T16:31:51.646 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-09-05T16:31:51.665 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-09-05T16:31:51.665 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-09-05T16:31:51.887 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-09-05T16:31:51.888 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-09-05T16:31:52.481 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-09-05T16:31:52.554 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-09-05T16:31:52.606 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-09-05T16:31:52.652 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-09-05T16:31:52.691 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-09-05T16:31:52.757 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-09-05T16:31:52.890 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-09-05T16:31:52.993 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-09-05T16:31:53.037 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-09-05T16:31:53.065 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-09-05T16:31:53.098 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-4650.gac9df2d6.el9 10/12 2024-09-05T16:31:56.234 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-4650.gac9df2d6.el 11/12 2024-09-05T16:31:56.294 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-4650.gac9df2d6.el9.noa 12/12 2024-09-05T16:31:58.169 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-4650.gac9df2d6.el9.noa 12/12 2024-09-05T16:31:58.169 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-4650.gac9df2d6.el 1/12 2024-09-05T16:31:58.169 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-4650.gac9df2d6.el9.noa 2/12 2024-09-05T16:31:58.170 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-4650.gac9df2d6.el9 3/12 2024-09-05T16:31:58.170 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-09-05T16:31:58.170 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-09-05T16:31:58.170 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-09-05T16:31:58.170 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-09-05T16:31:58.170 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-09-05T16:31:58.170 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-09-05T16:31:58.170 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-09-05T16:31:58.170 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-09-05T16:31:58.467 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-09-05T16:31:58.467 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:31:58.467 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-09-05T16:31:58.467 INFO:teuthology.orchestra.run.smithi087.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-05T16:31:58.467 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-grafana-dashboards-2:19.3.0-4650.gac9df2d6.el9.noarch 2024-09-05T16:31:58.467 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-dashboard-2:19.3.0-4650.gac9df2d6.el9.noarch 2024-09-05T16:31:58.467 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-prometheus-alerts-2:19.3.0-4650.gac9df2d6.el9.noarch 2024-09-05T16:31:58.467 INFO:teuthology.orchestra.run.smithi087.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-05T16:31:58.467 INFO:teuthology.orchestra.run.smithi087.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-05T16:31:58.467 INFO:teuthology.orchestra.run.smithi087.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-05T16:31:58.467 INFO:teuthology.orchestra.run.smithi087.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-05T16:31:58.468 INFO:teuthology.orchestra.run.smithi087.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-05T16:31:58.468 INFO:teuthology.orchestra.run.smithi087.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-05T16:31:58.468 INFO:teuthology.orchestra.run.smithi087.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-05T16:31:58.468 INFO:teuthology.orchestra.run.smithi087.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-05T16:31:58.468 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:31:58.468 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-09-05T16:31:58.714 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-05T16:31:59.271 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:08 ago on Thu 05 Sep 2024 04:30:51 PM UTC. 2024-09-05T16:31:59.410 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-09-05T16:31:59.411 INFO:teuthology.orchestra.run.smithi087.stdout:======================================================================================= 2024-09-05T16:31:59.411 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repository Size 2024-09-05T16:31:59.411 INFO:teuthology.orchestra.run.smithi087.stdout:======================================================================================= 2024-09-05T16:31:59.411 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-09-05T16:31:59.411 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-4650.gac9df2d6.el9 ceph-noarch 7.4 M 2024-09-05T16:31:59.412 INFO:teuthology.orchestra.run.smithi087.stdout:Installing dependencies: 2024-09-05T16:31:59.412 INFO:teuthology.orchestra.run.smithi087.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-05T16:31:59.412 INFO:teuthology.orchestra.run.smithi087.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-05T16:31:59.412 INFO:teuthology.orchestra.run.smithi087.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-05T16:31:59.412 INFO:teuthology.orchestra.run.smithi087.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-05T16:31:59.412 INFO:teuthology.orchestra.run.smithi087.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-05T16:31:59.412 INFO:teuthology.orchestra.run.smithi087.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-05T16:31:59.412 INFO:teuthology.orchestra.run.smithi087.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-05T16:31:59.412 INFO:teuthology.orchestra.run.smithi087.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-05T16:31:59.412 INFO:teuthology.orchestra.run.smithi087.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-05T16:31:59.412 INFO:teuthology.orchestra.run.smithi087.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-05T16:31:59.412 INFO:teuthology.orchestra.run.smithi087.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-05T16:31:59.413 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:31:59.413 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-09-05T16:31:59.413 INFO:teuthology.orchestra.run.smithi087.stdout:======================================================================================= 2024-09-05T16:31:59.413 INFO:teuthology.orchestra.run.smithi087.stdout:Install 12 Packages 2024-09-05T16:31:59.413 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:31:59.414 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 38 M 2024-09-05T16:31:59.414 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 212 M 2024-09-05T16:31:59.414 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-09-05T16:31:59.710 INFO:teuthology.orchestra.run.smithi087.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.3 MB/s | 191 kB 00:00 2024-09-05T16:31:59.743 INFO:teuthology.orchestra.run.smithi087.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 4.3 MB/s | 801 kB 00:00 2024-09-05T16:31:59.793 INFO:teuthology.orchestra.run.smithi087.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 392 kB/s | 33 kB 00:00 2024-09-05T16:31:59.835 INFO:teuthology.orchestra.run.smithi087.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 439 kB/s | 18 kB 00:00 2024-09-05T16:31:59.877 INFO:teuthology.orchestra.run.smithi087.stdout:(5/12): openblas-0.3.26-2.el9.x86_64.rpm 926 kB/s | 38 kB 00:00 2024-09-05T16:32:00.036 INFO:teuthology.orchestra.run.smithi087.stdout:(6/12): ceph-mgr-diskprediction-local-19.3.0-46 16 MB/s | 7.4 MB 00:00 2024-09-05T16:32:00.169 INFO:teuthology.orchestra.run.smithi087.stdout:(7/12): python3-devel-3.9.19-8.el9.x86_64.rpm 1.8 MB/s | 245 kB 00:00 2024-09-05T16:32:00.528 INFO:teuthology.orchestra.run.smithi087.stdout:(8/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 3.8 MB/s | 3.0 MB 00:00 2024-09-05T16:32:00.620 INFO:teuthology.orchestra.run.smithi087.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.7 MB/s | 250 kB 00:00 2024-09-05T16:32:01.079 INFO:teuthology.orchestra.run.smithi087.stdout:(10/12): openblas-openmp-0.3.26-2.el9.x86_64.rp 4.1 MB/s | 4.9 MB 00:01 2024-09-05T16:32:01.545 INFO:teuthology.orchestra.run.smithi087.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 3.7 MB/s | 5.1 MB 00:01 2024-09-05T16:32:04.887 INFO:teuthology.orchestra.run.smithi087.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 3.8 MB/s | 16 MB 00:04 2024-09-05T16:32:04.888 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-09-05T16:32:04.888 INFO:teuthology.orchestra.run.smithi087.stdout:Total 7.0 MB/s | 38 MB 00:05 2024-09-05T16:32:05.247 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-09-05T16:32:05.273 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-09-05T16:32:05.273 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-09-05T16:32:05.685 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-09-05T16:32:05.685 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-09-05T16:32:06.436 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-09-05T16:32:06.516 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-05T16:32:06.551 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-05T16:32:06.648 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-05T16:32:06.694 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-05T16:32:06.877 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-05T16:32:06.910 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-05T16:32:06.985 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-05T16:32:07.623 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-05T16:32:07.738 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-05T16:32:08.949 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-05T16:32:09.980 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-05T16:32:10.000 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-4650.gac9df 12/12 2024-09-05T16:32:12.440 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-4650.gac9df 12/12 2024-09-05T16:32:12.440 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-4650.gac9df 1/12 2024-09-05T16:32:12.440 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-05T16:32:12.440 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-05T16:32:12.440 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-05T16:32:12.440 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-05T16:32:12.440 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-05T16:32:12.440 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-05T16:32:12.441 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-05T16:32:12.441 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-05T16:32:12.441 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-05T16:32:12.441 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-05T16:32:12.826 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-05T16:32:12.826 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:32:12.826 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-09-05T16:32:12.826 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-diskprediction-local-2:19.3.0-4650.gac9df2d6.el9.noarch 2024-09-05T16:32:12.826 INFO:teuthology.orchestra.run.smithi087.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-05T16:32:12.826 INFO:teuthology.orchestra.run.smithi087.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-05T16:32:12.826 INFO:teuthology.orchestra.run.smithi087.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-05T16:32:12.827 INFO:teuthology.orchestra.run.smithi087.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-05T16:32:12.827 INFO:teuthology.orchestra.run.smithi087.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-05T16:32:12.827 INFO:teuthology.orchestra.run.smithi087.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-05T16:32:12.827 INFO:teuthology.orchestra.run.smithi087.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-05T16:32:12.827 INFO:teuthology.orchestra.run.smithi087.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-05T16:32:12.827 INFO:teuthology.orchestra.run.smithi087.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-05T16:32:12.827 INFO:teuthology.orchestra.run.smithi087.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-05T16:32:12.827 INFO:teuthology.orchestra.run.smithi087.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-05T16:32:12.827 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:32:12.827 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-09-05T16:32:13.212 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-mgr-rook 2024-09-05T16:32:13.773 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:22 ago on Thu 05 Sep 2024 04:30:51 PM UTC. 2024-09-05T16:32:13.908 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-09-05T16:32:13.909 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-05T16:32:13.910 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repository Size 2024-09-05T16:32:13.910 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-05T16:32:13.910 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-09-05T16:32:13.910 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-rook noarch 2:19.3.0-4650.gac9df2d6.el9 ceph-noarch 49 k 2024-09-05T16:32:13.910 INFO:teuthology.orchestra.run.smithi087.stdout:Installing dependencies: 2024-09-05T16:32:13.910 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-05T16:32:13.910 INFO:teuthology.orchestra.run.smithi087.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-05T16:32:13.910 INFO:teuthology.orchestra.run.smithi087.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-05T16:32:13.910 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-05T16:32:13.910 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-05T16:32:13.910 INFO:teuthology.orchestra.run.smithi087.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-05T16:32:13.910 INFO:teuthology.orchestra.run.smithi087.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-05T16:32:13.910 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-05T16:32:13.911 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-05T16:32:13.911 INFO:teuthology.orchestra.run.smithi087.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-05T16:32:13.911 INFO:teuthology.orchestra.run.smithi087.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-05T16:32:13.911 INFO:teuthology.orchestra.run.smithi087.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-05T16:32:13.911 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:32:13.911 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-09-05T16:32:13.911 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-05T16:32:13.911 INFO:teuthology.orchestra.run.smithi087.stdout:Install 13 Packages 2024-09-05T16:32:13.911 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:32:13.912 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 2.2 M 2024-09-05T16:32:13.912 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 26 M 2024-09-05T16:32:13.912 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-09-05T16:32:14.482 INFO:teuthology.orchestra.run.smithi087.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 225 kB/s | 26 kB 00:00 2024-09-05T16:32:14.499 INFO:teuthology.orchestra.run.smithi087.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 147 kB/s | 19 kB 00:00 2024-09-05T16:32:14.516 INFO:teuthology.orchestra.run.smithi087.stdout:(3/13): ceph-mgr-rook-19.3.0-4650.gac9df2d6.el9 334 kB/s | 49 kB 00:00 2024-09-05T16:32:14.642 INFO:teuthology.orchestra.run.smithi087.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.1 MB/s | 159 kB 00:00 2024-09-05T16:32:14.666 INFO:teuthology.orchestra.run.smithi087.stdout:(5/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.3 MB/s | 222 kB 00:00 2024-09-05T16:32:14.692 INFO:teuthology.orchestra.run.smithi087.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 54 kB 00:00 2024-09-05T16:32:14.717 INFO:teuthology.orchestra.run.smithi087.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 643 kB/s | 32 kB 00:00 2024-09-05T16:32:14.734 INFO:teuthology.orchestra.run.smithi087.stdout:(8/13): python3-certifi-2023.05.07-4.el9.noarch 351 kB/s | 14 kB 00:00 2024-09-05T16:32:14.759 INFO:teuthology.orchestra.run.smithi087.stdout:(9/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.1 MB/s | 279 kB 00:00 2024-09-05T16:32:14.792 INFO:teuthology.orchestra.run.smithi087.stdout:(10/13): python3-google-auth-2.34.0-1.el9.noarc 2.9 MB/s | 220 kB 00:00 2024-09-05T16:32:14.818 INFO:teuthology.orchestra.run.smithi087.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 3.8 MB/s | 90 kB 00:00 2024-09-05T16:32:14.835 INFO:teuthology.orchestra.run.smithi087.stdout:(12/13): python3-rsa-4.9-2.el9.noarch.rpm 785 kB/s | 59 kB 00:00 2024-09-05T16:32:14.877 INFO:teuthology.orchestra.run.smithi087.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 7.2 MB/s | 1.0 MB 00:00 2024-09-05T16:32:14.879 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-09-05T16:32:14.879 INFO:teuthology.orchestra.run.smithi087.stdout:Total 2.3 MB/s | 2.2 MB 00:00 2024-09-05T16:32:14.944 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-09-05T16:32:14.959 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-09-05T16:32:14.960 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-09-05T16:32:15.045 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-09-05T16:32:15.046 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-09-05T16:32:15.556 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-09-05T16:32:15.717 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-05T16:32:15.769 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-05T16:32:15.825 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-05T16:32:15.870 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-05T16:32:15.912 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-05T16:32:16.013 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-05T16:32:16.115 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-05T16:32:16.171 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-05T16:32:16.876 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-05T16:32:16.926 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-05T16:32:16.960 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-05T16:32:17.008 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-05T16:32:17.030 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-mgr-rook-2:19.3.0-4650.gac9df2d6.el9.noarch 13/13 2024-09-05T16:32:17.819 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-4650.gac9df2d6.el9.noarch 13/13 2024-09-05T16:32:17.819 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-mgr-rook-2:19.3.0-4650.gac9df2d6.el9.noarch 1/13 2024-09-05T16:32:17.819 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-05T16:32:17.819 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-05T16:32:17.819 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-05T16:32:17.819 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-05T16:32:17.819 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-05T16:32:17.819 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-05T16:32:17.820 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-05T16:32:17.820 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-05T16:32:17.820 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-05T16:32:17.820 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-05T16:32:17.820 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-05T16:32:18.140 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-05T16:32:18.140 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:32:18.140 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-09-05T16:32:18.140 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-rook-2:19.3.0-4650.gac9df2d6.el9.noarch 2024-09-05T16:32:18.140 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-05T16:32:18.140 INFO:teuthology.orchestra.run.smithi087.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-05T16:32:18.140 INFO:teuthology.orchestra.run.smithi087.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-05T16:32:18.140 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-05T16:32:18.140 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-05T16:32:18.140 INFO:teuthology.orchestra.run.smithi087.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-05T16:32:18.140 INFO:teuthology.orchestra.run.smithi087.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-05T16:32:18.140 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-05T16:32:18.141 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-05T16:32:18.141 INFO:teuthology.orchestra.run.smithi087.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-05T16:32:18.141 INFO:teuthology.orchestra.run.smithi087.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-05T16:32:18.141 INFO:teuthology.orchestra.run.smithi087.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-05T16:32:18.141 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:32:18.141 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-09-05T16:32:18.389 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-mgr-cephadm 2024-09-05T16:32:18.939 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:27 ago on Thu 05 Sep 2024 04:30:51 PM UTC. 2024-09-05T16:32:19.069 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-09-05T16:32:19.070 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-05T16:32:19.070 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repository Size 2024-09-05T16:32:19.070 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-05T16:32:19.071 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-09-05T16:32:19.071 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-cephadm noarch 2:19.3.0-4650.gac9df2d6.el9 ceph-noarch 154 k 2024-09-05T16:32:19.071 INFO:teuthology.orchestra.run.smithi087.stdout:Installing dependencies: 2024-09-05T16:32:19.071 INFO:teuthology.orchestra.run.smithi087.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-05T16:32:19.071 INFO:teuthology.orchestra.run.smithi087.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-05T16:32:19.071 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-05T16:32:19.071 INFO:teuthology.orchestra.run.smithi087.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-05T16:32:19.071 INFO:teuthology.orchestra.run.smithi087.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-05T16:32:19.071 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:32:19.071 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-09-05T16:32:19.071 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-05T16:32:19.071 INFO:teuthology.orchestra.run.smithi087.stdout:Install 6 Packages 2024-09-05T16:32:19.071 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:32:19.072 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 7.0 M 2024-09-05T16:32:19.072 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 33 M 2024-09-05T16:32:19.072 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-09-05T16:32:19.744 INFO:teuthology.orchestra.run.smithi087.stdout:(1/6): ceph-mgr-cephadm-19.3.0-4650.gac9df2d6.e 722 kB/s | 154 kB 00:00 2024-09-05T16:32:19.811 INFO:teuthology.orchestra.run.smithi087.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 9.0 MB/s | 548 kB 00:00 2024-09-05T16:32:19.836 INFO:teuthology.orchestra.run.smithi087.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 2.3 MB/s | 58 kB 00:00 2024-09-05T16:32:19.861 INFO:teuthology.orchestra.run.smithi087.stdout:(4/6): python3-typing-extensions-4.12.2-2.el9.n 3.0 MB/s | 74 kB 00:00 2024-09-05T16:32:20.303 INFO:teuthology.orchestra.run.smithi087.stdout:(5/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 321 kB/s | 248 kB 00:00 2024-09-05T16:32:20.687 INFO:teuthology.orchestra.run.smithi087.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 5.2 MB/s | 6.0 MB 00:01 2024-09-05T16:32:20.689 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-09-05T16:32:20.689 INFO:teuthology.orchestra.run.smithi087.stdout:Total 4.3 MB/s | 7.0 MB 00:01 2024-09-05T16:32:20.762 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-09-05T16:32:20.774 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-09-05T16:32:20.775 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-09-05T16:32:20.873 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-09-05T16:32:20.874 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-09-05T16:32:21.176 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-09-05T16:32:21.350 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-05T16:32:21.396 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-05T16:32:21.913 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-05T16:32:21.998 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-05T16:32:22.068 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-05T16:32:22.092 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-4650.gac9df2d6.el9.noarch 6/6 2024-09-05T16:32:22.816 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-4650.gac9df2d6.el9.noarch 6/6 2024-09-05T16:32:22.817 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-4650.gac9df2d6.el9.noarch 1/6 2024-09-05T16:32:22.817 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-05T16:32:22.817 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-05T16:32:22.817 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-05T16:32:22.817 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-05T16:32:23.072 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-05T16:32:23.072 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:32:23.072 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-09-05T16:32:23.072 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-cephadm-2:19.3.0-4650.gac9df2d6.el9.noarch 2024-09-05T16:32:23.072 INFO:teuthology.orchestra.run.smithi087.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-05T16:32:23.073 INFO:teuthology.orchestra.run.smithi087.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-05T16:32:23.073 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-05T16:32:23.073 INFO:teuthology.orchestra.run.smithi087.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-05T16:32:23.073 INFO:teuthology.orchestra.run.smithi087.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-05T16:32:23.073 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:32:23.073 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-09-05T16:32:23.315 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-fuse 2024-09-05T16:32:23.860 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:32 ago on Thu 05 Sep 2024 04:30:51 PM UTC. 2024-09-05T16:32:23.991 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-09-05T16:32:23.992 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-05T16:32:23.992 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repository Size 2024-09-05T16:32:23.992 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-05T16:32:23.993 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-09-05T16:32:23.993 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-fuse x86_64 2:19.3.0-4650.gac9df2d6.el9 ceph 900 k 2024-09-05T16:32:23.993 INFO:teuthology.orchestra.run.smithi087.stdout:Installing dependencies: 2024-09-05T16:32:23.993 INFO:teuthology.orchestra.run.smithi087.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-05T16:32:23.993 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:32:23.993 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-09-05T16:32:23.993 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-05T16:32:23.993 INFO:teuthology.orchestra.run.smithi087.stdout:Install 2 Packages 2024-09-05T16:32:23.993 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:32:23.993 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 979 k 2024-09-05T16:32:23.993 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 2.9 M 2024-09-05T16:32:23.993 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-09-05T16:32:24.388 INFO:teuthology.orchestra.run.smithi087.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 591 kB/s | 80 kB 00:00 2024-09-05T16:32:24.554 INFO:teuthology.orchestra.run.smithi087.stdout:(2/2): ceph-fuse-19.3.0-4650.gac9df2d6.el9.x86_ 2.9 MB/s | 900 kB 00:00 2024-09-05T16:32:24.555 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-09-05T16:32:24.555 INFO:teuthology.orchestra.run.smithi087.stdout:Total 1.7 MB/s | 979 kB 00:00 2024-09-05T16:32:24.567 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-09-05T16:32:24.589 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-09-05T16:32:24.589 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-09-05T16:32:24.668 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-09-05T16:32:24.669 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-09-05T16:32:24.855 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-09-05T16:32:24.991 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-05T16:32:25.052 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-fuse-2:19.3.0-4650.gac9df2d6.el9.x86_64 2/2 2024-09-05T16:32:25.745 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-fuse-2:19.3.0-4650.gac9df2d6.el9.x86_64 2/2 2024-09-05T16:32:25.746 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-fuse-2:19.3.0-4650.gac9df2d6.el9.x86_64 1/2 2024-09-05T16:32:26.031 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-05T16:32:26.032 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:32:26.032 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-09-05T16:32:26.032 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-fuse-2:19.3.0-4650.gac9df2d6.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-05T16:32:26.032 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:32:26.032 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-09-05T16:32:26.276 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-volume 2024-09-05T16:32:26.839 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:35 ago on Thu 05 Sep 2024 04:30:51 PM UTC. 2024-09-05T16:32:26.975 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-09-05T16:32:26.976 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-05T16:32:26.976 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repository Size 2024-09-05T16:32:26.976 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-05T16:32:26.976 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-09-05T16:32:26.976 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-volume noarch 2:19.3.0-4650.gac9df2d6.el9 ceph-noarch 285 k 2024-09-05T16:32:26.976 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:32:26.976 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-09-05T16:32:26.976 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-05T16:32:26.976 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2024-09-05T16:32:26.976 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:32:26.976 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 285 k 2024-09-05T16:32:26.977 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 1.3 M 2024-09-05T16:32:26.977 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-09-05T16:32:27.217 INFO:teuthology.orchestra.run.smithi087.stdout:ceph-volume-19.3.0-4650.gac9df2d6.el9.noarch.rp 1.2 MB/s | 285 kB 00:00 2024-09-05T16:32:27.217 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-09-05T16:32:27.218 INFO:teuthology.orchestra.run.smithi087.stdout:Total 1.2 MB/s | 285 kB 00:00 2024-09-05T16:32:27.218 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-09-05T16:32:27.226 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-09-05T16:32:27.226 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-09-05T16:32:27.249 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-09-05T16:32:27.249 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-09-05T16:32:27.463 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-09-05T16:32:27.535 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-volume-2:19.3.0-4650.gac9df2d6.el9.noarch 1/1 2024-09-05T16:32:27.552 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-volume-2:19.3.0-4650.gac9df2d6.el9.noarch 1/1 2024-09-05T16:32:27.552 INFO:teuthology.orchestra.run.smithi087.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-05T16:32:27.552 INFO:teuthology.orchestra.run.smithi087.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-05T16:32:27.552 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:32:28.485 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-volume-2:19.3.0-4650.gac9df2d6.el9.noarch 1/1 2024-09-05T16:32:28.485 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:32:28.485 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-09-05T16:32:28.485 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-volume-2:19.3.0-4650.gac9df2d6.el9.noarch 2024-09-05T16:32:28.485 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:32:28.486 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-09-05T16:32:28.798 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install librados-devel 2024-09-05T16:32:29.356 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:38 ago on Thu 05 Sep 2024 04:30:51 PM UTC. 2024-09-05T16:32:29.495 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-09-05T16:32:29.495 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-05T16:32:29.496 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repo Size 2024-09-05T16:32:29.496 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-05T16:32:29.496 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-09-05T16:32:29.496 INFO:teuthology.orchestra.run.smithi087.stdout: librados-devel x86_64 2:19.3.0-4650.gac9df2d6.el9 ceph 127 k 2024-09-05T16:32:29.496 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:32:29.496 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-09-05T16:32:29.496 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-05T16:32:29.496 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2024-09-05T16:32:29.496 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:32:29.496 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 127 k 2024-09-05T16:32:29.496 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 456 k 2024-09-05T16:32:29.496 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-09-05T16:32:29.688 INFO:teuthology.orchestra.run.smithi087.stdout:librados-devel-19.3.0-4650.gac9df2d6.el9.x86_64 662 kB/s | 127 kB 00:00 2024-09-05T16:32:29.689 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-09-05T16:32:29.689 INFO:teuthology.orchestra.run.smithi087.stdout:Total 656 kB/s | 127 kB 00:00 2024-09-05T16:32:29.689 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-09-05T16:32:29.695 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-09-05T16:32:29.695 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-09-05T16:32:29.758 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-09-05T16:32:29.759 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-09-05T16:32:29.908 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-09-05T16:32:29.992 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : librados-devel-2:19.3.0-4650.gac9df2d6.el9.x86_64 1/1 2024-09-05T16:32:30.439 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: librados-devel-2:19.3.0-4650.gac9df2d6.el9.x86_64 1/1 2024-09-05T16:32:30.698 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : librados-devel-2:19.3.0-4650.gac9df2d6.el9.x86_64 1/1 2024-09-05T16:32:30.698 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:32:30.698 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-09-05T16:32:30.698 INFO:teuthology.orchestra.run.smithi087.stdout: librados-devel-2:19.3.0-4650.gac9df2d6.el9.x86_64 2024-09-05T16:32:30.698 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:32:30.698 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-09-05T16:32:30.951 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install libcephfs2 2024-09-05T16:32:31.517 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:40 ago on Thu 05 Sep 2024 04:30:51 PM UTC. 2024-09-05T16:32:31.607 INFO:teuthology.orchestra.run.smithi087.stdout:Package libcephfs2-2:19.3.0-4650.gac9df2d6.el9.x86_64 is already installed. 2024-09-05T16:32:31.651 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-09-05T16:32:31.652 INFO:teuthology.orchestra.run.smithi087.stdout:Nothing to do. 2024-09-05T16:32:31.652 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-09-05T16:32:31.726 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install libcephfs-devel 2024-09-05T16:32:32.274 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:41 ago on Thu 05 Sep 2024 04:30:51 PM UTC. 2024-09-05T16:32:32.405 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-09-05T16:32:32.405 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-05T16:32:32.406 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repo Size 2024-09-05T16:32:32.406 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-05T16:32:32.406 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-09-05T16:32:32.406 INFO:teuthology.orchestra.run.smithi087.stdout: libcephfs-devel x86_64 2:19.3.0-4650.gac9df2d6.el9 ceph 32 k 2024-09-05T16:32:32.406 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:32:32.406 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-09-05T16:32:32.406 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-05T16:32:32.406 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2024-09-05T16:32:32.406 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:32:32.406 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 32 k 2024-09-05T16:32:32.406 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 142 k 2024-09-05T16:32:32.406 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-09-05T16:32:32.574 INFO:teuthology.orchestra.run.smithi087.stdout:libcephfs-devel-19.3.0-4650.gac9df2d6.el9.x86_6 189 kB/s | 32 kB 00:00 2024-09-05T16:32:32.575 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-09-05T16:32:32.575 INFO:teuthology.orchestra.run.smithi087.stdout:Total 187 kB/s | 32 kB 00:00 2024-09-05T16:32:32.575 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-09-05T16:32:32.579 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-09-05T16:32:32.579 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-09-05T16:32:32.582 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-09-05T16:32:32.583 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-09-05T16:32:32.661 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-09-05T16:32:32.753 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libcephfs-devel-2:19.3.0-4650.gac9df2d6.el9.x86_64 1/1 2024-09-05T16:32:33.160 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: libcephfs-devel-2:19.3.0-4650.gac9df2d6.el9.x86_64 1/1 2024-09-05T16:32:33.370 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libcephfs-devel-2:19.3.0-4650.gac9df2d6.el9.x86_64 1/1 2024-09-05T16:32:33.370 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:32:33.370 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-09-05T16:32:33.370 INFO:teuthology.orchestra.run.smithi087.stdout: libcephfs-devel-2:19.3.0-4650.gac9df2d6.el9.x86_64 2024-09-05T16:32:33.370 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:32:33.370 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-09-05T16:32:33.562 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install librados2 2024-09-05T16:32:34.110 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:43 ago on Thu 05 Sep 2024 04:30:51 PM UTC. 2024-09-05T16:32:34.197 INFO:teuthology.orchestra.run.smithi087.stdout:Package librados2-2:19.3.0-4650.gac9df2d6.el9.x86_64 is already installed. 2024-09-05T16:32:34.240 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-09-05T16:32:34.241 INFO:teuthology.orchestra.run.smithi087.stdout:Nothing to do. 2024-09-05T16:32:34.241 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-09-05T16:32:34.313 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install librbd1 2024-09-05T16:32:34.859 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:43 ago on Thu 05 Sep 2024 04:30:51 PM UTC. 2024-09-05T16:32:34.948 INFO:teuthology.orchestra.run.smithi087.stdout:Package librbd1-2:19.3.0-4650.gac9df2d6.el9.x86_64 is already installed. 2024-09-05T16:32:34.990 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-09-05T16:32:34.991 INFO:teuthology.orchestra.run.smithi087.stdout:Nothing to do. 2024-09-05T16:32:34.991 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-09-05T16:32:35.074 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install python3-rados 2024-09-05T16:32:35.623 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:44 ago on Thu 05 Sep 2024 04:30:51 PM UTC. 2024-09-05T16:32:35.714 INFO:teuthology.orchestra.run.smithi087.stdout:Package python3-rados-2:19.3.0-4650.gac9df2d6.el9.x86_64 is already installed. 2024-09-05T16:32:35.756 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-09-05T16:32:35.757 INFO:teuthology.orchestra.run.smithi087.stdout:Nothing to do. 2024-09-05T16:32:35.758 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-09-05T16:32:35.833 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install python3-rgw 2024-09-05T16:32:36.389 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:45 ago on Thu 05 Sep 2024 04:30:51 PM UTC. 2024-09-05T16:32:36.479 INFO:teuthology.orchestra.run.smithi087.stdout:Package python3-rgw-2:19.3.0-4650.gac9df2d6.el9.x86_64 is already installed. 2024-09-05T16:32:36.522 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-09-05T16:32:36.523 INFO:teuthology.orchestra.run.smithi087.stdout:Nothing to do. 2024-09-05T16:32:36.523 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-09-05T16:32:36.586 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install python3-cephfs 2024-09-05T16:32:37.141 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:46 ago on Thu 05 Sep 2024 04:30:51 PM UTC. 2024-09-05T16:32:37.230 INFO:teuthology.orchestra.run.smithi087.stdout:Package python3-cephfs-2:19.3.0-4650.gac9df2d6.el9.x86_64 is already installed. 2024-09-05T16:32:37.272 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-09-05T16:32:37.273 INFO:teuthology.orchestra.run.smithi087.stdout:Nothing to do. 2024-09-05T16:32:37.273 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-09-05T16:32:37.335 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install python3-rbd 2024-09-05T16:32:37.888 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:46 ago on Thu 05 Sep 2024 04:30:51 PM UTC. 2024-09-05T16:32:37.977 INFO:teuthology.orchestra.run.smithi087.stdout:Package python3-rbd-2:19.3.0-4650.gac9df2d6.el9.x86_64 is already installed. 2024-09-05T16:32:38.019 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-09-05T16:32:38.021 INFO:teuthology.orchestra.run.smithi087.stdout:Nothing to do. 2024-09-05T16:32:38.021 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-09-05T16:32:38.096 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install rbd-fuse 2024-09-05T16:32:38.641 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:47 ago on Thu 05 Sep 2024 04:30:51 PM UTC. 2024-09-05T16:32:38.773 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-09-05T16:32:38.774 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-05T16:32:38.775 INFO:teuthology.orchestra.run.smithi087.stdout: Package Architecture Version Repository Size 2024-09-05T16:32:38.775 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-05T16:32:38.775 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-09-05T16:32:38.775 INFO:teuthology.orchestra.run.smithi087.stdout: rbd-fuse x86_64 2:19.3.0-4650.gac9df2d6.el9 ceph 85 k 2024-09-05T16:32:38.775 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:32:38.775 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-09-05T16:32:38.775 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-05T16:32:38.776 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2024-09-05T16:32:38.776 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:32:38.776 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 85 k 2024-09-05T16:32:38.776 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 231 k 2024-09-05T16:32:38.776 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-09-05T16:32:38.969 INFO:teuthology.orchestra.run.smithi087.stdout:rbd-fuse-19.3.0-4650.gac9df2d6.el9.x86_64.rpm 438 kB/s | 85 kB 00:00 2024-09-05T16:32:38.970 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-09-05T16:32:38.970 INFO:teuthology.orchestra.run.smithi087.stdout:Total 435 kB/s | 85 kB 00:00 2024-09-05T16:32:38.970 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-09-05T16:32:38.977 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-09-05T16:32:38.977 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-09-05T16:32:39.039 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-09-05T16:32:39.040 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-09-05T16:32:39.187 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-09-05T16:32:39.273 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : rbd-fuse-2:19.3.0-4650.gac9df2d6.el9.x86_64 1/1 2024-09-05T16:32:39.658 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: rbd-fuse-2:19.3.0-4650.gac9df2d6.el9.x86_64 1/1 2024-09-05T16:32:39.890 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : rbd-fuse-2:19.3.0-4650.gac9df2d6.el9.x86_64 1/1 2024-09-05T16:32:39.890 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:32:39.890 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-09-05T16:32:39.890 INFO:teuthology.orchestra.run.smithi087.stdout: rbd-fuse-2:19.3.0-4650.gac9df2d6.el9.x86_64 2024-09-05T16:32:39.890 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:32:39.890 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-09-05T16:32:40.163 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install rbd-mirror 2024-09-05T16:32:40.725 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:49 ago on Thu 05 Sep 2024 04:30:51 PM UTC. 2024-09-05T16:32:40.860 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-09-05T16:32:40.860 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-05T16:32:40.860 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repo Size 2024-09-05T16:32:40.861 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-05T16:32:40.861 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-09-05T16:32:40.861 INFO:teuthology.orchestra.run.smithi087.stdout: rbd-mirror x86_64 2:19.3.0-4650.gac9df2d6.el9 ceph 3.2 M 2024-09-05T16:32:40.861 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:32:40.861 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-09-05T16:32:40.861 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-05T16:32:40.861 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2024-09-05T16:32:40.861 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:32:40.861 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 3.2 M 2024-09-05T16:32:40.861 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 14 M 2024-09-05T16:32:40.861 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-09-05T16:32:41.205 INFO:teuthology.orchestra.run.smithi087.stdout:rbd-mirror-19.3.0-4650.gac9df2d6.el9.x86_64.rpm 9.2 MB/s | 3.2 MB 00:00 2024-09-05T16:32:41.206 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-09-05T16:32:41.206 INFO:teuthology.orchestra.run.smithi087.stdout:Total 9.2 MB/s | 3.2 MB 00:00 2024-09-05T16:32:41.206 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-09-05T16:32:41.215 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-09-05T16:32:41.215 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-09-05T16:32:41.296 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-09-05T16:32:41.297 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-09-05T16:32:41.730 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-09-05T16:32:41.798 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : rbd-mirror-2:19.3.0-4650.gac9df2d6.el9.x86_64 1/1 2024-09-05T16:32:41.823 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: rbd-mirror-2:19.3.0-4650.gac9df2d6.el9.x86_64 1/1 2024-09-05T16:32:41.823 INFO:teuthology.orchestra.run.smithi087.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-05T16:32:41.823 INFO:teuthology.orchestra.run.smithi087.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-05T16:32:41.823 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-05T16:32:41.823 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-05T16:32:41.823 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:32:42.862 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : rbd-mirror-2:19.3.0-4650.gac9df2d6.el9.x86_64 1/1 2024-09-05T16:32:42.862 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:32:42.862 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-09-05T16:32:42.862 INFO:teuthology.orchestra.run.smithi087.stdout: rbd-mirror-2:19.3.0-4650.gac9df2d6.el9.x86_64 2024-09-05T16:32:42.862 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:32:42.862 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-09-05T16:32:43.103 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install rbd-nbd 2024-09-05T16:32:43.669 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:52 ago on Thu 05 Sep 2024 04:30:51 PM UTC. 2024-09-05T16:32:43.805 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-09-05T16:32:43.806 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-05T16:32:43.806 INFO:teuthology.orchestra.run.smithi087.stdout: Package Architecture Version Repository Size 2024-09-05T16:32:43.806 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-05T16:32:43.806 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-09-05T16:32:43.806 INFO:teuthology.orchestra.run.smithi087.stdout: rbd-nbd x86_64 2:19.3.0-4650.gac9df2d6.el9 ceph 171 k 2024-09-05T16:32:43.806 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:32:43.806 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-09-05T16:32:43.806 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-05T16:32:43.806 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2024-09-05T16:32:43.806 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:32:43.806 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 171 k 2024-09-05T16:32:43.807 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 490 k 2024-09-05T16:32:43.807 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-09-05T16:32:44.033 INFO:teuthology.orchestra.run.smithi087.stdout:rbd-nbd-19.3.0-4650.gac9df2d6.el9.x86_64.rpm 757 kB/s | 171 kB 00:00 2024-09-05T16:32:44.034 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-09-05T16:32:44.034 INFO:teuthology.orchestra.run.smithi087.stdout:Total 752 kB/s | 171 kB 00:00 2024-09-05T16:32:44.034 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-09-05T16:32:44.041 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-09-05T16:32:44.041 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-09-05T16:32:44.105 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-09-05T16:32:44.106 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-09-05T16:32:44.255 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-09-05T16:32:44.353 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : rbd-nbd-2:19.3.0-4650.gac9df2d6.el9.x86_64 1/1 2024-09-05T16:32:44.759 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: rbd-nbd-2:19.3.0-4650.gac9df2d6.el9.x86_64 1/1 2024-09-05T16:32:45.001 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : rbd-nbd-2:19.3.0-4650.gac9df2d6.el9.x86_64 1/1 2024-09-05T16:32:45.001 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:32:45.001 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-09-05T16:32:45.001 INFO:teuthology.orchestra.run.smithi087.stdout: rbd-nbd-2:19.3.0-4650.gac9df2d6.el9.x86_64 2024-09-05T16:32:45.001 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:32:45.001 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-09-05T16:32:45.242 DEBUG:teuthology.parallel:result is None 2024-09-05T16:32:45.243 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=ac9df2d66a4c8e4e8634a3fe39168f4f4683bc03 2024-09-05T16:32:45.418 DEBUG:teuthology.orchestra.run.smithi087:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-05T16:32:45.439 INFO:teuthology.orchestra.run.smithi087.stdout:19.3.0-4650.gac9df2d6.el9 2024-09-05T16:32:45.440 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4650.gac9df2d6.el9 2024-09-05T16:32:45.440 INFO:teuthology.task.install:The correct ceph version 19.3.0-4650.gac9df2d6 is installed. 2024-09-05T16:32:45.442 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-05T16:32:45.442 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-09-05T16:32:45.442 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-05T16:32:45.504 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-05T16:32:45.504 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-09-05T16:32:45.504 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd of=/usr/bin/daemon-helper 2024-09-05T16:32:45.568 DEBUG:teuthology.orchestra.run.smithi087:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-05T16:32:45.633 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-05T16:32:45.634 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-09-05T16:32:45.634 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-05T16:32:45.698 DEBUG:teuthology.orchestra.run.smithi087:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-05T16:32:45.763 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-05T16:32:45.763 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-09-05T16:32:45.763 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd of=/usr/bin/stdin-killer 2024-09-05T16:32:45.828 DEBUG:teuthology.orchestra.run.smithi087:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-05T16:32:45.893 INFO:teuthology.run_tasks:Running task ceph... 2024-09-05T16:32:45.979 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-05T16:32:45.979 DEBUG:teuthology.orchestra.run.smithi087:> sudo chmod 777 /var/log/ceph 2024-09-05T16:32:46.003 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-05T16:32:46.003 DEBUG:teuthology.orchestra.run.smithi087:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-05T16:32:46.069 INFO:tasks.ceph:Creating extra log directories... 2024-09-05T16:32:46.069 DEBUG:teuthology.orchestra.run.smithi087:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-05T16:32:46.156 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-09-05T16:32:46.156 INFO:tasks.ceph:config {'conf': {'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore write v2 random': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(OBJECT_', '\\(SLOW_OPS\\)', 'slow request', '\\(POOL_APP_NOT_ENABLED\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', '\\(MON_DOWN\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-05T16:32:46.156 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/skanta-2024-09-05_12:54:40-rados-wip-bharath4-testing-2024-09-02-0957-distro-default-smithi/7890779', 'branch': 'wip-bharath4-testing-2024-09-02-0957', 'description': 'rados/singleton/{all/ec-lost-unfound mon_election/classic msgr-failures/few msgr/async-v2only objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-hybrid} rados supported-random-distro$/{centos_latest}}', 'email': 'skanta@redhat.com', 'first_in_suite': False, 'job_id': '7890779', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'skanta-2024-09-05_12:54:40-rados-wip-bharath4-testing-2024-09-02-0957-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-bharath4-testing-2024-09-02-0957'}, 'ceph': {'conf': {'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore write v2 random': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', '\\(MON_DOWN\\)'], 'sha1': 'ac9df2d66a4c8e4e8634a3fe39168f4f4683bc03'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'ac9df2d66a4c8e4e8634a3fe39168f4f4683bc03'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-bharath4-testing-2024-09-02-0957', 'sha1': 'ac9df2d66a4c8e4e8634a3fe39168f4f4683bc03'}}, 'owner': 'scheduled_skanta@teuthology', 'priority': 99, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3']], 'seed': 5171, 'sha1': 'ac9df2d66a4c8e4e8634a3fe39168f4f4683bc03', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rados', 'suite_branch': 'wip-bharath4-testing-2024-09-02-0957', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_ac9df2d66a4c8e4e8634a3fe39168f4f4683bc03/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': 'ac9df2d66a4c8e4e8634a3fe39168f4f4683bc03', 'targets': {'smithi087.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEBEONroSHW8yf0u1X9DR7rduLLOszGC4pmixy6wxEg7QaYnz4xpcLgxvC7B4TbUV5WQw5IYbxwGn5Dsq0/QvF8='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'create_rbd_pool': False, 'log-ignorelist': ['objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(OBJECT_', '\\(SLOW_OPS\\)', 'slow request', '\\(POOL_APP_NOT_ENABLED\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', '\\(MON_DOWN\\)'], 'pre-mgr-commands': ['sudo ceph config set mgr mgr_pool false --force'], 'conf': {'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore write v2 random': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'flavor': 'default', 'fs': 'xfs', 'sha1': 'ac9df2d66a4c8e4e8634a3fe39168f4f4683bc03', 'cluster': 'ceph'}}, {'ec_lost_unfound': None}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-05_12:54:40', 'tube': 'smithi', 'user': 'skanta', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-05T16:32:46.157 DEBUG:teuthology.orchestra.run.smithi087:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-05T16:32:46.190 DEBUG:teuthology.orchestra.run.smithi087:> sudo install -d -m0777 -- /var/run/ceph 2024-09-05T16:32:46.255 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-09-05T16:32:46.255 DEBUG:teuthology.orchestra.run.smithi087:> dd if=/scratch_devs of=/dev/stdout 2024-09-05T16:32:46.311 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-05T16:32:46.311 DEBUG:teuthology.orchestra.run.smithi087:> stat /dev/vg_nvme/lv_1 2024-09-05T16:32:46.366 INFO:teuthology.orchestra.run.smithi087.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-05T16:32:46.366 INFO:teuthology.orchestra.run.smithi087.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-05T16:32:46.366 INFO:teuthology.orchestra.run.smithi087.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-09-05T16:32:46.366 INFO:teuthology.orchestra.run.smithi087.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-05T16:32:46.367 INFO:teuthology.orchestra.run.smithi087.stdout:Context: system_u:object_r:device_t:s0 2024-09-05T16:32:46.367 INFO:teuthology.orchestra.run.smithi087.stdout:Access: 2024-09-05 16:32:42.077679144 +0000 2024-09-05T16:32:46.367 INFO:teuthology.orchestra.run.smithi087.stdout:Modify: 2024-09-05 16:30:12.025729337 +0000 2024-09-05T16:32:46.367 INFO:teuthology.orchestra.run.smithi087.stdout:Change: 2024-09-05 16:30:12.025729337 +0000 2024-09-05T16:32:46.367 INFO:teuthology.orchestra.run.smithi087.stdout: Birth: 2024-09-05 16:30:12.025729337 +0000 2024-09-05T16:32:46.367 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-05T16:32:46.429 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records in 2024-09-05T16:32:46.433 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records out 2024-09-05T16:32:46.434 INFO:teuthology.orchestra.run.smithi087.stderr:512 bytes copied, 0.000228012 s, 2.2 MB/s 2024-09-05T16:32:46.435 DEBUG:teuthology.orchestra.run.smithi087:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-05T16:32:46.491 DEBUG:teuthology.orchestra.run.smithi087:> stat /dev/vg_nvme/lv_2 2024-09-05T16:32:46.546 INFO:teuthology.orchestra.run.smithi087.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-05T16:32:46.547 INFO:teuthology.orchestra.run.smithi087.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-05T16:32:46.547 INFO:teuthology.orchestra.run.smithi087.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-09-05T16:32:46.547 INFO:teuthology.orchestra.run.smithi087.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-05T16:32:46.547 INFO:teuthology.orchestra.run.smithi087.stdout:Context: system_u:object_r:device_t:s0 2024-09-05T16:32:46.547 INFO:teuthology.orchestra.run.smithi087.stdout:Access: 2024-09-05 16:32:42.078679121 +0000 2024-09-05T16:32:46.547 INFO:teuthology.orchestra.run.smithi087.stdout:Modify: 2024-09-05 16:30:12.341721886 +0000 2024-09-05T16:32:46.547 INFO:teuthology.orchestra.run.smithi087.stdout:Change: 2024-09-05 16:30:12.341721886 +0000 2024-09-05T16:32:46.547 INFO:teuthology.orchestra.run.smithi087.stdout: Birth: 2024-09-05 16:30:12.341721886 +0000 2024-09-05T16:32:46.547 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-05T16:32:46.609 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records in 2024-09-05T16:32:46.609 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records out 2024-09-05T16:32:46.609 INFO:teuthology.orchestra.run.smithi087.stderr:512 bytes copied, 0.000184135 s, 2.8 MB/s 2024-09-05T16:32:46.610 DEBUG:teuthology.orchestra.run.smithi087:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-05T16:32:46.666 DEBUG:teuthology.orchestra.run.smithi087:> stat /dev/vg_nvme/lv_3 2024-09-05T16:32:46.721 INFO:teuthology.orchestra.run.smithi087.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-05T16:32:46.721 INFO:teuthology.orchestra.run.smithi087.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-05T16:32:46.721 INFO:teuthology.orchestra.run.smithi087.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-09-05T16:32:46.722 INFO:teuthology.orchestra.run.smithi087.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-05T16:32:46.722 INFO:teuthology.orchestra.run.smithi087.stdout:Context: system_u:object_r:device_t:s0 2024-09-05T16:32:46.722 INFO:teuthology.orchestra.run.smithi087.stdout:Access: 2024-09-05 16:32:42.078679121 +0000 2024-09-05T16:32:46.722 INFO:teuthology.orchestra.run.smithi087.stdout:Modify: 2024-09-05 16:30:12.669714152 +0000 2024-09-05T16:32:46.722 INFO:teuthology.orchestra.run.smithi087.stdout:Change: 2024-09-05 16:30:12.669714152 +0000 2024-09-05T16:32:46.722 INFO:teuthology.orchestra.run.smithi087.stdout: Birth: 2024-09-05 16:30:12.669714152 +0000 2024-09-05T16:32:46.722 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-05T16:32:46.784 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records in 2024-09-05T16:32:46.784 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records out 2024-09-05T16:32:46.785 INFO:teuthology.orchestra.run.smithi087.stderr:512 bytes copied, 0.000231565 s, 2.2 MB/s 2024-09-05T16:32:46.785 DEBUG:teuthology.orchestra.run.smithi087:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-05T16:32:46.843 DEBUG:teuthology.orchestra.run.smithi087:> stat /dev/vg_nvme/lv_4 2024-09-05T16:32:46.898 INFO:teuthology.orchestra.run.smithi087.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-05T16:32:46.898 INFO:teuthology.orchestra.run.smithi087.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-05T16:32:46.898 INFO:teuthology.orchestra.run.smithi087.stdout:Device: 5h/5d Inode: 721 Links: 1 2024-09-05T16:32:46.898 INFO:teuthology.orchestra.run.smithi087.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-05T16:32:46.898 INFO:teuthology.orchestra.run.smithi087.stdout:Context: system_u:object_r:device_t:s0 2024-09-05T16:32:46.899 INFO:teuthology.orchestra.run.smithi087.stdout:Access: 2024-09-05 16:32:42.078679121 +0000 2024-09-05T16:32:46.899 INFO:teuthology.orchestra.run.smithi087.stdout:Modify: 2024-09-05 16:30:12.991706560 +0000 2024-09-05T16:32:46.899 INFO:teuthology.orchestra.run.smithi087.stdout:Change: 2024-09-05 16:30:12.991706560 +0000 2024-09-05T16:32:46.899 INFO:teuthology.orchestra.run.smithi087.stdout: Birth: 2024-09-05 16:30:12.991706560 +0000 2024-09-05T16:32:46.899 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-05T16:32:46.961 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records in 2024-09-05T16:32:46.961 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records out 2024-09-05T16:32:46.961 INFO:teuthology.orchestra.run.smithi087.stderr:512 bytes copied, 0.000181338 s, 2.8 MB/s 2024-09-05T16:32:46.962 DEBUG:teuthology.orchestra.run.smithi087:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-05T16:32:47.018 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-05T16:32:47.018 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi087.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}} 2024-09-05T16:32:47.018 INFO:tasks.ceph:Generating config... 2024-09-05T16:32:47.020 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-09-05T16:32:47.020 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-09-05T16:32:47.020 INFO:tasks.ceph:[global] ms bind msgr1 = False 2024-09-05T16:32:47.020 INFO:tasks.ceph:[global] ms bind msgr2 = True 2024-09-05T16:32:47.020 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-09-05T16:32:47.020 INFO:tasks.ceph:[global] ms type = async 2024-09-05T16:32:47.020 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-05T16:32:47.020 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-05T16:32:47.020 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-05T16:32:47.020 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-05T16:32:47.020 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-05T16:32:47.020 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-09-05T16:32:47.020 INFO:tasks.ceph:[osd] bluefs allocator = hybrid 2024-09-05T16:32:47.021 INFO:tasks.ceph:[osd] bluestore allocator = hybrid 2024-09-05T16:32:47.021 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-09-05T16:32:47.021 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-09-05T16:32:47.021 INFO:tasks.ceph:[osd] bluestore write v2 random = True 2024-09-05T16:32:47.021 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-09-05T16:32:47.021 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-09-05T16:32:47.021 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-09-05T16:32:47.021 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-05T16:32:47.021 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-05T16:32:47.021 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-09-05T16:32:47.021 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-09-05T16:32:47.021 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-09-05T16:32:47.021 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-09-05T16:32:47.021 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-09-05T16:32:47.022 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-09-05T16:32:47.022 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-09-05T16:32:47.022 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-09-05T16:32:47.022 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-09-05T16:32:47.022 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-09-05T16:32:47.022 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-09-05T16:32:47.022 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-09-05T16:32:47.022 INFO:tasks.ceph:Setting up mon.a... 2024-09-05T16:32:47.022 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-09-05T16:32:47.104 INFO:teuthology.orchestra.run.smithi087.stdout:creating /etc/ceph/ceph.keyring 2024-09-05T16:32:47.106 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-09-05T16:32:47.148 DEBUG:teuthology.orchestra.run.smithi087:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-05T16:32:47.212 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.87'), ('mon.b', '[v2:172.21.15.87:3301,v1:172.21.15.87:6790]'), ('mon.c', '[v2:172.21.15.87:3302,v1:172.21.15.87:6791]')] 2024-09-05T16:32:47.213 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.87,[v2:172.21.15.87:3301,v1:172.21.15.87:6790],[v2:172.21.15.87:3302,v1:172.21.15.87:6791]', 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore write v2 random': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon scrub interval': 300}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}} 2024-09-05T16:32:47.214 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-09-05T16:32:47.214 DEBUG:teuthology.orchestra.run.smithi087:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-05T16:32:47.269 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.87 --addv b '[v2:172.21.15.87:3301,v1:172.21.15.87:6790]' --addv c '[v2:172.21.15.87:3302,v1:172.21.15.87:6791]' --print /home/ubuntu/cephtest/ceph.monmap 2024-09-05T16:32:47.356 INFO:teuthology.orchestra.run.smithi087.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-09-05T16:32:47.356 INFO:teuthology.orchestra.run.smithi087.stdout:monmaptool: generated fsid f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc 2024-09-05T16:32:47.356 INFO:teuthology.orchestra.run.smithi087.stdout:setting min_mon_release = quincy 2024-09-05T16:32:47.356 INFO:teuthology.orchestra.run.smithi087.stdout:epoch 0 2024-09-05T16:32:47.356 INFO:teuthology.orchestra.run.smithi087.stdout:fsid f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc 2024-09-05T16:32:47.356 INFO:teuthology.orchestra.run.smithi087.stdout:last_changed 2024-09-05T16:32:47.362483+0000 2024-09-05T16:32:47.357 INFO:teuthology.orchestra.run.smithi087.stdout:created 2024-09-05T16:32:47.362483+0000 2024-09-05T16:32:47.357 INFO:teuthology.orchestra.run.smithi087.stdout:min_mon_release 17 (quincy) 2024-09-05T16:32:47.357 INFO:teuthology.orchestra.run.smithi087.stdout:election_strategy: 1 2024-09-05T16:32:47.357 INFO:teuthology.orchestra.run.smithi087.stdout:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-09-05T16:32:47.357 INFO:teuthology.orchestra.run.smithi087.stdout:1: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.b 2024-09-05T16:32:47.357 INFO:teuthology.orchestra.run.smithi087.stdout:2: [v2:172.21.15.87:3302/0,v1:172.21.15.87:6791/0] mon.c 2024-09-05T16:32:47.357 INFO:teuthology.orchestra.run.smithi087.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-09-05T16:32:47.358 DEBUG:teuthology.orchestra.run.smithi087:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-05T16:32:47.373 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc... 2024-09-05T16:32:47.374 DEBUG:teuthology.orchestra.run.smithi087:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-05T16:32:47.456 INFO:teuthology.orchestra.run.smithi087.stdout:[global] 2024-09-05T16:32:47.456 INFO:teuthology.orchestra.run.smithi087.stdout: chdir = "" 2024-09-05T16:32:47.456 INFO:teuthology.orchestra.run.smithi087.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-05T16:32:47.456 INFO:teuthology.orchestra.run.smithi087.stdout: auth supported = cephx 2024-09-05T16:32:47.456 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:32:47.457 INFO:teuthology.orchestra.run.smithi087.stdout: filestore xattr use omap = true 2024-09-05T16:32:47.457 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:32:47.457 INFO:teuthology.orchestra.run.smithi087.stdout: mon clock drift allowed = 1.000 2024-09-05T16:32:47.457 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:32:47.457 INFO:teuthology.orchestra.run.smithi087.stdout: osd crush chooseleaf type = 0 2024-09-05T16:32:47.457 INFO:teuthology.orchestra.run.smithi087.stdout: auth debug = true 2024-09-05T16:32:47.457 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:32:47.457 INFO:teuthology.orchestra.run.smithi087.stdout: ms die on old message = true 2024-09-05T16:32:47.457 INFO:teuthology.orchestra.run.smithi087.stdout: ms die on bug = true 2024-09-05T16:32:47.457 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:32:47.457 INFO:teuthology.orchestra.run.smithi087.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-05T16:32:47.457 INFO:teuthology.orchestra.run.smithi087.stdout: mon pg warn max object skew = 0 2024-09-05T16:32:47.457 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:32:47.458 INFO:teuthology.orchestra.run.smithi087.stdout: # disable pg_autoscaler by default for new pools 2024-09-05T16:32:47.458 INFO:teuthology.orchestra.run.smithi087.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-05T16:32:47.458 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:32:47.458 INFO:teuthology.orchestra.run.smithi087.stdout: osd pool default size = 2 2024-09-05T16:32:47.458 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:32:47.458 INFO:teuthology.orchestra.run.smithi087.stdout: mon osd allow primary affinity = true 2024-09-05T16:32:47.458 INFO:teuthology.orchestra.run.smithi087.stdout: mon osd allow pg remap = true 2024-09-05T16:32:47.458 INFO:teuthology.orchestra.run.smithi087.stdout: mon warn on legacy crush tunables = false 2024-09-05T16:32:47.458 INFO:teuthology.orchestra.run.smithi087.stdout: mon warn on crush straw calc version zero = false 2024-09-05T16:32:47.458 INFO:teuthology.orchestra.run.smithi087.stdout: mon warn on no sortbitwise = false 2024-09-05T16:32:47.458 INFO:teuthology.orchestra.run.smithi087.stdout: mon warn on osd down out interval zero = false 2024-09-05T16:32:47.458 INFO:teuthology.orchestra.run.smithi087.stdout: mon warn on too few osds = false 2024-09-05T16:32:47.458 INFO:teuthology.orchestra.run.smithi087.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-05T16:32:47.458 INFO:teuthology.orchestra.run.smithi087.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-05T16:32:47.459 INFO:teuthology.orchestra.run.smithi087.stdout: mon_allow_pool_size_one = true 2024-09-05T16:32:47.459 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:32:47.459 INFO:teuthology.orchestra.run.smithi087.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-05T16:32:47.459 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:32:47.459 INFO:teuthology.orchestra.run.smithi087.stdout: osd default data pool replay window = 5 2024-09-05T16:32:47.459 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:32:47.459 INFO:teuthology.orchestra.run.smithi087.stdout: mon allow pool delete = true 2024-09-05T16:32:47.459 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:32:47.459 INFO:teuthology.orchestra.run.smithi087.stdout: mon cluster log file level = debug 2024-09-05T16:32:47.459 INFO:teuthology.orchestra.run.smithi087.stdout: debug asserts on shutdown = true 2024-09-05T16:32:47.459 INFO:teuthology.orchestra.run.smithi087.stdout: mon health detail to clog = false 2024-09-05T16:32:47.459 INFO:teuthology.orchestra.run.smithi087.stdout: mon host = "172.21.15.87,[v2:172.21.15.87:3301,v1:172.21.15.87:6790],[v2:172.21.15.87:3302,v1:172.21.15.87:6791]" 2024-09-05T16:32:47.459 INFO:teuthology.orchestra.run.smithi087.stdout: mon client directed command retry = 5 2024-09-05T16:32:47.459 INFO:teuthology.orchestra.run.smithi087.stdout: mon election default strategy = 1 2024-09-05T16:32:47.460 INFO:teuthology.orchestra.run.smithi087.stdout: ms bind msgr1 = False 2024-09-05T16:32:47.460 INFO:teuthology.orchestra.run.smithi087.stdout: ms bind msgr2 = True 2024-09-05T16:32:47.460 INFO:teuthology.orchestra.run.smithi087.stdout: ms inject socket failures = 5000 2024-09-05T16:32:47.460 INFO:teuthology.orchestra.run.smithi087.stdout: ms type = async 2024-09-05T16:32:47.460 INFO:teuthology.orchestra.run.smithi087.stdout: fsid = f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc 2024-09-05T16:32:47.460 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:32:47.460 INFO:teuthology.orchestra.run.smithi087.stdout:[osd] 2024-09-05T16:32:47.460 INFO:teuthology.orchestra.run.smithi087.stdout: osd journal size = 100 2024-09-05T16:32:47.460 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:32:47.460 INFO:teuthology.orchestra.run.smithi087.stdout: osd scrub load threshold = 5.0 2024-09-05T16:32:47.460 INFO:teuthology.orchestra.run.smithi087.stdout: osd scrub max interval = 600 2024-09-05T16:32:47.460 INFO:teuthology.orchestra.run.smithi087.stdout: osd mclock profile = high_recovery_ops 2024-09-05T16:32:47.460 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:32:47.460 INFO:teuthology.orchestra.run.smithi087.stdout: osd recover clone overlap = true 2024-09-05T16:32:47.461 INFO:teuthology.orchestra.run.smithi087.stdout: osd recovery max chunk = 1048576 2024-09-05T16:32:47.461 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:32:47.461 INFO:teuthology.orchestra.run.smithi087.stdout: osd debug shutdown = true 2024-09-05T16:32:47.461 INFO:teuthology.orchestra.run.smithi087.stdout: osd debug op order = true 2024-09-05T16:32:47.461 INFO:teuthology.orchestra.run.smithi087.stdout: osd debug verify stray on activate = true 2024-09-05T16:32:47.461 INFO:teuthology.orchestra.run.smithi087.stdout: osd debug trim objects = true 2024-09-05T16:32:47.461 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:32:47.461 INFO:teuthology.orchestra.run.smithi087.stdout: osd open classes on start = true 2024-09-05T16:32:47.461 INFO:teuthology.orchestra.run.smithi087.stdout: osd debug pg log writeout = true 2024-09-05T16:32:47.461 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:32:47.461 INFO:teuthology.orchestra.run.smithi087.stdout: osd deep scrub update digest min age = 30 2024-09-05T16:32:47.461 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:32:47.461 INFO:teuthology.orchestra.run.smithi087.stdout: osd map max advance = 10 2024-09-05T16:32:47.462 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:32:47.462 INFO:teuthology.orchestra.run.smithi087.stdout: journal zero on create = true 2024-09-05T16:32:47.462 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:32:47.462 INFO:teuthology.orchestra.run.smithi087.stdout: filestore ondisk finisher threads = 3 2024-09-05T16:32:47.462 INFO:teuthology.orchestra.run.smithi087.stdout: filestore apply finisher threads = 3 2024-09-05T16:32:47.462 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:32:47.462 INFO:teuthology.orchestra.run.smithi087.stdout: bdev debug aio = true 2024-09-05T16:32:47.462 INFO:teuthology.orchestra.run.smithi087.stdout: osd debug misdirected ops = true 2024-09-05T16:32:47.462 INFO:teuthology.orchestra.run.smithi087.stdout: bluefs allocator = hybrid 2024-09-05T16:32:47.462 INFO:teuthology.orchestra.run.smithi087.stdout: bluestore allocator = hybrid 2024-09-05T16:32:47.462 INFO:teuthology.orchestra.run.smithi087.stdout: bluestore block size = 96636764160 2024-09-05T16:32:47.462 INFO:teuthology.orchestra.run.smithi087.stdout: bluestore fsck on mount = True 2024-09-05T16:32:47.462 INFO:teuthology.orchestra.run.smithi087.stdout: bluestore write v2 random = True 2024-09-05T16:32:47.462 INFO:teuthology.orchestra.run.smithi087.stdout: bluestore zero block detection = True 2024-09-05T16:32:47.462 INFO:teuthology.orchestra.run.smithi087.stdout: debug bluefs = 20 2024-09-05T16:32:47.463 INFO:teuthology.orchestra.run.smithi087.stdout: debug bluestore = 20 2024-09-05T16:32:47.463 INFO:teuthology.orchestra.run.smithi087.stdout: debug ms = 1 2024-09-05T16:32:47.463 INFO:teuthology.orchestra.run.smithi087.stdout: debug osd = 20 2024-09-05T16:32:47.463 INFO:teuthology.orchestra.run.smithi087.stdout: debug rocksdb = 10 2024-09-05T16:32:47.463 INFO:teuthology.orchestra.run.smithi087.stdout: mon osd backfillfull_ratio = 0.85 2024-09-05T16:32:47.463 INFO:teuthology.orchestra.run.smithi087.stdout: mon osd full ratio = 0.9 2024-09-05T16:32:47.463 INFO:teuthology.orchestra.run.smithi087.stdout: mon osd nearfull ratio = 0.8 2024-09-05T16:32:47.463 INFO:teuthology.orchestra.run.smithi087.stdout: osd debug verify cached snaps = True 2024-09-05T16:32:47.463 INFO:teuthology.orchestra.run.smithi087.stdout: osd debug verify missing on start = True 2024-09-05T16:32:47.463 INFO:teuthology.orchestra.run.smithi087.stdout: osd failsafe full ratio = 0.95 2024-09-05T16:32:47.463 INFO:teuthology.orchestra.run.smithi087.stdout: osd mclock override recovery settings = True 2024-09-05T16:32:47.463 INFO:teuthology.orchestra.run.smithi087.stdout: osd objectstore = bluestore 2024-09-05T16:32:47.463 INFO:teuthology.orchestra.run.smithi087.stdout: osd op queue = debug_random 2024-09-05T16:32:47.463 INFO:teuthology.orchestra.run.smithi087.stdout: osd op queue cut off = debug_random 2024-09-05T16:32:47.463 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:32:47.464 INFO:teuthology.orchestra.run.smithi087.stdout:[mgr] 2024-09-05T16:32:47.464 INFO:teuthology.orchestra.run.smithi087.stdout: debug ms = 1 2024-09-05T16:32:47.464 INFO:teuthology.orchestra.run.smithi087.stdout: debug mgr = 20 2024-09-05T16:32:47.464 INFO:teuthology.orchestra.run.smithi087.stdout: debug mon = 20 2024-09-05T16:32:47.464 INFO:teuthology.orchestra.run.smithi087.stdout: debug auth = 20 2024-09-05T16:32:47.464 INFO:teuthology.orchestra.run.smithi087.stdout: mon reweight min pgs per osd = 4 2024-09-05T16:32:47.464 INFO:teuthology.orchestra.run.smithi087.stdout: mon reweight min bytes per osd = 10 2024-09-05T16:32:47.464 INFO:teuthology.orchestra.run.smithi087.stdout: mgr/telemetry/nag = false 2024-09-05T16:32:47.464 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:32:47.464 INFO:teuthology.orchestra.run.smithi087.stdout:[mon] 2024-09-05T16:32:47.464 INFO:teuthology.orchestra.run.smithi087.stdout: debug ms = 1 2024-09-05T16:32:47.464 INFO:teuthology.orchestra.run.smithi087.stdout: debug mon = 20 2024-09-05T16:32:47.464 INFO:teuthology.orchestra.run.smithi087.stdout: debug paxos = 20 2024-09-05T16:32:47.465 INFO:teuthology.orchestra.run.smithi087.stdout: debug auth = 20 2024-09-05T16:32:47.465 INFO:teuthology.orchestra.run.smithi087.stdout: mon data avail warn = 5 2024-09-05T16:32:47.465 INFO:teuthology.orchestra.run.smithi087.stdout: mon mgr mkfs grace = 240 2024-09-05T16:32:47.465 INFO:teuthology.orchestra.run.smithi087.stdout: mon reweight min pgs per osd = 4 2024-09-05T16:32:47.465 INFO:teuthology.orchestra.run.smithi087.stdout: mon osd reporter subtree level = osd 2024-09-05T16:32:47.465 INFO:teuthology.orchestra.run.smithi087.stdout: mon osd prime pg temp = true 2024-09-05T16:32:47.465 INFO:teuthology.orchestra.run.smithi087.stdout: mon reweight min bytes per osd = 10 2024-09-05T16:32:47.465 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:32:47.465 INFO:teuthology.orchestra.run.smithi087.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-05T16:32:47.465 INFO:teuthology.orchestra.run.smithi087.stdout: auth mon ticket ttl = 660 # 11m 2024-09-05T16:32:47.465 INFO:teuthology.orchestra.run.smithi087.stdout: auth service ticket ttl = 240 # 4m 2024-09-05T16:32:47.465 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:32:47.465 INFO:teuthology.orchestra.run.smithi087.stdout: # don't complain about insecure global_id in the test suite 2024-09-05T16:32:47.465 INFO:teuthology.orchestra.run.smithi087.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-05T16:32:47.466 INFO:teuthology.orchestra.run.smithi087.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-05T16:32:47.466 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:32:47.466 INFO:teuthology.orchestra.run.smithi087.stdout: # 1m isn't quite enough 2024-09-05T16:32:47.466 INFO:teuthology.orchestra.run.smithi087.stdout: mon_down_mkfs_grace = 2m 2024-09-05T16:32:47.466 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:32:47.466 INFO:teuthology.orchestra.run.smithi087.stdout: mon_warn_on_filestore_osds = false 2024-09-05T16:32:47.466 INFO:teuthology.orchestra.run.smithi087.stdout: mon scrub interval = 300 2024-09-05T16:32:47.466 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:32:47.466 INFO:teuthology.orchestra.run.smithi087.stdout:[client] 2024-09-05T16:32:47.466 INFO:teuthology.orchestra.run.smithi087.stdout: rgw cache enabled = true 2024-09-05T16:32:47.466 INFO:teuthology.orchestra.run.smithi087.stdout: rgw enable ops log = true 2024-09-05T16:32:47.466 INFO:teuthology.orchestra.run.smithi087.stdout: rgw enable usage log = true 2024-09-05T16:32:47.466 INFO:teuthology.orchestra.run.smithi087.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-05T16:32:47.466 INFO:teuthology.orchestra.run.smithi087.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-05T16:32:47.467 INFO:teuthology.orchestra.run.smithi087.stdout:[mon.a] 2024-09-05T16:32:47.467 INFO:teuthology.orchestra.run.smithi087.stdout:[mon.b] 2024-09-05T16:32:47.467 INFO:teuthology.orchestra.run.smithi087.stdout:[mon.c] 2024-09-05T16:32:47.468 INFO:tasks.ceph:Creating admin key on mon.a... 2024-09-05T16:32:47.468 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-09-05T16:32:47.563 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-05T16:32:47.563 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-09-05T16:32:47.563 DEBUG:teuthology.orchestra.run.smithi087:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-09-05T16:32:47.578 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-09-05T16:32:47.578 DEBUG:teuthology.orchestra.run.smithi087:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-09-05T16:32:47.633 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi087.front.sepia.ceph.com 2024-09-05T16:32:47.633 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-09-05T16:32:47.633 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-05T16:32:47.633 DEBUG:teuthology.orchestra.run.smithi087:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-05T16:32:47.707 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-09-05T16:32:47.707 DEBUG:teuthology.orchestra.run.smithi087:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-05T16:32:47.763 INFO:tasks.ceph:Setting up mon nodes... 2024-09-05T16:32:47.763 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-05T16:32:47.763 DEBUG:teuthology.orchestra.run.smithi087:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-09-05T16:32:47.853 INFO:teuthology.orchestra.run.smithi087.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-09-05T16:32:47.856 INFO:tasks.ceph:Setting up mds nodes... 2024-09-05T16:32:47.856 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-05T16:32:47.856 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-05T16:32:47.856 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi087.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}} 2024-09-05T16:32:47.857 DEBUG:teuthology.orchestra.run.smithi087:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-09-05T16:32:47.881 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-05T16:32:47.881 INFO:tasks.ceph:role: osd.0 2024-09-05T16:32:47.881 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi087.front.sepia.ceph.com 2024-09-05T16:32:47.882 DEBUG:teuthology.orchestra.run.smithi087:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-05T16:32:47.953 INFO:teuthology.orchestra.run.smithi087.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-05T16:32:47.953 INFO:teuthology.orchestra.run.smithi087.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-05T16:32:47.953 INFO:teuthology.orchestra.run.smithi087.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-05T16:32:47.953 INFO:teuthology.orchestra.run.smithi087.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-05T16:32:47.954 INFO:teuthology.orchestra.run.smithi087.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-05T16:32:47.954 INFO:teuthology.orchestra.run.smithi087.stdout: = sunit=0 swidth=0 blks 2024-09-05T16:32:47.954 INFO:teuthology.orchestra.run.smithi087.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-05T16:32:47.954 INFO:teuthology.orchestra.run.smithi087.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-05T16:32:47.954 INFO:teuthology.orchestra.run.smithi087.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-05T16:32:47.954 INFO:teuthology.orchestra.run.smithi087.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-05T16:32:48.478 INFO:teuthology.orchestra.run.smithi087.stdout:Discarding blocks...Done. 2024-09-05T16:32:48.480 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi087.front.sepia.ceph.com -o noatime 2024-09-05T16:32:48.480 DEBUG:teuthology.orchestra.run.smithi087:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-09-05T16:32:48.587 DEBUG:teuthology.orchestra.run.smithi087:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-09-05T16:32:48.618 DEBUG:teuthology.orchestra.run.smithi087:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-09-05T16:32:48.683 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-05T16:32:48.683 INFO:tasks.ceph:role: osd.1 2024-09-05T16:32:48.684 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi087.front.sepia.ceph.com 2024-09-05T16:32:48.684 DEBUG:teuthology.orchestra.run.smithi087:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-05T16:32:48.757 INFO:teuthology.orchestra.run.smithi087.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-05T16:32:48.757 INFO:teuthology.orchestra.run.smithi087.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-05T16:32:48.757 INFO:teuthology.orchestra.run.smithi087.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-05T16:32:48.757 INFO:teuthology.orchestra.run.smithi087.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-05T16:32:48.757 INFO:teuthology.orchestra.run.smithi087.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-05T16:32:48.757 INFO:teuthology.orchestra.run.smithi087.stdout: = sunit=0 swidth=0 blks 2024-09-05T16:32:48.757 INFO:teuthology.orchestra.run.smithi087.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-05T16:32:48.757 INFO:teuthology.orchestra.run.smithi087.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-05T16:32:48.757 INFO:teuthology.orchestra.run.smithi087.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-05T16:32:48.757 INFO:teuthology.orchestra.run.smithi087.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-05T16:32:49.226 INFO:teuthology.orchestra.run.smithi087.stdout:Discarding blocks...Done. 2024-09-05T16:32:49.228 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi087.front.sepia.ceph.com -o noatime 2024-09-05T16:32:49.228 DEBUG:teuthology.orchestra.run.smithi087:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-09-05T16:32:49.334 DEBUG:teuthology.orchestra.run.smithi087:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-09-05T16:32:49.363 DEBUG:teuthology.orchestra.run.smithi087:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-09-05T16:32:49.429 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-05T16:32:49.429 INFO:tasks.ceph:role: osd.2 2024-09-05T16:32:49.429 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi087.front.sepia.ceph.com 2024-09-05T16:32:49.429 DEBUG:teuthology.orchestra.run.smithi087:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-05T16:32:49.501 INFO:teuthology.orchestra.run.smithi087.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-05T16:32:49.502 INFO:teuthology.orchestra.run.smithi087.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-05T16:32:49.502 INFO:teuthology.orchestra.run.smithi087.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-05T16:32:49.502 INFO:teuthology.orchestra.run.smithi087.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-05T16:32:49.502 INFO:teuthology.orchestra.run.smithi087.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-05T16:32:49.502 INFO:teuthology.orchestra.run.smithi087.stdout: = sunit=0 swidth=0 blks 2024-09-05T16:32:49.502 INFO:teuthology.orchestra.run.smithi087.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-05T16:32:49.502 INFO:teuthology.orchestra.run.smithi087.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-05T16:32:49.502 INFO:teuthology.orchestra.run.smithi087.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-05T16:32:49.502 INFO:teuthology.orchestra.run.smithi087.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-05T16:32:50.026 INFO:teuthology.orchestra.run.smithi087.stdout:Discarding blocks...Done. 2024-09-05T16:32:50.029 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi087.front.sepia.ceph.com -o noatime 2024-09-05T16:32:50.029 DEBUG:teuthology.orchestra.run.smithi087:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-09-05T16:32:50.173 DEBUG:teuthology.orchestra.run.smithi087:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-09-05T16:32:50.203 DEBUG:teuthology.orchestra.run.smithi087:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-09-05T16:32:50.268 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-05T16:32:50.268 INFO:tasks.ceph:role: osd.3 2024-09-05T16:32:50.268 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi087.front.sepia.ceph.com 2024-09-05T16:32:50.268 DEBUG:teuthology.orchestra.run.smithi087:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-09-05T16:32:50.340 INFO:teuthology.orchestra.run.smithi087.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-09-05T16:32:50.340 INFO:teuthology.orchestra.run.smithi087.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-05T16:32:50.340 INFO:teuthology.orchestra.run.smithi087.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-05T16:32:50.340 INFO:teuthology.orchestra.run.smithi087.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-05T16:32:50.340 INFO:teuthology.orchestra.run.smithi087.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-05T16:32:50.340 INFO:teuthology.orchestra.run.smithi087.stdout: = sunit=0 swidth=0 blks 2024-09-05T16:32:50.340 INFO:teuthology.orchestra.run.smithi087.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-05T16:32:50.340 INFO:teuthology.orchestra.run.smithi087.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-05T16:32:50.340 INFO:teuthology.orchestra.run.smithi087.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-05T16:32:50.340 INFO:teuthology.orchestra.run.smithi087.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-05T16:32:50.854 INFO:teuthology.orchestra.run.smithi087.stdout:Discarding blocks...Done. 2024-09-05T16:32:50.856 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi087.front.sepia.ceph.com -o noatime 2024-09-05T16:32:50.856 DEBUG:teuthology.orchestra.run.smithi087:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-09-05T16:32:50.979 DEBUG:teuthology.orchestra.run.smithi087:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-09-05T16:32:51.011 DEBUG:teuthology.orchestra.run.smithi087:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-05T16:32:51.109 INFO:teuthology.orchestra.run.smithi087.stderr:2024-09-05T16:32:51.114+0000 7f143ad41640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-09-05T16:32:51.109 INFO:teuthology.orchestra.run.smithi087.stderr:2024-09-05T16:32:51.115+0000 7f143ad41640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-09-05T16:32:51.109 INFO:teuthology.orchestra.run.smithi087.stderr:2024-09-05T16:32:51.115+0000 7f143ad41640 -1 bdev(0x558f04761800 /var/lib/ceph/osd/ceph-0/block) open stat got: (1) Operation not permitted 2024-09-05T16:32:51.110 INFO:teuthology.orchestra.run.smithi087.stderr:2024-09-05T16:32:51.115+0000 7f143ad41640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-09-05T16:32:54.026 DEBUG:teuthology.orchestra.run.smithi087:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-05T16:32:54.054 DEBUG:teuthology.orchestra.run.smithi087:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-05T16:32:54.143 INFO:teuthology.orchestra.run.smithi087.stderr:2024-09-05T16:32:54.148+0000 7f6e78831640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-09-05T16:32:54.143 INFO:teuthology.orchestra.run.smithi087.stderr:2024-09-05T16:32:54.148+0000 7f6e78831640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-09-05T16:32:54.143 INFO:teuthology.orchestra.run.smithi087.stderr:2024-09-05T16:32:54.148+0000 7f6e78831640 -1 bdev(0x5583648f1800 /var/lib/ceph/osd/ceph-1/block) open stat got: (1) Operation not permitted 2024-09-05T16:32:54.143 INFO:teuthology.orchestra.run.smithi087.stderr:2024-09-05T16:32:54.148+0000 7f6e78831640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-09-05T16:32:57.794 DEBUG:teuthology.orchestra.run.smithi087:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-05T16:32:57.823 DEBUG:teuthology.orchestra.run.smithi087:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-05T16:32:57.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-09-05T16:32:57.918+0000 7fd383bcc640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-09-05T16:32:57.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-09-05T16:32:57.919+0000 7fd383bcc640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-09-05T16:32:57.914 INFO:teuthology.orchestra.run.smithi087.stderr:2024-09-05T16:32:57.919+0000 7fd383bcc640 -1 bdev(0x55f7a3e93800 /var/lib/ceph/osd/ceph-2/block) open stat got: (1) Operation not permitted 2024-09-05T16:32:57.914 INFO:teuthology.orchestra.run.smithi087.stderr:2024-09-05T16:32:57.919+0000 7fd383bcc640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-09-05T16:33:00.825 DEBUG:teuthology.orchestra.run.smithi087:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-05T16:33:00.852 DEBUG:teuthology.orchestra.run.smithi087:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-05T16:33:00.939 INFO:teuthology.orchestra.run.smithi087.stderr:2024-09-05T16:33:00.945+0000 7f8bc21c1640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-09-05T16:33:00.940 INFO:teuthology.orchestra.run.smithi087.stderr:2024-09-05T16:33:00.945+0000 7f8bc21c1640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-09-05T16:33:00.940 INFO:teuthology.orchestra.run.smithi087.stderr:2024-09-05T16:33:00.945+0000 7f8bc21c1640 -1 bdev(0x55af4bc95800 /var/lib/ceph/osd/ceph-3/block) open stat got: (1) Operation not permitted 2024-09-05T16:33:00.940 INFO:teuthology.orchestra.run.smithi087.stderr:2024-09-05T16:33:00.945+0000 7f8bc21c1640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-09-05T16:33:04.112 DEBUG:teuthology.orchestra.run.smithi087:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-05T16:33:04.142 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-05T16:33:04.142 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-09-05T16:33:04.142 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-09-05T16:33:04.210 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-09-05T16:33:04.210 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-09-05T16:33:04.277 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-09-05T16:33:04.277 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-09-05T16:33:04.343 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-09-05T16:33:04.343 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-09-05T16:33:04.409 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-09-05T16:33:04.409 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-09-05T16:33:04.474 INFO:tasks.ceph:Adding keys to all mons... 2024-09-05T16:33:04.475 DEBUG:teuthology.orchestra.run.smithi087:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-05T16:33:04.538 INFO:teuthology.orchestra.run.smithi087.stdout:[mgr.x] 2024-09-05T16:33:04.538 INFO:teuthology.orchestra.run.smithi087.stdout: key = AQAv3dlmZfxGMxAAkijBdZQ4HviOzR6tNh1erw== 2024-09-05T16:33:04.538 INFO:teuthology.orchestra.run.smithi087.stdout:[osd.0] 2024-09-05T16:33:04.539 INFO:teuthology.orchestra.run.smithi087.stdout: key = AQAz3dlm06rlBhAATDq814U5IGbaItGrL3apww== 2024-09-05T16:33:04.539 INFO:teuthology.orchestra.run.smithi087.stdout:[osd.1] 2024-09-05T16:33:04.539 INFO:teuthology.orchestra.run.smithi087.stdout: key = AQA23dlmd7rmCBAABIJAmQ/mQ9qC5cbMTHZEzA== 2024-09-05T16:33:04.539 INFO:teuthology.orchestra.run.smithi087.stdout:[osd.2] 2024-09-05T16:33:04.539 INFO:teuthology.orchestra.run.smithi087.stdout: key = AQA53dlmWg7RNhAAkUnvZWca32nhJV1vVhQSpg== 2024-09-05T16:33:04.539 INFO:teuthology.orchestra.run.smithi087.stdout:[osd.3] 2024-09-05T16:33:04.540 INFO:teuthology.orchestra.run.smithi087.stdout: key = AQA83dlmuVhlOBAAbrSqeNqD8AoRJEB9cADSYg== 2024-09-05T16:33:04.541 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-05T16:33:04.633 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-05T16:33:04.683 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-05T16:33:04.734 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-05T16:33:04.786 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-05T16:33:04.838 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-05T16:33:04.838 DEBUG:teuthology.orchestra.run.smithi087:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-09-05T16:33:04.863 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-05T16:33:04.960 DEBUG:teuthology.orchestra.run.smithi087:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-09-05T16:33:04.985 DEBUG:teuthology.orchestra.run.smithi087:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-09-05T16:33:05.050 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-05T16:33:05.146 DEBUG:teuthology.orchestra.run.smithi087:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-09-05T16:33:05.170 DEBUG:teuthology.orchestra.run.smithi087:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-09-05T16:33:05.234 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-05T16:33:05.331 DEBUG:teuthology.orchestra.run.smithi087:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-09-05T16:33:05.355 DEBUG:teuthology.orchestra.run.smithi087:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-05T16:33:05.410 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-09-05T16:33:05.411 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-05T16:33:05.411 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-05T16:33:05.453 INFO:tasks.ceph.mon.a:Started 2024-09-05T16:33:05.453 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-05T16:33:05.453 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-05T16:33:05.455 INFO:tasks.ceph.mon.b:Started 2024-09-05T16:33:05.455 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-05T16:33:05.456 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-05T16:33:05.457 INFO:tasks.ceph.mon.c:Started 2024-09-05T16:33:05.458 DEBUG:teuthology.orchestra.run.smithi087:> sudo ceph config set mgr mgr_pool false --force 2024-09-05T16:33:16.055 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-09-05T16:33:16.055 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-09-05T16:33:16.055 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-09-05T16:33:16.058 INFO:tasks.ceph.mgr.x:Started 2024-09-05T16:33:16.058 DEBUG:tasks.ceph:set 0 configs 2024-09-05T16:33:16.058 INFO:tasks.ceph:Setting crush tunables to default 2024-09-05T16:33:16.058 DEBUG:teuthology.orchestra.run.smithi087:> sudo ceph --cluster ceph osd crush tunables default 2024-09-05T16:33:16.232 INFO:teuthology.orchestra.run.smithi087.stderr:adjusted tunables profile to default 2024-09-05T16:33:16.243 INFO:tasks.ceph:check_enable_crimson: False 2024-09-05T16:33:16.243 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-09-05T16:33:16.243 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-09-05T16:33:16.243 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-09-05T16:33:16.268 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-09-05T16:33:16.268 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-09-05T16:33:16.332 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-09-05T16:33:16.332 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-09-05T16:33:16.396 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-09-05T16:33:16.397 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-09-05T16:33:16.462 DEBUG:teuthology.orchestra.run.smithi087:> sudo ceph --cluster ceph osd new f76bac77-d755-450f-9fbf-fb53a6033c24 0 2024-09-05T16:33:16.676 INFO:teuthology.orchestra.run.smithi087.stdout:0 2024-09-05T16:33:16.687 DEBUG:teuthology.orchestra.run.smithi087:> sudo ceph --cluster ceph osd new 941684ea-0764-4182-94cc-136349df4f23 1 2024-09-05T16:33:20.632 INFO:teuthology.orchestra.run.smithi087.stdout:1 2024-09-05T16:33:20.648 DEBUG:teuthology.orchestra.run.smithi087:> sudo ceph --cluster ceph osd new 17135934-ecb7-445a-b1dc-27f55da9f8ca 2 2024-09-05T16:33:20.826 INFO:teuthology.orchestra.run.smithi087.stdout:2 2024-09-05T16:33:20.837 DEBUG:teuthology.orchestra.run.smithi087:> sudo ceph --cluster ceph osd new 098f9cbd-3da5-4ee7-8439-610b8cae774e 3 2024-09-05T16:33:21.017 INFO:teuthology.orchestra.run.smithi087.stdout:3 2024-09-05T16:33:21.028 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-05T16:33:21.028 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-09-05T16:33:21.030 INFO:tasks.ceph.osd.0:Started 2024-09-05T16:33:21.031 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-05T16:33:21.031 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-09-05T16:33:21.033 INFO:tasks.ceph.osd.1:Started 2024-09-05T16:33:21.033 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-05T16:33:21.033 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-09-05T16:33:21.035 INFO:tasks.ceph.osd.2:Started 2024-09-05T16:33:21.035 INFO:tasks.ceph.osd.3:Restarting daemon 2024-09-05T16:33:21.035 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-09-05T16:33:21.037 INFO:tasks.ceph.osd.3:Started 2024-09-05T16:33:21.038 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-05T16:33:21.243 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:33:21.243 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":6,"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","created":"2024-09-05T16:33:15.770967+0000","modified":"2024-09-05T16:33:21.020289+0000","last_up_change":"0.000000","last_in_change":"2024-09-05T16:33:21.020289+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f76bac77-d755-450f-9fbf-fb53a6033c24","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"941684ea-0764-4182-94cc-136349df4f23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"17135934-ecb7-445a-b1dc-27f55da9f8ca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"098f9cbd-3da5-4ee7-8439-610b8cae774e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-05T16:33:21.254 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-09-05T16:33:21.254 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-05T16:33:21.658 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-05T16:33:21.816 INFO:teuthology.misc.health.smithi087.stdout: 2024-09-05T16:33:21.816 INFO:teuthology.misc.health.smithi087.stdout:{"epoch":6,"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","created":"2024-09-05T16:33:15.770967+0000","modified":"2024-09-05T16:33:21.020289+0000","last_up_change":"0.000000","last_in_change":"2024-09-05T16:33:21.020289+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f76bac77-d755-450f-9fbf-fb53a6033c24","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"941684ea-0764-4182-94cc-136349df4f23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"17135934-ecb7-445a-b1dc-27f55da9f8ca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"098f9cbd-3da5-4ee7-8439-610b8cae774e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-05T16:33:21.826 DEBUG:teuthology.misc:0 of 4 OSDs are up 2024-09-05T16:33:22.505 INFO:tasks.ceph.osd.2.smithi087.stderr:2024-09-05T16:33:22.510+0000 7f1de8a11640 -1 Falling back to public interface 2024-09-05T16:33:22.741 INFO:tasks.ceph.osd.1.smithi087.stderr:2024-09-05T16:33:22.745+0000 7fd4fb169640 -1 Falling back to public interface 2024-09-05T16:33:22.741 INFO:tasks.ceph.osd.3.smithi087.stderr:2024-09-05T16:33:22.745+0000 7f4032a1c640 -1 Falling back to public interface 2024-09-05T16:33:22.748 INFO:tasks.ceph.osd.0.smithi087.stderr:2024-09-05T16:33:22.753+0000 7f2b71c0e640 -1 Falling back to public interface 2024-09-05T16:33:25.776 INFO:tasks.ceph.mgr.x.smithi087.stderr:2024-09-05T16:33:25.782+0000 7fb355ac6640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-09-05T16:33:25.778 INFO:tasks.ceph.mgr.x.smithi087.stderr:2024-09-05T16:33:25.784+0000 7fb355ac6640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-09-05T16:33:26.542 INFO:tasks.ceph.osd.0.smithi087.stderr:2024-09-05T16:33:26.548+0000 7f2b71c0e640 -1 osd.0 0 log_to_monitors true 2024-09-05T16:33:26.800 INFO:tasks.ceph.osd.1.smithi087.stderr:2024-09-05T16:33:26.806+0000 7fd4fb169640 -1 osd.1 0 log_to_monitors true 2024-09-05T16:33:27.078 INFO:tasks.ceph.osd.2.smithi087.stderr:2024-09-05T16:33:27.084+0000 7f1de8a11640 -1 osd.2 0 log_to_monitors true 2024-09-05T16:33:27.083 INFO:tasks.ceph.osd.3.smithi087.stderr:2024-09-05T16:33:27.089+0000 7f4032a1c640 -1 osd.3 0 log_to_monitors true 2024-09-05T16:33:27.792 INFO:tasks.ceph.osd.0.smithi087.stderr:2024-09-05T16:33:27.797+0000 7f2b6d486640 -1 osd.0 0 waiting for initial osdmap 2024-09-05T16:33:27.794 INFO:tasks.ceph.osd.0.smithi087.stderr:2024-09-05T16:33:27.800+0000 7f2b692ad640 -1 osd.0 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-05T16:33:28.229 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-05T16:33:28.501 INFO:tasks.ceph.osd.0.smithi087.stderr:2024-09-05T16:33:28.506+0000 7f2b64ea0640 -1 osd.0 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-05T16:33:28.515 INFO:teuthology.misc.health.smithi087.stdout: 2024-09-05T16:33:28.515 INFO:teuthology.misc.health.smithi087.stdout:{"epoch":8,"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","created":"2024-09-05T16:33:15.770967+0000","modified":"2024-09-05T16:33:27.782329+0000","last_up_change":"0.000000","last_in_change":"2024-09-05T16:33:21.020289+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f76bac77-d755-450f-9fbf-fb53a6033c24","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"941684ea-0764-4182-94cc-136349df4f23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"17135934-ecb7-445a-b1dc-27f55da9f8ca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"098f9cbd-3da5-4ee7-8439-610b8cae774e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-05T16:33:28.526 DEBUG:teuthology.misc:0 of 4 OSDs are up 2024-09-05T16:33:28.794 INFO:tasks.ceph.osd.2.smithi087.stderr:2024-09-05T16:33:28.800+0000 7f1de4392640 -1 osd.2 0 waiting for initial osdmap 2024-09-05T16:33:28.957 INFO:tasks.ceph.osd.2.smithi087.stderr:2024-09-05T16:33:28.962+0000 7f1de01b9640 -1 osd.2 9 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-05T16:33:30.007 INFO:tasks.ceph.osd.1.smithi087.stderr:2024-09-05T16:33:30.013+0000 7fd4f790c640 -1 osd.1 0 waiting for initial osdmap 2024-09-05T16:33:30.011 INFO:tasks.ceph.osd.1.smithi087.stderr:2024-09-05T16:33:30.017+0000 7fd4eef17640 -1 osd.1 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-05T16:33:30.052 INFO:tasks.ceph.osd.3.smithi087.stderr:2024-09-05T16:33:30.057+0000 7f402eb21640 -1 osd.3 0 waiting for initial osdmap 2024-09-05T16:33:30.055 INFO:tasks.ceph.osd.3.smithi087.stderr:2024-09-05T16:33:30.061+0000 7f402612c640 -1 osd.3 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-05T16:33:34.929 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-05T16:33:35.213 INFO:teuthology.misc.health.smithi087.stdout: 2024-09-05T16:33:35.213 INFO:teuthology.misc.health.smithi087.stdout:{"epoch":12,"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","created":"2024-09-05T16:33:15.770967+0000","modified":"2024-09-05T16:33:32.060239+0000","last_up_change":"2024-09-05T16:33:31.016726+0000","last_in_change":"2024-09-05T16:33:21.020289+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f76bac77-d755-450f-9fbf-fb53a6033c24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6812","nonce":2286506651}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6813","nonce":2286506651}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6815","nonce":2286506651}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6814","nonce":2286506651}]},"public_addr":"172.21.15.87:6812/2286506651","cluster_addr":"172.21.15.87:6813/2286506651","heartbeat_back_addr":"172.21.15.87:6815/2286506651","heartbeat_front_addr":"172.21.15.87:6814/2286506651","state":["exists","up"]},{"osd":1,"uuid":"941684ea-0764-4182-94cc-136349df4f23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6804","nonce":3459114360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6805","nonce":3459114360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6809","nonce":3459114360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6807","nonce":3459114360}]},"public_addr":"172.21.15.87:6804/3459114360","cluster_addr":"172.21.15.87:6805/3459114360","heartbeat_back_addr":"172.21.15.87:6809/3459114360","heartbeat_front_addr":"172.21.15.87:6807/3459114360","state":["exists","up"]},{"osd":2,"uuid":"17135934-ecb7-445a-b1dc-27f55da9f8ca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6800","nonce":4234108552}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6801","nonce":4234108552}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6803","nonce":4234108552}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6802","nonce":4234108552}]},"public_addr":"172.21.15.87:6800/4234108552","cluster_addr":"172.21.15.87:6801/4234108552","heartbeat_back_addr":"172.21.15.87:6803/4234108552","heartbeat_front_addr":"172.21.15.87:6802/4234108552","state":["exists","up"]},{"osd":3,"uuid":"098f9cbd-3da5-4ee7-8439-610b8cae774e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6806","nonce":3527623826}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6808","nonce":3527623826}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6811","nonce":3527623826}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6810","nonce":3527623826}]},"public_addr":"172.21.15.87:6806/3527623826","cluster_addr":"172.21.15.87:6808/3527623826","heartbeat_back_addr":"172.21.15.87:6811/3527623826","heartbeat_front_addr":"172.21.15.87:6810/3527623826","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-05T16:33:27.513154+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-05T16:33:27.800689+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-05T16:33:28.127983+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-05T16:33:28.046083+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-05T16:33:35.224 DEBUG:teuthology.misc:4 of 4 OSDs are up 2024-09-05T16:33:35.224 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-09-05T16:33:35.225 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config log 1 --format=json 2024-09-05T16:33:35.225 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-05T16:33:35.509 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:33:35.521 INFO:teuthology.orchestra.run.smithi087.stdout:[{"version":4,"timestamp":"2024-09-05T16:33:30.058666+0000","name":"","changes":[{"name":"osd.3/osd_mclock_max_capacity_iops_ssd","new_value":"6648.999978"}]}] 2024-09-05T16:33:35.521 INFO:tasks.ceph_manager:config epoch is 4 2024-09-05T16:33:35.522 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-05T16:33:35.522 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-09-05T16:33:35.522 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-09-05T16:33:35.852 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:33:35.863 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":4,"flags":0,"active_gid":24099,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6816","nonce":1489638964}]},"active_addr":"172.21.15.87:6816/1489638964","active_change":"2024-09-05T16:33:24.771722+0000","active_mgr_features":4540719139924082687,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:1.26.1","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.87:0","nonce":623774439}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.87:0","nonce":381785353}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.87:0","nonce":118693329}]}]} 2024-09-05T16:33:35.868 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-09-05T16:33:35.868 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-05T16:33:35.868 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-05T16:33:36.151 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:33:36.151 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":12,"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","created":"2024-09-05T16:33:15.770967+0000","modified":"2024-09-05T16:33:32.060239+0000","last_up_change":"2024-09-05T16:33:31.016726+0000","last_in_change":"2024-09-05T16:33:21.020289+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f76bac77-d755-450f-9fbf-fb53a6033c24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6812","nonce":2286506651}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6813","nonce":2286506651}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6815","nonce":2286506651}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6814","nonce":2286506651}]},"public_addr":"172.21.15.87:6812/2286506651","cluster_addr":"172.21.15.87:6813/2286506651","heartbeat_back_addr":"172.21.15.87:6815/2286506651","heartbeat_front_addr":"172.21.15.87:6814/2286506651","state":["exists","up"]},{"osd":1,"uuid":"941684ea-0764-4182-94cc-136349df4f23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6804","nonce":3459114360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6805","nonce":3459114360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6809","nonce":3459114360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6807","nonce":3459114360}]},"public_addr":"172.21.15.87:6804/3459114360","cluster_addr":"172.21.15.87:6805/3459114360","heartbeat_back_addr":"172.21.15.87:6809/3459114360","heartbeat_front_addr":"172.21.15.87:6807/3459114360","state":["exists","up"]},{"osd":2,"uuid":"17135934-ecb7-445a-b1dc-27f55da9f8ca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6800","nonce":4234108552}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6801","nonce":4234108552}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6803","nonce":4234108552}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6802","nonce":4234108552}]},"public_addr":"172.21.15.87:6800/4234108552","cluster_addr":"172.21.15.87:6801/4234108552","heartbeat_back_addr":"172.21.15.87:6803/4234108552","heartbeat_front_addr":"172.21.15.87:6802/4234108552","state":["exists","up"]},{"osd":3,"uuid":"098f9cbd-3da5-4ee7-8439-610b8cae774e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6806","nonce":3527623826}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6808","nonce":3527623826}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6811","nonce":3527623826}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6810","nonce":3527623826}]},"public_addr":"172.21.15.87:6806/3527623826","cluster_addr":"172.21.15.87:6808/3527623826","heartbeat_back_addr":"172.21.15.87:6811/3527623826","heartbeat_front_addr":"172.21.15.87:6810/3527623826","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-05T16:33:27.513154+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-05T16:33:27.800689+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-05T16:33:28.127983+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-05T16:33:28.046083+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-05T16:33:36.163 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-05T16:33:36.163 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-05T16:33:36.444 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:33:36.444 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":12,"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","created":"2024-09-05T16:33:15.770967+0000","modified":"2024-09-05T16:33:32.060239+0000","last_up_change":"2024-09-05T16:33:31.016726+0000","last_in_change":"2024-09-05T16:33:21.020289+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f76bac77-d755-450f-9fbf-fb53a6033c24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6812","nonce":2286506651}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6813","nonce":2286506651}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6815","nonce":2286506651}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6814","nonce":2286506651}]},"public_addr":"172.21.15.87:6812/2286506651","cluster_addr":"172.21.15.87:6813/2286506651","heartbeat_back_addr":"172.21.15.87:6815/2286506651","heartbeat_front_addr":"172.21.15.87:6814/2286506651","state":["exists","up"]},{"osd":1,"uuid":"941684ea-0764-4182-94cc-136349df4f23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6804","nonce":3459114360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6805","nonce":3459114360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6809","nonce":3459114360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6807","nonce":3459114360}]},"public_addr":"172.21.15.87:6804/3459114360","cluster_addr":"172.21.15.87:6805/3459114360","heartbeat_back_addr":"172.21.15.87:6809/3459114360","heartbeat_front_addr":"172.21.15.87:6807/3459114360","state":["exists","up"]},{"osd":2,"uuid":"17135934-ecb7-445a-b1dc-27f55da9f8ca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6800","nonce":4234108552}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6801","nonce":4234108552}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6803","nonce":4234108552}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6802","nonce":4234108552}]},"public_addr":"172.21.15.87:6800/4234108552","cluster_addr":"172.21.15.87:6801/4234108552","heartbeat_back_addr":"172.21.15.87:6803/4234108552","heartbeat_front_addr":"172.21.15.87:6802/4234108552","state":["exists","up"]},{"osd":3,"uuid":"098f9cbd-3da5-4ee7-8439-610b8cae774e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6806","nonce":3527623826}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6808","nonce":3527623826}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6811","nonce":3527623826}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6810","nonce":3527623826}]},"public_addr":"172.21.15.87:6806/3527623826","cluster_addr":"172.21.15.87:6808/3527623826","heartbeat_back_addr":"172.21.15.87:6811/3527623826","heartbeat_front_addr":"172.21.15.87:6810/3527623826","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-05T16:33:27.513154+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-05T16:33:27.800689+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-05T16:33:28.127983+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-05T16:33:28.046083+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-05T16:33:36.457 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-05T16:33:36.457 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-05T16:33:36.458 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-05T16:33:36.458 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-05T16:33:36.598 INFO:teuthology.orchestra.run.smithi087.stdout:42949672964 2024-09-05T16:33:36.598 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-05T16:33:36.599 INFO:teuthology.orchestra.run.smithi087.stdout:47244640260 2024-09-05T16:33:36.600 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-05T16:33:36.617 INFO:teuthology.orchestra.run.smithi087.stdout:47244640261 2024-09-05T16:33:36.617 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-05T16:33:36.618 INFO:teuthology.orchestra.run.smithi087.stdout:38654705669 2024-09-05T16:33:36.619 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-05T16:33:36.910 INFO:teuthology.orchestra.run.smithi087.stdout:47244640260 2024-09-05T16:33:36.923 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640260 for osd.3 2024-09-05T16:33:36.924 DEBUG:teuthology.parallel:result is None 2024-09-05T16:33:36.968 INFO:teuthology.orchestra.run.smithi087.stdout:47244640261 2024-09-05T16:33:36.980 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640261 got 47244640261 for osd.1 2024-09-05T16:33:36.980 DEBUG:teuthology.parallel:result is None 2024-09-05T16:33:37.012 INFO:teuthology.orchestra.run.smithi087.stdout:42949672964 2024-09-05T16:33:37.025 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672964 got 42949672964 for osd.2 2024-09-05T16:33:37.025 DEBUG:teuthology.parallel:result is None 2024-09-05T16:33:37.062 INFO:teuthology.orchestra.run.smithi087.stdout:38654705669 2024-09-05T16:33:37.074 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705669 got 38654705669 for osd.0 2024-09-05T16:33:37.075 DEBUG:teuthology.parallel:result is None 2024-09-05T16:33:37.075 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-05T16:33:37.075 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:33:37.395 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:33:37.395 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:33:37.408 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2024-09-05T16:33:36.778145+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":108544,"kb_used_data":416,"kb_used_omap":6,"kb_used_meta":108025,"kb_avail":377378816,"statfs":{"total":386547056640,"available":386435907584,"internally_reserved":0,"allocated":425984,"data_stored":91828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6378,"internal_metadata":110618390},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[],"pool_stats":[],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640260,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27136,"kb_used_data":104,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344704,"statfs":{"total":96636764160,"available":96608976896,"internally_reserved":0,"allocated":106496,"data_stored":22957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640261,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27136,"kb_used_data":104,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344704,"statfs":{"total":96636764160,"available":96608976896,"internally_reserved":0,"allocated":106496,"data_stored":22957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949672964,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27136,"kb_used_data":104,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344704,"statfs":{"total":96636764160,"available":96608976896,"internally_reserved":0,"allocated":106496,"data_stored":22957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705669,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27136,"kb_used_data":104,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344704,"statfs":{"total":96636764160,"available":96608976896,"internally_reserved":0,"allocated":106496,"data_stored":22957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-09-05T16:33:37.408 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:33:37.686 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:33:37.686 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:33:37.698 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2024-09-05T16:33:36.778145+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":108544,"kb_used_data":416,"kb_used_omap":6,"kb_used_meta":108025,"kb_avail":377378816,"statfs":{"total":386547056640,"available":386435907584,"internally_reserved":0,"allocated":425984,"data_stored":91828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6378,"internal_metadata":110618390},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[],"pool_stats":[],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640260,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27136,"kb_used_data":104,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344704,"statfs":{"total":96636764160,"available":96608976896,"internally_reserved":0,"allocated":106496,"data_stored":22957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640261,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27136,"kb_used_data":104,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344704,"statfs":{"total":96636764160,"available":96608976896,"internally_reserved":0,"allocated":106496,"data_stored":22957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949672964,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27136,"kb_used_data":104,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344704,"statfs":{"total":96636764160,"available":96608976896,"internally_reserved":0,"allocated":106496,"data_stored":22957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705669,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27136,"kb_used_data":104,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344704,"statfs":{"total":96636764160,"available":96608976896,"internally_reserved":0,"allocated":106496,"data_stored":22957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-09-05T16:33:37.698 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-05T16:33:37.698 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-05T16:33:37.698 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-09-05T16:33:37.699 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-05T16:33:38.019 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:33:38.019 INFO:teuthology.orchestra.run.smithi087.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-05T16:33:38.031 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-09-05T16:33:38.031 INFO:teuthology.run_tasks:Running task ec_lost_unfound... 2024-09-05T16:33:38.059 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-05T16:33:38.342 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:33:38.342 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":12,"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","created":"2024-09-05T16:33:15.770967+0000","modified":"2024-09-05T16:33:32.060239+0000","last_up_change":"2024-09-05T16:33:31.016726+0000","last_in_change":"2024-09-05T16:33:21.020289+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f76bac77-d755-450f-9fbf-fb53a6033c24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6812","nonce":2286506651}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6813","nonce":2286506651}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6815","nonce":2286506651}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6814","nonce":2286506651}]},"public_addr":"172.21.15.87:6812/2286506651","cluster_addr":"172.21.15.87:6813/2286506651","heartbeat_back_addr":"172.21.15.87:6815/2286506651","heartbeat_front_addr":"172.21.15.87:6814/2286506651","state":["exists","up"]},{"osd":1,"uuid":"941684ea-0764-4182-94cc-136349df4f23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6804","nonce":3459114360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6805","nonce":3459114360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6809","nonce":3459114360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6807","nonce":3459114360}]},"public_addr":"172.21.15.87:6804/3459114360","cluster_addr":"172.21.15.87:6805/3459114360","heartbeat_back_addr":"172.21.15.87:6809/3459114360","heartbeat_front_addr":"172.21.15.87:6807/3459114360","state":["exists","up"]},{"osd":2,"uuid":"17135934-ecb7-445a-b1dc-27f55da9f8ca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6800","nonce":4234108552}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6801","nonce":4234108552}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6803","nonce":4234108552}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6802","nonce":4234108552}]},"public_addr":"172.21.15.87:6800/4234108552","cluster_addr":"172.21.15.87:6801/4234108552","heartbeat_back_addr":"172.21.15.87:6803/4234108552","heartbeat_front_addr":"172.21.15.87:6802/4234108552","state":["exists","up"]},{"osd":3,"uuid":"098f9cbd-3da5-4ee7-8439-610b8cae774e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6806","nonce":3527623826}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6808","nonce":3527623826}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6811","nonce":3527623826}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6810","nonce":3527623826}]},"public_addr":"172.21.15.87:6806/3527623826","cluster_addr":"172.21.15.87:6808/3527623826","heartbeat_back_addr":"172.21.15.87:6811/3527623826","heartbeat_front_addr":"172.21.15.87:6810/3527623826","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-05T16:33:27.513154+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-05T16:33:27.800689+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-05T16:33:28.127983+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-05T16:33:28.046083+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-05T16:33:38.354 INFO:tasks.ec_lost_unfound.ceph_manager:[] 2024-09-05T16:33:38.354 INFO:tasks.ec_lost_unfound.ceph_manager:waiting for clean 2024-09-05T16:33:38.355 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:33:38.632 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:33:38.633 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:33:38.645 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2024-09-05T16:33:36.778145+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":108544,"kb_used_data":416,"kb_used_omap":6,"kb_used_meta":108025,"kb_avail":377378816,"statfs":{"total":386547056640,"available":386435907584,"internally_reserved":0,"allocated":425984,"data_stored":91828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6378,"internal_metadata":110618390},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[],"pool_stats":[],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640260,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27136,"kb_used_data":104,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344704,"statfs":{"total":96636764160,"available":96608976896,"internally_reserved":0,"allocated":106496,"data_stored":22957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640261,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27136,"kb_used_data":104,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344704,"statfs":{"total":96636764160,"available":96608976896,"internally_reserved":0,"allocated":106496,"data_stored":22957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949672964,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27136,"kb_used_data":104,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344704,"statfs":{"total":96636764160,"available":96608976896,"internally_reserved":0,"allocated":106496,"data_stored":22957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705669,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27136,"kb_used_data":104,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344704,"statfs":{"total":96636764160,"available":96608976896,"internally_reserved":0,"allocated":106496,"data_stored":22957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-09-05T16:33:38.645 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:33:38.925 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:33:38.926 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:33:38.938 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2024-09-05T16:33:38.778429+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":108544,"kb_used_data":416,"kb_used_omap":6,"kb_used_meta":108025,"kb_avail":377378816,"statfs":{"total":386547056640,"available":386435907584,"internally_reserved":0,"allocated":425984,"data_stored":91828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6378,"internal_metadata":110618390},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[],"pool_stats":[],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640261,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27136,"kb_used_data":104,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344704,"statfs":{"total":96636764160,"available":96608976896,"internally_reserved":0,"allocated":106496,"data_stored":22957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640262,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27136,"kb_used_data":104,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344704,"statfs":{"total":96636764160,"available":96608976896,"internally_reserved":0,"allocated":106496,"data_stored":22957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949672964,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27136,"kb_used_data":104,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344704,"statfs":{"total":96636764160,"available":96608976896,"internally_reserved":0,"allocated":106496,"data_stored":22957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705669,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27136,"kb_used_data":104,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344704,"statfs":{"total":96636764160,"available":96608976896,"internally_reserved":0,"allocated":106496,"data_stored":22957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-09-05T16:33:38.938 INFO:tasks.ec_lost_unfound.ceph_manager:clean! 2024-09-05T16:33:38.939 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set lost_unfound k=2 m=2 crush-failure-domain=osd 2024-09-05T16:33:39.856 INFO:tasks.ec_lost_unfound.ceph_manager:creating pool_name unique_pool_0 2024-09-05T16:33:39.856 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create unique_pool_0 16 16 erasure lost_unfound 2024-09-05T16:33:41.854 INFO:teuthology.orchestra.run.smithi087.stderr:pool 'unique_pool_0' created 2024-09-05T16:33:41.876 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set unique_pool_0 min_size 2 2024-09-05T16:33:42.864 INFO:teuthology.orchestra.run.smithi087.stderr:set pool 1 min_size to 2 2024-09-05T16:33:42.883 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2024-09-05T16:33:43.857 INFO:teuthology.orchestra.run.smithi087.stderr:enabled application 'rados' on pool 'unique_pool_0' 2024-09-05T16:33:44.891 INFO:tasks.util.rados:rados -p unique_pool_0 put dummy /etc/group 2024-09-05T16:33:44.891 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put dummy /etc/group 2024-09-05T16:33:44.950 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-05T16:33:44.951 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-05T16:33:45.086 INFO:teuthology.orchestra.run.smithi087.stdout:47244640264 2024-09-05T16:33:45.087 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-05T16:33:45.087 INFO:teuthology.orchestra.run.smithi087.stdout:38654705672 2024-09-05T16:33:45.088 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-05T16:33:45.417 INFO:teuthology.orchestra.run.smithi087.stdout:47244640263 2024-09-05T16:33:45.423 INFO:teuthology.orchestra.run.smithi087.stdout:38654705670 2024-09-05T16:33:45.430 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 47244640264 got 47244640263 for osd.1 2024-09-05T16:33:45.436 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 38654705672 got 38654705670 for osd.0 2024-09-05T16:33:46.431 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-05T16:33:46.437 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-05T16:33:46.757 INFO:teuthology.orchestra.run.smithi087.stdout:47244640263 2024-09-05T16:33:46.770 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 47244640264 got 47244640263 for osd.1 2024-09-05T16:33:46.793 INFO:teuthology.orchestra.run.smithi087.stdout:38654705670 2024-09-05T16:33:46.806 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 38654705672 got 38654705670 for osd.0 2024-09-05T16:33:47.772 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-05T16:33:47.808 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-05T16:33:48.100 INFO:teuthology.orchestra.run.smithi087.stdout:47244640264 2024-09-05T16:33:48.113 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 47244640264 got 47244640264 for osd.1 2024-09-05T16:33:48.113 DEBUG:teuthology.parallel:result is None 2024-09-05T16:33:48.136 INFO:teuthology.orchestra.run.smithi087.stdout:38654705672 2024-09-05T16:33:48.150 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 38654705672 got 38654705672 for osd.0 2024-09-05T16:33:48.150 DEBUG:teuthology.parallel:result is None 2024-09-05T16:33:48.150 INFO:tasks.ec_lost_unfound.ceph_manager:waiting for recovery to complete 2024-09-05T16:33:48.151 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:33:48.469 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:33:48.470 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:33:48.482 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-09-05T16:33:46.780042+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":44,"acting":44,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":3,"kb":377487360,"kb_used":108688,"kb_used_data":560,"kb_used_omap":6,"kb_used_meta":108025,"kb_avail":377378672,"statfs":{"total":386547056640,"available":386435760128,"internally_reserved":0,"allocated":573440,"data_stored":151141,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6378,"internal_metadata":110618390},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-05T16:33:41.856813+0000","last_change":"2024-09-05T16:33:41.856813+0000","last_active":"2024-09-05T16:33:41.856813+0000","last_peered":"2024-09-05T16:33:41.856813+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-05T16:33:41.856813+0000","last_undegraded":"2024-09-05T16:33:41.856813+0000","last_fullsized":"2024-09-05T16:33:41.856813+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.e","version":"0'0","reported_seq":16,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-05T16:33:44.957193+0000","last_change":"2024-09-05T16:33:43.890250+0000","last_active":"2024-09-05T16:33:44.957193+0000","last_peered":"2024-09-05T16:33:44.957193+0000","last_clean":"2024-09-05T16:33:44.957193+0000","last_became_active":"2024-09-05T16:33:43.889890+0000","last_became_peered":"2024-09-05T16:33:43.889890+0000","last_unstale":"2024-09-05T16:33:44.957193+0000","last_undegraded":"2024-09-05T16:33:44.957193+0000","last_fullsized":"2024-09-05T16:33:44.957193+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T03:29:01.015741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"0'0","reported_seq":16,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-05T16:33:44.871516+0000","last_change":"2024-09-05T16:33:43.889084+0000","last_active":"2024-09-05T16:33:44.871516+0000","last_peered":"2024-09-05T16:33:44.871516+0000","last_clean":"2024-09-05T16:33:44.871516+0000","last_became_active":"2024-09-05T16:33:43.888465+0000","last_became_peered":"2024-09-05T16:33:43.888465+0000","last_unstale":"2024-09-05T16:33:44.871516+0000","last_undegraded":"2024-09-05T16:33:44.871516+0000","last_fullsized":"2024-09-05T16:33:44.871516+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:27:25.722702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-05T16:33:44.957203+0000","last_change":"2024-09-05T16:33:43.885432+0000","last_active":"2024-09-05T16:33:44.957203+0000","last_peered":"2024-09-05T16:33:44.957203+0000","last_clean":"2024-09-05T16:33:44.957203+0000","last_became_active":"2024-09-05T16:33:43.884906+0000","last_became_peered":"2024-09-05T16:33:43.884906+0000","last_unstale":"2024-09-05T16:33:44.957203+0000","last_undegraded":"2024-09-05T16:33:44.957203+0000","last_fullsized":"2024-09-05T16:33:44.957203+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T00:51:10.166665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-05T16:33:41.856813+0000","last_change":"2024-09-05T16:33:41.856813+0000","last_active":"2024-09-05T16:33:41.856813+0000","last_peered":"2024-09-05T16:33:41.856813+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-05T16:33:41.856813+0000","last_undegraded":"2024-09-05T16:33:41.856813+0000","last_fullsized":"2024-09-05T16:33:41.856813+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-05T16:33:41.856813+0000","last_change":"2024-09-05T16:33:41.856813+0000","last_active":"2024-09-05T16:33:41.856813+0000","last_peered":"2024-09-05T16:33:41.856813+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-05T16:33:41.856813+0000","last_undegraded":"2024-09-05T16:33:41.856813+0000","last_fullsized":"2024-09-05T16:33:41.856813+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":16,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-05T16:33:44.869419+0000","last_change":"2024-09-05T16:33:43.879776+0000","last_active":"2024-09-05T16:33:44.869419+0000","last_peered":"2024-09-05T16:33:44.869419+0000","last_clean":"2024-09-05T16:33:44.869419+0000","last_became_active":"2024-09-05T16:33:43.879346+0000","last_became_peered":"2024-09-05T16:33:43.879346+0000","last_unstale":"2024-09-05T16:33:44.869419+0000","last_undegraded":"2024-09-05T16:33:44.869419+0000","last_fullsized":"2024-09-05T16:33:44.869419+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T03:11:38.159719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":16,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-05T16:33:44.956973+0000","last_change":"2024-09-05T16:33:43.891112+0000","last_active":"2024-09-05T16:33:44.956973+0000","last_peered":"2024-09-05T16:33:44.956973+0000","last_clean":"2024-09-05T16:33:44.956973+0000","last_became_active":"2024-09-05T16:33:43.890413+0000","last_became_peered":"2024-09-05T16:33:43.890413+0000","last_unstale":"2024-09-05T16:33:44.956973+0000","last_undegraded":"2024-09-05T16:33:44.956973+0000","last_fullsized":"2024-09-05T16:33:44.956973+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T02:51:58.162036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-05T16:33:41.856813+0000","last_change":"2024-09-05T16:33:41.856813+0000","last_active":"2024-09-05T16:33:41.856813+0000","last_peered":"2024-09-05T16:33:41.856813+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-05T16:33:41.856813+0000","last_undegraded":"2024-09-05T16:33:41.856813+0000","last_fullsized":"2024-09-05T16:33:41.856813+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":16,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-05T16:33:44.871591+0000","last_change":"2024-09-05T16:33:43.886478+0000","last_active":"2024-09-05T16:33:44.871591+0000","last_peered":"2024-09-05T16:33:44.871591+0000","last_clean":"2024-09-05T16:33:44.871591+0000","last_became_active":"2024-09-05T16:33:43.885565+0000","last_became_peered":"2024-09-05T16:33:43.885565+0000","last_unstale":"2024-09-05T16:33:44.871591+0000","last_undegraded":"2024-09-05T16:33:44.871591+0000","last_fullsized":"2024-09-05T16:33:44.871591+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:44:19.491731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-05T16:33:41.856813+0000","last_change":"2024-09-05T16:33:41.856813+0000","last_active":"2024-09-05T16:33:41.856813+0000","last_peered":"2024-09-05T16:33:41.856813+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-05T16:33:41.856813+0000","last_undegraded":"2024-09-05T16:33:41.856813+0000","last_fullsized":"2024-09-05T16:33:41.856813+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-05T16:33:44.956944+0000","last_change":"2024-09-05T16:33:43.890813+0000","last_active":"2024-09-05T16:33:44.956944+0000","last_peered":"2024-09-05T16:33:44.956944+0000","last_clean":"2024-09-05T16:33:44.956944+0000","last_became_active":"2024-09-05T16:33:43.889961+0000","last_became_peered":"2024-09-05T16:33:43.889961+0000","last_unstale":"2024-09-05T16:33:44.956944+0000","last_undegraded":"2024-09-05T16:33:44.956944+0000","last_fullsized":"2024-09-05T16:33:44.956944+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T03:38:03.746292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"0'0","reported_seq":16,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-05T16:33:44.957048+0000","last_change":"2024-09-05T16:33:43.893959+0000","last_active":"2024-09-05T16:33:44.957048+0000","last_peered":"2024-09-05T16:33:44.957048+0000","last_clean":"2024-09-05T16:33:44.957048+0000","last_became_active":"2024-09-05T16:33:43.893550+0000","last_became_peered":"2024-09-05T16:33:43.893550+0000","last_unstale":"2024-09-05T16:33:44.957048+0000","last_undegraded":"2024-09-05T16:33:44.957048+0000","last_fullsized":"2024-09-05T16:33:44.957048+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T03:13:31.149229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"0'0","reported_seq":16,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-05T16:33:44.956984+0000","last_change":"2024-09-05T16:33:43.886333+0000","last_active":"2024-09-05T16:33:44.956984+0000","last_peered":"2024-09-05T16:33:44.956984+0000","last_clean":"2024-09-05T16:33:44.956984+0000","last_became_active":"2024-09-05T16:33:43.885840+0000","last_became_peered":"2024-09-05T16:33:43.885840+0000","last_unstale":"2024-09-05T16:33:44.956984+0000","last_undegraded":"2024-09-05T16:33:44.956984+0000","last_fullsized":"2024-09-05T16:33:44.956984+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:45:13.094481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"0'0","reported_seq":16,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-05T16:33:44.957113+0000","last_change":"2024-09-05T16:33:43.881131+0000","last_active":"2024-09-05T16:33:44.957113+0000","last_peered":"2024-09-05T16:33:44.957113+0000","last_clean":"2024-09-05T16:33:44.957113+0000","last_became_active":"2024-09-05T16:33:43.880598+0000","last_became_peered":"2024-09-05T16:33:43.880598+0000","last_unstale":"2024-09-05T16:33:44.957113+0000","last_undegraded":"2024-09-05T16:33:44.957113+0000","last_fullsized":"2024-09-05T16:33:44.957113+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T17:55:21.440218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"0'0","reported_seq":16,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-05T16:33:44.957183+0000","last_change":"2024-09-05T16:33:43.890336+0000","last_active":"2024-09-05T16:33:44.957183+0000","last_peered":"2024-09-05T16:33:44.957183+0000","last_clean":"2024-09-05T16:33:44.957183+0000","last_became_active":"2024-09-05T16:33:43.889915+0000","last_became_peered":"2024-09-05T16:33:43.889915+0000","last_unstale":"2024-09-05T16:33:44.957183+0000","last_undegraded":"2024-09-05T16:33:44.957183+0000","last_fullsized":"2024-09-05T16:33:44.957183+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:42:02.617765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":12288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":44,"acting":44,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640262,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27152,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344688,"statfs":{"total":96636764160,"available":96608960512,"internally_reserved":0,"allocated":122880,"data_stored":29265,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640264,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27180,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344660,"statfs":{"total":96636764160,"available":96608931840,"internally_reserved":0,"allocated":147456,"data_stored":39495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949672966,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27176,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344664,"statfs":{"total":96636764160,"available":96608935936,"internally_reserved":0,"allocated":147456,"data_stored":39495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705672,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27180,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344660,"statfs":{"total":96636764160,"available":96608931840,"internally_reserved":0,"allocated":155648,"data_stored":42886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:33:48.483 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:33:48.763 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:33:48.764 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:33:48.776 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-09-05T16:33:46.780042+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":44,"acting":44,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":3,"kb":377487360,"kb_used":108688,"kb_used_data":560,"kb_used_omap":6,"kb_used_meta":108025,"kb_avail":377378672,"statfs":{"total":386547056640,"available":386435760128,"internally_reserved":0,"allocated":573440,"data_stored":151141,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6378,"internal_metadata":110618390},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-05T16:33:41.856813+0000","last_change":"2024-09-05T16:33:41.856813+0000","last_active":"2024-09-05T16:33:41.856813+0000","last_peered":"2024-09-05T16:33:41.856813+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-05T16:33:41.856813+0000","last_undegraded":"2024-09-05T16:33:41.856813+0000","last_fullsized":"2024-09-05T16:33:41.856813+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.e","version":"0'0","reported_seq":16,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-05T16:33:44.957193+0000","last_change":"2024-09-05T16:33:43.890250+0000","last_active":"2024-09-05T16:33:44.957193+0000","last_peered":"2024-09-05T16:33:44.957193+0000","last_clean":"2024-09-05T16:33:44.957193+0000","last_became_active":"2024-09-05T16:33:43.889890+0000","last_became_peered":"2024-09-05T16:33:43.889890+0000","last_unstale":"2024-09-05T16:33:44.957193+0000","last_undegraded":"2024-09-05T16:33:44.957193+0000","last_fullsized":"2024-09-05T16:33:44.957193+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T03:29:01.015741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"0'0","reported_seq":16,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-05T16:33:44.871516+0000","last_change":"2024-09-05T16:33:43.889084+0000","last_active":"2024-09-05T16:33:44.871516+0000","last_peered":"2024-09-05T16:33:44.871516+0000","last_clean":"2024-09-05T16:33:44.871516+0000","last_became_active":"2024-09-05T16:33:43.888465+0000","last_became_peered":"2024-09-05T16:33:43.888465+0000","last_unstale":"2024-09-05T16:33:44.871516+0000","last_undegraded":"2024-09-05T16:33:44.871516+0000","last_fullsized":"2024-09-05T16:33:44.871516+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:27:25.722702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-05T16:33:44.957203+0000","last_change":"2024-09-05T16:33:43.885432+0000","last_active":"2024-09-05T16:33:44.957203+0000","last_peered":"2024-09-05T16:33:44.957203+0000","last_clean":"2024-09-05T16:33:44.957203+0000","last_became_active":"2024-09-05T16:33:43.884906+0000","last_became_peered":"2024-09-05T16:33:43.884906+0000","last_unstale":"2024-09-05T16:33:44.957203+0000","last_undegraded":"2024-09-05T16:33:44.957203+0000","last_fullsized":"2024-09-05T16:33:44.957203+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T00:51:10.166665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-05T16:33:41.856813+0000","last_change":"2024-09-05T16:33:41.856813+0000","last_active":"2024-09-05T16:33:41.856813+0000","last_peered":"2024-09-05T16:33:41.856813+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-05T16:33:41.856813+0000","last_undegraded":"2024-09-05T16:33:41.856813+0000","last_fullsized":"2024-09-05T16:33:41.856813+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-05T16:33:41.856813+0000","last_change":"2024-09-05T16:33:41.856813+0000","last_active":"2024-09-05T16:33:41.856813+0000","last_peered":"2024-09-05T16:33:41.856813+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-05T16:33:41.856813+0000","last_undegraded":"2024-09-05T16:33:41.856813+0000","last_fullsized":"2024-09-05T16:33:41.856813+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":16,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-05T16:33:44.869419+0000","last_change":"2024-09-05T16:33:43.879776+0000","last_active":"2024-09-05T16:33:44.869419+0000","last_peered":"2024-09-05T16:33:44.869419+0000","last_clean":"2024-09-05T16:33:44.869419+0000","last_became_active":"2024-09-05T16:33:43.879346+0000","last_became_peered":"2024-09-05T16:33:43.879346+0000","last_unstale":"2024-09-05T16:33:44.869419+0000","last_undegraded":"2024-09-05T16:33:44.869419+0000","last_fullsized":"2024-09-05T16:33:44.869419+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T03:11:38.159719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":16,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-05T16:33:44.956973+0000","last_change":"2024-09-05T16:33:43.891112+0000","last_active":"2024-09-05T16:33:44.956973+0000","last_peered":"2024-09-05T16:33:44.956973+0000","last_clean":"2024-09-05T16:33:44.956973+0000","last_became_active":"2024-09-05T16:33:43.890413+0000","last_became_peered":"2024-09-05T16:33:43.890413+0000","last_unstale":"2024-09-05T16:33:44.956973+0000","last_undegraded":"2024-09-05T16:33:44.956973+0000","last_fullsized":"2024-09-05T16:33:44.956973+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T02:51:58.162036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-05T16:33:41.856813+0000","last_change":"2024-09-05T16:33:41.856813+0000","last_active":"2024-09-05T16:33:41.856813+0000","last_peered":"2024-09-05T16:33:41.856813+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-05T16:33:41.856813+0000","last_undegraded":"2024-09-05T16:33:41.856813+0000","last_fullsized":"2024-09-05T16:33:41.856813+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":16,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-05T16:33:44.871591+0000","last_change":"2024-09-05T16:33:43.886478+0000","last_active":"2024-09-05T16:33:44.871591+0000","last_peered":"2024-09-05T16:33:44.871591+0000","last_clean":"2024-09-05T16:33:44.871591+0000","last_became_active":"2024-09-05T16:33:43.885565+0000","last_became_peered":"2024-09-05T16:33:43.885565+0000","last_unstale":"2024-09-05T16:33:44.871591+0000","last_undegraded":"2024-09-05T16:33:44.871591+0000","last_fullsized":"2024-09-05T16:33:44.871591+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:44:19.491731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-05T16:33:41.856813+0000","last_change":"2024-09-05T16:33:41.856813+0000","last_active":"2024-09-05T16:33:41.856813+0000","last_peered":"2024-09-05T16:33:41.856813+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-05T16:33:41.856813+0000","last_undegraded":"2024-09-05T16:33:41.856813+0000","last_fullsized":"2024-09-05T16:33:41.856813+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-05T16:33:44.956944+0000","last_change":"2024-09-05T16:33:43.890813+0000","last_active":"2024-09-05T16:33:44.956944+0000","last_peered":"2024-09-05T16:33:44.956944+0000","last_clean":"2024-09-05T16:33:44.956944+0000","last_became_active":"2024-09-05T16:33:43.889961+0000","last_became_peered":"2024-09-05T16:33:43.889961+0000","last_unstale":"2024-09-05T16:33:44.956944+0000","last_undegraded":"2024-09-05T16:33:44.956944+0000","last_fullsized":"2024-09-05T16:33:44.956944+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T03:38:03.746292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"0'0","reported_seq":16,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-05T16:33:44.957048+0000","last_change":"2024-09-05T16:33:43.893959+0000","last_active":"2024-09-05T16:33:44.957048+0000","last_peered":"2024-09-05T16:33:44.957048+0000","last_clean":"2024-09-05T16:33:44.957048+0000","last_became_active":"2024-09-05T16:33:43.893550+0000","last_became_peered":"2024-09-05T16:33:43.893550+0000","last_unstale":"2024-09-05T16:33:44.957048+0000","last_undegraded":"2024-09-05T16:33:44.957048+0000","last_fullsized":"2024-09-05T16:33:44.957048+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T03:13:31.149229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"0'0","reported_seq":16,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-05T16:33:44.956984+0000","last_change":"2024-09-05T16:33:43.886333+0000","last_active":"2024-09-05T16:33:44.956984+0000","last_peered":"2024-09-05T16:33:44.956984+0000","last_clean":"2024-09-05T16:33:44.956984+0000","last_became_active":"2024-09-05T16:33:43.885840+0000","last_became_peered":"2024-09-05T16:33:43.885840+0000","last_unstale":"2024-09-05T16:33:44.956984+0000","last_undegraded":"2024-09-05T16:33:44.956984+0000","last_fullsized":"2024-09-05T16:33:44.956984+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:45:13.094481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"0'0","reported_seq":16,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-05T16:33:44.957113+0000","last_change":"2024-09-05T16:33:43.881131+0000","last_active":"2024-09-05T16:33:44.957113+0000","last_peered":"2024-09-05T16:33:44.957113+0000","last_clean":"2024-09-05T16:33:44.957113+0000","last_became_active":"2024-09-05T16:33:43.880598+0000","last_became_peered":"2024-09-05T16:33:43.880598+0000","last_unstale":"2024-09-05T16:33:44.957113+0000","last_undegraded":"2024-09-05T16:33:44.957113+0000","last_fullsized":"2024-09-05T16:33:44.957113+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T17:55:21.440218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"0'0","reported_seq":16,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-05T16:33:44.957183+0000","last_change":"2024-09-05T16:33:43.890336+0000","last_active":"2024-09-05T16:33:44.957183+0000","last_peered":"2024-09-05T16:33:44.957183+0000","last_clean":"2024-09-05T16:33:44.957183+0000","last_became_active":"2024-09-05T16:33:43.889915+0000","last_became_peered":"2024-09-05T16:33:43.889915+0000","last_unstale":"2024-09-05T16:33:44.957183+0000","last_undegraded":"2024-09-05T16:33:44.957183+0000","last_fullsized":"2024-09-05T16:33:44.957183+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:42:02.617765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":12288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":44,"acting":44,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640262,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27152,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344688,"statfs":{"total":96636764160,"available":96608960512,"internally_reserved":0,"allocated":122880,"data_stored":29265,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640264,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27180,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344660,"statfs":{"total":96636764160,"available":96608931840,"internally_reserved":0,"allocated":147456,"data_stored":39495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949672966,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27176,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344664,"statfs":{"total":96636764160,"available":96608935936,"internally_reserved":0,"allocated":147456,"data_stored":39495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705672,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27180,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344660,"statfs":{"total":96636764160,"available":96608931840,"internally_reserved":0,"allocated":155648,"data_stored":42886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:33:48.777 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:33:49.059 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:33:49.059 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:33:49.070 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2024-09-05T16:33:48.780285+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":37,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":108724,"kb_used_data":608,"kb_used_omap":6,"kb_used_meta":108025,"kb_avail":377378636,"statfs":{"total":386547056640,"available":386435723264,"internally_reserved":0,"allocated":622592,"data_stored":176345,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6378,"internal_metadata":110618390},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.f","version":"0'0","reported_seq":16,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-05T16:33:44.870837+0000","last_change":"2024-09-05T16:33:43.881028+0000","last_active":"2024-09-05T16:33:44.870837+0000","last_peered":"2024-09-05T16:33:44.870837+0000","last_clean":"2024-09-05T16:33:44.870837+0000","last_became_active":"2024-09-05T16:33:43.880638+0000","last_became_peered":"2024-09-05T16:33:43.880638+0000","last_unstale":"2024-09-05T16:33:44.870837+0000","last_undegraded":"2024-09-05T16:33:44.870837+0000","last_fullsized":"2024-09-05T16:33:44.870837+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T17:05:19.681724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"0'0","reported_seq":16,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-05T16:33:44.957193+0000","last_change":"2024-09-05T16:33:43.890250+0000","last_active":"2024-09-05T16:33:44.957193+0000","last_peered":"2024-09-05T16:33:44.957193+0000","last_clean":"2024-09-05T16:33:44.957193+0000","last_became_active":"2024-09-05T16:33:43.889890+0000","last_became_peered":"2024-09-05T16:33:43.889890+0000","last_unstale":"2024-09-05T16:33:44.957193+0000","last_undegraded":"2024-09-05T16:33:44.957193+0000","last_fullsized":"2024-09-05T16:33:44.957193+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T03:29:01.015741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"0'0","reported_seq":16,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-05T16:33:44.871516+0000","last_change":"2024-09-05T16:33:43.889084+0000","last_active":"2024-09-05T16:33:44.871516+0000","last_peered":"2024-09-05T16:33:44.871516+0000","last_clean":"2024-09-05T16:33:44.871516+0000","last_became_active":"2024-09-05T16:33:43.888465+0000","last_became_peered":"2024-09-05T16:33:43.888465+0000","last_unstale":"2024-09-05T16:33:44.871516+0000","last_undegraded":"2024-09-05T16:33:44.871516+0000","last_fullsized":"2024-09-05T16:33:44.871516+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:27:25.722702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-05T16:33:44.957203+0000","last_change":"2024-09-05T16:33:43.885432+0000","last_active":"2024-09-05T16:33:44.957203+0000","last_peered":"2024-09-05T16:33:44.957203+0000","last_clean":"2024-09-05T16:33:44.957203+0000","last_became_active":"2024-09-05T16:33:43.884906+0000","last_became_peered":"2024-09-05T16:33:43.884906+0000","last_unstale":"2024-09-05T16:33:44.957203+0000","last_undegraded":"2024-09-05T16:33:44.957203+0000","last_fullsized":"2024-09-05T16:33:44.957203+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T00:51:10.166665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":16,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-05T16:33:44.870465+0000","last_change":"2024-09-05T16:33:43.882907+0000","last_active":"2024-09-05T16:33:44.870465+0000","last_peered":"2024-09-05T16:33:44.870465+0000","last_clean":"2024-09-05T16:33:44.870465+0000","last_became_active":"2024-09-05T16:33:43.882032+0000","last_became_peered":"2024-09-05T16:33:43.882032+0000","last_unstale":"2024-09-05T16:33:44.870465+0000","last_undegraded":"2024-09-05T16:33:44.870465+0000","last_fullsized":"2024-09-05T16:33:44.870465+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:40:44.485781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"0'0","reported_seq":16,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-05T16:33:44.870528+0000","last_change":"2024-09-05T16:33:43.884160+0000","last_active":"2024-09-05T16:33:44.870528+0000","last_peered":"2024-09-05T16:33:44.870528+0000","last_clean":"2024-09-05T16:33:44.870528+0000","last_became_active":"2024-09-05T16:33:43.883424+0000","last_became_peered":"2024-09-05T16:33:43.883424+0000","last_unstale":"2024-09-05T16:33:44.870528+0000","last_undegraded":"2024-09-05T16:33:44.870528+0000","last_fullsized":"2024-09-05T16:33:44.870528+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T21:41:54.053804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":16,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-05T16:33:44.869419+0000","last_change":"2024-09-05T16:33:43.879776+0000","last_active":"2024-09-05T16:33:44.869419+0000","last_peered":"2024-09-05T16:33:44.869419+0000","last_clean":"2024-09-05T16:33:44.869419+0000","last_became_active":"2024-09-05T16:33:43.879346+0000","last_became_peered":"2024-09-05T16:33:43.879346+0000","last_unstale":"2024-09-05T16:33:44.869419+0000","last_undegraded":"2024-09-05T16:33:44.869419+0000","last_fullsized":"2024-09-05T16:33:44.869419+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T03:11:38.159719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":16,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-05T16:33:44.956973+0000","last_change":"2024-09-05T16:33:43.891112+0000","last_active":"2024-09-05T16:33:44.956973+0000","last_peered":"2024-09-05T16:33:44.956973+0000","last_clean":"2024-09-05T16:33:44.956973+0000","last_became_active":"2024-09-05T16:33:43.890413+0000","last_became_peered":"2024-09-05T16:33:43.890413+0000","last_unstale":"2024-09-05T16:33:44.956973+0000","last_undegraded":"2024-09-05T16:33:44.956973+0000","last_fullsized":"2024-09-05T16:33:44.956973+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T02:51:58.162036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"18'1","reported_seq":17,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-05T16:33:44.953958+0000","last_change":"2024-09-05T16:33:43.884357+0000","last_active":"2024-09-05T16:33:44.953958+0000","last_peered":"2024-09-05T16:33:44.953958+0000","last_clean":"2024-09-05T16:33:44.953958+0000","last_became_active":"2024-09-05T16:33:43.883972+0000","last_became_peered":"2024-09-05T16:33:43.883972+0000","last_unstale":"2024-09-05T16:33:44.953958+0000","last_undegraded":"2024-09-05T16:33:44.953958+0000","last_fullsized":"2024-09-05T16:33:44.953958+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T01:14:35.925787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":16,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-05T16:33:44.871591+0000","last_change":"2024-09-05T16:33:43.886478+0000","last_active":"2024-09-05T16:33:44.871591+0000","last_peered":"2024-09-05T16:33:44.871591+0000","last_clean":"2024-09-05T16:33:44.871591+0000","last_became_active":"2024-09-05T16:33:43.885565+0000","last_became_peered":"2024-09-05T16:33:43.885565+0000","last_unstale":"2024-09-05T16:33:44.871591+0000","last_undegraded":"2024-09-05T16:33:44.871591+0000","last_fullsized":"2024-09-05T16:33:44.871591+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:44:19.491731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":16,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-05T16:33:44.870728+0000","last_change":"2024-09-05T16:33:43.880726+0000","last_active":"2024-09-05T16:33:44.870728+0000","last_peered":"2024-09-05T16:33:44.870728+0000","last_clean":"2024-09-05T16:33:44.870728+0000","last_became_active":"2024-09-05T16:33:43.880349+0000","last_became_peered":"2024-09-05T16:33:43.880349+0000","last_unstale":"2024-09-05T16:33:44.870728+0000","last_undegraded":"2024-09-05T16:33:44.870728+0000","last_fullsized":"2024-09-05T16:33:44.870728+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T04:15:04.743842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-05T16:33:44.956944+0000","last_change":"2024-09-05T16:33:43.890813+0000","last_active":"2024-09-05T16:33:44.956944+0000","last_peered":"2024-09-05T16:33:44.956944+0000","last_clean":"2024-09-05T16:33:44.956944+0000","last_became_active":"2024-09-05T16:33:43.889961+0000","last_became_peered":"2024-09-05T16:33:43.889961+0000","last_unstale":"2024-09-05T16:33:44.956944+0000","last_undegraded":"2024-09-05T16:33:44.956944+0000","last_fullsized":"2024-09-05T16:33:44.956944+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T03:38:03.746292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"0'0","reported_seq":16,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-05T16:33:44.957048+0000","last_change":"2024-09-05T16:33:43.893959+0000","last_active":"2024-09-05T16:33:44.957048+0000","last_peered":"2024-09-05T16:33:44.957048+0000","last_clean":"2024-09-05T16:33:44.957048+0000","last_became_active":"2024-09-05T16:33:43.893550+0000","last_became_peered":"2024-09-05T16:33:43.893550+0000","last_unstale":"2024-09-05T16:33:44.957048+0000","last_undegraded":"2024-09-05T16:33:44.957048+0000","last_fullsized":"2024-09-05T16:33:44.957048+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T03:13:31.149229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"0'0","reported_seq":16,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-05T16:33:44.956984+0000","last_change":"2024-09-05T16:33:43.886333+0000","last_active":"2024-09-05T16:33:44.956984+0000","last_peered":"2024-09-05T16:33:44.956984+0000","last_clean":"2024-09-05T16:33:44.956984+0000","last_became_active":"2024-09-05T16:33:43.885840+0000","last_became_peered":"2024-09-05T16:33:43.885840+0000","last_unstale":"2024-09-05T16:33:44.956984+0000","last_undegraded":"2024-09-05T16:33:44.956984+0000","last_fullsized":"2024-09-05T16:33:44.956984+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:45:13.094481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"0'0","reported_seq":16,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-05T16:33:44.957113+0000","last_change":"2024-09-05T16:33:43.881131+0000","last_active":"2024-09-05T16:33:44.957113+0000","last_peered":"2024-09-05T16:33:44.957113+0000","last_clean":"2024-09-05T16:33:44.957113+0000","last_became_active":"2024-09-05T16:33:43.880598+0000","last_became_peered":"2024-09-05T16:33:43.880598+0000","last_unstale":"2024-09-05T16:33:44.957113+0000","last_undegraded":"2024-09-05T16:33:44.957113+0000","last_fullsized":"2024-09-05T16:33:44.957113+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T17:55:21.440218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"0'0","reported_seq":16,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-05T16:33:44.957183+0000","last_change":"2024-09-05T16:33:43.890336+0000","last_active":"2024-09-05T16:33:44.957183+0000","last_peered":"2024-09-05T16:33:44.957183+0000","last_clean":"2024-09-05T16:33:44.957183+0000","last_became_active":"2024-09-05T16:33:43.889915+0000","last_became_peered":"2024-09-05T16:33:43.889915+0000","last_unstale":"2024-09-05T16:33:44.957183+0000","last_undegraded":"2024-09-05T16:33:44.957183+0000","last_fullsized":"2024-09-05T16:33:44.957183+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:42:02.617765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640263,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":156,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":159744,"data_stored":46982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640265,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":156,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":159744,"data_stored":46982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949672966,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27176,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344664,"statfs":{"total":96636764160,"available":96608935936,"internally_reserved":0,"allocated":147456,"data_stored":39495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705672,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27180,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344660,"statfs":{"total":96636764160,"available":96608931840,"internally_reserved":0,"allocated":155648,"data_stored":42886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:33:52.073 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:33:52.365 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:33:52.366 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:33:52.377 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2024-09-05T16:33:50.780707+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":53,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":108744,"kb_used_data":624,"kb_used_omap":6,"kb_used_meta":108025,"kb_avail":377378616,"statfs":{"total":386547056640,"available":386435702784,"internally_reserved":0,"allocated":638976,"data_stored":187928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6378,"internal_metadata":110618390},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000422"},"pg_stats":[{"pgid":"1.f","version":"0'0","reported_seq":16,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-05T16:33:44.870837+0000","last_change":"2024-09-05T16:33:43.881028+0000","last_active":"2024-09-05T16:33:44.870837+0000","last_peered":"2024-09-05T16:33:44.870837+0000","last_clean":"2024-09-05T16:33:44.870837+0000","last_became_active":"2024-09-05T16:33:43.880638+0000","last_became_peered":"2024-09-05T16:33:43.880638+0000","last_unstale":"2024-09-05T16:33:44.870837+0000","last_undegraded":"2024-09-05T16:33:44.870837+0000","last_fullsized":"2024-09-05T16:33:44.870837+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T17:05:19.681724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"0'0","reported_seq":16,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-05T16:33:44.957193+0000","last_change":"2024-09-05T16:33:43.890250+0000","last_active":"2024-09-05T16:33:44.957193+0000","last_peered":"2024-09-05T16:33:44.957193+0000","last_clean":"2024-09-05T16:33:44.957193+0000","last_became_active":"2024-09-05T16:33:43.889890+0000","last_became_peered":"2024-09-05T16:33:43.889890+0000","last_unstale":"2024-09-05T16:33:44.957193+0000","last_undegraded":"2024-09-05T16:33:44.957193+0000","last_fullsized":"2024-09-05T16:33:44.957193+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T03:29:01.015741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"0'0","reported_seq":16,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-05T16:33:44.871516+0000","last_change":"2024-09-05T16:33:43.889084+0000","last_active":"2024-09-05T16:33:44.871516+0000","last_peered":"2024-09-05T16:33:44.871516+0000","last_clean":"2024-09-05T16:33:44.871516+0000","last_became_active":"2024-09-05T16:33:43.888465+0000","last_became_peered":"2024-09-05T16:33:43.888465+0000","last_unstale":"2024-09-05T16:33:44.871516+0000","last_undegraded":"2024-09-05T16:33:44.871516+0000","last_fullsized":"2024-09-05T16:33:44.871516+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:27:25.722702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-05T16:33:44.957203+0000","last_change":"2024-09-05T16:33:43.885432+0000","last_active":"2024-09-05T16:33:44.957203+0000","last_peered":"2024-09-05T16:33:44.957203+0000","last_clean":"2024-09-05T16:33:44.957203+0000","last_became_active":"2024-09-05T16:33:43.884906+0000","last_became_peered":"2024-09-05T16:33:43.884906+0000","last_unstale":"2024-09-05T16:33:44.957203+0000","last_undegraded":"2024-09-05T16:33:44.957203+0000","last_fullsized":"2024-09-05T16:33:44.957203+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T00:51:10.166665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":16,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-05T16:33:44.870465+0000","last_change":"2024-09-05T16:33:43.882907+0000","last_active":"2024-09-05T16:33:44.870465+0000","last_peered":"2024-09-05T16:33:44.870465+0000","last_clean":"2024-09-05T16:33:44.870465+0000","last_became_active":"2024-09-05T16:33:43.882032+0000","last_became_peered":"2024-09-05T16:33:43.882032+0000","last_unstale":"2024-09-05T16:33:44.870465+0000","last_undegraded":"2024-09-05T16:33:44.870465+0000","last_fullsized":"2024-09-05T16:33:44.870465+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:40:44.485781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"0'0","reported_seq":16,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-05T16:33:44.870528+0000","last_change":"2024-09-05T16:33:43.884160+0000","last_active":"2024-09-05T16:33:44.870528+0000","last_peered":"2024-09-05T16:33:44.870528+0000","last_clean":"2024-09-05T16:33:44.870528+0000","last_became_active":"2024-09-05T16:33:43.883424+0000","last_became_peered":"2024-09-05T16:33:43.883424+0000","last_unstale":"2024-09-05T16:33:44.870528+0000","last_undegraded":"2024-09-05T16:33:44.870528+0000","last_fullsized":"2024-09-05T16:33:44.870528+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T21:41:54.053804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":16,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-05T16:33:44.869419+0000","last_change":"2024-09-05T16:33:43.879776+0000","last_active":"2024-09-05T16:33:44.869419+0000","last_peered":"2024-09-05T16:33:44.869419+0000","last_clean":"2024-09-05T16:33:44.869419+0000","last_became_active":"2024-09-05T16:33:43.879346+0000","last_became_peered":"2024-09-05T16:33:43.879346+0000","last_unstale":"2024-09-05T16:33:44.869419+0000","last_undegraded":"2024-09-05T16:33:44.869419+0000","last_fullsized":"2024-09-05T16:33:44.869419+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T03:11:38.159719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":16,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-05T16:33:44.956973+0000","last_change":"2024-09-05T16:33:43.891112+0000","last_active":"2024-09-05T16:33:44.956973+0000","last_peered":"2024-09-05T16:33:44.956973+0000","last_clean":"2024-09-05T16:33:44.956973+0000","last_became_active":"2024-09-05T16:33:43.890413+0000","last_became_peered":"2024-09-05T16:33:43.890413+0000","last_unstale":"2024-09-05T16:33:44.956973+0000","last_undegraded":"2024-09-05T16:33:44.956973+0000","last_fullsized":"2024-09-05T16:33:44.956973+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T02:51:58.162036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"18'1","reported_seq":17,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-05T16:33:44.953958+0000","last_change":"2024-09-05T16:33:43.884357+0000","last_active":"2024-09-05T16:33:44.953958+0000","last_peered":"2024-09-05T16:33:44.953958+0000","last_clean":"2024-09-05T16:33:44.953958+0000","last_became_active":"2024-09-05T16:33:43.883972+0000","last_became_peered":"2024-09-05T16:33:43.883972+0000","last_unstale":"2024-09-05T16:33:44.953958+0000","last_undegraded":"2024-09-05T16:33:44.953958+0000","last_fullsized":"2024-09-05T16:33:44.953958+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T01:14:35.925787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":16,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-05T16:33:44.871591+0000","last_change":"2024-09-05T16:33:43.886478+0000","last_active":"2024-09-05T16:33:44.871591+0000","last_peered":"2024-09-05T16:33:44.871591+0000","last_clean":"2024-09-05T16:33:44.871591+0000","last_became_active":"2024-09-05T16:33:43.885565+0000","last_became_peered":"2024-09-05T16:33:43.885565+0000","last_unstale":"2024-09-05T16:33:44.871591+0000","last_undegraded":"2024-09-05T16:33:44.871591+0000","last_fullsized":"2024-09-05T16:33:44.871591+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:44:19.491731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":16,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-05T16:33:44.870728+0000","last_change":"2024-09-05T16:33:43.880726+0000","last_active":"2024-09-05T16:33:44.870728+0000","last_peered":"2024-09-05T16:33:44.870728+0000","last_clean":"2024-09-05T16:33:44.870728+0000","last_became_active":"2024-09-05T16:33:43.880349+0000","last_became_peered":"2024-09-05T16:33:43.880349+0000","last_unstale":"2024-09-05T16:33:44.870728+0000","last_undegraded":"2024-09-05T16:33:44.870728+0000","last_fullsized":"2024-09-05T16:33:44.870728+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T04:15:04.743842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-05T16:33:44.956944+0000","last_change":"2024-09-05T16:33:43.890813+0000","last_active":"2024-09-05T16:33:44.956944+0000","last_peered":"2024-09-05T16:33:44.956944+0000","last_clean":"2024-09-05T16:33:44.956944+0000","last_became_active":"2024-09-05T16:33:43.889961+0000","last_became_peered":"2024-09-05T16:33:43.889961+0000","last_unstale":"2024-09-05T16:33:44.956944+0000","last_undegraded":"2024-09-05T16:33:44.956944+0000","last_fullsized":"2024-09-05T16:33:44.956944+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T03:38:03.746292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"0'0","reported_seq":16,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-05T16:33:44.957048+0000","last_change":"2024-09-05T16:33:43.893959+0000","last_active":"2024-09-05T16:33:44.957048+0000","last_peered":"2024-09-05T16:33:44.957048+0000","last_clean":"2024-09-05T16:33:44.957048+0000","last_became_active":"2024-09-05T16:33:43.893550+0000","last_became_peered":"2024-09-05T16:33:43.893550+0000","last_unstale":"2024-09-05T16:33:44.957048+0000","last_undegraded":"2024-09-05T16:33:44.957048+0000","last_fullsized":"2024-09-05T16:33:44.957048+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T03:13:31.149229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"0'0","reported_seq":16,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-05T16:33:44.956984+0000","last_change":"2024-09-05T16:33:43.886333+0000","last_active":"2024-09-05T16:33:44.956984+0000","last_peered":"2024-09-05T16:33:44.956984+0000","last_clean":"2024-09-05T16:33:44.956984+0000","last_became_active":"2024-09-05T16:33:43.885840+0000","last_became_peered":"2024-09-05T16:33:43.885840+0000","last_unstale":"2024-09-05T16:33:44.956984+0000","last_undegraded":"2024-09-05T16:33:44.956984+0000","last_fullsized":"2024-09-05T16:33:44.956984+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:45:13.094481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"0'0","reported_seq":16,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-05T16:33:44.957113+0000","last_change":"2024-09-05T16:33:43.881131+0000","last_active":"2024-09-05T16:33:44.957113+0000","last_peered":"2024-09-05T16:33:44.957113+0000","last_clean":"2024-09-05T16:33:44.957113+0000","last_became_active":"2024-09-05T16:33:43.880598+0000","last_became_peered":"2024-09-05T16:33:43.880598+0000","last_unstale":"2024-09-05T16:33:44.957113+0000","last_undegraded":"2024-09-05T16:33:44.957113+0000","last_fullsized":"2024-09-05T16:33:44.957113+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T17:55:21.440218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"0'0","reported_seq":16,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-05T16:33:44.957183+0000","last_change":"2024-09-05T16:33:43.890336+0000","last_active":"2024-09-05T16:33:44.957183+0000","last_peered":"2024-09-05T16:33:44.957183+0000","last_clean":"2024-09-05T16:33:44.957183+0000","last_became_active":"2024-09-05T16:33:43.889915+0000","last_became_peered":"2024-09-05T16:33:43.889915+0000","last_unstale":"2024-09-05T16:33:44.957183+0000","last_undegraded":"2024-09-05T16:33:44.957183+0000","last_fullsized":"2024-09-05T16:33:44.957183+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:42:02.617765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640263,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":156,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":159744,"data_stored":46982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640265,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":156,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":159744,"data_stored":46982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949672967,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27192,"kb_used_data":156,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344648,"statfs":{"total":96636764160,"available":96608919552,"internally_reserved":0,"allocated":159744,"data_stored":46982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705673,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":156,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":159744,"data_stored":46982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:33:52.378 INFO:tasks.ec_lost_unfound.ceph_manager:recovered! 2024-09-05T16:33:52.379 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_1 /etc/group 2024-09-05T16:33:52.379 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_1 /etc/group 2024-09-05T16:33:52.430 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_1 /etc/group 2024-09-05T16:33:52.431 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_1 /etc/group 2024-09-05T16:33:52.481 INFO:tasks.util.rados:rados -p unique_pool_0 rm existed_1 2024-09-05T16:33:52.481 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 rm existed_1 2024-09-05T16:33:52.533 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_2 /etc/group 2024-09-05T16:33:52.534 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_2 /etc/group 2024-09-05T16:33:52.584 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_2 /etc/group 2024-09-05T16:33:52.585 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_2 /etc/group 2024-09-05T16:33:52.636 INFO:tasks.util.rados:rados -p unique_pool_0 rm existed_2 2024-09-05T16:33:52.636 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 rm existed_2 2024-09-05T16:33:52.685 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_3 /etc/group 2024-09-05T16:33:52.686 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_3 /etc/group 2024-09-05T16:33:52.737 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_3 /etc/group 2024-09-05T16:33:52.737 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_3 /etc/group 2024-09-05T16:33:52.788 INFO:tasks.util.rados:rados -p unique_pool_0 rm existed_3 2024-09-05T16:33:52.788 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 rm existed_3 2024-09-05T16:33:52.838 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_4 /etc/group 2024-09-05T16:33:52.838 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_4 /etc/group 2024-09-05T16:33:52.890 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_4 /etc/group 2024-09-05T16:33:52.890 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_4 /etc/group 2024-09-05T16:33:52.941 INFO:tasks.util.rados:rados -p unique_pool_0 rm existed_4 2024-09-05T16:33:52.942 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 rm existed_4 2024-09-05T16:33:52.991 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_5 /etc/group 2024-09-05T16:33:52.991 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_5 /etc/group 2024-09-05T16:33:53.041 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_5 /etc/group 2024-09-05T16:33:53.042 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_5 /etc/group 2024-09-05T16:33:53.093 INFO:tasks.util.rados:rados -p unique_pool_0 rm existed_5 2024-09-05T16:33:53.093 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 rm existed_5 2024-09-05T16:33:53.145 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_6 /etc/group 2024-09-05T16:33:53.145 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_6 /etc/group 2024-09-05T16:33:53.197 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_6 /etc/group 2024-09-05T16:33:53.197 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_6 /etc/group 2024-09-05T16:33:53.247 INFO:tasks.util.rados:rados -p unique_pool_0 rm existed_6 2024-09-05T16:33:53.247 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 rm existed_6 2024-09-05T16:33:53.299 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_7 /etc/group 2024-09-05T16:33:53.300 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_7 /etc/group 2024-09-05T16:33:53.351 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_7 /etc/group 2024-09-05T16:33:53.351 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_7 /etc/group 2024-09-05T16:33:53.401 INFO:tasks.util.rados:rados -p unique_pool_0 rm existed_7 2024-09-05T16:33:53.401 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 rm existed_7 2024-09-05T16:33:53.451 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_8 /etc/group 2024-09-05T16:33:53.451 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_8 /etc/group 2024-09-05T16:33:53.501 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_8 /etc/group 2024-09-05T16:33:53.502 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_8 /etc/group 2024-09-05T16:33:53.552 INFO:tasks.util.rados:rados -p unique_pool_0 rm existed_8 2024-09-05T16:33:53.552 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 rm existed_8 2024-09-05T16:33:53.601 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_9 /etc/group 2024-09-05T16:33:53.602 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_9 /etc/group 2024-09-05T16:33:53.654 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_9 /etc/group 2024-09-05T16:33:53.654 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_9 /etc/group 2024-09-05T16:33:53.704 INFO:tasks.util.rados:rados -p unique_pool_0 rm existed_9 2024-09-05T16:33:53.704 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 rm existed_9 2024-09-05T16:33:53.754 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 injectargs '--osd-recovery-delay-start 1000 --osd-min-pg-log-entries 100000000' 2024-09-05T16:33:53.875 INFO:teuthology.orchestra.run.smithi087.stdout:{} 2024-09-05T16:33:53.875 INFO:teuthology.orchestra.run.smithi087.stderr:osd_min_pg_log_entries = '' (not observed, change may require restart) osd_recovery_delay_start = '' 2024-09-05T16:33:53.886 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-09-05T16:33:53.886 INFO:teuthology.orchestra.run:waiting for 300 2024-09-05T16:33:53.922 INFO:tasks.ceph.osd.0:Stopped 2024-09-05T16:33:53.923 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd down 0 2024-09-05T16:33:54.255 INFO:tasks.ceph.mon.a.smithi087.stderr:2024-09-05T16:33:54.261+0000 7efd556cf640 -1 mon.a@0(leader).osd e18 definitely_dead 0 2024-09-05T16:33:54.897 INFO:teuthology.orchestra.run.smithi087.stderr:marked down osd.0. 2024-09-05T16:33:54.914 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-09-05T16:33:54.914 INFO:teuthology.orchestra.run:waiting for 300 2024-09-05T16:33:54.958 INFO:tasks.ceph.osd.3:Stopped 2024-09-05T16:33:54.958 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd down 3 2024-09-05T16:33:55.253 INFO:tasks.ceph.mon.a.smithi087.stderr:2024-09-05T16:33:55.259+0000 7efd556cf640 -1 mon.a@0(leader).osd e19 definitely_dead 0 2024-09-05T16:33:55.898 INFO:teuthology.orchestra.run.smithi087.stderr:marked down osd.3. 2024-09-05T16:33:55.915 INFO:tasks.util.rados:rados -p unique_pool_0 put new_1 /etc/group 2024-09-05T16:33:55.915 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put new_1 /etc/group 2024-09-05T16:33:56.912 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_1 /etc/group 2024-09-05T16:33:56.912 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_1 /etc/group 2024-09-05T16:33:56.966 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_1 /etc/group 2024-09-05T16:33:56.967 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_1 /etc/group 2024-09-05T16:33:57.018 INFO:tasks.util.rados:rados -p unique_pool_0 put new_2 /etc/group 2024-09-05T16:33:57.018 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put new_2 /etc/group 2024-09-05T16:33:57.069 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_2 /etc/group 2024-09-05T16:33:57.069 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_2 /etc/group 2024-09-05T16:33:57.119 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_2 /etc/group 2024-09-05T16:33:57.119 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_2 /etc/group 2024-09-05T16:33:57.170 INFO:tasks.util.rados:rados -p unique_pool_0 put new_3 /etc/group 2024-09-05T16:33:57.170 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put new_3 /etc/group 2024-09-05T16:33:57.221 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_3 /etc/group 2024-09-05T16:33:57.221 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_3 /etc/group 2024-09-05T16:33:57.271 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_3 /etc/group 2024-09-05T16:33:57.271 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_3 /etc/group 2024-09-05T16:33:57.322 INFO:tasks.util.rados:rados -p unique_pool_0 put new_4 /etc/group 2024-09-05T16:33:57.322 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put new_4 /etc/group 2024-09-05T16:33:57.372 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_4 /etc/group 2024-09-05T16:33:57.373 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_4 /etc/group 2024-09-05T16:33:57.423 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_4 /etc/group 2024-09-05T16:33:57.423 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_4 /etc/group 2024-09-05T16:33:57.474 INFO:tasks.util.rados:rados -p unique_pool_0 put new_5 /etc/group 2024-09-05T16:33:57.474 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put new_5 /etc/group 2024-09-05T16:33:57.524 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_5 /etc/group 2024-09-05T16:33:57.524 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_5 /etc/group 2024-09-05T16:33:57.575 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_5 /etc/group 2024-09-05T16:33:57.575 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_5 /etc/group 2024-09-05T16:33:57.626 INFO:tasks.util.rados:rados -p unique_pool_0 put new_6 /etc/group 2024-09-05T16:33:57.626 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put new_6 /etc/group 2024-09-05T16:33:57.677 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_6 /etc/group 2024-09-05T16:33:57.677 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_6 /etc/group 2024-09-05T16:33:57.728 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_6 /etc/group 2024-09-05T16:33:57.728 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_6 /etc/group 2024-09-05T16:33:57.780 INFO:tasks.util.rados:rados -p unique_pool_0 put new_7 /etc/group 2024-09-05T16:33:57.780 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put new_7 /etc/group 2024-09-05T16:33:57.830 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_7 /etc/group 2024-09-05T16:33:57.831 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_7 /etc/group 2024-09-05T16:33:57.881 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_7 /etc/group 2024-09-05T16:33:57.881 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_7 /etc/group 2024-09-05T16:33:57.932 INFO:tasks.util.rados:rados -p unique_pool_0 put new_8 /etc/group 2024-09-05T16:33:57.932 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put new_8 /etc/group 2024-09-05T16:33:57.983 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_8 /etc/group 2024-09-05T16:33:57.983 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_8 /etc/group 2024-09-05T16:33:58.035 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_8 /etc/group 2024-09-05T16:33:58.035 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_8 /etc/group 2024-09-05T16:33:58.085 INFO:tasks.util.rados:rados -p unique_pool_0 put new_9 /etc/group 2024-09-05T16:33:58.085 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put new_9 /etc/group 2024-09-05T16:33:58.135 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_9 /etc/group 2024-09-05T16:33:58.136 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_9 /etc/group 2024-09-05T16:33:58.186 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_9 /etc/group 2024-09-05T16:33:58.186 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_9 /etc/group 2024-09-05T16:33:58.236 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-05T16:33:58.236 INFO:teuthology.orchestra.run:waiting for 300 2024-09-05T16:33:58.267 INFO:tasks.ceph.osd.1:Stopped 2024-09-05T16:33:58.267 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd down 1 2024-09-05T16:33:58.598 INFO:tasks.ceph.mon.a.smithi087.stderr:2024-09-05T16:33:58.603+0000 7efd556cf640 -1 mon.a@0(leader).osd e21 definitely_dead 0 2024-09-05T16:33:58.917 INFO:teuthology.orchestra.run.smithi087.stderr:marked down osd.1. 2024-09-05T16:33:58.931 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd lost 1 --yes-i-really-mean-it 2024-09-05T16:33:59.922 INFO:teuthology.orchestra.run.smithi087.stderr:marked osd lost in epoch 22 2024-09-05T16:33:59.939 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-05T16:33:59.939 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-09-05T16:33:59.942 INFO:tasks.ceph.osd.0:Started 2024-09-05T16:33:59.942 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-05T16:34:00.046 INFO:teuthology.orchestra.run.smithi087.stderr:no valid command found; 10 closest matches: 2024-09-05T16:34:00.046 INFO:teuthology.orchestra.run.smithi087.stderr:0 2024-09-05T16:34:00.046 INFO:teuthology.orchestra.run.smithi087.stderr:1 2024-09-05T16:34:00.046 INFO:teuthology.orchestra.run.smithi087.stderr:2 2024-09-05T16:34:00.046 INFO:teuthology.orchestra.run.smithi087.stderr:abort 2024-09-05T16:34:00.046 INFO:teuthology.orchestra.run.smithi087.stderr:assert 2024-09-05T16:34:00.046 INFO:teuthology.orchestra.run.smithi087.stderr:config diff 2024-09-05T16:34:00.046 INFO:teuthology.orchestra.run.smithi087.stderr:config diff get 2024-09-05T16:34:00.046 INFO:teuthology.orchestra.run.smithi087.stderr:config get 2024-09-05T16:34:00.046 INFO:teuthology.orchestra.run.smithi087.stderr:config help [] 2024-09-05T16:34:00.046 INFO:teuthology.orchestra.run.smithi087.stderr:config set ... 2024-09-05T16:34:00.047 INFO:teuthology.orchestra.run.smithi087.stderr:admin_socket: invalid command 2024-09-05T16:34:00.048 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-05T16:34:00.049 INFO:tasks.ec_lost_unfound.ceph_manager:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2024-09-05T16:34:00.870 INFO:tasks.ceph.osd.0.smithi087.stderr:2024-09-05T16:34:00.875+0000 7fa9e9ee2640 -1 Falling back to public interface 2024-09-05T16:34:04.373 INFO:tasks.ceph.osd.0.smithi087.stderr:2024-09-05T16:34:04.378+0000 7fa9e9ee2640 -1 osd.0 18 log_to_monitors true 2024-09-05T16:34:04.938 INFO:tasks.ceph.osd.0.smithi087.stderr:2024-09-05T16:34:04.943+0000 7fa9e1c98640 -1 osd.0 18 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-05T16:34:05.050 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-05T16:34:05.165 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-09-05T16:34:05.165 INFO:teuthology.orchestra.run.smithi087.stdout: "ops": [], 2024-09-05T16:34:05.166 INFO:teuthology.orchestra.run.smithi087.stdout: "num_ops": 0 2024-09-05T16:34:05.166 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-09-05T16:34:05.174 INFO:tasks.ec_lost_unfound.ceph_manager:waiting for osd.0 to be up 2024-09-05T16:34:05.174 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-05T16:34:05.462 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:34:05.462 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":24,"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","created":"2024-09-05T16:33:15.770967+0000","modified":"2024-09-05T16:34:04.934012+0000","last_up_change":"2024-09-05T16:33:58.913778+0000","last_in_change":"2024-09-05T16:33:21.020289+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"unique_pool_0","create_time":"2024-09-05T16:33:40.865276+0000","flags":1,"flags_names":"hashpspool","type":3,"size":4,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"lost_unfound","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"f76bac77-d755-450f-9fbf-fb53a6033c24","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":16,"down_at":19,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6812","nonce":2286506651}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6813","nonce":2286506651}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6815","nonce":2286506651}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6814","nonce":2286506651}]},"public_addr":"172.21.15.87:6812/2286506651","cluster_addr":"172.21.15.87:6813/2286506651","heartbeat_back_addr":"172.21.15.87:6815/2286506651","heartbeat_front_addr":"172.21.15.87:6814/2286506651","state":["exists"]},{"osd":1,"uuid":"941684ea-0764-4182-94cc-136349df4f23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":20,"down_at":22,"lost_at":22,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6804","nonce":3459114360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6805","nonce":3459114360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6809","nonce":3459114360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6807","nonce":3459114360}]},"public_addr":"172.21.15.87:6804/3459114360","cluster_addr":"172.21.15.87:6805/3459114360","heartbeat_back_addr":"172.21.15.87:6809/3459114360","heartbeat_front_addr":"172.21.15.87:6807/3459114360","state":["exists"]},{"osd":2,"uuid":"17135934-ecb7-445a-b1dc-27f55da9f8ca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6800","nonce":4234108552}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6801","nonce":4234108552}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6803","nonce":4234108552}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6802","nonce":4234108552}]},"public_addr":"172.21.15.87:6800/4234108552","cluster_addr":"172.21.15.87:6801/4234108552","heartbeat_back_addr":"172.21.15.87:6803/4234108552","heartbeat_front_addr":"172.21.15.87:6802/4234108552","state":["exists","up"]},{"osd":3,"uuid":"098f9cbd-3da5-4ee7-8439-610b8cae774e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":19,"down_at":20,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6806","nonce":3527623826}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6808","nonce":3527623826}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6811","nonce":3527623826}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6810","nonce":3527623826}]},"public_addr":"172.21.15.87:6806/3527623826","cluster_addr":"172.21.15.87:6808/3527623826","heartbeat_back_addr":"172.21.15.87:6811/3527623826","heartbeat_front_addr":"172.21.15.87:6810/3527623826","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-05T16:33:27.513154+0000","dead_epoch":19},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-05T16:33:27.800689+0000","dead_epoch":22},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-05T16:33:28.127983+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-05T16:33:28.046083+0000","dead_epoch":20}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"lost_unfound":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-05T16:34:08.475 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-05T16:34:08.768 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:34:08.768 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":26,"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","created":"2024-09-05T16:33:15.770967+0000","modified":"2024-09-05T16:34:06.939578+0000","last_up_change":"2024-09-05T16:34:05.938461+0000","last_in_change":"2024-09-05T16:33:21.020289+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"unique_pool_0","create_time":"2024-09-05T16:33:40.865276+0000","flags":1,"flags_names":"hashpspool","type":3,"size":4,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"lost_unfound","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"f76bac77-d755-450f-9fbf-fb53a6033c24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":18,"up_from":25,"up_thru":25,"down_at":19,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6804","nonce":67704509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6805","nonce":67704509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6807","nonce":67704509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6806","nonce":67704509}]},"public_addr":"172.21.15.87:6804/67704509","cluster_addr":"172.21.15.87:6805/67704509","heartbeat_back_addr":"172.21.15.87:6807/67704509","heartbeat_front_addr":"172.21.15.87:6806/67704509","state":["exists","up"]},{"osd":1,"uuid":"941684ea-0764-4182-94cc-136349df4f23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":20,"down_at":22,"lost_at":22,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6804","nonce":3459114360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6805","nonce":3459114360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6809","nonce":3459114360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6807","nonce":3459114360}]},"public_addr":"172.21.15.87:6804/3459114360","cluster_addr":"172.21.15.87:6805/3459114360","heartbeat_back_addr":"172.21.15.87:6809/3459114360","heartbeat_front_addr":"172.21.15.87:6807/3459114360","state":["exists"]},{"osd":2,"uuid":"17135934-ecb7-445a-b1dc-27f55da9f8ca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6800","nonce":4234108552}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6801","nonce":4234108552}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6803","nonce":4234108552}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6802","nonce":4234108552}]},"public_addr":"172.21.15.87:6800/4234108552","cluster_addr":"172.21.15.87:6801/4234108552","heartbeat_back_addr":"172.21.15.87:6803/4234108552","heartbeat_front_addr":"172.21.15.87:6802/4234108552","state":["exists","up"]},{"osd":3,"uuid":"098f9cbd-3da5-4ee7-8439-610b8cae774e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":19,"down_at":20,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6806","nonce":3527623826}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6808","nonce":3527623826}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6811","nonce":3527623826}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6810","nonce":3527623826}]},"public_addr":"172.21.15.87:6806/3527623826","cluster_addr":"172.21.15.87:6808/3527623826","heartbeat_back_addr":"172.21.15.87:6811/3527623826","heartbeat_front_addr":"172.21.15.87:6810/3527623826","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-05T16:33:27.513154+0000","dead_epoch":19},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-05T16:33:27.800689+0000","dead_epoch":22},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-05T16:33:28.127983+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-05T16:33:28.046083+0000","dead_epoch":20}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"lost_unfound":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-05T16:34:08.781 INFO:tasks.ec_lost_unfound.ceph_manager:osd.0 is up 2024-09-05T16:34:08.781 INFO:tasks.ceph.osd.3:Restarting daemon 2024-09-05T16:34:08.781 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-09-05T16:34:08.783 INFO:tasks.ceph.osd.3:Started 2024-09-05T16:34:08.783 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-05T16:34:08.880 INFO:teuthology.orchestra.run.smithi087.stderr:no valid command found; 10 closest matches: 2024-09-05T16:34:08.881 INFO:teuthology.orchestra.run.smithi087.stderr:0 2024-09-05T16:34:08.881 INFO:teuthology.orchestra.run.smithi087.stderr:1 2024-09-05T16:34:08.881 INFO:teuthology.orchestra.run.smithi087.stderr:2 2024-09-05T16:34:08.881 INFO:teuthology.orchestra.run.smithi087.stderr:abort 2024-09-05T16:34:08.881 INFO:teuthology.orchestra.run.smithi087.stderr:assert 2024-09-05T16:34:08.881 INFO:teuthology.orchestra.run.smithi087.stderr:config diff 2024-09-05T16:34:08.881 INFO:teuthology.orchestra.run.smithi087.stderr:config diff get 2024-09-05T16:34:08.881 INFO:teuthology.orchestra.run.smithi087.stderr:config get 2024-09-05T16:34:08.881 INFO:teuthology.orchestra.run.smithi087.stderr:config help [] 2024-09-05T16:34:08.881 INFO:teuthology.orchestra.run.smithi087.stderr:config set ... 2024-09-05T16:34:08.881 INFO:teuthology.orchestra.run.smithi087.stderr:admin_socket: invalid command 2024-09-05T16:34:08.883 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-05T16:34:08.883 INFO:tasks.ec_lost_unfound.ceph_manager:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-09-05T16:34:09.711 INFO:tasks.ceph.osd.3.smithi087.stderr:2024-09-05T16:34:09.717+0000 7f31d262c640 -1 Falling back to public interface 2024-09-05T16:34:10.828 INFO:tasks.ceph.mon.a.smithi087.stderr:2024-09-05T16:34:10.834+0000 7efd57ed4640 -1 log_channel(cluster) log [ERR] : Health check failed: Possible data damage: 14 pgs recovery_unfound (PG_DAMAGED) 2024-09-05T16:34:13.749 INFO:tasks.ceph.osd.3.smithi087.stderr:2024-09-05T16:34:13.755+0000 7f31d262c640 -1 osd.3 19 log_to_monitors true 2024-09-05T16:34:13.884 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-05T16:34:13.897 INFO:tasks.ceph.osd.3.smithi087.stderr:2024-09-05T16:34:13.904+0000 7f31c9a61640 -1 osd.3 19 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-05T16:34:13.990 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-09-05T16:34:13.990 INFO:teuthology.orchestra.run.smithi087.stdout: "ops": [], 2024-09-05T16:34:13.990 INFO:teuthology.orchestra.run.smithi087.stdout: "num_ops": 0 2024-09-05T16:34:13.990 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-09-05T16:34:13.998 INFO:tasks.ec_lost_unfound.ceph_manager:waiting for osd.3 to be up 2024-09-05T16:34:13.998 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-05T16:34:14.322 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:34:14.322 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":27,"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","created":"2024-09-05T16:33:15.770967+0000","modified":"2024-09-05T16:34:13.889790+0000","last_up_change":"2024-09-05T16:34:05.938461+0000","last_in_change":"2024-09-05T16:33:21.020289+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"unique_pool_0","create_time":"2024-09-05T16:33:40.865276+0000","flags":1,"flags_names":"hashpspool","type":3,"size":4,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"lost_unfound","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"f76bac77-d755-450f-9fbf-fb53a6033c24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":18,"up_from":25,"up_thru":25,"down_at":19,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6804","nonce":67704509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6805","nonce":67704509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6807","nonce":67704509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6806","nonce":67704509}]},"public_addr":"172.21.15.87:6804/67704509","cluster_addr":"172.21.15.87:6805/67704509","heartbeat_back_addr":"172.21.15.87:6807/67704509","heartbeat_front_addr":"172.21.15.87:6806/67704509","state":["exists","up"]},{"osd":1,"uuid":"941684ea-0764-4182-94cc-136349df4f23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":20,"down_at":22,"lost_at":22,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6804","nonce":3459114360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6805","nonce":3459114360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6809","nonce":3459114360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6807","nonce":3459114360}]},"public_addr":"172.21.15.87:6804/3459114360","cluster_addr":"172.21.15.87:6805/3459114360","heartbeat_back_addr":"172.21.15.87:6809/3459114360","heartbeat_front_addr":"172.21.15.87:6807/3459114360","state":["exists"]},{"osd":2,"uuid":"17135934-ecb7-445a-b1dc-27f55da9f8ca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6800","nonce":4234108552}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6801","nonce":4234108552}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6803","nonce":4234108552}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6802","nonce":4234108552}]},"public_addr":"172.21.15.87:6800/4234108552","cluster_addr":"172.21.15.87:6801/4234108552","heartbeat_back_addr":"172.21.15.87:6803/4234108552","heartbeat_front_addr":"172.21.15.87:6802/4234108552","state":["exists","up"]},{"osd":3,"uuid":"098f9cbd-3da5-4ee7-8439-610b8cae774e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":19,"down_at":20,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6806","nonce":3527623826}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6808","nonce":3527623826}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6811","nonce":3527623826}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6810","nonce":3527623826}]},"public_addr":"172.21.15.87:6806/3527623826","cluster_addr":"172.21.15.87:6808/3527623826","heartbeat_back_addr":"172.21.15.87:6811/3527623826","heartbeat_front_addr":"172.21.15.87:6810/3527623826","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-05T16:33:27.513154+0000","dead_epoch":19},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-05T16:33:27.800689+0000","dead_epoch":22},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-05T16:33:28.127983+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-05T16:33:28.046083+0000","dead_epoch":20}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"lost_unfound":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-05T16:34:16.904 INFO:tasks.ceph.mon.a.smithi087.stderr:2024-09-05T16:34:16.911+0000 7efd57ed4640 -1 log_channel(cluster) log [ERR] : Health check update: Possible data damage: 5 pgs recovery_unfound (PG_DAMAGED) 2024-09-05T16:34:17.335 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-05T16:34:17.631 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:34:17.631 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":29,"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","created":"2024-09-05T16:33:15.770967+0000","modified":"2024-09-05T16:34:15.897173+0000","last_up_change":"2024-09-05T16:34:14.895894+0000","last_in_change":"2024-09-05T16:33:21.020289+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"unique_pool_0","create_time":"2024-09-05T16:33:40.865276+0000","flags":1,"flags_names":"hashpspool","type":3,"size":4,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"lost_unfound","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"f76bac77-d755-450f-9fbf-fb53a6033c24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":18,"up_from":25,"up_thru":28,"down_at":19,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6804","nonce":67704509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6805","nonce":67704509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6807","nonce":67704509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6806","nonce":67704509}]},"public_addr":"172.21.15.87:6804/67704509","cluster_addr":"172.21.15.87:6805/67704509","heartbeat_back_addr":"172.21.15.87:6807/67704509","heartbeat_front_addr":"172.21.15.87:6806/67704509","state":["exists","up"]},{"osd":1,"uuid":"941684ea-0764-4182-94cc-136349df4f23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":20,"down_at":22,"lost_at":22,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6804","nonce":3459114360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6805","nonce":3459114360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6809","nonce":3459114360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6807","nonce":3459114360}]},"public_addr":"172.21.15.87:6804/3459114360","cluster_addr":"172.21.15.87:6805/3459114360","heartbeat_back_addr":"172.21.15.87:6809/3459114360","heartbeat_front_addr":"172.21.15.87:6807/3459114360","state":["exists"]},{"osd":2,"uuid":"17135934-ecb7-445a-b1dc-27f55da9f8ca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6800","nonce":4234108552}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6801","nonce":4234108552}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6803","nonce":4234108552}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6802","nonce":4234108552}]},"public_addr":"172.21.15.87:6800/4234108552","cluster_addr":"172.21.15.87:6801/4234108552","heartbeat_back_addr":"172.21.15.87:6803/4234108552","heartbeat_front_addr":"172.21.15.87:6802/4234108552","state":["exists","up"]},{"osd":3,"uuid":"098f9cbd-3da5-4ee7-8439-610b8cae774e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":11,"last_clean_end":19,"up_from":28,"up_thru":28,"down_at":20,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6808","nonce":1182532508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6809","nonce":1182532508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6811","nonce":1182532508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6810","nonce":1182532508}]},"public_addr":"172.21.15.87:6808/1182532508","cluster_addr":"172.21.15.87:6809/1182532508","heartbeat_back_addr":"172.21.15.87:6811/1182532508","heartbeat_front_addr":"172.21.15.87:6810/1182532508","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-05T16:33:27.513154+0000","dead_epoch":19},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-05T16:33:27.800689+0000","dead_epoch":22},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-05T16:33:28.127983+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-05T16:33:28.046083+0000","dead_epoch":20}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"lost_unfound":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-05T16:34:17.644 INFO:tasks.ec_lost_unfound.ceph_manager:osd.3 is up 2024-09-05T16:34:17.644 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-05T16:34:17.645 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-05T16:34:17.645 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-05T16:34:17.782 INFO:teuthology.orchestra.run.smithi087.stdout:107374182404 2024-09-05T16:34:17.782 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-05T16:34:17.786 INFO:teuthology.orchestra.run.smithi087.stdout:120259084290 2024-09-05T16:34:17.786 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-05T16:34:17.787 INFO:teuthology.orchestra.run.smithi087.stdout:42949672973 2024-09-05T16:34:17.787 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-05T16:34:18.087 INFO:teuthology.orchestra.run.smithi087.stdout:107374182403 2024-09-05T16:34:18.100 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 107374182404 got 107374182403 for osd.0 2024-09-05T16:34:18.128 INFO:teuthology.orchestra.run.smithi087.stdout:1 2024-09-05T16:34:18.132 INFO:teuthology.orchestra.run.smithi087.stdout:42949672972 2024-09-05T16:34:18.140 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 120259084290 got 1 for osd.3 2024-09-05T16:34:18.144 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 42949672973 got 42949672972 for osd.2 2024-09-05T16:34:19.102 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-05T16:34:19.141 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-05T16:34:19.145 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-05T16:34:19.418 INFO:teuthology.orchestra.run.smithi087.stdout:107374182404 2024-09-05T16:34:19.430 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 107374182404 got 107374182404 for osd.0 2024-09-05T16:34:19.431 DEBUG:teuthology.parallel:result is None 2024-09-05T16:34:19.454 INFO:teuthology.orchestra.run.smithi087.stdout:120259084290 2024-09-05T16:34:19.456 INFO:teuthology.orchestra.run.smithi087.stdout:42949672973 2024-09-05T16:34:19.467 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 120259084290 got 120259084290 for osd.3 2024-09-05T16:34:19.467 DEBUG:teuthology.parallel:result is None 2024-09-05T16:34:19.469 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 42949672973 got 42949672973 for osd.2 2024-09-05T16:34:19.469 DEBUG:teuthology.parallel:result is None 2024-09-05T16:34:19.469 INFO:tasks.ec_lost_unfound.ceph_manager:waiting till active 2024-09-05T16:34:19.469 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:34:19.789 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:34:19.789 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:34:19.801 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":51,"stamp":"2024-09-05T16:34:18.786540+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":188608,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":82,"num_objects_misplaced":0,"num_objects_unfound":27,"num_objects_dirty":28,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":55,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":110212,"kb_used_data":1120,"kb_used_omap":11,"kb_used_meta":108980,"kb_avail":377377148,"statfs":{"total":386547056640,"available":386434199552,"internally_reserved":0,"allocated":1146880,"data_stored":528253,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11990,"internal_metadata":111595818},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":3,"num_objects_missing":3,"num_objects_degraded":-1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001515"},"pg_stats":[{"pgid":"1.f","version":"0'0","reported_seq":29,"reported_epoch":29,"state":"active+undersized","last_fresh":"2024-09-05T16:34:15.914062+0000","last_change":"2024-09-05T16:34:15.913051+0000","last_active":"2024-09-05T16:34:15.914062+0000","last_peered":"2024-09-05T16:34:15.914062+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:15.913051+0000","last_became_peered":"2024-09-05T16:34:15.913051+0000","last_unstale":"2024-09-05T16:34:15.914062+0000","last_undegraded":"2024-09-05T16:34:15.914062+0000","last_fullsized":"2024-09-05T16:34:15.905613+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T19:25:44.252388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2147483647,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"21'4","reported_seq":44,"reported_epoch":29,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-05T16:34:15.920679+0000","last_change":"2024-09-05T16:34:15.920955+0000","last_active":"2024-09-05T16:34:15.920679+0000","last_peered":"2024-09-05T16:34:15.920679+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:15.910788+0000","last_became_peered":"2024-09-05T16:34:15.910788+0000","last_unstale":"2024-09-05T16:34:15.920679+0000","last_undegraded":"2024-09-05T16:34:15.903622+0000","last_fullsized":"2024-09-05T16:34:15.903245+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["2(3)"],"object_location_counts":[{"shards":"2(3)","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"21'3","reported_seq":56,"reported_epoch":29,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-05T16:34:15.922091+0000","last_change":"2024-09-05T16:34:15.922327+0000","last_active":"2024-09-05T16:34:15.922091+0000","last_peered":"2024-09-05T16:34:15.922091+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:15.917951+0000","last_became_peered":"2024-09-05T16:34:15.917951+0000","last_unstale":"2024-09-05T16:34:15.922091+0000","last_undegraded":"2024-09-05T16:34:15.904811+0000","last_fullsized":"2024-09-05T16:34:15.904154+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":1,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,2147483647,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)"],"object_location_counts":[{"shards":"2(0)","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"21'4","reported_seq":44,"reported_epoch":29,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-05T16:34:15.924002+0000","last_change":"2024-09-05T16:34:15.924273+0000","last_active":"2024-09-05T16:34:15.924002+0000","last_peered":"2024-09-05T16:34:15.924002+0000","last_clean":"2024-09-05T16:33:53.148717+0000","last_became_active":"2024-09-05T16:34:15.912196+0000","last_became_peered":"2024-09-05T16:34:15.912196+0000","last_unstale":"2024-09-05T16:34:15.924002+0000","last_undegraded":"2024-09-05T16:34:15.905010+0000","last_fullsized":"2024-09-05T16:34:15.904610+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["2(2)"],"object_location_counts":[{"shards":"2(2)","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"21'3","reported_seq":32,"reported_epoch":29,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-05T16:34:15.918995+0000","last_change":"2024-09-05T16:34:15.919374+0000","last_active":"2024-09-05T16:34:15.918995+0000","last_peered":"2024-09-05T16:34:15.918995+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:15.910648+0000","last_became_peered":"2024-09-05T16:34:15.910648+0000","last_unstale":"2024-09-05T16:34:15.918995+0000","last_undegraded":"2024-09-05T16:34:15.904665+0000","last_fullsized":"2024-09-05T16:34:15.903277+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2147483647,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["2(3)"],"object_location_counts":[{"shards":"2(3)","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"0'0","reported_seq":29,"reported_epoch":29,"state":"active+undersized","last_fresh":"2024-09-05T16:34:15.914637+0000","last_change":"2024-09-05T16:34:15.913477+0000","last_active":"2024-09-05T16:34:15.914637+0000","last_peered":"2024-09-05T16:34:15.914637+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:15.913477+0000","last_became_peered":"2024-09-05T16:34:15.913477+0000","last_unstale":"2024-09-05T16:34:15.914637+0000","last_undegraded":"2024-09-05T16:34:15.914637+0000","last_fullsized":"2024-09-05T16:34:15.906631+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T01:56:46.762011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,2147483647],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"21'6","reported_seq":46,"reported_epoch":29,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-05T16:34:15.925143+0000","last_change":"2024-09-05T16:34:15.925438+0000","last_active":"2024-09-05T16:34:15.925143+0000","last_peered":"2024-09-05T16:34:15.925143+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:15.913479+0000","last_became_peered":"2024-09-05T16:34:15.913479+0000","last_unstale":"2024-09-05T16:34:15.925143+0000","last_undegraded":"2024-09-05T16:34:15.905955+0000","last_fullsized":"2024-09-05T16:34:15.905617+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2147483647,2],"acting":[0,3,2147483647,2],"avail_no_missing":["2(3)"],"object_location_counts":[{"shards":"2(3)","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"21'5","reported_seq":45,"reported_epoch":29,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-05T16:34:15.916874+0000","last_change":"2024-09-05T16:34:15.917364+0000","last_active":"2024-09-05T16:34:15.916874+0000","last_peered":"2024-09-05T16:34:15.916874+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:15.910570+0000","last_became_peered":"2024-09-05T16:34:15.910570+0000","last_unstale":"2024-09-05T16:34:15.916874+0000","last_undegraded":"2024-09-05T16:34:15.904563+0000","last_fullsized":"2024-09-05T16:34:15.903952+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["2(2)"],"object_location_counts":[{"shards":"2(2)","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"21'9","reported_seq":37,"reported_epoch":29,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-05T16:34:15.927234+0000","last_change":"2024-09-05T16:34:15.927619+0000","last_active":"2024-09-05T16:34:15.927234+0000","last_peered":"2024-09-05T16:34:15.927234+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:15.914297+0000","last_became_peered":"2024-09-05T16:34:15.914297+0000","last_unstale":"2024-09-05T16:34:15.927234+0000","last_undegraded":"2024-09-05T16:34:15.906625+0000","last_fullsized":"2024-09-05T16:34:15.906328+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":3,"num_objects_missing":3,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":3,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,2147483647],"acting":[3,0,2,2147483647],"avail_no_missing":["2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1},{"shards":"2(2)","objects":3}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"21'2","reported_seq":54,"reported_epoch":29,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-05T16:34:15.926463+0000","last_change":"2024-09-05T16:34:15.926681+0000","last_active":"2024-09-05T16:34:15.926463+0000","last_peered":"2024-09-05T16:34:15.926463+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:15.918296+0000","last_became_peered":"2024-09-05T16:34:15.918296+0000","last_unstale":"2024-09-05T16:34:15.926463+0000","last_undegraded":"2024-09-05T16:34:15.905187+0000","last_fullsized":"2024-09-05T16:34:15.904834+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,2147483647],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)"],"object_location_counts":[{"shards":"2(0)","objects":2}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"21'5","reported_seq":33,"reported_epoch":29,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-05T16:34:15.925674+0000","last_change":"2024-09-05T16:34:15.925969+0000","last_active":"2024-09-05T16:34:15.925674+0000","last_peered":"2024-09-05T16:34:15.925674+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:15.914271+0000","last_became_peered":"2024-09-05T16:34:15.914271+0000","last_unstale":"2024-09-05T16:34:15.925674+0000","last_undegraded":"2024-09-05T16:34:15.905903+0000","last_fullsized":"2024-09-05T16:34:15.905357+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":3,"num_objects_missing":3,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":3,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2147483647,2],"acting":[3,0,2147483647,2],"avail_no_missing":["2(3)"],"object_location_counts":[{"shards":"2(3)","objects":3}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"21'1","reported_seq":16,"reported_epoch":29,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-05T16:34:15.923439+0000","last_change":"2024-09-05T16:34:15.923439+0000","last_active":"2024-09-05T16:34:15.923439+0000","last_peered":"2024-09-05T16:34:15.923439+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:15.913627+0000","last_became_peered":"2024-09-05T16:34:15.913627+0000","last_unstale":"2024-09-05T16:34:15.923439+0000","last_undegraded":"2024-09-05T16:34:15.906286+0000","last_fullsized":"2024-09-05T16:34:15.906044+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":1,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["2(2)"],"object_location_counts":[{"shards":"2(2)","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"21'1","reported_seq":54,"reported_epoch":29,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-05T16:34:15.920018+0000","last_change":"2024-09-05T16:34:15.920326+0000","last_active":"2024-09-05T16:34:15.920018+0000","last_peered":"2024-09-05T16:34:15.920018+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:15.917839+0000","last_became_peered":"2024-09-05T16:34:15.917839+0000","last_unstale":"2024-09-05T16:34:15.920018+0000","last_undegraded":"2024-09-05T16:34:15.904338+0000","last_fullsized":"2024-09-05T16:34:15.904018+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":1,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)"],"object_location_counts":[{"shards":"2(1)","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"21'4","reported_seq":32,"reported_epoch":29,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-05T16:34:15.921542+0000","last_change":"2024-09-05T16:34:15.921542+0000","last_active":"2024-09-05T16:34:15.921542+0000","last_peered":"2024-09-05T16:34:15.921542+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:15.913175+0000","last_became_peered":"2024-09-05T16:34:15.913175+0000","last_unstale":"2024-09-05T16:34:15.921542+0000","last_undegraded":"2024-09-05T16:34:15.905345+0000","last_fullsized":"2024-09-05T16:34:15.905038+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["2(2)"],"object_location_counts":[{"shards":"2(2)","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"21'3","reported_seq":57,"reported_epoch":29,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-05T16:34:15.924563+0000","last_change":"2024-09-05T16:34:15.924816+0000","last_active":"2024-09-05T16:34:15.924563+0000","last_peered":"2024-09-05T16:34:15.924563+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:15.917968+0000","last_became_peered":"2024-09-05T16:34:15.917968+0000","last_unstale":"2024-09-05T16:34:15.924563+0000","last_undegraded":"2024-09-05T16:34:15.904385+0000","last_fullsized":"2024-09-05T16:34:15.904040+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)"],"object_location_counts":[{"shards":"2(1)","objects":2}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"21'5","reported_seq":45,"reported_epoch":29,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-05T16:34:15.922625+0000","last_change":"2024-09-05T16:34:15.922928+0000","last_active":"2024-09-05T16:34:15.922625+0000","last_peered":"2024-09-05T16:34:15.922625+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:15.910947+0000","last_became_peered":"2024-09-05T16:34:15.910947+0000","last_unstale":"2024-09-05T16:34:15.922625+0000","last_undegraded":"2024-09-05T16:34:15.905253+0000","last_fullsized":"2024-09-05T16:34:15.904912+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["2(2)"],"object_location_counts":[{"shards":"2(2)","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":188608,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":82,"num_objects_misplaced":0,"num_objects_unfound":27,"num_objects_dirty":28,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":55,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":237568,"data_stored":237568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084290,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27828,"kb_used_data":280,"kb_used_omap":4,"kb_used_meta":27515,"kb_avail":94344012,"statfs":{"total":96636764160,"available":96608268288,"internally_reserved":0,"allocated":286720,"data_stored":121263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":28176083},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640267,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":212992,"data_stored":90736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949672973,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27380,"kb_used_data":352,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344460,"statfs":{"total":96636764160,"available":96608727040,"internally_reserved":0,"allocated":360448,"data_stored":194991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182404,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27764,"kb_used_data":280,"kb_used_omap":4,"kb_used_meta":27451,"kb_avail":94344076,"statfs":{"total":96636764160,"available":96608333824,"internally_reserved":0,"allocated":286720,"data_stored":121263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":28110541},"hb_peers":[2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":40960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":40960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":114688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":40960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:34:19.802 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:34:20.150 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:34:20.151 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_ERR","checks":{"OBJECT_UNFOUND":{"severity":"HEALTH_WARN","summary":{"message":"27/28 objects unfound (96.429%)","count":27},"muted":false},"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DAMAGED":{"severity":"HEALTH_ERR","summary":{"message":"Possible data damage: 14 pgs recovery_unfound","count":14},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 82/112 objects degraded (73.214%), 14 pgs degraded","count":14},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":63,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":29,"num_osds":4,"num_up_osds":3,"osd_up_since":1725554054,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_unfound+undersized+degraded","count":14},{"state_name":"active+undersized","count":2}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":188608,"bytes_used":112857088,"bytes_avail":386434199552,"bytes_total":386547056640,"degraded_objects":82,"degraded_total":112,"degraded_ratio":0.7321428571428571,"unfound_objects":27,"unfound_total":28,"unfound_ratio":0.9642857142857143},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":1,"modified":"2024-09-05T16:33:15.771822+0000","services":{}},"progress_events":{}} 2024-09-05T16:34:20.163 INFO:tasks.ec_lost_unfound.ceph_manager:{'fsid': 'f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc', 'health': {'status': 'HEALTH_ERR', 'checks': {'OBJECT_UNFOUND': {'severity': 'HEALTH_WARN', 'summary': {'message': '27/28 objects unfound (96.429%)', 'count': 27}, 'muted': False}, 'OSD_DOWN': {'severity': 'HEALTH_WARN', 'summary': {'message': '1 osds down', 'count': 1}, 'muted': False}, 'PG_DAMAGED': {'severity': 'HEALTH_ERR', 'summary': {'message': 'Possible data damage: 14 pgs recovery_unfound', 'count': 14}, 'muted': False}, 'PG_DEGRADED': {'severity': 'HEALTH_WARN', 'summary': {'message': 'Degraded data redundancy: 82/112 objects degraded (73.214%), 14 pgs degraded', 'count': 14}, 'muted': False}}, 'mutes': []}, 'election_epoch': 10, 'quorum': [0, 1, 2], 'quorum_names': ['a', 'b', 'c'], 'quorum_age': 63, 'monmap': {'epoch': 1, 'min_mon_release_name': 'squid', 'num_mons': 3}, 'osdmap': {'epoch': 29, 'num_osds': 4, 'num_up_osds': 3, 'osd_up_since': 1725554054, 'num_in_osds': 4, 'osd_in_since': 1725554001, 'num_remapped_pgs': 0}, 'pgmap': {'pgs_by_state': [{'state_name': 'active+recovery_unfound+undersized+degraded', 'count': 14}, {'state_name': 'active+undersized', 'count': 2}], 'num_pgs': 16, 'num_pools': 1, 'num_objects': 28, 'data_bytes': 188608, 'bytes_used': 112857088, 'bytes_avail': 386434199552, 'bytes_total': 386547056640, 'degraded_objects': 82, 'degraded_total': 112, 'degraded_ratio': 0.7321428571428571, 'unfound_objects': 27, 'unfound_total': 28, 'unfound_ratio': 0.9642857142857143}, 'fsmap': {'epoch': 1, 'btime': '2024-09-05T16:33:15:768647+0000', 'by_rank': [], 'up:standby': 0}, 'mgrmap': {'available': True, 'num_standbys': 0, 'modules': ['iostat', 'nfs', 'restful'], 'services': {}}, 'servicemap': {'epoch': 1, 'modified': '2024-09-05T16:33:15.771822+0000', 'services': {}}, 'progress_events': {}} 2024-09-05T16:34:20.163 INFO:tasks.ec_lost_unfound.ceph_manager:active! 2024-09-05T16:34:20.163 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-05T16:34:20.164 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-05T16:34:20.164 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-05T16:34:20.300 INFO:teuthology.orchestra.run.smithi087.stdout:42949672975 2024-09-05T16:34:20.300 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-05T16:34:20.304 INFO:teuthology.orchestra.run.smithi087.stdout:107374182406 2024-09-05T16:34:20.304 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-05T16:34:20.305 INFO:teuthology.orchestra.run.smithi087.stdout:120259084292 2024-09-05T16:34:20.305 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-05T16:34:20.603 INFO:teuthology.orchestra.run.smithi087.stdout:42949672973 2024-09-05T16:34:20.616 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 42949672975 got 42949672973 for osd.2 2024-09-05T16:34:20.657 INFO:teuthology.orchestra.run.smithi087.stdout:120259084290 2024-09-05T16:34:20.661 INFO:teuthology.orchestra.run.smithi087.stdout:107374182404 2024-09-05T16:34:20.671 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 120259084292 got 120259084290 for osd.3 2024-09-05T16:34:20.674 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 107374182406 got 107374182404 for osd.0 2024-09-05T16:34:21.617 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-05T16:34:21.671 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-05T16:34:21.675 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-05T16:34:21.926 INFO:teuthology.orchestra.run.smithi087.stdout:42949672975 2024-09-05T16:34:21.940 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 42949672975 got 42949672975 for osd.2 2024-09-05T16:34:21.940 DEBUG:teuthology.parallel:result is None 2024-09-05T16:34:21.977 INFO:teuthology.orchestra.run.smithi087.stdout:120259084292 2024-09-05T16:34:21.990 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 120259084292 got 120259084292 for osd.3 2024-09-05T16:34:21.990 DEBUG:teuthology.parallel:result is None 2024-09-05T16:34:22.020 INFO:teuthology.orchestra.run.smithi087.stdout:107374182406 2024-09-05T16:34:22.033 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 107374182406 got 107374182406 for osd.0 2024-09-05T16:34:22.033 DEBUG:teuthology.parallel:result is None 2024-09-05T16:34:22.033 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:34:22.425 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:34:22.426 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_ERR","checks":{"OBJECT_UNFOUND":{"severity":"HEALTH_WARN","summary":{"message":"27/28 objects unfound (96.429%)","count":27},"muted":false},"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DAMAGED":{"severity":"HEALTH_ERR","summary":{"message":"Possible data damage: 14 pgs recovery_unfound","count":14},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 82/112 objects degraded (73.214%), 14 pgs degraded","count":14},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":65,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":29,"num_osds":4,"num_up_osds":3,"osd_up_since":1725554054,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_unfound+undersized+degraded","count":14},{"state_name":"active+undersized","count":2}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":188608,"bytes_used":112857088,"bytes_avail":386434199552,"bytes_total":386547056640,"degraded_objects":82,"degraded_total":112,"degraded_ratio":0.7321428571428571,"unfound_objects":27,"unfound_total":28,"unfound_ratio":0.9642857142857143},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":1,"modified":"2024-09-05T16:33:15.771822+0000","services":{}},"progress_events":{}} 2024-09-05T16:34:22.438 INFO:tasks.ec_lost_unfound.ceph_manager:{'fsid': 'f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc', 'health': {'status': 'HEALTH_ERR', 'checks': {'OBJECT_UNFOUND': {'severity': 'HEALTH_WARN', 'summary': {'message': '27/28 objects unfound (96.429%)', 'count': 27}, 'muted': False}, 'OSD_DOWN': {'severity': 'HEALTH_WARN', 'summary': {'message': '1 osds down', 'count': 1}, 'muted': False}, 'PG_DAMAGED': {'severity': 'HEALTH_ERR', 'summary': {'message': 'Possible data damage: 14 pgs recovery_unfound', 'count': 14}, 'muted': False}, 'PG_DEGRADED': {'severity': 'HEALTH_WARN', 'summary': {'message': 'Degraded data redundancy: 82/112 objects degraded (73.214%), 14 pgs degraded', 'count': 14}, 'muted': False}}, 'mutes': []}, 'election_epoch': 10, 'quorum': [0, 1, 2], 'quorum_names': ['a', 'b', 'c'], 'quorum_age': 65, 'monmap': {'epoch': 1, 'min_mon_release_name': 'squid', 'num_mons': 3}, 'osdmap': {'epoch': 29, 'num_osds': 4, 'num_up_osds': 3, 'osd_up_since': 1725554054, 'num_in_osds': 4, 'osd_in_since': 1725554001, 'num_remapped_pgs': 0}, 'pgmap': {'pgs_by_state': [{'state_name': 'active+recovery_unfound+undersized+degraded', 'count': 14}, {'state_name': 'active+undersized', 'count': 2}], 'num_pgs': 16, 'num_pools': 1, 'num_objects': 28, 'data_bytes': 188608, 'bytes_used': 112857088, 'bytes_avail': 386434199552, 'bytes_total': 386547056640, 'degraded_objects': 82, 'degraded_total': 112, 'degraded_ratio': 0.7321428571428571, 'unfound_objects': 27, 'unfound_total': 28, 'unfound_ratio': 0.9642857142857143}, 'fsmap': {'epoch': 1, 'btime': '2024-09-05T16:33:15:768647+0000', 'by_rank': [], 'up:standby': 0}, 'mgrmap': {'available': True, 'num_standbys': 0, 'modules': ['iostat', 'nfs', 'restful'], 'services': {}}, 'servicemap': {'epoch': 1, 'modified': '2024-09-05T16:33:15.771822+0000', 'services': {}}, 'progress_events': {}} 2024-09-05T16:34:22.438 INFO:tasks.ec_lost_unfound:there are 27 unfound objects 2024-09-05T16:34:22.438 DEBUG:teuthology.orchestra.run.smithi087:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.admin -b 4096 -p unique_pool_0 -t 20 bench 240 write' 2024-09-05T16:34:22.490 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stderr:INFO: op_size has been rounded to 8192 2024-09-05T16:34:22.490 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout:hints = 1 2024-09-05T16:34:22.490 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout:Maintaining 20 concurrent writes of 8192 bytes to objects of size 8192 for up to 240 seconds or 0 objects 2024-09-05T16:34:22.490 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout:Object prefix: benchmark_data_smithi087_45591 2024-09-05T16:34:22.490 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-05T16:34:22.490 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 0 0 0 0 0 0 - 0 2024-09-05T16:34:23.490 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 1 20 1694 1674 13.0724 13.0781 0.0119868 0.0118881 2024-09-05T16:34:24.490 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 2 20 3341 3321 12.9693 12.8672 0.0169625 0.0120017 2024-09-05T16:34:25.490 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 3 20 4923 4903 12.7657 12.3594 0.0162364 0.0121923 2024-09-05T16:34:25.541 INFO:tasks.ceph.mon.a.smithi087.stderr:2024-09-05T16:34:25.546+0000 7efd57ed4640 -1 log_channel(cluster) log [ERR] : Health check update: Possible data damage: 14 pgs recovery_unfound (PG_DAMAGED) 2024-09-05T16:34:26.490 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 4 20 6477 6457 12.609 12.1406 0.010577 0.0123648 2024-09-05T16:34:27.492 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 5 20 8000 7980 12.4667 11.8984 0.0272812 0.0125116 2024-09-05T16:34:28.491 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 6 20 9469 9449 12.3015 11.4766 0.0135256 0.012681 2024-09-05T16:34:29.491 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 7 20 10862 10842 12.0987 10.8828 0.0124288 0.0128996 2024-09-05T16:34:30.491 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 8 20 12245 12225 11.9369 10.8047 0.0128739 0.0130727 2024-09-05T16:34:31.491 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 9 20 13605 13585 11.791 10.625 0.0186741 0.0132382 2024-09-05T16:34:32.442 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:34:32.491 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 10 20 14906 14886 11.6283 10.1641 0.0128535 0.0134184 2024-09-05T16:34:32.996 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:34:32.996 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:34:33.017 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":58,"stamp":"2024-09-05T16:34:32.790717+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":89522368,"num_objects":10933,"num_object_clones":0,"num_object_copies":43732,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":10987,"num_objects_misplaced":0,"num_objects_unfound":27,"num_objects_dirty":10933,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10960,"num_write_kb":87562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10960,"ondisk_log_size":10960,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":316532,"kb_used_data":129772,"kb_used_omap":11,"kb_used_meta":186676,"kb_avail":377170828,"statfs":{"total":386547056640,"available":386222927872,"internally_reserved":0,"allocated":132886528,"data_stored":132267901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11990,"internal_metadata":191156522},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,3,11,13,16],"upper_bound":32},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":89333760,"num_objects":10905,"num_object_clones":0,"num_object_copies":43620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10905,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10905,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10905,"num_write_kb":87240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003491"},"pg_stats":[{"pgid":"1.f","version":"29'639","reported_seq":668,"reported_epoch":29,"state":"active+undersized+degraded","last_fresh":"2024-09-05T16:34:28.920671+0000","last_change":"2024-09-05T16:34:22.526044+0000","last_active":"2024-09-05T16:34:28.920671+0000","last_peered":"2024-09-05T16:34:28.920671+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:15.913051+0000","last_became_peered":"2024-09-05T16:34:15.913051+0000","last_unstale":"2024-09-05T16:34:28.920671+0000","last_undegraded":"2024-09-05T16:34:22.517240+0000","last_fullsized":"2024-09-05T16:34:15.905613+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":639,"log_dups_size":0,"ondisk_log_size":639,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T19:25:44.252388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5234688,"num_objects":639,"num_object_clones":0,"num_object_copies":2556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":639,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":639,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":639,"num_write_kb":5112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2147483647,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":639}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"29'744","reported_seq":784,"reported_epoch":29,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-05T16:34:29.947804+0000","last_change":"2024-09-05T16:34:15.920955+0000","last_active":"2024-09-05T16:34:29.947804+0000","last_peered":"2024-09-05T16:34:29.947804+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:15.910788+0000","last_became_peered":"2024-09-05T16:34:15.910788+0000","last_unstale":"2024-09-05T16:34:29.947804+0000","last_undegraded":"2024-09-05T16:34:15.903622+0000","last_fullsized":"2024-09-05T16:34:15.903245+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":744,"log_dups_size":0,"ondisk_log_size":744,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6075552,"num_objects":742,"num_object_clones":0,"num_object_copies":2968,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":746,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":742,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":744,"num_write_kb":5948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":740},{"shards":"2(3)","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"29'731","reported_seq":784,"reported_epoch":29,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-05T16:34:29.997581+0000","last_change":"2024-09-05T16:34:15.922327+0000","last_active":"2024-09-05T16:34:29.997581+0000","last_peered":"2024-09-05T16:34:29.997581+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:15.917951+0000","last_became_peered":"2024-09-05T16:34:15.917951+0000","last_unstale":"2024-09-05T16:34:29.997581+0000","last_undegraded":"2024-09-05T16:34:15.904811+0000","last_fullsized":"2024-09-05T16:34:15.904154+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":731,"log_dups_size":0,"ondisk_log_size":731,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5970512,"num_objects":729,"num_object_clones":0,"num_object_copies":2916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":731,"num_objects_misplaced":0,"num_objects_unfound":1,"num_objects_dirty":729,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":731,"num_write_kb":5838,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,2147483647,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":728},{"shards":"2(0)","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"29'730","reported_seq":770,"reported_epoch":29,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-05T16:34:29.946524+0000","last_change":"2024-09-05T16:34:15.924273+0000","last_active":"2024-09-05T16:34:29.946524+0000","last_peered":"2024-09-05T16:34:29.946524+0000","last_clean":"2024-09-05T16:33:53.148717+0000","last_became_active":"2024-09-05T16:34:15.912196+0000","last_became_peered":"2024-09-05T16:34:15.912196+0000","last_unstale":"2024-09-05T16:34:29.946524+0000","last_undegraded":"2024-09-05T16:34:15.905010+0000","last_fullsized":"2024-09-05T16:34:15.904610+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":730,"log_dups_size":0,"ondisk_log_size":730,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5960864,"num_objects":728,"num_object_clones":0,"num_object_copies":2912,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":732,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":728,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":730,"num_write_kb":5829,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":726},{"shards":"2(2)","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"29'650","reported_seq":679,"reported_epoch":29,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-05T16:34:28.910994+0000","last_change":"2024-09-05T16:34:15.919374+0000","last_active":"2024-09-05T16:34:28.910994+0000","last_peered":"2024-09-05T16:34:28.910994+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:15.910648+0000","last_became_peered":"2024-09-05T16:34:15.910648+0000","last_unstale":"2024-09-05T16:34:28.910994+0000","last_undegraded":"2024-09-05T16:34:15.904665+0000","last_fullsized":"2024-09-05T16:34:15.903277+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":650,"log_dups_size":0,"ondisk_log_size":650,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5313696,"num_objects":649,"num_object_clones":0,"num_object_copies":2596,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":653,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":649,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":650,"num_write_kb":5197,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2147483647,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":647},{"shards":"2(3)","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"29'622","reported_seq":651,"reported_epoch":29,"state":"active+undersized+degraded","last_fresh":"2024-09-05T16:34:28.917278+0000","last_change":"2024-09-05T16:34:22.533654+0000","last_active":"2024-09-05T16:34:28.917278+0000","last_peered":"2024-09-05T16:34:28.917278+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:15.913477+0000","last_became_peered":"2024-09-05T16:34:15.913477+0000","last_unstale":"2024-09-05T16:34:28.917278+0000","last_undegraded":"2024-09-05T16:34:22.533443+0000","last_fullsized":"2024-09-05T16:34:15.906631+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":622,"log_dups_size":0,"ondisk_log_size":622,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T01:56:46.762011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5095424,"num_objects":622,"num_object_clones":0,"num_object_copies":2488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":622,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":622,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":622,"num_write_kb":4976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,2147483647],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":622}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"29'651","reported_seq":691,"reported_epoch":29,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-05T16:34:29.956070+0000","last_change":"2024-09-05T16:34:15.925438+0000","last_active":"2024-09-05T16:34:29.956070+0000","last_peered":"2024-09-05T16:34:29.956070+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:15.913479+0000","last_became_peered":"2024-09-05T16:34:15.913479+0000","last_unstale":"2024-09-05T16:34:29.956070+0000","last_undegraded":"2024-09-05T16:34:15.905955+0000","last_fullsized":"2024-09-05T16:34:15.905617+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":651,"log_dups_size":0,"ondisk_log_size":651,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5297312,"num_objects":647,"num_object_clones":0,"num_object_copies":2588,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":651,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":647,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":651,"num_write_kb":5188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2147483647,2],"acting":[0,3,2147483647,2],"avail_no_missing":["2(3)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":645},{"shards":"2(3)","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"29'726","reported_seq":766,"reported_epoch":29,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-05T16:34:29.946885+0000","last_change":"2024-09-05T16:34:15.917364+0000","last_active":"2024-09-05T16:34:29.946885+0000","last_peered":"2024-09-05T16:34:29.946885+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:15.910570+0000","last_became_peered":"2024-09-05T16:34:15.910570+0000","last_unstale":"2024-09-05T16:34:29.946885+0000","last_undegraded":"2024-09-05T16:34:15.904563+0000","last_fullsized":"2024-09-05T16:34:15.903952+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":726,"log_dups_size":0,"ondisk_log_size":726,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5919904,"num_objects":723,"num_object_clones":0,"num_object_copies":2892,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":727,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":723,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":726,"num_write_kb":5796,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":721},{"shards":"2(2)","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"29'625","reported_seq":653,"reported_epoch":29,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-05T16:34:28.910914+0000","last_change":"2024-09-05T16:34:15.927619+0000","last_active":"2024-09-05T16:34:28.910914+0000","last_peered":"2024-09-05T16:34:28.910914+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:15.914297+0000","last_became_peered":"2024-09-05T16:34:15.914297+0000","last_unstale":"2024-09-05T16:34:28.910914+0000","last_undegraded":"2024-09-05T16:34:15.906625+0000","last_fullsized":"2024-09-05T16:34:15.906328+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":625,"log_dups_size":0,"ondisk_log_size":625,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5073216,"num_objects":620,"num_object_clones":0,"num_object_copies":2480,"num_objects_missing_on_primary":3,"num_objects_missing":3,"num_objects_degraded":626,"num_objects_misplaced":0,"num_objects_unfound":3,"num_objects_dirty":620,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":625,"num_write_kb":4977,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,2147483647],"acting":[3,0,2,2147483647],"avail_no_missing":["2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":617},{"shards":"2(2)","objects":3}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"29'750","reported_seq":802,"reported_epoch":29,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-05T16:34:29.970619+0000","last_change":"2024-09-05T16:34:15.926681+0000","last_active":"2024-09-05T16:34:29.970619+0000","last_peered":"2024-09-05T16:34:29.970619+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:15.918296+0000","last_became_peered":"2024-09-05T16:34:15.918296+0000","last_unstale":"2024-09-05T16:34:29.970619+0000","last_undegraded":"2024-09-05T16:34:15.905187+0000","last_fullsized":"2024-09-05T16:34:15.904834+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":750,"log_dups_size":0,"ondisk_log_size":750,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6141088,"num_objects":750,"num_object_clones":0,"num_object_copies":3000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":754,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":750,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":750,"num_write_kb":5998,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,2147483647],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":748},{"shards":"2(0)","objects":2}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"29'661","reported_seq":689,"reported_epoch":29,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-05T16:34:28.902405+0000","last_change":"2024-09-05T16:34:15.925969+0000","last_active":"2024-09-05T16:34:28.902405+0000","last_peered":"2024-09-05T16:34:28.902405+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:15.914271+0000","last_became_peered":"2024-09-05T16:34:15.914271+0000","last_unstale":"2024-09-05T16:34:28.902405+0000","last_undegraded":"2024-09-05T16:34:15.905903+0000","last_fullsized":"2024-09-05T16:34:15.905357+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":661,"log_dups_size":0,"ondisk_log_size":661,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5394160,"num_objects":659,"num_object_clones":0,"num_object_copies":2636,"num_objects_missing_on_primary":3,"num_objects_missing":3,"num_objects_degraded":665,"num_objects_misplaced":0,"num_objects_unfound":3,"num_objects_dirty":659,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":661,"num_write_kb":5276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2147483647,2],"acting":[3,0,2147483647,2],"avail_no_missing":["2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":656},{"shards":"2(3)","objects":3}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"29'597","reported_seq":609,"reported_epoch":29,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-05T16:34:28.918341+0000","last_change":"2024-09-05T16:34:15.923439+0000","last_active":"2024-09-05T16:34:28.918341+0000","last_peered":"2024-09-05T16:34:28.918341+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:15.913627+0000","last_became_peered":"2024-09-05T16:34:15.913627+0000","last_unstale":"2024-09-05T16:34:28.918341+0000","last_undegraded":"2024-09-05T16:34:15.906286+0000","last_fullsized":"2024-09-05T16:34:15.906044+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":597,"log_dups_size":0,"ondisk_log_size":597,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4889168,"num_objects":597,"num_object_clones":0,"num_object_copies":2388,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":599,"num_objects_misplaced":0,"num_objects_unfound":1,"num_objects_dirty":597,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":597,"num_write_kb":4775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":596},{"shards":"2(2)","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"29'726","reported_seq":777,"reported_epoch":29,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-05T16:34:29.998900+0000","last_change":"2024-09-05T16:34:15.920326+0000","last_active":"2024-09-05T16:34:29.998900+0000","last_peered":"2024-09-05T16:34:29.998900+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:15.917839+0000","last_became_peered":"2024-09-05T16:34:15.917839+0000","last_unstale":"2024-09-05T16:34:29.998900+0000","last_undegraded":"2024-09-05T16:34:15.904338+0000","last_fullsized":"2024-09-05T16:34:15.904018+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":726,"log_dups_size":0,"ondisk_log_size":726,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5945936,"num_objects":726,"num_object_clones":0,"num_object_copies":2904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":728,"num_objects_misplaced":0,"num_objects_unfound":1,"num_objects_dirty":726,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":726,"num_write_kb":5807,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":725},{"shards":"2(1)","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"29'655","reported_seq":683,"reported_epoch":29,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-05T16:34:28.884047+0000","last_change":"2024-09-05T16:34:15.921542+0000","last_active":"2024-09-05T16:34:28.884047+0000","last_peered":"2024-09-05T16:34:28.884047+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:15.913175+0000","last_became_peered":"2024-09-05T16:34:15.913175+0000","last_unstale":"2024-09-05T16:34:28.884047+0000","last_undegraded":"2024-09-05T16:34:15.905345+0000","last_fullsized":"2024-09-05T16:34:15.905038+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":655,"log_dups_size":0,"ondisk_log_size":655,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5346464,"num_objects":653,"num_object_clones":0,"num_object_copies":2612,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":657,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":653,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":655,"num_write_kb":5236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":651},{"shards":"2(2)","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"29'742","reported_seq":796,"reported_epoch":29,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-05T16:34:29.986939+0000","last_change":"2024-09-05T16:34:15.924816+0000","last_active":"2024-09-05T16:34:29.986939+0000","last_peered":"2024-09-05T16:34:29.986939+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:15.917968+0000","last_became_peered":"2024-09-05T16:34:15.917968+0000","last_unstale":"2024-09-05T16:34:29.986939+0000","last_undegraded":"2024-09-05T16:34:15.904385+0000","last_fullsized":"2024-09-05T16:34:15.904040+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":742,"log_dups_size":0,"ondisk_log_size":742,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6067360,"num_objects":741,"num_object_clones":0,"num_object_copies":2964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":745,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":741,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":742,"num_write_kb":5933,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":739},{"shards":"2(1)","objects":2}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"29'711","reported_seq":751,"reported_epoch":29,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-05T16:34:29.941784+0000","last_change":"2024-09-05T16:34:15.922928+0000","last_active":"2024-09-05T16:34:29.941784+0000","last_peered":"2024-09-05T16:34:29.941784+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:15.910947+0000","last_became_peered":"2024-09-05T16:34:15.910947+0000","last_unstale":"2024-09-05T16:34:29.941784+0000","last_undegraded":"2024-09-05T16:34:15.905253+0000","last_fullsized":"2024-09-05T16:34:15.904912+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":711,"log_dups_size":0,"ondisk_log_size":711,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5797024,"num_objects":708,"num_object_clones":0,"num_object_copies":2832,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":712,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":708,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":711,"num_write_kb":5676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":706},{"shards":"2(2)","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":89522368,"num_objects":10933,"num_object_clones":0,"num_object_copies":43732,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":10987,"num_objects_misplaced":0,"num_objects_unfound":27,"num_objects_dirty":10933,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10960,"num_write_kb":87562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":136019968,"data_stored":136019968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10960,"ondisk_log_size":10960,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084294,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":83548,"kb_used_data":39372,"kb_used_omap":4,"kb_used_meta":44155,"kb_avail":94288292,"statfs":{"total":96636764160,"available":96551211008,"internally_reserved":0,"allocated":40316928,"data_stored":40151471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":45215443},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,3,4,5,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640267,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":212992,"data_stored":90736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949672977,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":103736,"kb_used_data":46184,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94268104,"statfs":{"total":96636764160,"available":96530538496,"internally_reserved":0,"allocated":47292416,"data_stored":47126959,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,4,7],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182408,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":102008,"kb_used_data":44008,"kb_used_omap":4,"kb_used_meta":57979,"kb_avail":94269832,"statfs":{"total":96636764160,"available":96532307968,"internally_reserved":0,"allocated":45064192,"data_stored":44898735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":59371213},"hb_peers":[2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,4,6],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":47177728,"data_stored":47177728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":40960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":47480832,"data_stored":47480832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":41320448,"data_stored":41320448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:34:33.018 INFO:tasks.ec_lost_unfound:no unfound in 1.f 2024-09-05T16:34:33.018 INFO:tasks.ec_lost_unfound:listing missing/lost in 1.e state active+recovery_unfound+undersized+degraded 2024-09-05T16:34:33.018 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 1.e list_unfound '{}' 2024-09-05T16:34:33.239 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-09-05T16:34:33.239 INFO:teuthology.orchestra.run.smithi087.stdout: "offset": { 2024-09-05T16:34:33.239 INFO:teuthology.orchestra.run.smithi087.stdout: "oid": "", 2024-09-05T16:34:33.239 INFO:teuthology.orchestra.run.smithi087.stdout: "key": "", 2024-09-05T16:34:33.239 INFO:teuthology.orchestra.run.smithi087.stdout: "snapid": 0, 2024-09-05T16:34:33.239 INFO:teuthology.orchestra.run.smithi087.stdout: "hash": 0, 2024-09-05T16:34:33.239 INFO:teuthology.orchestra.run.smithi087.stdout: "max": 0, 2024-09-05T16:34:33.239 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": -9223372036854775808, 2024-09-05T16:34:33.239 INFO:teuthology.orchestra.run.smithi087.stdout: "namespace": "" 2024-09-05T16:34:33.240 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:33.240 INFO:teuthology.orchestra.run.smithi087.stdout: "num_missing": 2, 2024-09-05T16:34:33.240 INFO:teuthology.orchestra.run.smithi087.stdout: "num_unfound": 2, 2024-09-05T16:34:33.240 INFO:teuthology.orchestra.run.smithi087.stdout: "objects": [ 2024-09-05T16:34:33.240 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:33.240 INFO:teuthology.orchestra.run.smithi087.stdout: "oid": { 2024-09-05T16:34:33.240 INFO:teuthology.orchestra.run.smithi087.stdout: "oid": "existing_4", 2024-09-05T16:34:33.240 INFO:teuthology.orchestra.run.smithi087.stdout: "key": "", 2024-09-05T16:34:33.240 INFO:teuthology.orchestra.run.smithi087.stdout: "snapid": -2, 2024-09-05T16:34:33.240 INFO:teuthology.orchestra.run.smithi087.stdout: "hash": 1224592654, 2024-09-05T16:34:33.240 INFO:teuthology.orchestra.run.smithi087.stdout: "max": 0, 2024-09-05T16:34:33.240 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 1, 2024-09-05T16:34:33.240 INFO:teuthology.orchestra.run.smithi087.stdout: "namespace": "" 2024-09-05T16:34:33.241 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:33.241 INFO:teuthology.orchestra.run.smithi087.stdout: "need": "21'3", 2024-09-05T16:34:33.241 INFO:teuthology.orchestra.run.smithi087.stdout: "have": "18'1", 2024-09-05T16:34:33.241 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": "none", 2024-09-05T16:34:33.241 INFO:teuthology.orchestra.run.smithi087.stdout: "clean_regions": "clean_offsets: [(6736, 18446744073709544879)], clean_omap: true, new_object: false", 2024-09-05T16:34:33.241 INFO:teuthology.orchestra.run.smithi087.stdout: "locations": [ 2024-09-05T16:34:33.241 INFO:teuthology.orchestra.run.smithi087.stdout: "2(3)" 2024-09-05T16:34:33.241 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-09-05T16:34:33.241 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:33.241 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:33.241 INFO:teuthology.orchestra.run.smithi087.stdout: "oid": { 2024-09-05T16:34:33.241 INFO:teuthology.orchestra.run.smithi087.stdout: "oid": "existing_6", 2024-09-05T16:34:33.242 INFO:teuthology.orchestra.run.smithi087.stdout: "key": "", 2024-09-05T16:34:33.242 INFO:teuthology.orchestra.run.smithi087.stdout: "snapid": -2, 2024-09-05T16:34:33.242 INFO:teuthology.orchestra.run.smithi087.stdout: "hash": 3101443614, 2024-09-05T16:34:33.242 INFO:teuthology.orchestra.run.smithi087.stdout: "max": 0, 2024-09-05T16:34:33.242 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 1, 2024-09-05T16:34:33.242 INFO:teuthology.orchestra.run.smithi087.stdout: "namespace": "" 2024-09-05T16:34:33.242 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:33.242 INFO:teuthology.orchestra.run.smithi087.stdout: "need": "21'4", 2024-09-05T16:34:33.242 INFO:teuthology.orchestra.run.smithi087.stdout: "have": "18'2", 2024-09-05T16:34:33.242 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": "none", 2024-09-05T16:34:33.242 INFO:teuthology.orchestra.run.smithi087.stdout: "clean_regions": "clean_offsets: [(6736, 18446744073709544879)], clean_omap: true, new_object: false", 2024-09-05T16:34:33.242 INFO:teuthology.orchestra.run.smithi087.stdout: "locations": [ 2024-09-05T16:34:33.242 INFO:teuthology.orchestra.run.smithi087.stdout: "2(3)" 2024-09-05T16:34:33.242 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-09-05T16:34:33.242 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-09-05T16:34:33.243 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-09-05T16:34:33.243 INFO:teuthology.orchestra.run.smithi087.stdout: "state": "NotRecovering", 2024-09-05T16:34:33.243 INFO:teuthology.orchestra.run.smithi087.stdout: "available_might_have_unfound": true, 2024-09-05T16:34:33.243 INFO:teuthology.orchestra.run.smithi087.stdout: "might_have_unfound": [ 2024-09-05T16:34:33.243 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:33.243 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": "1(0)", 2024-09-05T16:34:33.243 INFO:teuthology.orchestra.run.smithi087.stdout: "status": "osd is down" 2024-09-05T16:34:33.243 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-09-05T16:34:33.243 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-09-05T16:34:33.243 INFO:teuthology.orchestra.run.smithi087.stdout: "more": false 2024-09-05T16:34:33.243 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-09-05T16:34:33.257 INFO:tasks.ec_lost_unfound:{'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 2, 'num_unfound': 2, 'objects': [{'oid': {'oid': 'existing_4', 'key': '', 'snapid': -2, 'hash': 1224592654, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "21'3", 'have': "18'1", 'flags': 'none', 'clean_regions': 'clean_offsets: [(6736, 18446744073709544879)], clean_omap: true, new_object: false', 'locations': ['2(3)']}, {'oid': {'oid': 'existing_6', 'key': '', 'snapid': -2, 'hash': 3101443614, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "21'4", 'have': "18'2", 'flags': 'none', 'clean_regions': 'clean_offsets: [(6736, 18446744073709544879)], clean_omap: true, new_object: false', 'locations': ['2(3)']}], 'state': 'NotRecovering', 'available_might_have_unfound': True, 'might_have_unfound': [{'osd': '1(0)', 'status': 'osd is down'}]} 2024-09-05T16:34:33.257 INFO:tasks.ec_lost_unfound:reverting unfound in 1.e 2024-09-05T16:34:33.257 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 1.e mark_unfound_lost delete 2024-09-05T16:34:33.488 INFO:teuthology.orchestra.run.smithi087.stderr:pg has 2 objects unfound and apparently lost marking 2024-09-05T16:34:33.491 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 11 20 16010 15990 11.3552 8.625 0.0193286 0.0137437 2024-09-05T16:34:33.507 INFO:tasks.ec_lost_unfound:listing missing/lost in 1.d state active+recovery_unfound+undersized+degraded 2024-09-05T16:34:33.507 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 1.d list_unfound '{}' 2024-09-05T16:34:33.744 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-09-05T16:34:33.744 INFO:teuthology.orchestra.run.smithi087.stdout: "offset": { 2024-09-05T16:34:33.744 INFO:teuthology.orchestra.run.smithi087.stdout: "oid": "", 2024-09-05T16:34:33.744 INFO:teuthology.orchestra.run.smithi087.stdout: "key": "", 2024-09-05T16:34:33.744 INFO:teuthology.orchestra.run.smithi087.stdout: "snapid": 0, 2024-09-05T16:34:33.744 INFO:teuthology.orchestra.run.smithi087.stdout: "hash": 0, 2024-09-05T16:34:33.744 INFO:teuthology.orchestra.run.smithi087.stdout: "max": 0, 2024-09-05T16:34:33.744 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": -9223372036854775808, 2024-09-05T16:34:33.744 INFO:teuthology.orchestra.run.smithi087.stdout: "namespace": "" 2024-09-05T16:34:33.745 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:33.745 INFO:teuthology.orchestra.run.smithi087.stdout: "num_missing": 1, 2024-09-05T16:34:33.745 INFO:teuthology.orchestra.run.smithi087.stdout: "num_unfound": 1, 2024-09-05T16:34:33.745 INFO:teuthology.orchestra.run.smithi087.stdout: "objects": [ 2024-09-05T16:34:33.745 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:33.745 INFO:teuthology.orchestra.run.smithi087.stdout: "oid": { 2024-09-05T16:34:33.745 INFO:teuthology.orchestra.run.smithi087.stdout: "oid": "existed_7", 2024-09-05T16:34:33.745 INFO:teuthology.orchestra.run.smithi087.stdout: "key": "", 2024-09-05T16:34:33.745 INFO:teuthology.orchestra.run.smithi087.stdout: "snapid": -2, 2024-09-05T16:34:33.745 INFO:teuthology.orchestra.run.smithi087.stdout: "hash": 2358579165, 2024-09-05T16:34:33.745 INFO:teuthology.orchestra.run.smithi087.stdout: "max": 0, 2024-09-05T16:34:33.745 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 1, 2024-09-05T16:34:33.746 INFO:teuthology.orchestra.run.smithi087.stdout: "namespace": "" 2024-09-05T16:34:33.746 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:33.746 INFO:teuthology.orchestra.run.smithi087.stdout: "need": "21'3", 2024-09-05T16:34:33.746 INFO:teuthology.orchestra.run.smithi087.stdout: "have": "0'0", 2024-09-05T16:34:33.746 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": "none", 2024-09-05T16:34:33.746 INFO:teuthology.orchestra.run.smithi087.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2024-09-05T16:34:33.746 INFO:teuthology.orchestra.run.smithi087.stdout: "locations": [ 2024-09-05T16:34:33.746 INFO:teuthology.orchestra.run.smithi087.stdout: "2(0)" 2024-09-05T16:34:33.746 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-09-05T16:34:33.746 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-09-05T16:34:33.746 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-09-05T16:34:33.746 INFO:teuthology.orchestra.run.smithi087.stdout: "state": "NotRecovering", 2024-09-05T16:34:33.746 INFO:teuthology.orchestra.run.smithi087.stdout: "available_might_have_unfound": true, 2024-09-05T16:34:33.747 INFO:teuthology.orchestra.run.smithi087.stdout: "might_have_unfound": [ 2024-09-05T16:34:33.747 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:33.747 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": "1(1)", 2024-09-05T16:34:33.747 INFO:teuthology.orchestra.run.smithi087.stdout: "status": "osd is down" 2024-09-05T16:34:33.747 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-09-05T16:34:33.747 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-09-05T16:34:33.747 INFO:teuthology.orchestra.run.smithi087.stdout: "more": false 2024-09-05T16:34:33.748 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-09-05T16:34:33.768 INFO:tasks.ec_lost_unfound:{'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 1, 'num_unfound': 1, 'objects': [{'oid': {'oid': 'existed_7', 'key': '', 'snapid': -2, 'hash': 2358579165, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "21'3", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: false, new_object: true', 'locations': ['2(0)']}], 'state': 'NotRecovering', 'available_might_have_unfound': True, 'might_have_unfound': [{'osd': '1(1)', 'status': 'osd is down'}]} 2024-09-05T16:34:33.768 INFO:tasks.ec_lost_unfound:reverting unfound in 1.d 2024-09-05T16:34:33.769 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 1.d mark_unfound_lost delete 2024-09-05T16:34:34.031 INFO:teuthology.orchestra.run.smithi087.stderr:pg has 1 objects unfound and apparently lost marking 2024-09-05T16:34:34.058 INFO:tasks.ec_lost_unfound:listing missing/lost in 1.0 state active+recovery_unfound+undersized+degraded 2024-09-05T16:34:34.059 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 1.0 list_unfound '{}' 2024-09-05T16:34:34.277 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-09-05T16:34:34.277 INFO:teuthology.orchestra.run.smithi087.stdout: "offset": { 2024-09-05T16:34:34.277 INFO:teuthology.orchestra.run.smithi087.stdout: "oid": "", 2024-09-05T16:34:34.277 INFO:teuthology.orchestra.run.smithi087.stdout: "key": "", 2024-09-05T16:34:34.277 INFO:teuthology.orchestra.run.smithi087.stdout: "snapid": 0, 2024-09-05T16:34:34.277 INFO:teuthology.orchestra.run.smithi087.stdout: "hash": 0, 2024-09-05T16:34:34.278 INFO:teuthology.orchestra.run.smithi087.stdout: "max": 0, 2024-09-05T16:34:34.278 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": -9223372036854775808, 2024-09-05T16:34:34.278 INFO:teuthology.orchestra.run.smithi087.stdout: "namespace": "" 2024-09-05T16:34:34.278 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:34.278 INFO:teuthology.orchestra.run.smithi087.stdout: "num_missing": 2, 2024-09-05T16:34:34.278 INFO:teuthology.orchestra.run.smithi087.stdout: "num_unfound": 2, 2024-09-05T16:34:34.278 INFO:teuthology.orchestra.run.smithi087.stdout: "objects": [ 2024-09-05T16:34:34.278 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:34.278 INFO:teuthology.orchestra.run.smithi087.stdout: "oid": { 2024-09-05T16:34:34.278 INFO:teuthology.orchestra.run.smithi087.stdout: "oid": "new_5", 2024-09-05T16:34:34.278 INFO:teuthology.orchestra.run.smithi087.stdout: "key": "", 2024-09-05T16:34:34.279 INFO:teuthology.orchestra.run.smithi087.stdout: "snapid": -2, 2024-09-05T16:34:34.279 INFO:teuthology.orchestra.run.smithi087.stdout: "hash": 4248945696, 2024-09-05T16:34:34.279 INFO:teuthology.orchestra.run.smithi087.stdout: "max": 0, 2024-09-05T16:34:34.279 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 1, 2024-09-05T16:34:34.279 INFO:teuthology.orchestra.run.smithi087.stdout: "namespace": "" 2024-09-05T16:34:34.279 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:34.279 INFO:teuthology.orchestra.run.smithi087.stdout: "need": "21'3", 2024-09-05T16:34:34.279 INFO:teuthology.orchestra.run.smithi087.stdout: "have": "0'0", 2024-09-05T16:34:34.279 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": "none", 2024-09-05T16:34:34.280 INFO:teuthology.orchestra.run.smithi087.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2024-09-05T16:34:34.280 INFO:teuthology.orchestra.run.smithi087.stdout: "locations": [ 2024-09-05T16:34:34.280 INFO:teuthology.orchestra.run.smithi087.stdout: "2(2)" 2024-09-05T16:34:34.280 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-09-05T16:34:34.280 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:34.280 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:34.280 INFO:teuthology.orchestra.run.smithi087.stdout: "oid": { 2024-09-05T16:34:34.280 INFO:teuthology.orchestra.run.smithi087.stdout: "oid": "existed_5", 2024-09-05T16:34:34.280 INFO:teuthology.orchestra.run.smithi087.stdout: "key": "", 2024-09-05T16:34:34.280 INFO:teuthology.orchestra.run.smithi087.stdout: "snapid": -2, 2024-09-05T16:34:34.280 INFO:teuthology.orchestra.run.smithi087.stdout: "hash": 4212789968, 2024-09-05T16:34:34.280 INFO:teuthology.orchestra.run.smithi087.stdout: "max": 0, 2024-09-05T16:34:34.280 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 1, 2024-09-05T16:34:34.281 INFO:teuthology.orchestra.run.smithi087.stdout: "namespace": "" 2024-09-05T16:34:34.281 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:34.281 INFO:teuthology.orchestra.run.smithi087.stdout: "need": "21'4", 2024-09-05T16:34:34.281 INFO:teuthology.orchestra.run.smithi087.stdout: "have": "0'0", 2024-09-05T16:34:34.281 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": "none", 2024-09-05T16:34:34.281 INFO:teuthology.orchestra.run.smithi087.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2024-09-05T16:34:34.281 INFO:teuthology.orchestra.run.smithi087.stdout: "locations": [ 2024-09-05T16:34:34.281 INFO:teuthology.orchestra.run.smithi087.stdout: "2(2)" 2024-09-05T16:34:34.281 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-09-05T16:34:34.281 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-09-05T16:34:34.281 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-09-05T16:34:34.282 INFO:teuthology.orchestra.run.smithi087.stdout: "state": "NotRecovering", 2024-09-05T16:34:34.282 INFO:teuthology.orchestra.run.smithi087.stdout: "available_might_have_unfound": true, 2024-09-05T16:34:34.282 INFO:teuthology.orchestra.run.smithi087.stdout: "might_have_unfound": [ 2024-09-05T16:34:34.282 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:34.282 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": "1(0)", 2024-09-05T16:34:34.282 INFO:teuthology.orchestra.run.smithi087.stdout: "status": "osd is down" 2024-09-05T16:34:34.282 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-09-05T16:34:34.282 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-09-05T16:34:34.282 INFO:teuthology.orchestra.run.smithi087.stdout: "more": false 2024-09-05T16:34:34.283 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-09-05T16:34:34.300 INFO:tasks.ec_lost_unfound:{'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 2, 'num_unfound': 2, 'objects': [{'oid': {'oid': 'new_5', 'key': '', 'snapid': -2, 'hash': 4248945696, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "21'3", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: false, new_object: true', 'locations': ['2(2)']}, {'oid': {'oid': 'existed_5', 'key': '', 'snapid': -2, 'hash': 4212789968, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "21'4", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: false, new_object: true', 'locations': ['2(2)']}], 'state': 'NotRecovering', 'available_might_have_unfound': True, 'might_have_unfound': [{'osd': '1(0)', 'status': 'osd is down'}]} 2024-09-05T16:34:34.300 INFO:tasks.ec_lost_unfound:reverting unfound in 1.0 2024-09-05T16:34:34.301 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 1.0 mark_unfound_lost delete 2024-09-05T16:34:34.492 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 12 20 17124 17104 11.1333 8.70312 0.0174867 0.0140196 2024-09-05T16:34:34.567 INFO:teuthology.orchestra.run.smithi087.stderr:pg has 2 objects unfound and apparently lost marking 2024-09-05T16:34:34.589 INFO:tasks.ec_lost_unfound:listing missing/lost in 1.1 state active+recovery_unfound+undersized+degraded 2024-09-05T16:34:34.590 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 1.1 list_unfound '{}' 2024-09-05T16:34:34.812 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-09-05T16:34:34.813 INFO:teuthology.orchestra.run.smithi087.stdout: "offset": { 2024-09-05T16:34:34.813 INFO:teuthology.orchestra.run.smithi087.stdout: "oid": "", 2024-09-05T16:34:34.813 INFO:teuthology.orchestra.run.smithi087.stdout: "key": "", 2024-09-05T16:34:34.813 INFO:teuthology.orchestra.run.smithi087.stdout: "snapid": 0, 2024-09-05T16:34:34.813 INFO:teuthology.orchestra.run.smithi087.stdout: "hash": 0, 2024-09-05T16:34:34.813 INFO:teuthology.orchestra.run.smithi087.stdout: "max": 0, 2024-09-05T16:34:34.813 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": -9223372036854775808, 2024-09-05T16:34:34.813 INFO:teuthology.orchestra.run.smithi087.stdout: "namespace": "" 2024-09-05T16:34:34.813 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:34.813 INFO:teuthology.orchestra.run.smithi087.stdout: "num_missing": 2, 2024-09-05T16:34:34.813 INFO:teuthology.orchestra.run.smithi087.stdout: "num_unfound": 2, 2024-09-05T16:34:34.814 INFO:teuthology.orchestra.run.smithi087.stdout: "objects": [ 2024-09-05T16:34:34.814 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:34.814 INFO:teuthology.orchestra.run.smithi087.stdout: "oid": { 2024-09-05T16:34:34.814 INFO:teuthology.orchestra.run.smithi087.stdout: "oid": "existing_3", 2024-09-05T16:34:34.814 INFO:teuthology.orchestra.run.smithi087.stdout: "key": "", 2024-09-05T16:34:34.814 INFO:teuthology.orchestra.run.smithi087.stdout: "snapid": -2, 2024-09-05T16:34:34.814 INFO:teuthology.orchestra.run.smithi087.stdout: "hash": 459768081, 2024-09-05T16:34:34.814 INFO:teuthology.orchestra.run.smithi087.stdout: "max": 0, 2024-09-05T16:34:34.814 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 1, 2024-09-05T16:34:34.814 INFO:teuthology.orchestra.run.smithi087.stdout: "namespace": "" 2024-09-05T16:34:34.814 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:34.814 INFO:teuthology.orchestra.run.smithi087.stdout: "need": "21'3", 2024-09-05T16:34:34.814 INFO:teuthology.orchestra.run.smithi087.stdout: "have": "18'1", 2024-09-05T16:34:34.815 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": "none", 2024-09-05T16:34:34.815 INFO:teuthology.orchestra.run.smithi087.stdout: "clean_regions": "clean_offsets: [(6736, 18446744073709544879)], clean_omap: true, new_object: false", 2024-09-05T16:34:34.815 INFO:teuthology.orchestra.run.smithi087.stdout: "locations": [ 2024-09-05T16:34:34.815 INFO:teuthology.orchestra.run.smithi087.stdout: "2(3)" 2024-09-05T16:34:34.815 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-09-05T16:34:34.815 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:34.815 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:34.815 INFO:teuthology.orchestra.run.smithi087.stdout: "oid": { 2024-09-05T16:34:34.815 INFO:teuthology.orchestra.run.smithi087.stdout: "oid": "new_3", 2024-09-05T16:34:34.815 INFO:teuthology.orchestra.run.smithi087.stdout: "key": "", 2024-09-05T16:34:34.815 INFO:teuthology.orchestra.run.smithi087.stdout: "snapid": -2, 2024-09-05T16:34:34.815 INFO:teuthology.orchestra.run.smithi087.stdout: "hash": 480221105, 2024-09-05T16:34:34.815 INFO:teuthology.orchestra.run.smithi087.stdout: "max": 0, 2024-09-05T16:34:34.816 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 1, 2024-09-05T16:34:34.816 INFO:teuthology.orchestra.run.smithi087.stdout: "namespace": "" 2024-09-05T16:34:34.816 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:34.816 INFO:teuthology.orchestra.run.smithi087.stdout: "need": "21'2", 2024-09-05T16:34:34.816 INFO:teuthology.orchestra.run.smithi087.stdout: "have": "0'0", 2024-09-05T16:34:34.816 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": "none", 2024-09-05T16:34:34.816 INFO:teuthology.orchestra.run.smithi087.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2024-09-05T16:34:34.816 INFO:teuthology.orchestra.run.smithi087.stdout: "locations": [ 2024-09-05T16:34:34.816 INFO:teuthology.orchestra.run.smithi087.stdout: "2(3)" 2024-09-05T16:34:34.816 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-09-05T16:34:34.816 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-09-05T16:34:34.816 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-09-05T16:34:34.816 INFO:teuthology.orchestra.run.smithi087.stdout: "state": "NotRecovering", 2024-09-05T16:34:34.817 INFO:teuthology.orchestra.run.smithi087.stdout: "available_might_have_unfound": true, 2024-09-05T16:34:34.817 INFO:teuthology.orchestra.run.smithi087.stdout: "might_have_unfound": [ 2024-09-05T16:34:34.817 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:34.817 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": "1(1)", 2024-09-05T16:34:34.817 INFO:teuthology.orchestra.run.smithi087.stdout: "status": "osd is down" 2024-09-05T16:34:34.817 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-09-05T16:34:34.817 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-09-05T16:34:34.817 INFO:teuthology.orchestra.run.smithi087.stdout: "more": false 2024-09-05T16:34:34.817 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-09-05T16:34:34.833 INFO:tasks.ec_lost_unfound:{'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 2, 'num_unfound': 2, 'objects': [{'oid': {'oid': 'existing_3', 'key': '', 'snapid': -2, 'hash': 459768081, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "21'3", 'have': "18'1", 'flags': 'none', 'clean_regions': 'clean_offsets: [(6736, 18446744073709544879)], clean_omap: true, new_object: false', 'locations': ['2(3)']}, {'oid': {'oid': 'new_3', 'key': '', 'snapid': -2, 'hash': 480221105, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "21'2", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: false, new_object: true', 'locations': ['2(3)']}], 'state': 'NotRecovering', 'available_might_have_unfound': True, 'might_have_unfound': [{'osd': '1(1)', 'status': 'osd is down'}]} 2024-09-05T16:34:34.833 INFO:tasks.ec_lost_unfound:reverting unfound in 1.1 2024-09-05T16:34:34.833 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 1.1 mark_unfound_lost delete 2024-09-05T16:34:35.100 INFO:teuthology.orchestra.run.smithi087.stderr:pg has 2 objects unfound and apparently lost marking 2024-09-05T16:34:35.125 INFO:tasks.ec_lost_unfound:no unfound in 1.c 2024-09-05T16:34:35.125 INFO:tasks.ec_lost_unfound:listing missing/lost in 1.2 state active+recovery_unfound+undersized+degraded 2024-09-05T16:34:35.125 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 1.2 list_unfound '{}' 2024-09-05T16:34:35.343 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-09-05T16:34:35.343 INFO:teuthology.orchestra.run.smithi087.stdout: "offset": { 2024-09-05T16:34:35.343 INFO:teuthology.orchestra.run.smithi087.stdout: "oid": "", 2024-09-05T16:34:35.343 INFO:teuthology.orchestra.run.smithi087.stdout: "key": "", 2024-09-05T16:34:35.343 INFO:teuthology.orchestra.run.smithi087.stdout: "snapid": 0, 2024-09-05T16:34:35.343 INFO:teuthology.orchestra.run.smithi087.stdout: "hash": 0, 2024-09-05T16:34:35.343 INFO:teuthology.orchestra.run.smithi087.stdout: "max": 0, 2024-09-05T16:34:35.343 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": -9223372036854775808, 2024-09-05T16:34:35.343 INFO:teuthology.orchestra.run.smithi087.stdout: "namespace": "" 2024-09-05T16:34:35.344 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:35.344 INFO:teuthology.orchestra.run.smithi087.stdout: "num_missing": 2, 2024-09-05T16:34:35.344 INFO:teuthology.orchestra.run.smithi087.stdout: "num_unfound": 2, 2024-09-05T16:34:35.344 INFO:teuthology.orchestra.run.smithi087.stdout: "objects": [ 2024-09-05T16:34:35.344 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:35.344 INFO:teuthology.orchestra.run.smithi087.stdout: "oid": { 2024-09-05T16:34:35.344 INFO:teuthology.orchestra.run.smithi087.stdout: "oid": "existed_6", 2024-09-05T16:34:35.344 INFO:teuthology.orchestra.run.smithi087.stdout: "key": "", 2024-09-05T16:34:35.344 INFO:teuthology.orchestra.run.smithi087.stdout: "snapid": -2, 2024-09-05T16:34:35.344 INFO:teuthology.orchestra.run.smithi087.stdout: "hash": 1708673474, 2024-09-05T16:34:35.344 INFO:teuthology.orchestra.run.smithi087.stdout: "max": 0, 2024-09-05T16:34:35.344 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 1, 2024-09-05T16:34:35.344 INFO:teuthology.orchestra.run.smithi087.stdout: "namespace": "" 2024-09-05T16:34:35.345 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:35.345 INFO:teuthology.orchestra.run.smithi087.stdout: "need": "21'5", 2024-09-05T16:34:35.345 INFO:teuthology.orchestra.run.smithi087.stdout: "have": "0'0", 2024-09-05T16:34:35.345 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": "none", 2024-09-05T16:34:35.345 INFO:teuthology.orchestra.run.smithi087.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2024-09-05T16:34:35.345 INFO:teuthology.orchestra.run.smithi087.stdout: "locations": [ 2024-09-05T16:34:35.345 INFO:teuthology.orchestra.run.smithi087.stdout: "2(3)" 2024-09-05T16:34:35.345 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-09-05T16:34:35.345 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:35.345 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:35.345 INFO:teuthology.orchestra.run.smithi087.stdout: "oid": { 2024-09-05T16:34:35.345 INFO:teuthology.orchestra.run.smithi087.stdout: "oid": "existed_9", 2024-09-05T16:34:35.345 INFO:teuthology.orchestra.run.smithi087.stdout: "key": "", 2024-09-05T16:34:35.346 INFO:teuthology.orchestra.run.smithi087.stdout: "snapid": -2, 2024-09-05T16:34:35.346 INFO:teuthology.orchestra.run.smithi087.stdout: "hash": 831402034, 2024-09-05T16:34:35.346 INFO:teuthology.orchestra.run.smithi087.stdout: "max": 0, 2024-09-05T16:34:35.346 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 1, 2024-09-05T16:34:35.346 INFO:teuthology.orchestra.run.smithi087.stdout: "namespace": "" 2024-09-05T16:34:35.346 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:35.346 INFO:teuthology.orchestra.run.smithi087.stdout: "need": "21'6", 2024-09-05T16:34:35.346 INFO:teuthology.orchestra.run.smithi087.stdout: "have": "0'0", 2024-09-05T16:34:35.346 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": "none", 2024-09-05T16:34:35.346 INFO:teuthology.orchestra.run.smithi087.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2024-09-05T16:34:35.346 INFO:teuthology.orchestra.run.smithi087.stdout: "locations": [ 2024-09-05T16:34:35.346 INFO:teuthology.orchestra.run.smithi087.stdout: "2(3)" 2024-09-05T16:34:35.346 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-09-05T16:34:35.347 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-09-05T16:34:35.347 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-09-05T16:34:35.347 INFO:teuthology.orchestra.run.smithi087.stdout: "state": "NotRecovering", 2024-09-05T16:34:35.347 INFO:teuthology.orchestra.run.smithi087.stdout: "available_might_have_unfound": true, 2024-09-05T16:34:35.347 INFO:teuthology.orchestra.run.smithi087.stdout: "might_have_unfound": [ 2024-09-05T16:34:35.347 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:35.347 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": "1(2)", 2024-09-05T16:34:35.347 INFO:teuthology.orchestra.run.smithi087.stdout: "status": "osd is down" 2024-09-05T16:34:35.347 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-09-05T16:34:35.347 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-09-05T16:34:35.347 INFO:teuthology.orchestra.run.smithi087.stdout: "more": false 2024-09-05T16:34:35.347 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-09-05T16:34:35.369 INFO:tasks.ec_lost_unfound:{'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 2, 'num_unfound': 2, 'objects': [{'oid': {'oid': 'existed_6', 'key': '', 'snapid': -2, 'hash': 1708673474, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "21'5", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: false, new_object: true', 'locations': ['2(3)']}, {'oid': {'oid': 'existed_9', 'key': '', 'snapid': -2, 'hash': 831402034, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "21'6", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: false, new_object: true', 'locations': ['2(3)']}], 'state': 'NotRecovering', 'available_might_have_unfound': True, 'might_have_unfound': [{'osd': '1(2)', 'status': 'osd is down'}]} 2024-09-05T16:34:35.369 INFO:tasks.ec_lost_unfound:reverting unfound in 1.2 2024-09-05T16:34:35.369 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 1.2 mark_unfound_lost delete 2024-09-05T16:34:35.492 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 13 20 18211 18191 10.9301 8.49219 0.0158425 0.0142836 2024-09-05T16:34:35.618 INFO:teuthology.orchestra.run.smithi087.stderr:pg has 2 objects unfound and apparently lost marking 2024-09-05T16:34:35.636 INFO:tasks.ec_lost_unfound:listing missing/lost in 1.3 state active+recovery_unfound+undersized+degraded 2024-09-05T16:34:35.637 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 1.3 list_unfound '{}' 2024-09-05T16:34:35.857 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-09-05T16:34:35.857 INFO:teuthology.orchestra.run.smithi087.stdout: "offset": { 2024-09-05T16:34:35.857 INFO:teuthology.orchestra.run.smithi087.stdout: "oid": "", 2024-09-05T16:34:35.857 INFO:teuthology.orchestra.run.smithi087.stdout: "key": "", 2024-09-05T16:34:35.857 INFO:teuthology.orchestra.run.smithi087.stdout: "snapid": 0, 2024-09-05T16:34:35.857 INFO:teuthology.orchestra.run.smithi087.stdout: "hash": 0, 2024-09-05T16:34:35.858 INFO:teuthology.orchestra.run.smithi087.stdout: "max": 0, 2024-09-05T16:34:35.858 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": -9223372036854775808, 2024-09-05T16:34:35.858 INFO:teuthology.orchestra.run.smithi087.stdout: "namespace": "" 2024-09-05T16:34:35.858 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:35.858 INFO:teuthology.orchestra.run.smithi087.stdout: "num_missing": 2, 2024-09-05T16:34:35.858 INFO:teuthology.orchestra.run.smithi087.stdout: "num_unfound": 2, 2024-09-05T16:34:35.858 INFO:teuthology.orchestra.run.smithi087.stdout: "objects": [ 2024-09-05T16:34:35.858 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:35.858 INFO:teuthology.orchestra.run.smithi087.stdout: "oid": { 2024-09-05T16:34:35.858 INFO:teuthology.orchestra.run.smithi087.stdout: "oid": "existing_8", 2024-09-05T16:34:35.858 INFO:teuthology.orchestra.run.smithi087.stdout: "key": "", 2024-09-05T16:34:35.858 INFO:teuthology.orchestra.run.smithi087.stdout: "snapid": -2, 2024-09-05T16:34:35.858 INFO:teuthology.orchestra.run.smithi087.stdout: "hash": 1901787523, 2024-09-05T16:34:35.858 INFO:teuthology.orchestra.run.smithi087.stdout: "max": 0, 2024-09-05T16:34:35.859 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 1, 2024-09-05T16:34:35.859 INFO:teuthology.orchestra.run.smithi087.stdout: "namespace": "" 2024-09-05T16:34:35.859 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:35.859 INFO:teuthology.orchestra.run.smithi087.stdout: "need": "21'5", 2024-09-05T16:34:35.859 INFO:teuthology.orchestra.run.smithi087.stdout: "have": "18'3", 2024-09-05T16:34:35.859 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": "none", 2024-09-05T16:34:35.859 INFO:teuthology.orchestra.run.smithi087.stdout: "clean_regions": "clean_offsets: [(6736, 18446744073709544879)], clean_omap: true, new_object: false", 2024-09-05T16:34:35.859 INFO:teuthology.orchestra.run.smithi087.stdout: "locations": [ 2024-09-05T16:34:35.859 INFO:teuthology.orchestra.run.smithi087.stdout: "2(2)" 2024-09-05T16:34:35.859 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-09-05T16:34:35.859 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:35.859 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:35.859 INFO:teuthology.orchestra.run.smithi087.stdout: "oid": { 2024-09-05T16:34:35.859 INFO:teuthology.orchestra.run.smithi087.stdout: "oid": "existed_1", 2024-09-05T16:34:35.860 INFO:teuthology.orchestra.run.smithi087.stdout: "key": "", 2024-09-05T16:34:35.860 INFO:teuthology.orchestra.run.smithi087.stdout: "snapid": -2, 2024-09-05T16:34:35.860 INFO:teuthology.orchestra.run.smithi087.stdout: "hash": 3122379363, 2024-09-05T16:34:35.860 INFO:teuthology.orchestra.run.smithi087.stdout: "max": 0, 2024-09-05T16:34:35.860 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 1, 2024-09-05T16:34:35.860 INFO:teuthology.orchestra.run.smithi087.stdout: "namespace": "" 2024-09-05T16:34:35.860 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:35.860 INFO:teuthology.orchestra.run.smithi087.stdout: "need": "21'4", 2024-09-05T16:34:35.860 INFO:teuthology.orchestra.run.smithi087.stdout: "have": "0'0", 2024-09-05T16:34:35.860 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": "none", 2024-09-05T16:34:35.860 INFO:teuthology.orchestra.run.smithi087.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2024-09-05T16:34:35.860 INFO:teuthology.orchestra.run.smithi087.stdout: "locations": [ 2024-09-05T16:34:35.860 INFO:teuthology.orchestra.run.smithi087.stdout: "2(2)" 2024-09-05T16:34:35.861 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-09-05T16:34:35.861 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-09-05T16:34:35.861 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-09-05T16:34:35.861 INFO:teuthology.orchestra.run.smithi087.stdout: "state": "NotRecovering", 2024-09-05T16:34:35.861 INFO:teuthology.orchestra.run.smithi087.stdout: "available_might_have_unfound": true, 2024-09-05T16:34:35.861 INFO:teuthology.orchestra.run.smithi087.stdout: "might_have_unfound": [ 2024-09-05T16:34:35.861 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:35.861 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": "1(0)", 2024-09-05T16:34:35.861 INFO:teuthology.orchestra.run.smithi087.stdout: "status": "osd is down" 2024-09-05T16:34:35.861 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-09-05T16:34:35.861 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-09-05T16:34:35.861 INFO:teuthology.orchestra.run.smithi087.stdout: "more": false 2024-09-05T16:34:35.861 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-09-05T16:34:35.885 INFO:tasks.ec_lost_unfound:{'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 2, 'num_unfound': 2, 'objects': [{'oid': {'oid': 'existing_8', 'key': '', 'snapid': -2, 'hash': 1901787523, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "21'5", 'have': "18'3", 'flags': 'none', 'clean_regions': 'clean_offsets: [(6736, 18446744073709544879)], clean_omap: true, new_object: false', 'locations': ['2(2)']}, {'oid': {'oid': 'existed_1', 'key': '', 'snapid': -2, 'hash': 3122379363, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "21'4", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: false, new_object: true', 'locations': ['2(2)']}], 'state': 'NotRecovering', 'available_might_have_unfound': True, 'might_have_unfound': [{'osd': '1(0)', 'status': 'osd is down'}]} 2024-09-05T16:34:35.886 INFO:tasks.ec_lost_unfound:reverting unfound in 1.3 2024-09-05T16:34:35.886 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 1.3 mark_unfound_lost delete 2024-09-05T16:34:36.148 INFO:teuthology.orchestra.run.smithi087.stderr:pg has 2 objects unfound and apparently lost marking 2024-09-05T16:34:36.172 INFO:tasks.ec_lost_unfound:listing missing/lost in 1.4 state active+recovery_unfound+undersized+degraded 2024-09-05T16:34:36.172 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 1.4 list_unfound '{}' 2024-09-05T16:34:36.397 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-09-05T16:34:36.397 INFO:teuthology.orchestra.run.smithi087.stdout: "offset": { 2024-09-05T16:34:36.397 INFO:teuthology.orchestra.run.smithi087.stdout: "oid": "", 2024-09-05T16:34:36.397 INFO:teuthology.orchestra.run.smithi087.stdout: "key": "", 2024-09-05T16:34:36.397 INFO:teuthology.orchestra.run.smithi087.stdout: "snapid": 0, 2024-09-05T16:34:36.397 INFO:teuthology.orchestra.run.smithi087.stdout: "hash": 0, 2024-09-05T16:34:36.398 INFO:teuthology.orchestra.run.smithi087.stdout: "max": 0, 2024-09-05T16:34:36.398 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": -9223372036854775808, 2024-09-05T16:34:36.398 INFO:teuthology.orchestra.run.smithi087.stdout: "namespace": "" 2024-09-05T16:34:36.398 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:36.398 INFO:teuthology.orchestra.run.smithi087.stdout: "num_missing": 3, 2024-09-05T16:34:36.398 INFO:teuthology.orchestra.run.smithi087.stdout: "num_unfound": 3, 2024-09-05T16:34:36.398 INFO:teuthology.orchestra.run.smithi087.stdout: "objects": [ 2024-09-05T16:34:36.398 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:36.398 INFO:teuthology.orchestra.run.smithi087.stdout: "oid": { 2024-09-05T16:34:36.398 INFO:teuthology.orchestra.run.smithi087.stdout: "oid": "existed_4", 2024-09-05T16:34:36.398 INFO:teuthology.orchestra.run.smithi087.stdout: "key": "", 2024-09-05T16:34:36.398 INFO:teuthology.orchestra.run.smithi087.stdout: "snapid": -2, 2024-09-05T16:34:36.398 INFO:teuthology.orchestra.run.smithi087.stdout: "hash": 873891716, 2024-09-05T16:34:36.399 INFO:teuthology.orchestra.run.smithi087.stdout: "max": 0, 2024-09-05T16:34:36.399 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 1, 2024-09-05T16:34:36.399 INFO:teuthology.orchestra.run.smithi087.stdout: "namespace": "" 2024-09-05T16:34:36.399 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:36.399 INFO:teuthology.orchestra.run.smithi087.stdout: "need": "21'8", 2024-09-05T16:34:36.399 INFO:teuthology.orchestra.run.smithi087.stdout: "have": "0'0", 2024-09-05T16:34:36.399 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": "none", 2024-09-05T16:34:36.399 INFO:teuthology.orchestra.run.smithi087.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2024-09-05T16:34:36.399 INFO:teuthology.orchestra.run.smithi087.stdout: "locations": [ 2024-09-05T16:34:36.399 INFO:teuthology.orchestra.run.smithi087.stdout: "2(2)" 2024-09-05T16:34:36.399 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-09-05T16:34:36.399 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:36.399 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:36.399 INFO:teuthology.orchestra.run.smithi087.stdout: "oid": { 2024-09-05T16:34:36.400 INFO:teuthology.orchestra.run.smithi087.stdout: "oid": "existing_7", 2024-09-05T16:34:36.400 INFO:teuthology.orchestra.run.smithi087.stdout: "key": "", 2024-09-05T16:34:36.400 INFO:teuthology.orchestra.run.smithi087.stdout: "snapid": -2, 2024-09-05T16:34:36.400 INFO:teuthology.orchestra.run.smithi087.stdout: "hash": 3996155460, 2024-09-05T16:34:36.400 INFO:teuthology.orchestra.run.smithi087.stdout: "max": 0, 2024-09-05T16:34:36.400 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 1, 2024-09-05T16:34:36.400 INFO:teuthology.orchestra.run.smithi087.stdout: "namespace": "" 2024-09-05T16:34:36.400 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:36.400 INFO:teuthology.orchestra.run.smithi087.stdout: "need": "21'9", 2024-09-05T16:34:36.400 INFO:teuthology.orchestra.run.smithi087.stdout: "have": "18'6", 2024-09-05T16:34:36.400 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": "none", 2024-09-05T16:34:36.400 INFO:teuthology.orchestra.run.smithi087.stdout: "clean_regions": "clean_offsets: [(6736, 18446744073709544879)], clean_omap: true, new_object: false", 2024-09-05T16:34:36.400 INFO:teuthology.orchestra.run.smithi087.stdout: "locations": [ 2024-09-05T16:34:36.400 INFO:teuthology.orchestra.run.smithi087.stdout: "2(2)" 2024-09-05T16:34:36.401 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-09-05T16:34:36.401 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:36.401 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:36.401 INFO:teuthology.orchestra.run.smithi087.stdout: "oid": { 2024-09-05T16:34:36.401 INFO:teuthology.orchestra.run.smithi087.stdout: "oid": "existed_2", 2024-09-05T16:34:36.401 INFO:teuthology.orchestra.run.smithi087.stdout: "key": "", 2024-09-05T16:34:36.401 INFO:teuthology.orchestra.run.smithi087.stdout: "snapid": -2, 2024-09-05T16:34:36.401 INFO:teuthology.orchestra.run.smithi087.stdout: "hash": 1767667508, 2024-09-05T16:34:36.401 INFO:teuthology.orchestra.run.smithi087.stdout: "max": 0, 2024-09-05T16:34:36.401 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 1, 2024-09-05T16:34:36.401 INFO:teuthology.orchestra.run.smithi087.stdout: "namespace": "" 2024-09-05T16:34:36.401 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:36.401 INFO:teuthology.orchestra.run.smithi087.stdout: "need": "21'7", 2024-09-05T16:34:36.402 INFO:teuthology.orchestra.run.smithi087.stdout: "have": "0'0", 2024-09-05T16:34:36.402 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": "none", 2024-09-05T16:34:36.402 INFO:teuthology.orchestra.run.smithi087.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2024-09-05T16:34:36.402 INFO:teuthology.orchestra.run.smithi087.stdout: "locations": [ 2024-09-05T16:34:36.402 INFO:teuthology.orchestra.run.smithi087.stdout: "2(2)" 2024-09-05T16:34:36.402 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-09-05T16:34:36.402 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-09-05T16:34:36.402 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-09-05T16:34:36.402 INFO:teuthology.orchestra.run.smithi087.stdout: "state": "NotRecovering", 2024-09-05T16:34:36.402 INFO:teuthology.orchestra.run.smithi087.stdout: "available_might_have_unfound": true, 2024-09-05T16:34:36.402 INFO:teuthology.orchestra.run.smithi087.stdout: "might_have_unfound": [ 2024-09-05T16:34:36.402 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:36.402 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": "1(3)", 2024-09-05T16:34:36.402 INFO:teuthology.orchestra.run.smithi087.stdout: "status": "osd is down" 2024-09-05T16:34:36.402 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-09-05T16:34:36.403 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-09-05T16:34:36.403 INFO:teuthology.orchestra.run.smithi087.stdout: "more": false 2024-09-05T16:34:36.403 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-09-05T16:34:36.425 INFO:tasks.ec_lost_unfound:{'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 3, 'num_unfound': 3, 'objects': [{'oid': {'oid': 'existed_4', 'key': '', 'snapid': -2, 'hash': 873891716, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "21'8", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: false, new_object: true', 'locations': ['2(2)']}, {'oid': {'oid': 'existing_7', 'key': '', 'snapid': -2, 'hash': 3996155460, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "21'9", 'have': "18'6", 'flags': 'none', 'clean_regions': 'clean_offsets: [(6736, 18446744073709544879)], clean_omap: true, new_object: false', 'locations': ['2(2)']}, {'oid': {'oid': 'existed_2', 'key': '', 'snapid': -2, 'hash': 1767667508, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "21'7", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: false, new_object: true', 'locations': ['2(2)']}], 'state': 'NotRecovering', 'available_might_have_unfound': True, 'might_have_unfound': [{'osd': '1(3)', 'status': 'osd is down'}]} 2024-09-05T16:34:36.425 INFO:tasks.ec_lost_unfound:reverting unfound in 1.4 2024-09-05T16:34:36.425 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 1.4 mark_unfound_lost delete 2024-09-05T16:34:36.492 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 14 20 19271 19251 10.7409 8.28125 0.0191105 0.0145355 2024-09-05T16:34:36.676 INFO:teuthology.orchestra.run.smithi087.stderr:pg has 3 objects unfound and apparently lost marking 2024-09-05T16:34:36.698 INFO:tasks.ec_lost_unfound:listing missing/lost in 1.5 state active+recovery_unfound+undersized+degraded 2024-09-05T16:34:36.699 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 1.5 list_unfound '{}' 2024-09-05T16:34:36.917 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-09-05T16:34:36.917 INFO:teuthology.orchestra.run.smithi087.stdout: "offset": { 2024-09-05T16:34:36.917 INFO:teuthology.orchestra.run.smithi087.stdout: "oid": "", 2024-09-05T16:34:36.917 INFO:teuthology.orchestra.run.smithi087.stdout: "key": "", 2024-09-05T16:34:36.918 INFO:teuthology.orchestra.run.smithi087.stdout: "snapid": 0, 2024-09-05T16:34:36.918 INFO:teuthology.orchestra.run.smithi087.stdout: "hash": 0, 2024-09-05T16:34:36.918 INFO:teuthology.orchestra.run.smithi087.stdout: "max": 0, 2024-09-05T16:34:36.918 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": -9223372036854775808, 2024-09-05T16:34:36.918 INFO:teuthology.orchestra.run.smithi087.stdout: "namespace": "" 2024-09-05T16:34:36.918 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:36.918 INFO:teuthology.orchestra.run.smithi087.stdout: "num_missing": 2, 2024-09-05T16:34:36.918 INFO:teuthology.orchestra.run.smithi087.stdout: "num_unfound": 2, 2024-09-05T16:34:36.918 INFO:teuthology.orchestra.run.smithi087.stdout: "objects": [ 2024-09-05T16:34:36.918 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:36.918 INFO:teuthology.orchestra.run.smithi087.stdout: "oid": { 2024-09-05T16:34:36.918 INFO:teuthology.orchestra.run.smithi087.stdout: "oid": "new_1", 2024-09-05T16:34:36.918 INFO:teuthology.orchestra.run.smithi087.stdout: "key": "", 2024-09-05T16:34:36.919 INFO:teuthology.orchestra.run.smithi087.stdout: "snapid": -2, 2024-09-05T16:34:36.919 INFO:teuthology.orchestra.run.smithi087.stdout: "hash": 26281365, 2024-09-05T16:34:36.919 INFO:teuthology.orchestra.run.smithi087.stdout: "max": 0, 2024-09-05T16:34:36.919 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 1, 2024-09-05T16:34:36.919 INFO:teuthology.orchestra.run.smithi087.stdout: "namespace": "" 2024-09-05T16:34:36.919 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:36.919 INFO:teuthology.orchestra.run.smithi087.stdout: "need": "21'1", 2024-09-05T16:34:36.919 INFO:teuthology.orchestra.run.smithi087.stdout: "have": "0'0", 2024-09-05T16:34:36.919 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": "none", 2024-09-05T16:34:36.919 INFO:teuthology.orchestra.run.smithi087.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2024-09-05T16:34:36.919 INFO:teuthology.orchestra.run.smithi087.stdout: "locations": [ 2024-09-05T16:34:36.919 INFO:teuthology.orchestra.run.smithi087.stdout: "2(0)" 2024-09-05T16:34:36.920 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-09-05T16:34:36.920 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:36.920 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:36.920 INFO:teuthology.orchestra.run.smithi087.stdout: "oid": { 2024-09-05T16:34:36.920 INFO:teuthology.orchestra.run.smithi087.stdout: "oid": "new_2", 2024-09-05T16:34:36.920 INFO:teuthology.orchestra.run.smithi087.stdout: "key": "", 2024-09-05T16:34:36.920 INFO:teuthology.orchestra.run.smithi087.stdout: "snapid": -2, 2024-09-05T16:34:36.920 INFO:teuthology.orchestra.run.smithi087.stdout: "hash": 3196340981, 2024-09-05T16:34:36.920 INFO:teuthology.orchestra.run.smithi087.stdout: "max": 0, 2024-09-05T16:34:36.920 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 1, 2024-09-05T16:34:36.920 INFO:teuthology.orchestra.run.smithi087.stdout: "namespace": "" 2024-09-05T16:34:36.920 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:36.921 INFO:teuthology.orchestra.run.smithi087.stdout: "need": "21'2", 2024-09-05T16:34:36.921 INFO:teuthology.orchestra.run.smithi087.stdout: "have": "0'0", 2024-09-05T16:34:36.921 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": "none", 2024-09-05T16:34:36.921 INFO:teuthology.orchestra.run.smithi087.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2024-09-05T16:34:36.921 INFO:teuthology.orchestra.run.smithi087.stdout: "locations": [ 2024-09-05T16:34:36.921 INFO:teuthology.orchestra.run.smithi087.stdout: "2(0)" 2024-09-05T16:34:36.921 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-09-05T16:34:36.921 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-09-05T16:34:36.921 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-09-05T16:34:36.921 INFO:teuthology.orchestra.run.smithi087.stdout: "state": "NotRecovering", 2024-09-05T16:34:36.921 INFO:teuthology.orchestra.run.smithi087.stdout: "available_might_have_unfound": true, 2024-09-05T16:34:36.921 INFO:teuthology.orchestra.run.smithi087.stdout: "might_have_unfound": [ 2024-09-05T16:34:36.921 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:36.922 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": "1(3)", 2024-09-05T16:34:36.922 INFO:teuthology.orchestra.run.smithi087.stdout: "status": "osd is down" 2024-09-05T16:34:36.922 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-09-05T16:34:36.922 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-09-05T16:34:36.922 INFO:teuthology.orchestra.run.smithi087.stdout: "more": false 2024-09-05T16:34:36.922 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-09-05T16:34:36.937 INFO:tasks.ec_lost_unfound:{'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 2, 'num_unfound': 2, 'objects': [{'oid': {'oid': 'new_1', 'key': '', 'snapid': -2, 'hash': 26281365, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "21'1", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: false, new_object: true', 'locations': ['2(0)']}, {'oid': {'oid': 'new_2', 'key': '', 'snapid': -2, 'hash': 3196340981, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "21'2", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: false, new_object: true', 'locations': ['2(0)']}], 'state': 'NotRecovering', 'available_might_have_unfound': True, 'might_have_unfound': [{'osd': '1(3)', 'status': 'osd is down'}]} 2024-09-05T16:34:36.937 INFO:tasks.ec_lost_unfound:reverting unfound in 1.5 2024-09-05T16:34:36.937 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 1.5 mark_unfound_lost delete 2024-09-05T16:34:36.951 INFO:tasks.ceph.mon.a.smithi087.stderr:2024-09-05T16:34:36.958+0000 7efd57ed4640 -1 log_channel(cluster) log [ERR] : Health check update: Possible data damage: 11 pgs recovery_unfound (PG_DAMAGED) 2024-09-05T16:34:37.179 INFO:teuthology.orchestra.run.smithi087.stderr:pg has 2 objects unfound and apparently lost marking 2024-09-05T16:34:37.200 INFO:tasks.ec_lost_unfound:listing missing/lost in 1.6 state active+recovery_unfound+undersized+degraded 2024-09-05T16:34:37.200 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 1.6 list_unfound '{}' 2024-09-05T16:34:37.479 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-09-05T16:34:37.479 INFO:teuthology.orchestra.run.smithi087.stdout: "offset": { 2024-09-05T16:34:37.479 INFO:teuthology.orchestra.run.smithi087.stdout: "oid": "", 2024-09-05T16:34:37.479 INFO:teuthology.orchestra.run.smithi087.stdout: "key": "", 2024-09-05T16:34:37.479 INFO:teuthology.orchestra.run.smithi087.stdout: "snapid": 0, 2024-09-05T16:34:37.479 INFO:teuthology.orchestra.run.smithi087.stdout: "hash": 0, 2024-09-05T16:34:37.479 INFO:teuthology.orchestra.run.smithi087.stdout: "max": 0, 2024-09-05T16:34:37.479 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": -9223372036854775808, 2024-09-05T16:34:37.479 INFO:teuthology.orchestra.run.smithi087.stdout: "namespace": "" 2024-09-05T16:34:37.480 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:37.480 INFO:teuthology.orchestra.run.smithi087.stdout: "num_missing": 3, 2024-09-05T16:34:37.480 INFO:teuthology.orchestra.run.smithi087.stdout: "num_unfound": 3, 2024-09-05T16:34:37.480 INFO:teuthology.orchestra.run.smithi087.stdout: "objects": [ 2024-09-05T16:34:37.480 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:37.480 INFO:teuthology.orchestra.run.smithi087.stdout: "oid": { 2024-09-05T16:34:37.480 INFO:teuthology.orchestra.run.smithi087.stdout: "oid": "existed_8", 2024-09-05T16:34:37.480 INFO:teuthology.orchestra.run.smithi087.stdout: "key": "", 2024-09-05T16:34:37.480 INFO:teuthology.orchestra.run.smithi087.stdout: "snapid": -2, 2024-09-05T16:34:37.480 INFO:teuthology.orchestra.run.smithi087.stdout: "hash": 1861424966, 2024-09-05T16:34:37.480 INFO:teuthology.orchestra.run.smithi087.stdout: "max": 0, 2024-09-05T16:34:37.480 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 1, 2024-09-05T16:34:37.480 INFO:teuthology.orchestra.run.smithi087.stdout: "namespace": "" 2024-09-05T16:34:37.480 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:37.481 INFO:teuthology.orchestra.run.smithi087.stdout: "need": "21'5", 2024-09-05T16:34:37.481 INFO:teuthology.orchestra.run.smithi087.stdout: "have": "0'0", 2024-09-05T16:34:37.481 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": "none", 2024-09-05T16:34:37.481 INFO:teuthology.orchestra.run.smithi087.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2024-09-05T16:34:37.481 INFO:teuthology.orchestra.run.smithi087.stdout: "locations": [ 2024-09-05T16:34:37.481 INFO:teuthology.orchestra.run.smithi087.stdout: "2(3)" 2024-09-05T16:34:37.481 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-09-05T16:34:37.481 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:37.481 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:37.481 INFO:teuthology.orchestra.run.smithi087.stdout: "oid": { 2024-09-05T16:34:37.481 INFO:teuthology.orchestra.run.smithi087.stdout: "oid": "new_6", 2024-09-05T16:34:37.481 INFO:teuthology.orchestra.run.smithi087.stdout: "key": "", 2024-09-05T16:34:37.481 INFO:teuthology.orchestra.run.smithi087.stdout: "snapid": -2, 2024-09-05T16:34:37.481 INFO:teuthology.orchestra.run.smithi087.stdout: "hash": 3910131350, 2024-09-05T16:34:37.482 INFO:teuthology.orchestra.run.smithi087.stdout: "max": 0, 2024-09-05T16:34:37.482 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 1, 2024-09-05T16:34:37.482 INFO:teuthology.orchestra.run.smithi087.stdout: "namespace": "" 2024-09-05T16:34:37.482 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:37.482 INFO:teuthology.orchestra.run.smithi087.stdout: "need": "21'4", 2024-09-05T16:34:37.482 INFO:teuthology.orchestra.run.smithi087.stdout: "have": "0'0", 2024-09-05T16:34:37.482 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": "none", 2024-09-05T16:34:37.482 INFO:teuthology.orchestra.run.smithi087.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2024-09-05T16:34:37.482 INFO:teuthology.orchestra.run.smithi087.stdout: "locations": [ 2024-09-05T16:34:37.482 INFO:teuthology.orchestra.run.smithi087.stdout: "2(3)" 2024-09-05T16:34:37.482 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-09-05T16:34:37.482 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:37.482 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:37.482 INFO:teuthology.orchestra.run.smithi087.stdout: "oid": { 2024-09-05T16:34:37.483 INFO:teuthology.orchestra.run.smithi087.stdout: "oid": "new_4", 2024-09-05T16:34:37.483 INFO:teuthology.orchestra.run.smithi087.stdout: "key": "", 2024-09-05T16:34:37.483 INFO:teuthology.orchestra.run.smithi087.stdout: "snapid": -2, 2024-09-05T16:34:37.483 INFO:teuthology.orchestra.run.smithi087.stdout: "hash": 2787839990, 2024-09-05T16:34:37.483 INFO:teuthology.orchestra.run.smithi087.stdout: "max": 0, 2024-09-05T16:34:37.483 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 1, 2024-09-05T16:34:37.483 INFO:teuthology.orchestra.run.smithi087.stdout: "namespace": "" 2024-09-05T16:34:37.483 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:37.483 INFO:teuthology.orchestra.run.smithi087.stdout: "need": "21'3", 2024-09-05T16:34:37.483 INFO:teuthology.orchestra.run.smithi087.stdout: "have": "0'0", 2024-09-05T16:34:37.483 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": "none", 2024-09-05T16:34:37.483 INFO:teuthology.orchestra.run.smithi087.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2024-09-05T16:34:37.483 INFO:teuthology.orchestra.run.smithi087.stdout: "locations": [ 2024-09-05T16:34:37.483 INFO:teuthology.orchestra.run.smithi087.stdout: "2(3)" 2024-09-05T16:34:37.484 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-09-05T16:34:37.484 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-09-05T16:34:37.484 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-09-05T16:34:37.484 INFO:teuthology.orchestra.run.smithi087.stdout: "state": "NotRecovering", 2024-09-05T16:34:37.484 INFO:teuthology.orchestra.run.smithi087.stdout: "available_might_have_unfound": true, 2024-09-05T16:34:37.484 INFO:teuthology.orchestra.run.smithi087.stdout: "might_have_unfound": [ 2024-09-05T16:34:37.484 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:37.484 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": "1(2)", 2024-09-05T16:34:37.484 INFO:teuthology.orchestra.run.smithi087.stdout: "status": "osd is down" 2024-09-05T16:34:37.484 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-09-05T16:34:37.484 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-09-05T16:34:37.484 INFO:teuthology.orchestra.run.smithi087.stdout: "more": false 2024-09-05T16:34:37.484 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-09-05T16:34:37.492 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 15 20 20329 20309 10.5758 8.26562 0.0134463 0.0147604 2024-09-05T16:34:37.504 INFO:tasks.ec_lost_unfound:{'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 3, 'num_unfound': 3, 'objects': [{'oid': {'oid': 'existed_8', 'key': '', 'snapid': -2, 'hash': 1861424966, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "21'5", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: false, new_object: true', 'locations': ['2(3)']}, {'oid': {'oid': 'new_6', 'key': '', 'snapid': -2, 'hash': 3910131350, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "21'4", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: false, new_object: true', 'locations': ['2(3)']}, {'oid': {'oid': 'new_4', 'key': '', 'snapid': -2, 'hash': 2787839990, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "21'3", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: false, new_object: true', 'locations': ['2(3)']}], 'state': 'NotRecovering', 'available_might_have_unfound': True, 'might_have_unfound': [{'osd': '1(2)', 'status': 'osd is down'}]} 2024-09-05T16:34:37.504 INFO:tasks.ec_lost_unfound:reverting unfound in 1.6 2024-09-05T16:34:37.504 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 1.6 mark_unfound_lost delete 2024-09-05T16:34:37.826 INFO:teuthology.orchestra.run.smithi087.stderr:pg has 3 objects unfound and apparently lost marking 2024-09-05T16:34:37.845 INFO:tasks.ec_lost_unfound:listing missing/lost in 1.7 state active+recovery_unfound+undersized+degraded 2024-09-05T16:34:37.845 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 1.7 list_unfound '{}' 2024-09-05T16:34:38.070 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-09-05T16:34:38.070 INFO:teuthology.orchestra.run.smithi087.stdout: "offset": { 2024-09-05T16:34:38.070 INFO:teuthology.orchestra.run.smithi087.stdout: "oid": "", 2024-09-05T16:34:38.070 INFO:teuthology.orchestra.run.smithi087.stdout: "key": "", 2024-09-05T16:34:38.070 INFO:teuthology.orchestra.run.smithi087.stdout: "snapid": 0, 2024-09-05T16:34:38.070 INFO:teuthology.orchestra.run.smithi087.stdout: "hash": 0, 2024-09-05T16:34:38.070 INFO:teuthology.orchestra.run.smithi087.stdout: "max": 0, 2024-09-05T16:34:38.070 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": -9223372036854775808, 2024-09-05T16:34:38.070 INFO:teuthology.orchestra.run.smithi087.stdout: "namespace": "" 2024-09-05T16:34:38.070 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:38.071 INFO:teuthology.orchestra.run.smithi087.stdout: "num_missing": 1, 2024-09-05T16:34:38.071 INFO:teuthology.orchestra.run.smithi087.stdout: "num_unfound": 1, 2024-09-05T16:34:38.071 INFO:teuthology.orchestra.run.smithi087.stdout: "objects": [ 2024-09-05T16:34:38.071 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:38.071 INFO:teuthology.orchestra.run.smithi087.stdout: "oid": { 2024-09-05T16:34:38.071 INFO:teuthology.orchestra.run.smithi087.stdout: "oid": "new_7", 2024-09-05T16:34:38.071 INFO:teuthology.orchestra.run.smithi087.stdout: "key": "", 2024-09-05T16:34:38.071 INFO:teuthology.orchestra.run.smithi087.stdout: "snapid": -2, 2024-09-05T16:34:38.071 INFO:teuthology.orchestra.run.smithi087.stdout: "hash": 3761017527, 2024-09-05T16:34:38.071 INFO:teuthology.orchestra.run.smithi087.stdout: "max": 0, 2024-09-05T16:34:38.071 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 1, 2024-09-05T16:34:38.071 INFO:teuthology.orchestra.run.smithi087.stdout: "namespace": "" 2024-09-05T16:34:38.071 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:38.071 INFO:teuthology.orchestra.run.smithi087.stdout: "need": "21'1", 2024-09-05T16:34:38.072 INFO:teuthology.orchestra.run.smithi087.stdout: "have": "0'0", 2024-09-05T16:34:38.072 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": "none", 2024-09-05T16:34:38.072 INFO:teuthology.orchestra.run.smithi087.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2024-09-05T16:34:38.072 INFO:teuthology.orchestra.run.smithi087.stdout: "locations": [ 2024-09-05T16:34:38.072 INFO:teuthology.orchestra.run.smithi087.stdout: "2(2)" 2024-09-05T16:34:38.072 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-09-05T16:34:38.072 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-09-05T16:34:38.072 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-09-05T16:34:38.072 INFO:teuthology.orchestra.run.smithi087.stdout: "state": "NotRecovering", 2024-09-05T16:34:38.072 INFO:teuthology.orchestra.run.smithi087.stdout: "available_might_have_unfound": true, 2024-09-05T16:34:38.072 INFO:teuthology.orchestra.run.smithi087.stdout: "might_have_unfound": [ 2024-09-05T16:34:38.072 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:38.072 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": "1(0)", 2024-09-05T16:34:38.072 INFO:teuthology.orchestra.run.smithi087.stdout: "status": "osd is down" 2024-09-05T16:34:38.073 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-09-05T16:34:38.073 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-09-05T16:34:38.073 INFO:teuthology.orchestra.run.smithi087.stdout: "more": false 2024-09-05T16:34:38.073 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-09-05T16:34:38.094 INFO:tasks.ec_lost_unfound:{'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 1, 'num_unfound': 1, 'objects': [{'oid': {'oid': 'new_7', 'key': '', 'snapid': -2, 'hash': 3761017527, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "21'1", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: false, new_object: true', 'locations': ['2(2)']}], 'state': 'NotRecovering', 'available_might_have_unfound': True, 'might_have_unfound': [{'osd': '1(0)', 'status': 'osd is down'}]} 2024-09-05T16:34:38.094 INFO:tasks.ec_lost_unfound:reverting unfound in 1.7 2024-09-05T16:34:38.095 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 1.7 mark_unfound_lost delete 2024-09-05T16:34:38.348 INFO:teuthology.orchestra.run.smithi087.stderr:pg has 1 objects unfound and apparently lost marking 2024-09-05T16:34:38.376 INFO:tasks.ec_lost_unfound:listing missing/lost in 1.8 state active+recovery_unfound+undersized+degraded 2024-09-05T16:34:38.376 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 1.8 list_unfound '{}' 2024-09-05T16:34:38.493 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 16 20 21259 21239 10.3689 7.26562 0.0218426 0.0150576 2024-09-05T16:34:38.607 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-09-05T16:34:38.607 INFO:teuthology.orchestra.run.smithi087.stdout: "offset": { 2024-09-05T16:34:38.607 INFO:teuthology.orchestra.run.smithi087.stdout: "oid": "", 2024-09-05T16:34:38.607 INFO:teuthology.orchestra.run.smithi087.stdout: "key": "", 2024-09-05T16:34:38.607 INFO:teuthology.orchestra.run.smithi087.stdout: "snapid": 0, 2024-09-05T16:34:38.607 INFO:teuthology.orchestra.run.smithi087.stdout: "hash": 0, 2024-09-05T16:34:38.607 INFO:teuthology.orchestra.run.smithi087.stdout: "max": 0, 2024-09-05T16:34:38.607 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": -9223372036854775808, 2024-09-05T16:34:38.607 INFO:teuthology.orchestra.run.smithi087.stdout: "namespace": "" 2024-09-05T16:34:38.607 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:38.608 INFO:teuthology.orchestra.run.smithi087.stdout: "num_missing": 1, 2024-09-05T16:34:38.608 INFO:teuthology.orchestra.run.smithi087.stdout: "num_unfound": 1, 2024-09-05T16:34:38.608 INFO:teuthology.orchestra.run.smithi087.stdout: "objects": [ 2024-09-05T16:34:38.608 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:38.608 INFO:teuthology.orchestra.run.smithi087.stdout: "oid": { 2024-09-05T16:34:38.608 INFO:teuthology.orchestra.run.smithi087.stdout: "oid": "new_8", 2024-09-05T16:34:38.608 INFO:teuthology.orchestra.run.smithi087.stdout: "key": "", 2024-09-05T16:34:38.608 INFO:teuthology.orchestra.run.smithi087.stdout: "snapid": -2, 2024-09-05T16:34:38.608 INFO:teuthology.orchestra.run.smithi087.stdout: "hash": 2730327960, 2024-09-05T16:34:38.608 INFO:teuthology.orchestra.run.smithi087.stdout: "max": 0, 2024-09-05T16:34:38.608 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 1, 2024-09-05T16:34:38.608 INFO:teuthology.orchestra.run.smithi087.stdout: "namespace": "" 2024-09-05T16:34:38.608 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:38.608 INFO:teuthology.orchestra.run.smithi087.stdout: "need": "21'1", 2024-09-05T16:34:38.609 INFO:teuthology.orchestra.run.smithi087.stdout: "have": "0'0", 2024-09-05T16:34:38.609 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": "none", 2024-09-05T16:34:38.609 INFO:teuthology.orchestra.run.smithi087.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2024-09-05T16:34:38.609 INFO:teuthology.orchestra.run.smithi087.stdout: "locations": [ 2024-09-05T16:34:38.609 INFO:teuthology.orchestra.run.smithi087.stdout: "2(1)" 2024-09-05T16:34:38.609 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-09-05T16:34:38.609 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-09-05T16:34:38.609 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-09-05T16:34:38.609 INFO:teuthology.orchestra.run.smithi087.stdout: "state": "NotRecovering", 2024-09-05T16:34:38.609 INFO:teuthology.orchestra.run.smithi087.stdout: "available_might_have_unfound": true, 2024-09-05T16:34:38.609 INFO:teuthology.orchestra.run.smithi087.stdout: "might_have_unfound": [ 2024-09-05T16:34:38.609 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:38.609 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": "1(0)", 2024-09-05T16:34:38.609 INFO:teuthology.orchestra.run.smithi087.stdout: "status": "osd is down" 2024-09-05T16:34:38.610 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-09-05T16:34:38.610 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-09-05T16:34:38.610 INFO:teuthology.orchestra.run.smithi087.stdout: "more": false 2024-09-05T16:34:38.610 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-09-05T16:34:38.637 INFO:tasks.ec_lost_unfound:{'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 1, 'num_unfound': 1, 'objects': [{'oid': {'oid': 'new_8', 'key': '', 'snapid': -2, 'hash': 2730327960, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "21'1", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: false, new_object: true', 'locations': ['2(1)']}], 'state': 'NotRecovering', 'available_might_have_unfound': True, 'might_have_unfound': [{'osd': '1(0)', 'status': 'osd is down'}]} 2024-09-05T16:34:38.637 INFO:tasks.ec_lost_unfound:reverting unfound in 1.8 2024-09-05T16:34:38.637 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 1.8 mark_unfound_lost delete 2024-09-05T16:34:38.869 INFO:teuthology.orchestra.run.smithi087.stderr:pg has 1 objects unfound and apparently lost marking 2024-09-05T16:34:38.904 INFO:tasks.ec_lost_unfound:listing missing/lost in 1.9 state active+recovery_unfound+undersized+degraded 2024-09-05T16:34:38.905 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 1.9 list_unfound '{}' 2024-09-05T16:34:39.149 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-09-05T16:34:39.149 INFO:teuthology.orchestra.run.smithi087.stdout: "offset": { 2024-09-05T16:34:39.149 INFO:teuthology.orchestra.run.smithi087.stdout: "oid": "", 2024-09-05T16:34:39.150 INFO:teuthology.orchestra.run.smithi087.stdout: "key": "", 2024-09-05T16:34:39.150 INFO:teuthology.orchestra.run.smithi087.stdout: "snapid": 0, 2024-09-05T16:34:39.150 INFO:teuthology.orchestra.run.smithi087.stdout: "hash": 0, 2024-09-05T16:34:39.150 INFO:teuthology.orchestra.run.smithi087.stdout: "max": 0, 2024-09-05T16:34:39.150 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": -9223372036854775808, 2024-09-05T16:34:39.150 INFO:teuthology.orchestra.run.smithi087.stdout: "namespace": "" 2024-09-05T16:34:39.150 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:39.150 INFO:teuthology.orchestra.run.smithi087.stdout: "num_missing": 2, 2024-09-05T16:34:39.150 INFO:teuthology.orchestra.run.smithi087.stdout: "num_unfound": 2, 2024-09-05T16:34:39.150 INFO:teuthology.orchestra.run.smithi087.stdout: "objects": [ 2024-09-05T16:34:39.150 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:39.150 INFO:teuthology.orchestra.run.smithi087.stdout: "oid": { 2024-09-05T16:34:39.150 INFO:teuthology.orchestra.run.smithi087.stdout: "oid": "existing_5", 2024-09-05T16:34:39.150 INFO:teuthology.orchestra.run.smithi087.stdout: "key": "", 2024-09-05T16:34:39.151 INFO:teuthology.orchestra.run.smithi087.stdout: "snapid": -2, 2024-09-05T16:34:39.151 INFO:teuthology.orchestra.run.smithi087.stdout: "hash": 2998549577, 2024-09-05T16:34:39.151 INFO:teuthology.orchestra.run.smithi087.stdout: "max": 0, 2024-09-05T16:34:39.151 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 1, 2024-09-05T16:34:39.151 INFO:teuthology.orchestra.run.smithi087.stdout: "namespace": "" 2024-09-05T16:34:39.151 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:39.151 INFO:teuthology.orchestra.run.smithi087.stdout: "need": "21'4", 2024-09-05T16:34:39.151 INFO:teuthology.orchestra.run.smithi087.stdout: "have": "18'2", 2024-09-05T16:34:39.151 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": "none", 2024-09-05T16:34:39.151 INFO:teuthology.orchestra.run.smithi087.stdout: "clean_regions": "clean_offsets: [(6736, 18446744073709544879)], clean_omap: true, new_object: false", 2024-09-05T16:34:39.151 INFO:teuthology.orchestra.run.smithi087.stdout: "locations": [ 2024-09-05T16:34:39.151 INFO:teuthology.orchestra.run.smithi087.stdout: "2(2)" 2024-09-05T16:34:39.151 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-09-05T16:34:39.151 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:39.152 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:39.152 INFO:teuthology.orchestra.run.smithi087.stdout: "oid": { 2024-09-05T16:34:39.152 INFO:teuthology.orchestra.run.smithi087.stdout: "oid": "existing_2", 2024-09-05T16:34:39.152 INFO:teuthology.orchestra.run.smithi087.stdout: "key": "", 2024-09-05T16:34:39.152 INFO:teuthology.orchestra.run.smithi087.stdout: "snapid": -2, 2024-09-05T16:34:39.152 INFO:teuthology.orchestra.run.smithi087.stdout: "hash": 4032093161, 2024-09-05T16:34:39.152 INFO:teuthology.orchestra.run.smithi087.stdout: "max": 0, 2024-09-05T16:34:39.152 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 1, 2024-09-05T16:34:39.152 INFO:teuthology.orchestra.run.smithi087.stdout: "namespace": "" 2024-09-05T16:34:39.152 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:39.152 INFO:teuthology.orchestra.run.smithi087.stdout: "need": "21'3", 2024-09-05T16:34:39.152 INFO:teuthology.orchestra.run.smithi087.stdout: "have": "18'1", 2024-09-05T16:34:39.152 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": "none", 2024-09-05T16:34:39.152 INFO:teuthology.orchestra.run.smithi087.stdout: "clean_regions": "clean_offsets: [(6736, 18446744073709544879)], clean_omap: true, new_object: false", 2024-09-05T16:34:39.153 INFO:teuthology.orchestra.run.smithi087.stdout: "locations": [ 2024-09-05T16:34:39.153 INFO:teuthology.orchestra.run.smithi087.stdout: "2(2)" 2024-09-05T16:34:39.153 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-09-05T16:34:39.153 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-09-05T16:34:39.153 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-09-05T16:34:39.153 INFO:teuthology.orchestra.run.smithi087.stdout: "state": "NotRecovering", 2024-09-05T16:34:39.153 INFO:teuthology.orchestra.run.smithi087.stdout: "available_might_have_unfound": true, 2024-09-05T16:34:39.153 INFO:teuthology.orchestra.run.smithi087.stdout: "might_have_unfound": [ 2024-09-05T16:34:39.153 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:39.153 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": "1(0)", 2024-09-05T16:34:39.153 INFO:teuthology.orchestra.run.smithi087.stdout: "status": "osd is down" 2024-09-05T16:34:39.153 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-09-05T16:34:39.153 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-09-05T16:34:39.153 INFO:teuthology.orchestra.run.smithi087.stdout: "more": false 2024-09-05T16:34:39.154 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-09-05T16:34:39.172 INFO:tasks.ec_lost_unfound:{'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 2, 'num_unfound': 2, 'objects': [{'oid': {'oid': 'existing_5', 'key': '', 'snapid': -2, 'hash': 2998549577, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "21'4", 'have': "18'2", 'flags': 'none', 'clean_regions': 'clean_offsets: [(6736, 18446744073709544879)], clean_omap: true, new_object: false', 'locations': ['2(2)']}, {'oid': {'oid': 'existing_2', 'key': '', 'snapid': -2, 'hash': 4032093161, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "21'3", 'have': "18'1", 'flags': 'none', 'clean_regions': 'clean_offsets: [(6736, 18446744073709544879)], clean_omap: true, new_object: false', 'locations': ['2(2)']}], 'state': 'NotRecovering', 'available_might_have_unfound': True, 'might_have_unfound': [{'osd': '1(0)', 'status': 'osd is down'}]} 2024-09-05T16:34:39.173 INFO:tasks.ec_lost_unfound:reverting unfound in 1.9 2024-09-05T16:34:39.173 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 1.9 mark_unfound_lost delete 2024-09-05T16:34:39.359 INFO:teuthology.orchestra.run.smithi087.stderr:pg has 2 objects unfound and apparently lost marking 2024-09-05T16:34:39.369 INFO:tasks.ec_lost_unfound:listing missing/lost in 1.a state active+recovery_unfound+undersized+degraded 2024-09-05T16:34:39.370 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 1.a list_unfound '{}' 2024-09-05T16:34:39.487 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-09-05T16:34:39.488 INFO:teuthology.orchestra.run.smithi087.stdout: "offset": { 2024-09-05T16:34:39.488 INFO:teuthology.orchestra.run.smithi087.stdout: "oid": "", 2024-09-05T16:34:39.488 INFO:teuthology.orchestra.run.smithi087.stdout: "key": "", 2024-09-05T16:34:39.488 INFO:teuthology.orchestra.run.smithi087.stdout: "snapid": 0, 2024-09-05T16:34:39.488 INFO:teuthology.orchestra.run.smithi087.stdout: "hash": 0, 2024-09-05T16:34:39.488 INFO:teuthology.orchestra.run.smithi087.stdout: "max": 0, 2024-09-05T16:34:39.488 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": -9223372036854775808, 2024-09-05T16:34:39.488 INFO:teuthology.orchestra.run.smithi087.stdout: "namespace": "" 2024-09-05T16:34:39.488 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:39.488 INFO:teuthology.orchestra.run.smithi087.stdout: "num_missing": 2, 2024-09-05T16:34:39.488 INFO:teuthology.orchestra.run.smithi087.stdout: "num_unfound": 2, 2024-09-05T16:34:39.489 INFO:teuthology.orchestra.run.smithi087.stdout: "objects": [ 2024-09-05T16:34:39.489 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:39.489 INFO:teuthology.orchestra.run.smithi087.stdout: "oid": { 2024-09-05T16:34:39.489 INFO:teuthology.orchestra.run.smithi087.stdout: "oid": "existing_9", 2024-09-05T16:34:39.489 INFO:teuthology.orchestra.run.smithi087.stdout: "key": "", 2024-09-05T16:34:39.489 INFO:teuthology.orchestra.run.smithi087.stdout: "snapid": -2, 2024-09-05T16:34:39.489 INFO:teuthology.orchestra.run.smithi087.stdout: "hash": 1217469130, 2024-09-05T16:34:39.489 INFO:teuthology.orchestra.run.smithi087.stdout: "max": 0, 2024-09-05T16:34:39.489 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 1, 2024-09-05T16:34:39.489 INFO:teuthology.orchestra.run.smithi087.stdout: "namespace": "" 2024-09-05T16:34:39.489 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:39.489 INFO:teuthology.orchestra.run.smithi087.stdout: "need": "21'3", 2024-09-05T16:34:39.489 INFO:teuthology.orchestra.run.smithi087.stdout: "have": "18'1", 2024-09-05T16:34:39.490 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": "none", 2024-09-05T16:34:39.490 INFO:teuthology.orchestra.run.smithi087.stdout: "clean_regions": "clean_offsets: [(6736, 18446744073709544879)], clean_omap: true, new_object: false", 2024-09-05T16:34:39.490 INFO:teuthology.orchestra.run.smithi087.stdout: "locations": [ 2024-09-05T16:34:39.490 INFO:teuthology.orchestra.run.smithi087.stdout: "2(1)" 2024-09-05T16:34:39.490 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-09-05T16:34:39.490 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:39.490 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:39.490 INFO:teuthology.orchestra.run.smithi087.stdout: "oid": { 2024-09-05T16:34:39.490 INFO:teuthology.orchestra.run.smithi087.stdout: "oid": "new_9", 2024-09-05T16:34:39.490 INFO:teuthology.orchestra.run.smithi087.stdout: "key": "", 2024-09-05T16:34:39.490 INFO:teuthology.orchestra.run.smithi087.stdout: "snapid": -2, 2024-09-05T16:34:39.490 INFO:teuthology.orchestra.run.smithi087.stdout: "hash": 170713514, 2024-09-05T16:34:39.490 INFO:teuthology.orchestra.run.smithi087.stdout: "max": 0, 2024-09-05T16:34:39.490 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 1, 2024-09-05T16:34:39.491 INFO:teuthology.orchestra.run.smithi087.stdout: "namespace": "" 2024-09-05T16:34:39.491 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:39.491 INFO:teuthology.orchestra.run.smithi087.stdout: "need": "21'2", 2024-09-05T16:34:39.491 INFO:teuthology.orchestra.run.smithi087.stdout: "have": "0'0", 2024-09-05T16:34:39.491 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": "none", 2024-09-05T16:34:39.491 INFO:teuthology.orchestra.run.smithi087.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2024-09-05T16:34:39.491 INFO:teuthology.orchestra.run.smithi087.stdout: "locations": [ 2024-09-05T16:34:39.491 INFO:teuthology.orchestra.run.smithi087.stdout: "2(1)" 2024-09-05T16:34:39.491 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-09-05T16:34:39.491 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-09-05T16:34:39.491 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-09-05T16:34:39.491 INFO:teuthology.orchestra.run.smithi087.stdout: "state": "NotRecovering", 2024-09-05T16:34:39.491 INFO:teuthology.orchestra.run.smithi087.stdout: "available_might_have_unfound": true, 2024-09-05T16:34:39.491 INFO:teuthology.orchestra.run.smithi087.stdout: "might_have_unfound": [ 2024-09-05T16:34:39.492 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:39.492 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": "1(0)", 2024-09-05T16:34:39.492 INFO:teuthology.orchestra.run.smithi087.stdout: "status": "osd is down" 2024-09-05T16:34:39.492 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-09-05T16:34:39.492 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-09-05T16:34:39.492 INFO:teuthology.orchestra.run.smithi087.stdout: "more": false 2024-09-05T16:34:39.492 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-09-05T16:34:39.492 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 17 20 22078 22058 10.1354 6.39844 0.00357551 0.0152374 2024-09-05T16:34:39.507 INFO:tasks.ec_lost_unfound:{'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 2, 'num_unfound': 2, 'objects': [{'oid': {'oid': 'existing_9', 'key': '', 'snapid': -2, 'hash': 1217469130, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "21'3", 'have': "18'1", 'flags': 'none', 'clean_regions': 'clean_offsets: [(6736, 18446744073709544879)], clean_omap: true, new_object: false', 'locations': ['2(1)']}, {'oid': {'oid': 'new_9', 'key': '', 'snapid': -2, 'hash': 170713514, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "21'2", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: false, new_object: true', 'locations': ['2(1)']}], 'state': 'NotRecovering', 'available_might_have_unfound': True, 'might_have_unfound': [{'osd': '1(0)', 'status': 'osd is down'}]} 2024-09-05T16:34:39.507 INFO:tasks.ec_lost_unfound:reverting unfound in 1.a 2024-09-05T16:34:39.508 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 1.a mark_unfound_lost delete 2024-09-05T16:34:39.756 INFO:teuthology.orchestra.run.smithi087.stderr:pg has 2 objects unfound and apparently lost marking 2024-09-05T16:34:39.784 INFO:tasks.ec_lost_unfound:listing missing/lost in 1.b state active+recovery_unfound+undersized+degraded 2024-09-05T16:34:39.784 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 1.b list_unfound '{}' 2024-09-05T16:34:40.021 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-09-05T16:34:40.021 INFO:teuthology.orchestra.run.smithi087.stdout: "offset": { 2024-09-05T16:34:40.021 INFO:teuthology.orchestra.run.smithi087.stdout: "oid": "", 2024-09-05T16:34:40.021 INFO:teuthology.orchestra.run.smithi087.stdout: "key": "", 2024-09-05T16:34:40.021 INFO:teuthology.orchestra.run.smithi087.stdout: "snapid": 0, 2024-09-05T16:34:40.021 INFO:teuthology.orchestra.run.smithi087.stdout: "hash": 0, 2024-09-05T16:34:40.021 INFO:teuthology.orchestra.run.smithi087.stdout: "max": 0, 2024-09-05T16:34:40.021 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": -9223372036854775808, 2024-09-05T16:34:40.021 INFO:teuthology.orchestra.run.smithi087.stdout: "namespace": "" 2024-09-05T16:34:40.021 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:40.021 INFO:teuthology.orchestra.run.smithi087.stdout: "num_missing": 2, 2024-09-05T16:34:40.022 INFO:teuthology.orchestra.run.smithi087.stdout: "num_unfound": 2, 2024-09-05T16:34:40.022 INFO:teuthology.orchestra.run.smithi087.stdout: "objects": [ 2024-09-05T16:34:40.022 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:40.022 INFO:teuthology.orchestra.run.smithi087.stdout: "oid": { 2024-09-05T16:34:40.022 INFO:teuthology.orchestra.run.smithi087.stdout: "oid": "existed_3", 2024-09-05T16:34:40.022 INFO:teuthology.orchestra.run.smithi087.stdout: "key": "", 2024-09-05T16:34:40.022 INFO:teuthology.orchestra.run.smithi087.stdout: "snapid": -2, 2024-09-05T16:34:40.022 INFO:teuthology.orchestra.run.smithi087.stdout: "hash": 943960171, 2024-09-05T16:34:40.022 INFO:teuthology.orchestra.run.smithi087.stdout: "max": 0, 2024-09-05T16:34:40.022 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 1, 2024-09-05T16:34:40.022 INFO:teuthology.orchestra.run.smithi087.stdout: "namespace": "" 2024-09-05T16:34:40.022 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:40.022 INFO:teuthology.orchestra.run.smithi087.stdout: "need": "21'5", 2024-09-05T16:34:40.022 INFO:teuthology.orchestra.run.smithi087.stdout: "have": "0'0", 2024-09-05T16:34:40.023 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": "none", 2024-09-05T16:34:40.023 INFO:teuthology.orchestra.run.smithi087.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2024-09-05T16:34:40.023 INFO:teuthology.orchestra.run.smithi087.stdout: "locations": [ 2024-09-05T16:34:40.023 INFO:teuthology.orchestra.run.smithi087.stdout: "2(2)" 2024-09-05T16:34:40.023 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-09-05T16:34:40.023 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:40.023 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:40.023 INFO:teuthology.orchestra.run.smithi087.stdout: "oid": { 2024-09-05T16:34:40.023 INFO:teuthology.orchestra.run.smithi087.stdout: "oid": "existing_1", 2024-09-05T16:34:40.023 INFO:teuthology.orchestra.run.smithi087.stdout: "key": "", 2024-09-05T16:34:40.023 INFO:teuthology.orchestra.run.smithi087.stdout: "snapid": -2, 2024-09-05T16:34:40.023 INFO:teuthology.orchestra.run.smithi087.stdout: "hash": 2885884443, 2024-09-05T16:34:40.023 INFO:teuthology.orchestra.run.smithi087.stdout: "max": 0, 2024-09-05T16:34:40.023 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 1, 2024-09-05T16:34:40.024 INFO:teuthology.orchestra.run.smithi087.stdout: "namespace": "" 2024-09-05T16:34:40.024 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:40.024 INFO:teuthology.orchestra.run.smithi087.stdout: "need": "21'4", 2024-09-05T16:34:40.024 INFO:teuthology.orchestra.run.smithi087.stdout: "have": "18'1", 2024-09-05T16:34:40.024 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": "none", 2024-09-05T16:34:40.024 INFO:teuthology.orchestra.run.smithi087.stdout: "clean_regions": "clean_offsets: [(6736, 18446744073709544879)], clean_omap: true, new_object: false", 2024-09-05T16:34:40.024 INFO:teuthology.orchestra.run.smithi087.stdout: "locations": [ 2024-09-05T16:34:40.024 INFO:teuthology.orchestra.run.smithi087.stdout: "2(2)" 2024-09-05T16:34:40.024 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-09-05T16:34:40.024 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-09-05T16:34:40.024 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-09-05T16:34:40.024 INFO:teuthology.orchestra.run.smithi087.stdout: "state": "NotRecovering", 2024-09-05T16:34:40.024 INFO:teuthology.orchestra.run.smithi087.stdout: "available_might_have_unfound": true, 2024-09-05T16:34:40.024 INFO:teuthology.orchestra.run.smithi087.stdout: "might_have_unfound": [ 2024-09-05T16:34:40.025 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:40.025 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": "1(0)", 2024-09-05T16:34:40.025 INFO:teuthology.orchestra.run.smithi087.stdout: "status": "osd is down" 2024-09-05T16:34:40.025 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-09-05T16:34:40.025 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-09-05T16:34:40.025 INFO:teuthology.orchestra.run.smithi087.stdout: "more": false 2024-09-05T16:34:40.025 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-09-05T16:34:40.042 INFO:tasks.ec_lost_unfound:{'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 2, 'num_unfound': 2, 'objects': [{'oid': {'oid': 'existed_3', 'key': '', 'snapid': -2, 'hash': 943960171, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "21'5", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: false, new_object: true', 'locations': ['2(2)']}, {'oid': {'oid': 'existing_1', 'key': '', 'snapid': -2, 'hash': 2885884443, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "21'4", 'have': "18'1", 'flags': 'none', 'clean_regions': 'clean_offsets: [(6736, 18446744073709544879)], clean_omap: true, new_object: false', 'locations': ['2(2)']}], 'state': 'NotRecovering', 'available_might_have_unfound': True, 'might_have_unfound': [{'osd': '1(0)', 'status': 'osd is down'}]} 2024-09-05T16:34:40.043 INFO:tasks.ec_lost_unfound:reverting unfound in 1.b 2024-09-05T16:34:40.043 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 1.b mark_unfound_lost delete 2024-09-05T16:34:40.295 INFO:teuthology.orchestra.run.smithi087.stderr:pg has 2 objects unfound and apparently lost marking 2024-09-05T16:34:40.318 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 debug kick_recovery_wq 5 2024-09-05T16:34:40.492 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 18 20 23087 23067 10.0102 7.88281 0.0153675 0.0155971 2024-09-05T16:34:40.551 INFO:teuthology.orchestra.run.smithi087.stderr:kicking recovery queue. set osd_recovery_delay_start to 5 2024-09-05T16:34:40.576 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 debug kick_recovery_wq 5 2024-09-05T16:34:40.804 INFO:teuthology.orchestra.run.smithi087.stderr:kicking recovery queue. set osd_recovery_delay_start to 5 2024-09-05T16:34:40.828 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 debug kick_recovery_wq 5 2024-09-05T16:34:41.069 INFO:teuthology.orchestra.run.smithi087.stderr:kicking recovery queue. set osd_recovery_delay_start to 5 2024-09-05T16:34:41.092 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-05T16:34:41.092 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-05T16:34:41.092 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-05T16:34:41.352 INFO:teuthology.orchestra.run.smithi087.stdout:42949672980 2024-09-05T16:34:41.352 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-05T16:34:41.355 INFO:teuthology.orchestra.run.smithi087.stdout:107374182411 2024-09-05T16:34:41.356 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-05T16:34:41.360 INFO:teuthology.orchestra.run.smithi087.stdout:120259084297 2024-09-05T16:34:41.360 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-05T16:34:41.492 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 19 20 23967 23947 9.84516 6.875 0.0199296 0.0158352 2024-09-05T16:34:41.948 INFO:teuthology.orchestra.run.smithi087.stdout:42949672979 2024-09-05T16:34:41.978 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 42949672980 got 42949672979 for osd.2 2024-09-05T16:34:42.012 INFO:teuthology.orchestra.run.smithi087.stdout:107374182410 2024-09-05T16:34:42.015 INFO:teuthology.orchestra.run.smithi087.stdout:120259084296 2024-09-05T16:34:42.040 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 107374182411 got 107374182410 for osd.0 2024-09-05T16:34:42.046 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 120259084297 got 120259084296 for osd.3 2024-09-05T16:34:42.493 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout:2024-09-05T16:34:42.500075+0000 min lat: 0.00357551 max lat: 0.220395 avg lat: 0.0161452 2024-09-05T16:34:42.493 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-05T16:34:42.493 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 20 20 24781 24761 9.67086 6.35938 0.021274 0.0161452 2024-09-05T16:34:42.980 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-05T16:34:43.041 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-05T16:34:43.047 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-05T16:34:43.493 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 21 20 25677 25657 9.54365 7 0.0278019 0.0163359 2024-09-05T16:34:43.564 INFO:teuthology.orchestra.run.smithi087.stdout:42949672980 2024-09-05T16:34:43.592 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 42949672980 got 42949672980 for osd.2 2024-09-05T16:34:43.592 DEBUG:teuthology.parallel:result is None 2024-09-05T16:34:43.615 INFO:teuthology.orchestra.run.smithi087.stdout:107374182411 2024-09-05T16:34:43.643 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 107374182411 got 107374182411 for osd.0 2024-09-05T16:34:43.644 DEBUG:teuthology.parallel:result is None 2024-09-05T16:34:43.667 INFO:teuthology.orchestra.run.smithi087.stdout:120259084297 2024-09-05T16:34:43.694 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 120259084297 got 120259084297 for osd.3 2024-09-05T16:34:43.694 DEBUG:teuthology.parallel:result is None 2024-09-05T16:34:43.694 INFO:tasks.ec_lost_unfound.ceph_manager:waiting for recovery to complete 2024-09-05T16:34:43.695 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:34:44.286 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:34:44.286 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:34:44.308 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":63,"stamp":"2024-09-05T16:34:42.793573+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":195420352,"num_objects":23860,"num_object_clones":0,"num_object_copies":95440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23860,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23860,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23887,"num_write_kb":190978,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":23914,"ondisk_log_size":23914,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":585848,"kb_used_data":281068,"kb_used_omap":11,"kb_used_meta":304692,"kb_avail":376901512,"statfs":{"total":386547056640,"available":385947148288,"internally_reserved":0,"allocated":287813632,"data_stored":287195005,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11990,"internal_metadata":312004906},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,3,5,20,10],"upper_bound":32},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":105897984,"num_objects":12927,"num_object_clones":0,"num_object_copies":51708,"num_objects_missing_on_primary":-21,"num_objects_missing":-21,"num_objects_degraded":12873,"num_objects_misplaced":0,"num_objects_unfound":-27,"num_objects_dirty":12927,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12927,"num_write_kb":103416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003440"},"pg_stats":[{"pgid":"1.f","version":"29'1543","reported_seq":1570,"reported_epoch":29,"state":"active+undersized+degraded","last_fresh":"2024-09-05T16:34:41.267490+0000","last_change":"2024-09-05T16:34:22.526044+0000","last_active":"2024-09-05T16:34:41.267490+0000","last_peered":"2024-09-05T16:34:41.267490+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:15.913051+0000","last_became_peered":"2024-09-05T16:34:15.913051+0000","last_unstale":"2024-09-05T16:34:41.267490+0000","last_undegraded":"2024-09-05T16:34:22.517240+0000","last_fullsized":"2024-09-05T16:34:15.905613+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":1543,"log_dups_size":0,"ondisk_log_size":1543,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T19:25:44.252388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12640256,"num_objects":1543,"num_object_clones":0,"num_object_copies":6172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1543,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1543,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1543,"num_write_kb":12344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2147483647,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1543}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"29'1523","reported_seq":1574,"reported_epoch":29,"state":"active+undersized+degraded","last_fresh":"2024-09-05T16:34:41.292140+0000","last_change":"2024-09-05T16:34:33.555748+0000","last_active":"2024-09-05T16:34:41.292140+0000","last_peered":"2024-09-05T16:34:41.292140+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:15.910788+0000","last_became_peered":"2024-09-05T16:34:15.910788+0000","last_unstale":"2024-09-05T16:34:41.292140+0000","last_undegraded":"2024-09-05T16:34:15.903622+0000","last_fullsized":"2024-09-05T16:34:15.903245+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":1523,"log_dups_size":0,"ondisk_log_size":1523,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-05T16:34:43.645989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12440736,"num_objects":1519,"num_object_clones":0,"num_object_copies":6076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1519,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1519,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1521,"num_write_kb":12164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1519}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"29'1491","reported_seq":1552,"reported_epoch":29,"state":"active+undersized+degraded","last_fresh":"2024-09-05T16:34:41.294991+0000","last_change":"2024-09-05T16:34:34.084629+0000","last_active":"2024-09-05T16:34:41.294991+0000","last_peered":"2024-09-05T16:34:41.294991+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:15.917951+0000","last_became_peered":"2024-09-05T16:34:15.917951+0000","last_unstale":"2024-09-05T16:34:41.294991+0000","last_undegraded":"2024-09-05T16:34:15.904811+0000","last_fullsized":"2024-09-05T16:34:15.904154+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":1491,"log_dups_size":0,"ondisk_log_size":1491,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-05T16:34:45.754218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12188240,"num_objects":1488,"num_object_clones":0,"num_object_copies":5952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1488,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1488,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1490,"num_write_kb":11910,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,2147483647,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1488}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"29'1510","reported_seq":1561,"reported_epoch":29,"state":"active+undersized+degraded","last_fresh":"2024-09-05T16:34:41.308209+0000","last_change":"2024-09-05T16:34:34.682384+0000","last_active":"2024-09-05T16:34:41.308209+0000","last_peered":"2024-09-05T16:34:41.308209+0000","last_clean":"2024-09-05T16:33:53.148717+0000","last_became_active":"2024-09-05T16:34:15.912196+0000","last_became_peered":"2024-09-05T16:34:15.912196+0000","last_unstale":"2024-09-05T16:34:41.308209+0000","last_undegraded":"2024-09-05T16:34:15.905010+0000","last_fullsized":"2024-09-05T16:34:15.904610+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":1510,"log_dups_size":0,"ondisk_log_size":1510,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-05T16:34:45.721510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12334240,"num_objects":1506,"num_object_clones":0,"num_object_copies":6024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1506,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1506,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1508,"num_write_kb":12053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1506}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"29'1446","reported_seq":1486,"reported_epoch":29,"state":"active+undersized+degraded","last_fresh":"2024-09-05T16:34:41.302784+0000","last_change":"2024-09-05T16:34:35.192823+0000","last_active":"2024-09-05T16:34:41.302784+0000","last_peered":"2024-09-05T16:34:41.302784+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:15.910648+0000","last_became_peered":"2024-09-05T16:34:15.910648+0000","last_unstale":"2024-09-05T16:34:41.302784+0000","last_undegraded":"2024-09-05T16:34:15.904665+0000","last_fullsized":"2024-09-05T16:34:15.903277+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":1446,"log_dups_size":0,"ondisk_log_size":1446,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-05T16:34:45.802206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11818144,"num_objects":1443,"num_object_clones":0,"num_object_copies":5772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1443,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1443,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1444,"num_write_kb":11549,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2147483647,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1443}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"29'1450","reported_seq":1479,"reported_epoch":29,"state":"active+undersized+degraded","last_fresh":"2024-09-05T16:34:41.263663+0000","last_change":"2024-09-05T16:34:22.533654+0000","last_active":"2024-09-05T16:34:41.263663+0000","last_peered":"2024-09-05T16:34:41.263663+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:15.913477+0000","last_became_peered":"2024-09-05T16:34:15.913477+0000","last_unstale":"2024-09-05T16:34:41.263663+0000","last_undegraded":"2024-09-05T16:34:22.533443+0000","last_fullsized":"2024-09-05T16:34:15.906631+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":1450,"log_dups_size":0,"ondisk_log_size":1450,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T01:56:46.762011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11878400,"num_objects":1450,"num_object_clones":0,"num_object_copies":5800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1450,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1450,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1450,"num_write_kb":11600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,2147483647],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1450}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"29'1379","reported_seq":1430,"reported_epoch":29,"state":"active+undersized+degraded","last_fresh":"2024-09-05T16:34:41.223931+0000","last_change":"2024-09-05T16:34:35.683668+0000","last_active":"2024-09-05T16:34:41.223931+0000","last_peered":"2024-09-05T16:34:41.223931+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:15.913479+0000","last_became_peered":"2024-09-05T16:34:15.913479+0000","last_unstale":"2024-09-05T16:34:41.223931+0000","last_undegraded":"2024-09-05T16:34:15.905955+0000","last_fullsized":"2024-09-05T16:34:15.905617+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":1379,"log_dups_size":0,"ondisk_log_size":1379,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-05T16:34:41.602576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11244704,"num_objects":1373,"num_object_clones":0,"num_object_copies":5492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1373,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1377,"num_write_kb":10996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2147483647,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1373}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"29'1463","reported_seq":1514,"reported_epoch":29,"state":"active+undersized+degraded","last_fresh":"2024-09-05T16:34:41.294121+0000","last_change":"2024-09-05T16:34:36.231100+0000","last_active":"2024-09-05T16:34:41.294121+0000","last_peered":"2024-09-05T16:34:41.294121+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:15.910570+0000","last_became_peered":"2024-09-05T16:34:15.910570+0000","last_unstale":"2024-09-05T16:34:41.294121+0000","last_undegraded":"2024-09-05T16:34:15.904563+0000","last_fullsized":"2024-09-05T16:34:15.903952+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":1463,"log_dups_size":0,"ondisk_log_size":1463,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-05T16:34:42.604538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11941024,"num_objects":1458,"num_object_clones":0,"num_object_copies":5832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1458,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1458,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1461,"num_write_kb":11676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1458}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"29'1510","reported_seq":1552,"reported_epoch":29,"state":"active+undersized+degraded","last_fresh":"2024-09-05T16:34:41.302518+0000","last_change":"2024-09-05T16:34:36.789538+0000","last_active":"2024-09-05T16:34:41.302518+0000","last_peered":"2024-09-05T16:34:41.302518+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:15.914297+0000","last_became_peered":"2024-09-05T16:34:15.914297+0000","last_unstale":"2024-09-05T16:34:41.302518+0000","last_undegraded":"2024-09-05T16:34:15.906625+0000","last_fullsized":"2024-09-05T16:34:15.906328+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":1510,"log_dups_size":0,"ondisk_log_size":1510,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-05T16:34:42.766823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12298560,"num_objects":1502,"num_object_clones":0,"num_object_copies":6008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1502,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1502,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1507,"num_write_kb":12033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,2147483647],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1502}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"29'1507","reported_seq":1570,"reported_epoch":29,"state":"active+undersized+degraded","last_fresh":"2024-09-05T16:34:41.286043+0000","last_change":"2024-09-05T16:34:37.301824+0000","last_active":"2024-09-05T16:34:41.286043+0000","last_peered":"2024-09-05T16:34:41.286043+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:15.918296+0000","last_became_peered":"2024-09-05T16:34:15.918296+0000","last_unstale":"2024-09-05T16:34:41.286043+0000","last_undegraded":"2024-09-05T16:34:15.905187+0000","last_fullsized":"2024-09-05T16:34:15.904834+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":1507,"log_dups_size":0,"ondisk_log_size":1507,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-05T16:34:42.778601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12326048,"num_objects":1505,"num_object_clones":0,"num_object_copies":6020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1505,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1505,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1505,"num_write_kb":12038,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,2147483647],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1505}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"29'1568","reported_seq":1610,"reported_epoch":29,"state":"active+undersized+degraded","last_fresh":"2024-09-05T16:34:41.303795+0000","last_change":"2024-09-05T16:34:37.985329+0000","last_active":"2024-09-05T16:34:41.303795+0000","last_peered":"2024-09-05T16:34:41.303795+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:15.914271+0000","last_became_peered":"2024-09-05T16:34:15.914271+0000","last_unstale":"2024-09-05T16:34:41.303795+0000","last_undegraded":"2024-09-05T16:34:15.905903+0000","last_fullsized":"2024-09-05T16:34:15.905357+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":1568,"log_dups_size":0,"ondisk_log_size":1568,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-05T16:34:43.785107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12799728,"num_objects":1563,"num_object_clones":0,"num_object_copies":6252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1563,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1563,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1565,"num_write_kb":12508,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2147483647,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1563}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"29'1482","reported_seq":1505,"reported_epoch":29,"state":"active+undersized+degraded","last_fresh":"2024-09-05T16:34:41.331007+0000","last_change":"2024-09-05T16:34:38.396192+0000","last_active":"2024-09-05T16:34:41.331007+0000","last_peered":"2024-09-05T16:34:41.331007+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:15.913627+0000","last_became_peered":"2024-09-05T16:34:15.913627+0000","last_unstale":"2024-09-05T16:34:41.331007+0000","last_undegraded":"2024-09-05T16:34:15.906286+0000","last_fullsized":"2024-09-05T16:34:15.906044+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":1482,"log_dups_size":0,"ondisk_log_size":1482,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-05T16:34:44.816449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12130896,"num_objects":1481,"num_object_clones":0,"num_object_copies":5924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1481,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1481,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1481,"num_write_kb":11847,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1481}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"29'1522","reported_seq":1581,"reported_epoch":29,"state":"active+undersized+degraded","last_fresh":"2024-09-05T16:34:41.328510+0000","last_change":"2024-09-05T16:34:38.929130+0000","last_active":"2024-09-05T16:34:41.328510+0000","last_peered":"2024-09-05T16:34:41.328510+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:15.917839+0000","last_became_peered":"2024-09-05T16:34:15.917839+0000","last_unstale":"2024-09-05T16:34:41.328510+0000","last_undegraded":"2024-09-05T16:34:15.904338+0000","last_fullsized":"2024-09-05T16:34:15.904018+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":1522,"log_dups_size":0,"ondisk_log_size":1522,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-05T16:34:44.765247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12458576,"num_objects":1521,"num_object_clones":0,"num_object_copies":6084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1521,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1521,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1521,"num_write_kb":12167,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1521}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"29'1532","reported_seq":1571,"reported_epoch":29,"state":"active+undersized+degraded","last_fresh":"2024-09-05T16:34:41.303385+0000","last_change":"2024-09-05T16:34:39.373308+0000","last_active":"2024-09-05T16:34:41.303385+0000","last_peered":"2024-09-05T16:34:41.303385+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:15.913175+0000","last_became_peered":"2024-09-05T16:34:15.913175+0000","last_unstale":"2024-09-05T16:34:41.303385+0000","last_undegraded":"2024-09-05T16:34:15.905345+0000","last_fullsized":"2024-09-05T16:34:15.905038+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":1532,"log_dups_size":0,"ondisk_log_size":1532,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12514464,"num_objects":1528,"num_object_clones":0,"num_object_copies":6112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1528,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1528,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1530,"num_write_kb":12236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1528}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"29'1498","reported_seq":1562,"reported_epoch":29,"state":"active+undersized+degraded","last_fresh":"2024-09-05T16:34:41.327948+0000","last_change":"2024-09-05T16:34:39.868440+0000","last_active":"2024-09-05T16:34:41.327948+0000","last_peered":"2024-09-05T16:34:41.327948+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:15.917968+0000","last_became_peered":"2024-09-05T16:34:15.917968+0000","last_unstale":"2024-09-05T16:34:41.327948+0000","last_undegraded":"2024-09-05T16:34:15.904385+0000","last_fullsized":"2024-09-05T16:34:15.904040+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":1498,"log_dups_size":0,"ondisk_log_size":1498,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12244128,"num_objects":1495,"num_object_clones":0,"num_object_copies":5980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1495,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1495,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1496,"num_write_kb":11965,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1495}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"29'1490","reported_seq":1540,"reported_epoch":29,"state":"active+undersized+degraded","last_fresh":"2024-09-05T16:34:41.276242+0000","last_change":"2024-09-05T16:34:40.418222+0000","last_active":"2024-09-05T16:34:41.276242+0000","last_peered":"2024-09-05T16:34:41.276242+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:15.910947+0000","last_became_peered":"2024-09-05T16:34:15.910947+0000","last_unstale":"2024-09-05T16:34:41.276242+0000","last_undegraded":"2024-09-05T16:34:15.905253+0000","last_fullsized":"2024-09-05T16:34:15.904912+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":1490,"log_dups_size":0,"ondisk_log_size":1490,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12162208,"num_objects":1485,"num_object_clones":0,"num_object_copies":5940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1485,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1485,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1488,"num_write_kb":11892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1485}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":195420352,"num_objects":23860,"num_object_clones":0,"num_object_copies":95440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23860,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23860,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23887,"num_write_kb":190978,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":293023744,"data_stored":293023744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":23914,"ondisk_log_size":23914,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084297,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":183784,"kb_used_data":93780,"kb_used_omap":4,"kb_used_meta":89979,"kb_avail":94188056,"statfs":{"total":96636764160,"available":96448569344,"internally_reserved":0,"allocated":96030720,"data_stored":95865263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":92139219},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,2,4,5],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640267,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":212992,"data_stored":90736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949672980,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":187240,"kb_used_data":93592,"kb_used_omap":1,"kb_used_meta":93630,"kb_avail":94184600,"statfs":{"total":96636764160,"available":96445030400,"internally_reserved":0,"allocated":95838208,"data_stored":95672751,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":95877573},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,13,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182411,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":187584,"kb_used_data":93488,"kb_used_omap":4,"kb_used_meta":94075,"kb_avail":94184256,"statfs":{"total":96636764160,"available":96444678144,"internally_reserved":0,"allocated":95731712,"data_stored":95566255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":96333517},"hb_peers":[2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,3,3,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":97640448,"data_stored":97640448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":40960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":97673216,"data_stored":97673216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":97669120,"data_stored":97669120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:34:44.309 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:34:44.493 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 22 20 26568 26548 9.42623 6.96094 0.0174555 0.0165659 2024-09-05T16:34:44.857 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:34:44.857 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:34:44.882 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":64,"stamp":"2024-09-05T16:34:44.794048+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":203555008,"num_objects":24853,"num_object_clones":0,"num_object_copies":99412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24853,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24853,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24880,"num_write_kb":198922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24907,"ondisk_log_size":24907,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":618392,"kb_used_data":291408,"kb_used_omap":11,"kb_used_meta":326900,"kb_avail":376868968,"statfs":{"total":386547056640,"available":385913823232,"internally_reserved":0,"allocated":298401792,"data_stored":297783165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11990,"internal_metadata":334745898},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,6,5,20,11,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":114032640,"num_objects":13920,"num_object_clones":0,"num_object_copies":55680,"num_objects_missing_on_primary":-21,"num_objects_missing":-21,"num_objects_degraded":13866,"num_objects_misplaced":0,"num_objects_unfound":-27,"num_objects_dirty":13920,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13920,"num_write_kb":111360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003330"},"pg_stats":[{"pgid":"1.f","version":"29'1666","reported_seq":1695,"reported_epoch":29,"state":"active+undersized+degraded","last_fresh":"2024-09-05T16:34:43.925349+0000","last_change":"2024-09-05T16:34:22.526044+0000","last_active":"2024-09-05T16:34:43.925349+0000","last_peered":"2024-09-05T16:34:43.925349+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:15.913051+0000","last_became_peered":"2024-09-05T16:34:15.913051+0000","last_unstale":"2024-09-05T16:34:43.925349+0000","last_undegraded":"2024-09-05T16:34:22.517240+0000","last_fullsized":"2024-09-05T16:34:15.905613+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":1666,"log_dups_size":0,"ondisk_log_size":1666,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T19:25:44.252388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13647872,"num_objects":1666,"num_object_clones":0,"num_object_copies":6664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1666,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1666,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1666,"num_write_kb":13328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2147483647,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1666}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"29'1523","reported_seq":1574,"reported_epoch":29,"state":"active+undersized+degraded","last_fresh":"2024-09-05T16:34:41.292140+0000","last_change":"2024-09-05T16:34:33.555748+0000","last_active":"2024-09-05T16:34:41.292140+0000","last_peered":"2024-09-05T16:34:41.292140+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:15.910788+0000","last_became_peered":"2024-09-05T16:34:15.910788+0000","last_unstale":"2024-09-05T16:34:41.292140+0000","last_undegraded":"2024-09-05T16:34:15.903622+0000","last_fullsized":"2024-09-05T16:34:15.903245+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":1523,"log_dups_size":0,"ondisk_log_size":1523,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-05T16:34:43.645989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12440736,"num_objects":1519,"num_object_clones":0,"num_object_copies":6076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1519,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1519,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1521,"num_write_kb":12164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1519}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"29'1491","reported_seq":1552,"reported_epoch":29,"state":"active+undersized+degraded","last_fresh":"2024-09-05T16:34:41.294991+0000","last_change":"2024-09-05T16:34:34.084629+0000","last_active":"2024-09-05T16:34:41.294991+0000","last_peered":"2024-09-05T16:34:41.294991+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:15.917951+0000","last_became_peered":"2024-09-05T16:34:15.917951+0000","last_unstale":"2024-09-05T16:34:41.294991+0000","last_undegraded":"2024-09-05T16:34:15.904811+0000","last_fullsized":"2024-09-05T16:34:15.904154+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":1491,"log_dups_size":0,"ondisk_log_size":1491,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-05T16:34:45.754218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12188240,"num_objects":1488,"num_object_clones":0,"num_object_copies":5952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1488,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1488,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1490,"num_write_kb":11910,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,2147483647,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1488}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"29'1510","reported_seq":1561,"reported_epoch":29,"state":"active+undersized+degraded","last_fresh":"2024-09-05T16:34:41.308209+0000","last_change":"2024-09-05T16:34:34.682384+0000","last_active":"2024-09-05T16:34:41.308209+0000","last_peered":"2024-09-05T16:34:41.308209+0000","last_clean":"2024-09-05T16:33:53.148717+0000","last_became_active":"2024-09-05T16:34:15.912196+0000","last_became_peered":"2024-09-05T16:34:15.912196+0000","last_unstale":"2024-09-05T16:34:41.308209+0000","last_undegraded":"2024-09-05T16:34:15.905010+0000","last_fullsized":"2024-09-05T16:34:15.904610+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":1510,"log_dups_size":0,"ondisk_log_size":1510,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-05T16:34:45.721510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12334240,"num_objects":1506,"num_object_clones":0,"num_object_copies":6024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1506,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1506,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1508,"num_write_kb":12053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1506}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"29'1568","reported_seq":1608,"reported_epoch":29,"state":"active+undersized+degraded","last_fresh":"2024-09-05T16:34:43.909970+0000","last_change":"2024-09-05T16:34:35.192823+0000","last_active":"2024-09-05T16:34:43.909970+0000","last_peered":"2024-09-05T16:34:43.909970+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:15.910648+0000","last_became_peered":"2024-09-05T16:34:15.910648+0000","last_unstale":"2024-09-05T16:34:43.909970+0000","last_undegraded":"2024-09-05T16:34:15.904665+0000","last_fullsized":"2024-09-05T16:34:15.903277+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":1568,"log_dups_size":0,"ondisk_log_size":1568,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-05T16:34:45.802206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12817568,"num_objects":1565,"num_object_clones":0,"num_object_copies":6260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1565,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1565,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1566,"num_write_kb":12525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2147483647,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1565}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"29'1600","reported_seq":1629,"reported_epoch":29,"state":"active+undersized+degraded","last_fresh":"2024-09-05T16:34:43.995586+0000","last_change":"2024-09-05T16:34:22.533654+0000","last_active":"2024-09-05T16:34:43.995586+0000","last_peered":"2024-09-05T16:34:43.995586+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:15.913477+0000","last_became_peered":"2024-09-05T16:34:15.913477+0000","last_unstale":"2024-09-05T16:34:43.995586+0000","last_undegraded":"2024-09-05T16:34:22.533443+0000","last_fullsized":"2024-09-05T16:34:15.906631+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":1600,"log_dups_size":0,"ondisk_log_size":1600,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T01:56:46.762011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13107200,"num_objects":1600,"num_object_clones":0,"num_object_copies":6400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1600,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1600,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1600,"num_write_kb":12800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,2147483647],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1600}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"29'1379","reported_seq":1430,"reported_epoch":29,"state":"active+undersized+degraded","last_fresh":"2024-09-05T16:34:41.223931+0000","last_change":"2024-09-05T16:34:35.683668+0000","last_active":"2024-09-05T16:34:41.223931+0000","last_peered":"2024-09-05T16:34:41.223931+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:15.913479+0000","last_became_peered":"2024-09-05T16:34:15.913479+0000","last_unstale":"2024-09-05T16:34:41.223931+0000","last_undegraded":"2024-09-05T16:34:15.905955+0000","last_fullsized":"2024-09-05T16:34:15.905617+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":1379,"log_dups_size":0,"ondisk_log_size":1379,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-05T16:34:41.602576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11244704,"num_objects":1373,"num_object_clones":0,"num_object_copies":5492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1373,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1377,"num_write_kb":10996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2147483647,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1373}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"29'1463","reported_seq":1514,"reported_epoch":29,"state":"active+undersized+degraded","last_fresh":"2024-09-05T16:34:41.294121+0000","last_change":"2024-09-05T16:34:36.231100+0000","last_active":"2024-09-05T16:34:41.294121+0000","last_peered":"2024-09-05T16:34:41.294121+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:15.910570+0000","last_became_peered":"2024-09-05T16:34:15.910570+0000","last_unstale":"2024-09-05T16:34:41.294121+0000","last_undegraded":"2024-09-05T16:34:15.904563+0000","last_fullsized":"2024-09-05T16:34:15.903952+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":1463,"log_dups_size":0,"ondisk_log_size":1463,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-05T16:34:42.604538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11941024,"num_objects":1458,"num_object_clones":0,"num_object_copies":5832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1458,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1458,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1461,"num_write_kb":11676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1458}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"29'1672","reported_seq":1714,"reported_epoch":29,"state":"active+undersized+degraded","last_fresh":"2024-09-05T16:34:43.974765+0000","last_change":"2024-09-05T16:34:36.789538+0000","last_active":"2024-09-05T16:34:43.974765+0000","last_peered":"2024-09-05T16:34:43.974765+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:15.914297+0000","last_became_peered":"2024-09-05T16:34:15.914297+0000","last_unstale":"2024-09-05T16:34:43.974765+0000","last_undegraded":"2024-09-05T16:34:15.906625+0000","last_fullsized":"2024-09-05T16:34:15.906328+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":1672,"log_dups_size":0,"ondisk_log_size":1672,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-05T16:34:47.825254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13625664,"num_objects":1664,"num_object_clones":0,"num_object_copies":6656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1664,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1664,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1669,"num_write_kb":13329,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,2147483647],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1664}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"29'1507","reported_seq":1570,"reported_epoch":29,"state":"active+undersized+degraded","last_fresh":"2024-09-05T16:34:41.286043+0000","last_change":"2024-09-05T16:34:37.301824+0000","last_active":"2024-09-05T16:34:41.286043+0000","last_peered":"2024-09-05T16:34:41.286043+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:15.918296+0000","last_became_peered":"2024-09-05T16:34:15.918296+0000","last_unstale":"2024-09-05T16:34:41.286043+0000","last_undegraded":"2024-09-05T16:34:15.905187+0000","last_fullsized":"2024-09-05T16:34:15.904834+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":1507,"log_dups_size":0,"ondisk_log_size":1507,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-05T16:34:42.778601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12326048,"num_objects":1505,"num_object_clones":0,"num_object_copies":6020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1505,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1505,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1505,"num_write_kb":12038,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,2147483647],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1505}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"29'1708","reported_seq":1750,"reported_epoch":29,"state":"active+undersized+degraded","last_fresh":"2024-09-05T16:34:43.917665+0000","last_change":"2024-09-05T16:34:37.985329+0000","last_active":"2024-09-05T16:34:43.917665+0000","last_peered":"2024-09-05T16:34:43.917665+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:15.914271+0000","last_became_peered":"2024-09-05T16:34:15.914271+0000","last_unstale":"2024-09-05T16:34:43.917665+0000","last_undegraded":"2024-09-05T16:34:15.905903+0000","last_fullsized":"2024-09-05T16:34:15.905357+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":1708,"log_dups_size":0,"ondisk_log_size":1708,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-05T16:34:48.861780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13946608,"num_objects":1703,"num_object_clones":0,"num_object_copies":6812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1703,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1703,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1705,"num_write_kb":13628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2147483647,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1703}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"29'1620","reported_seq":1643,"reported_epoch":29,"state":"active+undersized+degraded","last_fresh":"2024-09-05T16:34:43.950850+0000","last_change":"2024-09-05T16:34:38.396192+0000","last_active":"2024-09-05T16:34:43.950850+0000","last_peered":"2024-09-05T16:34:43.950850+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:15.913627+0000","last_became_peered":"2024-09-05T16:34:15.913627+0000","last_unstale":"2024-09-05T16:34:43.950850+0000","last_undegraded":"2024-09-05T16:34:15.906286+0000","last_fullsized":"2024-09-05T16:34:15.906044+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":1620,"log_dups_size":0,"ondisk_log_size":1620,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-05T16:34:44.816449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13261392,"num_objects":1619,"num_object_clones":0,"num_object_copies":6476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1619,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1619,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1619,"num_write_kb":12951,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1619}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"29'1522","reported_seq":1581,"reported_epoch":29,"state":"active+undersized+degraded","last_fresh":"2024-09-05T16:34:41.328510+0000","last_change":"2024-09-05T16:34:38.929130+0000","last_active":"2024-09-05T16:34:41.328510+0000","last_peered":"2024-09-05T16:34:41.328510+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:15.917839+0000","last_became_peered":"2024-09-05T16:34:15.917839+0000","last_unstale":"2024-09-05T16:34:41.328510+0000","last_undegraded":"2024-09-05T16:34:15.904338+0000","last_fullsized":"2024-09-05T16:34:15.904018+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":1522,"log_dups_size":0,"ondisk_log_size":1522,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-05T16:34:44.765247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12458576,"num_objects":1521,"num_object_clones":0,"num_object_copies":6084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1521,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1521,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1521,"num_write_kb":12167,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1521}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"29'1690","reported_seq":1728,"reported_epoch":29,"state":"active+undersized+degraded","last_fresh":"2024-09-05T16:34:43.981510+0000","last_change":"2024-09-05T16:34:39.373308+0000","last_active":"2024-09-05T16:34:43.981510+0000","last_peered":"2024-09-05T16:34:43.981510+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:15.913175+0000","last_became_peered":"2024-09-05T16:34:15.913175+0000","last_unstale":"2024-09-05T16:34:43.981510+0000","last_undegraded":"2024-09-05T16:34:15.905345+0000","last_fullsized":"2024-09-05T16:34:15.905038+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":1690,"log_dups_size":0,"ondisk_log_size":1690,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-05T16:34:46.841863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13808800,"num_objects":1686,"num_object_clones":0,"num_object_copies":6744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1686,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1686,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1688,"num_write_kb":13500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1686}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"29'1498","reported_seq":1562,"reported_epoch":29,"state":"active+undersized+degraded","last_fresh":"2024-09-05T16:34:41.327948+0000","last_change":"2024-09-05T16:34:39.868440+0000","last_active":"2024-09-05T16:34:41.327948+0000","last_peered":"2024-09-05T16:34:41.327948+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:15.917968+0000","last_became_peered":"2024-09-05T16:34:15.917968+0000","last_unstale":"2024-09-05T16:34:41.327948+0000","last_undegraded":"2024-09-05T16:34:15.904385+0000","last_fullsized":"2024-09-05T16:34:15.904040+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":1498,"log_dups_size":0,"ondisk_log_size":1498,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12244128,"num_objects":1495,"num_object_clones":0,"num_object_copies":5980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1495,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1495,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1496,"num_write_kb":11965,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1495}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"29'1490","reported_seq":1540,"reported_epoch":29,"state":"active+undersized+degraded","last_fresh":"2024-09-05T16:34:41.276242+0000","last_change":"2024-09-05T16:34:40.418222+0000","last_active":"2024-09-05T16:34:41.276242+0000","last_peered":"2024-09-05T16:34:41.276242+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:15.910947+0000","last_became_peered":"2024-09-05T16:34:15.910947+0000","last_unstale":"2024-09-05T16:34:41.276242+0000","last_undegraded":"2024-09-05T16:34:15.905253+0000","last_fullsized":"2024-09-05T16:34:15.904912+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":1490,"log_dups_size":0,"ondisk_log_size":1490,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12162208,"num_objects":1485,"num_object_clones":0,"num_object_copies":5940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1485,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1485,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1488,"num_write_kb":11892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1485}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":203555008,"num_objects":24853,"num_object_clones":0,"num_object_copies":99412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24853,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24853,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24880,"num_write_kb":198922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":302231552,"data_stored":302231552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24907,"ondisk_log_size":24907,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084298,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":216328,"kb_used_data":104120,"kb_used_omap":4,"kb_used_meta":112187,"kb_avail":94155512,"statfs":{"total":96636764160,"available":96415244288,"internally_reserved":0,"allocated":106618880,"data_stored":106453423,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":114880211},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,4,2,4,6,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640267,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":212992,"data_stored":90736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949672980,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":187240,"kb_used_data":93592,"kb_used_omap":1,"kb_used_meta":93630,"kb_avail":94184600,"statfs":{"total":96636764160,"available":96445030400,"internally_reserved":0,"allocated":95838208,"data_stored":95672751,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":95877573},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,13,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182411,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":187584,"kb_used_data":93488,"kb_used_omap":4,"kb_used_meta":94075,"kb_avail":94184256,"statfs":{"total":96636764160,"available":96444678144,"internally_reserved":0,"allocated":95731712,"data_stored":95566255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":96333517},"hb_peers":[2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,3,3,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":97640448,"data_stored":97640448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":40960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":97673216,"data_stored":97673216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":106876928,"data_stored":106876928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:34:44.883 INFO:tasks.ec_lost_unfound.ceph_manager:recovered! 2024-09-05T16:34:44.884 INFO:tasks.util.rados:rados -p unique_pool_0 get new_1 - 2024-09-05T16:34:44.884 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get new_1 - 2024-09-05T16:34:44.988 INFO:teuthology.orchestra.run.smithi087.stderr:error getting unique_pool_0/new_1: (2) No such file or directory 2024-09-05T16:34:44.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-05T16:34:45.000 INFO:tasks.util.rados:rados -p unique_pool_0 get existed_1 - 2024-09-05T16:34:45.000 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existed_1 - 2024-09-05T16:34:45.095 INFO:teuthology.orchestra.run.smithi087.stderr:error getting unique_pool_0/existed_1: (2) No such file or directory 2024-09-05T16:34:45.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-05T16:34:45.107 INFO:tasks.util.rados:rados -p unique_pool_0 get existing_1 - 2024-09-05T16:34:45.107 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existing_1 - 2024-09-05T16:34:45.197 INFO:teuthology.orchestra.run.smithi087.stderr:error getting unique_pool_0/existing_1: (2) No such file or directory 2024-09-05T16:34:45.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-05T16:34:45.201 INFO:tasks.util.rados:rados -p unique_pool_0 get new_2 - 2024-09-05T16:34:45.201 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get new_2 - 2024-09-05T16:34:45.293 INFO:teuthology.orchestra.run.smithi087.stderr:error getting unique_pool_0/new_2: (2) No such file or directory 2024-09-05T16:34:45.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-05T16:34:45.303 INFO:tasks.util.rados:rados -p unique_pool_0 get existed_2 - 2024-09-05T16:34:45.303 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existed_2 - 2024-09-05T16:34:45.424 INFO:teuthology.orchestra.run.smithi087.stderr:error getting unique_pool_0/existed_2: (2) No such file or directory 2024-09-05T16:34:45.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-05T16:34:45.433 INFO:tasks.util.rados:rados -p unique_pool_0 get existing_2 - 2024-09-05T16:34:45.433 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existing_2 - 2024-09-05T16:34:45.493 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 23 20 27501 27481 9.33329 7.28906 0.0241574 0.0167298 2024-09-05T16:34:45.537 INFO:teuthology.orchestra.run.smithi087.stderr:error getting unique_pool_0/existing_2: (2) No such file or directory 2024-09-05T16:34:45.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-05T16:34:45.551 INFO:tasks.util.rados:rados -p unique_pool_0 get new_3 - 2024-09-05T16:34:45.551 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get new_3 - 2024-09-05T16:34:45.646 INFO:teuthology.orchestra.run.smithi087.stderr:error getting unique_pool_0/new_3: (2) No such file or directory 2024-09-05T16:34:45.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-05T16:34:45.656 INFO:tasks.util.rados:rados -p unique_pool_0 get existed_3 - 2024-09-05T16:34:45.656 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existed_3 - 2024-09-05T16:34:45.765 INFO:teuthology.orchestra.run.smithi087.stderr:error getting unique_pool_0/existed_3: (2) No such file or directory 2024-09-05T16:34:45.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-05T16:34:45.774 INFO:tasks.util.rados:rados -p unique_pool_0 get existing_3 - 2024-09-05T16:34:45.774 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existing_3 - 2024-09-05T16:34:45.875 INFO:teuthology.orchestra.run.smithi087.stderr:error getting unique_pool_0/existing_3: (2) No such file or directory 2024-09-05T16:34:45.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-05T16:34:45.887 INFO:tasks.util.rados:rados -p unique_pool_0 get new_4 - 2024-09-05T16:34:45.887 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get new_4 - 2024-09-05T16:34:45.999 INFO:teuthology.orchestra.run.smithi087.stderr:error getting unique_pool_0/new_4: (2) No such file or directory 2024-09-05T16:34:46.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-05T16:34:46.006 INFO:tasks.util.rados:rados -p unique_pool_0 get existed_4 - 2024-09-05T16:34:46.006 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existed_4 - 2024-09-05T16:34:46.114 INFO:teuthology.orchestra.run.smithi087.stderr:error getting unique_pool_0/existed_4: (2) No such file or directory 2024-09-05T16:34:46.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-05T16:34:46.122 INFO:tasks.util.rados:rados -p unique_pool_0 get existing_4 - 2024-09-05T16:34:46.123 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existing_4 - 2024-09-05T16:34:46.226 INFO:teuthology.orchestra.run.smithi087.stderr:error getting unique_pool_0/existing_4: (2) No such file or directory 2024-09-05T16:34:46.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-05T16:34:46.235 INFO:tasks.util.rados:rados -p unique_pool_0 get new_5 - 2024-09-05T16:34:46.235 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get new_5 - 2024-09-05T16:34:46.339 INFO:teuthology.orchestra.run.smithi087.stderr:error getting unique_pool_0/new_5: (2) No such file or directory 2024-09-05T16:34:46.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-05T16:34:46.351 INFO:tasks.util.rados:rados -p unique_pool_0 get existed_5 - 2024-09-05T16:34:46.351 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existed_5 - 2024-09-05T16:34:46.446 INFO:teuthology.orchestra.run.smithi087.stderr:error getting unique_pool_0/existed_5: (2) No such file or directory 2024-09-05T16:34:46.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-05T16:34:46.455 INFO:tasks.util.rados:rados -p unique_pool_0 get existing_5 - 2024-09-05T16:34:46.456 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existing_5 - 2024-09-05T16:34:46.493 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 24 20 28406 28386 9.23898 7.07031 0.0204088 0.0168996 2024-09-05T16:34:46.563 INFO:teuthology.orchestra.run.smithi087.stderr:error getting unique_pool_0/existing_5: (2) No such file or directory 2024-09-05T16:34:46.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-05T16:34:46.576 INFO:tasks.util.rados:rados -p unique_pool_0 get new_6 - 2024-09-05T16:34:46.576 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get new_6 - 2024-09-05T16:34:46.713 INFO:teuthology.orchestra.run.smithi087.stderr:error getting unique_pool_0/new_6: (2) No such file or directory 2024-09-05T16:34:46.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-05T16:34:46.726 INFO:tasks.util.rados:rados -p unique_pool_0 get existed_6 - 2024-09-05T16:34:46.726 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existed_6 - 2024-09-05T16:34:46.816 INFO:teuthology.orchestra.run.smithi087.stderr:error getting unique_pool_0/existed_6: (2) No such file or directory 2024-09-05T16:34:46.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-05T16:34:46.824 INFO:tasks.util.rados:rados -p unique_pool_0 get existing_6 - 2024-09-05T16:34:46.824 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existing_6 - 2024-09-05T16:34:46.920 INFO:teuthology.orchestra.run.smithi087.stderr:error getting unique_pool_0/existing_6: (2) No such file or directory 2024-09-05T16:34:46.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-05T16:34:46.930 INFO:tasks.util.rados:rados -p unique_pool_0 get new_7 - 2024-09-05T16:34:46.930 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get new_7 - 2024-09-05T16:34:47.057 INFO:teuthology.orchestra.run.smithi087.stderr:error getting unique_pool_0/new_7: (2) No such file or directory 2024-09-05T16:34:47.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-05T16:34:47.069 INFO:tasks.util.rados:rados -p unique_pool_0 get existed_7 - 2024-09-05T16:34:47.069 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existed_7 - 2024-09-05T16:34:47.175 INFO:teuthology.orchestra.run.smithi087.stderr:error getting unique_pool_0/existed_7: (2) No such file or directory 2024-09-05T16:34:47.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-05T16:34:47.184 INFO:tasks.util.rados:rados -p unique_pool_0 get existing_7 - 2024-09-05T16:34:47.184 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existing_7 - 2024-09-05T16:34:47.289 INFO:teuthology.orchestra.run.smithi087.stderr:error getting unique_pool_0/existing_7: (2) No such file or directory 2024-09-05T16:34:47.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-05T16:34:47.298 INFO:tasks.util.rados:rados -p unique_pool_0 get new_8 - 2024-09-05T16:34:47.298 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get new_8 - 2024-09-05T16:34:47.410 INFO:teuthology.orchestra.run.smithi087.stderr:error getting unique_pool_0/new_8: (2) No such file or directory 2024-09-05T16:34:47.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-05T16:34:47.419 INFO:tasks.util.rados:rados -p unique_pool_0 get existed_8 - 2024-09-05T16:34:47.419 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existed_8 - 2024-09-05T16:34:47.493 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 25 20 29315 29295 9.15347 7.10156 0.0279484 0.0170621 2024-09-05T16:34:47.538 INFO:teuthology.orchestra.run.smithi087.stderr:error getting unique_pool_0/existed_8: (2) No such file or directory 2024-09-05T16:34:47.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-05T16:34:47.549 INFO:tasks.util.rados:rados -p unique_pool_0 get existing_8 - 2024-09-05T16:34:47.549 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existing_8 - 2024-09-05T16:34:47.649 INFO:teuthology.orchestra.run.smithi087.stderr:error getting unique_pool_0/existing_8: (2) No such file or directory 2024-09-05T16:34:47.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-05T16:34:47.658 INFO:tasks.util.rados:rados -p unique_pool_0 get new_9 - 2024-09-05T16:34:47.658 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get new_9 - 2024-09-05T16:34:47.754 INFO:teuthology.orchestra.run.smithi087.stderr:error getting unique_pool_0/new_9: (2) No such file or directory 2024-09-05T16:34:47.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-05T16:34:47.760 INFO:tasks.util.rados:rados -p unique_pool_0 get existed_9 - 2024-09-05T16:34:47.760 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existed_9 - 2024-09-05T16:34:47.861 INFO:teuthology.orchestra.run.smithi087.stderr:error getting unique_pool_0/existed_9: (2) No such file or directory 2024-09-05T16:34:47.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-05T16:34:47.870 INFO:tasks.util.rados:rados -p unique_pool_0 get existing_9 - 2024-09-05T16:34:47.870 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existing_9 - 2024-09-05T16:34:47.972 INFO:teuthology.orchestra.run.smithi087.stderr:error getting unique_pool_0/existing_9: (2) No such file or directory 2024-09-05T16:34:47.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-05T16:34:47.986 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-05T16:34:47.987 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-09-05T16:34:47.989 INFO:tasks.ceph.osd.1:Started 2024-09-05T16:34:47.989 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-05T16:34:48.174 INFO:teuthology.orchestra.run.smithi087.stderr:no valid command found; 10 closest matches: 2024-09-05T16:34:48.174 INFO:teuthology.orchestra.run.smithi087.stderr:0 2024-09-05T16:34:48.174 INFO:teuthology.orchestra.run.smithi087.stderr:1 2024-09-05T16:34:48.175 INFO:teuthology.orchestra.run.smithi087.stderr:2 2024-09-05T16:34:48.175 INFO:teuthology.orchestra.run.smithi087.stderr:abort 2024-09-05T16:34:48.175 INFO:teuthology.orchestra.run.smithi087.stderr:assert 2024-09-05T16:34:48.175 INFO:teuthology.orchestra.run.smithi087.stderr:config diff 2024-09-05T16:34:48.175 INFO:teuthology.orchestra.run.smithi087.stderr:config diff get 2024-09-05T16:34:48.175 INFO:teuthology.orchestra.run.smithi087.stderr:config get 2024-09-05T16:34:48.175 INFO:teuthology.orchestra.run.smithi087.stderr:config help [] 2024-09-05T16:34:48.175 INFO:teuthology.orchestra.run.smithi087.stderr:config set ... 2024-09-05T16:34:48.175 INFO:teuthology.orchestra.run.smithi087.stderr:admin_socket: invalid command 2024-09-05T16:34:48.179 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-05T16:34:48.179 INFO:tasks.ec_lost_unfound.ceph_manager:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-09-05T16:34:48.493 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 26 20 30236 30216 9.07814 7.19531 0.0133582 0.0172027 2024-09-05T16:34:49.493 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 27 20 31243 31223 9.03327 7.86719 0.0233677 0.0172875 2024-09-05T16:34:49.707 INFO:tasks.ceph.osd.1.smithi087.stderr:2024-09-05T16:34:49.713+0000 7fc52ce18640 -1 Falling back to public interface 2024-09-05T16:34:50.493 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 28 20 32240 32220 8.98881 7.78906 0.0199142 0.0173729 2024-09-05T16:34:51.493 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 29 20 33233 33213 8.94634 7.75781 0.0206725 0.0174568 2024-09-05T16:34:52.493 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 30 20 34178 34158 8.89421 7.38281 0.0175735 0.0175571 2024-09-05T16:34:53.180 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-05T16:34:53.359 INFO:teuthology.orchestra.run.smithi087.stderr:no valid command found; 10 closest matches: 2024-09-05T16:34:53.359 INFO:teuthology.orchestra.run.smithi087.stderr:0 2024-09-05T16:34:53.359 INFO:teuthology.orchestra.run.smithi087.stderr:1 2024-09-05T16:34:53.359 INFO:teuthology.orchestra.run.smithi087.stderr:2 2024-09-05T16:34:53.359 INFO:teuthology.orchestra.run.smithi087.stderr:abort 2024-09-05T16:34:53.359 INFO:teuthology.orchestra.run.smithi087.stderr:assert 2024-09-05T16:34:53.359 INFO:teuthology.orchestra.run.smithi087.stderr:bluestore allocator dump block 2024-09-05T16:34:53.359 INFO:teuthology.orchestra.run.smithi087.stderr:bluestore allocator fragmentation block 2024-09-05T16:34:53.359 INFO:teuthology.orchestra.run.smithi087.stderr:bluestore allocator fragmentation histogram block [] [] 2024-09-05T16:34:53.360 INFO:teuthology.orchestra.run.smithi087.stderr:bluestore allocator score block 2024-09-05T16:34:53.360 INFO:teuthology.orchestra.run.smithi087.stderr:config diff 2024-09-05T16:34:53.360 INFO:teuthology.orchestra.run.smithi087.stderr:admin_socket: invalid command 2024-09-05T16:34:53.362 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-05T16:34:53.363 INFO:tasks.ec_lost_unfound.ceph_manager:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-09-05T16:34:53.493 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 31 20 35058 35038 8.82906 6.875 0.0179363 0.0176878 2024-09-05T16:34:53.591 INFO:tasks.ceph.osd.1.smithi087.stderr:2024-09-05T16:34:53.597+0000 7fc52ce18640 -1 osd.1 21 log_to_monitors true 2024-09-05T16:34:53.896 INFO:tasks.ceph.osd.1.smithi087.stderr:2024-09-05T16:34:53.901+0000 7fc5245a1640 -1 osd.1 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-05T16:34:54.493 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 32 20 35973 35953 8.77652 7.14844 0.0173756 0.0177927 2024-09-05T16:34:55.494 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 33 20 36182 36162 8.56004 1.63281 0.0086828 0.0178142 2024-09-05T16:34:56.494 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 34 20 36182 36162 8.30826 0 - 0.0178142 2024-09-05T16:34:57.494 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 35 20 36466 36446 8.13427 1.10938 0.0437557 0.0191904 2024-09-05T16:34:58.364 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-05T16:34:58.494 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 36 20 36903 36883 8.00315 3.41406 0.0472765 0.0194963 2024-09-05T16:34:58.578 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-09-05T16:34:58.578 INFO:teuthology.orchestra.run.smithi087.stdout: "ops": [ 2024-09-05T16:34:58.578 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.578 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "MOSDECSubOpWrite(1.3s0 33/32 ECSubWrite(tid=16201, reqid=client.14960.0:36914, at_version=0'0, trim_to=0'0, roll_forward_to=33'2301))", 2024-09-05T16:34:58.578 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-09-05T16:34:58.580375+0000", 2024-09-05T16:34:58.578 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 0.0033796030000000001, 2024-09-05T16:34:58.578 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0034002390000000002, 2024-09-05T16:34:58.579 INFO:teuthology.orchestra.run.smithi087.stdout: "continuous": false, 2024-09-05T16:34:58.579 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-09-05T16:34:58.579 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "reached pg", 2024-09-05T16:34:58.579 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-09-05T16:34:58.579 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.579 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-09-05T16:34:58.579 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-09-05T16:34:58.580375+0000", 2024-09-05T16:34:58.579 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-09-05T16:34:58.579 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:58.579 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.579 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-09-05T16:34:58.579 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-09-05T16:34:58.580375+0000", 2024-09-05T16:34:58.580 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-09-05T16:34:58.580 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:58.580 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.580 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-09-05T16:34:58.580 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-09-05T16:34:58.580375+0000", 2024-09-05T16:34:58.580 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.3999999999999998e-07 2024-09-05T16:34:58.580 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:58.580 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.580 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-09-05T16:34:58.580 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-09-05T16:34:58.580382+0000", 2024-09-05T16:34:58.580 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.8199999999999999e-06 2024-09-05T16:34:58.580 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:58.580 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.581 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-09-05T16:34:58.581 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-09-05T16:34:58.580387+0000", 2024-09-05T16:34:58.581 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.8230000000000002e-06 2024-09-05T16:34:58.581 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:58.581 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.581 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-09-05T16:34:58.581 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-09-05T16:34:58.580392+0000", 2024-09-05T16:34:58.581 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.7260000000000002e-06 2024-09-05T16:34:58.581 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:58.581 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.581 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-09-05T16:34:58.581 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-09-05T16:34:58.581534+0000", 2024-09-05T16:34:58.582 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0011419749999999999 2024-09-05T16:34:58.582 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:58.582 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.582 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_started", 2024-09-05T16:34:58.582 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-09-05T16:34:58.581543+0000", 2024-09-05T16:34:58.582 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.4020000000000008e-06 2024-09-05T16:34:58.582 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-09-05T16:34:58.582 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-09-05T16:34:58.582 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-09-05T16:34:58.582 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:58.582 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.582 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "MOSDECSubOpWrite(1.8s0 33/32 ECSubWrite(tid=13138, reqid=client.14960.0:36938, at_version=33'2340, trim_to=0'0, roll_forward_to=33'2339))", 2024-09-05T16:34:58.582 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-09-05T16:34:58.580433+0000", 2024-09-05T16:34:58.582 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 0.0033212469999999998, 2024-09-05T16:34:58.583 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0033951599999999999, 2024-09-05T16:34:58.583 INFO:teuthology.orchestra.run.smithi087.stdout: "continuous": false, 2024-09-05T16:34:58.583 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-09-05T16:34:58.583 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "queued for pg", 2024-09-05T16:34:58.583 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-09-05T16:34:58.583 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.583 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-09-05T16:34:58.583 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-09-05T16:34:58.580433+0000", 2024-09-05T16:34:58.583 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-09-05T16:34:58.583 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:58.583 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.583 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-09-05T16:34:58.583 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-09-05T16:34:58.580433+0000", 2024-09-05T16:34:58.584 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-09-05T16:34:58.584 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:58.584 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.584 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-09-05T16:34:58.584 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-09-05T16:34:58.580434+0000", 2024-09-05T16:34:58.584 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.1899999999999998e-07 2024-09-05T16:34:58.584 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:58.584 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.584 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-09-05T16:34:58.584 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-09-05T16:34:58.580451+0000", 2024-09-05T16:34:58.584 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.7526000000000001e-05 2024-09-05T16:34:58.584 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:58.585 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.585 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-09-05T16:34:58.585 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-09-05T16:34:58.580459+0000", 2024-09-05T16:34:58.585 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.6359999999999994e-06 2024-09-05T16:34:58.585 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:58.585 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.585 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-09-05T16:34:58.585 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-09-05T16:34:58.580464+0000", 2024-09-05T16:34:58.585 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.4670000000000003e-06 2024-09-05T16:34:58.585 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-09-05T16:34:58.585 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-09-05T16:34:58.585 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-09-05T16:34:58.585 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:58.585 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.586 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "MOSDECSubOpWrite(1.ds1 33/32 ECSubWrite(tid=13137, reqid=client.14960.0:36934, at_version=33'2310, trim_to=0'0, roll_forward_to=33'2307))", 2024-09-05T16:34:58.586 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-09-05T16:34:58.580474+0000", 2024-09-05T16:34:58.586 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 0.0032807090000000001, 2024-09-05T16:34:58.586 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0033918279999999999, 2024-09-05T16:34:58.586 INFO:teuthology.orchestra.run.smithi087.stdout: "continuous": false, 2024-09-05T16:34:58.586 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-09-05T16:34:58.586 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "queued for pg", 2024-09-05T16:34:58.586 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-09-05T16:34:58.586 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.586 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-09-05T16:34:58.586 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-09-05T16:34:58.580474+0000", 2024-09-05T16:34:58.586 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-09-05T16:34:58.586 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:58.587 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.587 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-09-05T16:34:58.587 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-09-05T16:34:58.580474+0000", 2024-09-05T16:34:58.587 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-09-05T16:34:58.587 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:58.587 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.587 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-09-05T16:34:58.587 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-09-05T16:34:58.580474+0000", 2024-09-05T16:34:58.587 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.92e-07 2024-09-05T16:34:58.587 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:58.587 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.587 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-09-05T16:34:58.587 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-09-05T16:34:58.580489+0000", 2024-09-05T16:34:58.587 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.466e-05 2024-09-05T16:34:58.588 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:58.588 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.588 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-09-05T16:34:58.588 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-09-05T16:34:58.580494+0000", 2024-09-05T16:34:58.588 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.3040000000000004e-06 2024-09-05T16:34:58.588 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:58.588 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.588 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-09-05T16:34:58.588 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-09-05T16:34:58.580499+0000", 2024-09-05T16:34:58.588 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.8890000000000001e-06 2024-09-05T16:34:58.588 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-09-05T16:34:58.588 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-09-05T16:34:58.588 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-09-05T16:34:58.589 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:58.589 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.589 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "MOSDECSubOpWrite(1.4s3 33/32 ECSubWrite(tid=22584, reqid=client.14960.0:36917, at_version=0'0, trim_to=0'0, roll_forward_to=33'2308))", 2024-09-05T16:34:58.589 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-09-05T16:34:58.580511+0000", 2024-09-05T16:34:58.589 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 0.0032435459999999999, 2024-09-05T16:34:58.589 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0033888170000000001, 2024-09-05T16:34:58.589 INFO:teuthology.orchestra.run.smithi087.stdout: "continuous": false, 2024-09-05T16:34:58.589 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-09-05T16:34:58.589 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "reached pg", 2024-09-05T16:34:58.589 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-09-05T16:34:58.589 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.589 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-09-05T16:34:58.589 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-09-05T16:34:58.580511+0000", 2024-09-05T16:34:58.589 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-09-05T16:34:58.590 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:58.590 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.590 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-09-05T16:34:58.590 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-09-05T16:34:58.580511+0000", 2024-09-05T16:34:58.590 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-09-05T16:34:58.590 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:58.590 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.590 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-09-05T16:34:58.590 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-09-05T16:34:58.580511+0000", 2024-09-05T16:34:58.591 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.0899999999999997e-07 2024-09-05T16:34:58.591 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:58.591 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.591 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-09-05T16:34:58.591 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-09-05T16:34:58.580519+0000", 2024-09-05T16:34:58.591 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.0979999999999996e-06 2024-09-05T16:34:58.591 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:58.591 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.591 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-09-05T16:34:58.591 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-09-05T16:34:58.580524+0000", 2024-09-05T16:34:58.591 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.4310000000000002e-06 2024-09-05T16:34:58.591 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:58.591 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.591 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-09-05T16:34:58.592 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-09-05T16:34:58.580529+0000", 2024-09-05T16:34:58.592 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.8119999999999998e-06 2024-09-05T16:34:58.592 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:58.592 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.592 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-09-05T16:34:58.592 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-09-05T16:34:58.580970+0000", 2024-09-05T16:34:58.592 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000440801 2024-09-05T16:34:58.592 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:58.592 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.592 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_started", 2024-09-05T16:34:58.592 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-09-05T16:34:58.580979+0000", 2024-09-05T16:34:58.592 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.0589999999999992e-06 2024-09-05T16:34:58.592 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-09-05T16:34:58.592 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-09-05T16:34:58.593 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-09-05T16:34:58.593 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:58.593 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.593 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "MOSDECSubOpWrite(1.1s1 33/32 ECSubWrite(tid=22585, reqid=client.14960.0:36933, at_version=33'2215, trim_to=0'0, roll_forward_to=33'2214))", 2024-09-05T16:34:58.593 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-09-05T16:34:58.580540+0000", 2024-09-05T16:34:58.593 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 0.0032143839999999998, 2024-09-05T16:34:58.593 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0034042629999999998, 2024-09-05T16:34:58.593 INFO:teuthology.orchestra.run.smithi087.stdout: "continuous": false, 2024-09-05T16:34:58.593 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-09-05T16:34:58.593 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "queued for pg", 2024-09-05T16:34:58.593 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-09-05T16:34:58.593 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.593 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-09-05T16:34:58.594 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-09-05T16:34:58.580540+0000", 2024-09-05T16:34:58.594 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-09-05T16:34:58.594 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:58.594 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.594 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-09-05T16:34:58.594 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-09-05T16:34:58.580540+0000", 2024-09-05T16:34:58.594 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-09-05T16:34:58.594 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:58.594 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.594 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-09-05T16:34:58.594 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-09-05T16:34:58.580540+0000", 2024-09-05T16:34:58.594 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.7599999999999999e-07 2024-09-05T16:34:58.594 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:58.594 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.595 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-09-05T16:34:58.595 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-09-05T16:34:58.580554+0000", 2024-09-05T16:34:58.595 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.3540000000000001e-05 2024-09-05T16:34:58.595 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:58.595 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.595 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-09-05T16:34:58.595 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-09-05T16:34:58.580559+0000", 2024-09-05T16:34:58.595 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.3809999999999998e-06 2024-09-05T16:34:58.595 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:58.595 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.595 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-09-05T16:34:58.595 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-09-05T16:34:58.580564+0000", 2024-09-05T16:34:58.595 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.8810000000000004e-06 2024-09-05T16:34:58.596 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-09-05T16:34:58.596 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-09-05T16:34:58.596 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-09-05T16:34:58.596 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:58.596 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.596 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "MOSDECSubOpWrite(1.ds1 33/32 ECSubWrite(tid=13129, reqid=client.14960.0:36908, at_version=0'0, trim_to=0'0, roll_forward_to=33'2307))", 2024-09-05T16:34:58.596 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-09-05T16:34:58.553750+0000", 2024-09-05T16:34:58.596 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 0.030005153999999999, 2024-09-05T16:34:58.596 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.030231234999999999, 2024-09-05T16:34:58.596 INFO:teuthology.orchestra.run.smithi087.stdout: "continuous": false, 2024-09-05T16:34:58.596 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-09-05T16:34:58.596 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "reached pg", 2024-09-05T16:34:58.596 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-09-05T16:34:58.596 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.597 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-09-05T16:34:58.597 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-09-05T16:34:58.553750+0000", 2024-09-05T16:34:58.597 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-09-05T16:34:58.597 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:58.597 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.597 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-09-05T16:34:58.597 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-09-05T16:34:58.553750+0000", 2024-09-05T16:34:58.597 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-09-05T16:34:58.597 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:58.597 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.597 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-09-05T16:34:58.597 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-09-05T16:34:58.553751+0000", 2024-09-05T16:34:58.597 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.9900000000000009e-07 2024-09-05T16:34:58.597 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:58.598 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.598 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-09-05T16:34:58.598 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-09-05T16:34:58.553766+0000", 2024-09-05T16:34:58.598 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.5659e-05 2024-09-05T16:34:58.598 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:58.598 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.598 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-09-05T16:34:58.598 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-09-05T16:34:58.553778+0000", 2024-09-05T16:34:58.598 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.1792e-05 2024-09-05T16:34:58.598 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:58.598 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.598 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-09-05T16:34:58.598 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-09-05T16:34:58.553786+0000", 2024-09-05T16:34:58.598 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.3829999999999992e-06 2024-09-05T16:34:58.599 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:58.599 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.599 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-09-05T16:34:58.599 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-09-05T16:34:58.553862+0000", 2024-09-05T16:34:58.599 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.5628000000000002e-05 2024-09-05T16:34:58.599 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:58.599 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.599 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_started", 2024-09-05T16:34:58.599 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-09-05T16:34:58.553871+0000", 2024-09-05T16:34:58.599 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.2990000000000002e-06 2024-09-05T16:34:58.599 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-09-05T16:34:58.599 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-09-05T16:34:58.599 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-09-05T16:34:58.600 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:58.600 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.600 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "MOSDECSubOpWrite(1.ds1 33/32 ECSubWrite(tid=13130, reqid=client.14960.0:36920, at_version=33'2308, trim_to=0'0, roll_forward_to=33'2307))", 2024-09-05T16:34:58.600 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-09-05T16:34:58.565029+0000", 2024-09-05T16:34:58.600 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 0.018725296999999998, 2024-09-05T16:34:58.600 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.018993281000000001, 2024-09-05T16:34:58.600 INFO:teuthology.orchestra.run.smithi087.stdout: "continuous": false, 2024-09-05T16:34:58.600 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-09-05T16:34:58.600 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "reached pg", 2024-09-05T16:34:58.600 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-09-05T16:34:58.600 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.600 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-09-05T16:34:58.600 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-09-05T16:34:58.565029+0000", 2024-09-05T16:34:58.600 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-09-05T16:34:58.601 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:58.601 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.601 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-09-05T16:34:58.601 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-09-05T16:34:58.565029+0000", 2024-09-05T16:34:58.601 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-09-05T16:34:58.601 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:58.601 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.601 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-09-05T16:34:58.601 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-09-05T16:34:58.565030+0000", 2024-09-05T16:34:58.601 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.5199999999999996e-07 2024-09-05T16:34:58.601 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:58.601 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.601 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-09-05T16:34:58.601 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-09-05T16:34:58.565053+0000", 2024-09-05T16:34:58.602 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.3383999999999998e-05 2024-09-05T16:34:58.602 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:58.602 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.602 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-09-05T16:34:58.602 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-09-05T16:34:58.565063+0000", 2024-09-05T16:34:58.602 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.8230000000000006e-06 2024-09-05T16:34:58.602 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:58.602 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.602 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-09-05T16:34:58.602 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-09-05T16:34:58.565071+0000", 2024-09-05T16:34:58.602 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.7719999999999994e-06 2024-09-05T16:34:58.602 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:58.602 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.602 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-09-05T16:34:58.603 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-09-05T16:34:58.565167+0000", 2024-09-05T16:34:58.603 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.5985999999999999e-05 2024-09-05T16:34:58.603 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:58.603 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.603 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_started", 2024-09-05T16:34:58.603 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-09-05T16:34:58.565176+0000", 2024-09-05T16:34:58.603 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.7409999999999998e-06 2024-09-05T16:34:58.603 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-09-05T16:34:58.603 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-09-05T16:34:58.603 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-09-05T16:34:58.603 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:58.603 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.603 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "MOSDECSubOpWrite(1.8s0 33/32 ECSubWrite(tid=13131, reqid=client.14960.0:36913, at_version=0'0, trim_to=0'0, roll_forward_to=33'2339))", 2024-09-05T16:34:58.603 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-09-05T16:34:58.565097+0000", 2024-09-05T16:34:58.604 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 0.01865737, 2024-09-05T16:34:58.604 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.018967009, 2024-09-05T16:34:58.604 INFO:teuthology.orchestra.run.smithi087.stdout: "continuous": false, 2024-09-05T16:34:58.604 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-09-05T16:34:58.604 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "reached pg", 2024-09-05T16:34:58.604 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-09-05T16:34:58.604 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.604 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-09-05T16:34:58.604 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-09-05T16:34:58.565097+0000", 2024-09-05T16:34:58.604 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-09-05T16:34:58.604 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:58.604 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.604 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-09-05T16:34:58.604 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-09-05T16:34:58.565097+0000", 2024-09-05T16:34:58.605 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-09-05T16:34:58.605 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:58.605 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.605 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-09-05T16:34:58.605 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-09-05T16:34:58.565098+0000", 2024-09-05T16:34:58.605 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.3200000000000001e-07 2024-09-05T16:34:58.605 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:58.605 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.605 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-09-05T16:34:58.605 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-09-05T16:34:58.565106+0000", 2024-09-05T16:34:58.605 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.0970000000000006e-06 2024-09-05T16:34:58.605 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:58.605 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.605 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-09-05T16:34:58.606 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-09-05T16:34:58.565112+0000", 2024-09-05T16:34:58.606 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.0839999999999997e-06 2024-09-05T16:34:58.606 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:58.606 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.606 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-09-05T16:34:58.606 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-09-05T16:34:58.565117+0000", 2024-09-05T16:34:58.606 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.6740000000000002e-06 2024-09-05T16:34:58.606 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:58.606 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.606 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-09-05T16:34:58.606 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-09-05T16:34:58.567393+0000", 2024-09-05T16:34:58.606 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.002275783 2024-09-05T16:34:58.606 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:58.606 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.607 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_started", 2024-09-05T16:34:58.607 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-09-05T16:34:58.567402+0000", 2024-09-05T16:34:58.607 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.7819999999999999e-06 2024-09-05T16:34:58.607 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-09-05T16:34:58.607 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-09-05T16:34:58.607 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-09-05T16:34:58.607 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:58.607 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.607 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "MOSDECSubOpWrite(1.5s3 33/32 ECSubWrite(tid=13132, reqid=client.14960.0:36919, at_version=0'0, trim_to=0'0, roll_forward_to=33'2318))", 2024-09-05T16:34:58.607 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-09-05T16:34:58.565129+0000", 2024-09-05T16:34:58.607 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 0.018625189, 2024-09-05T16:34:58.607 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.018980032000000001, 2024-09-05T16:34:58.607 INFO:teuthology.orchestra.run.smithi087.stdout: "continuous": false, 2024-09-05T16:34:58.608 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-09-05T16:34:58.608 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "reached pg", 2024-09-05T16:34:58.608 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-09-05T16:34:58.608 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.608 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-09-05T16:34:58.608 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-09-05T16:34:58.565129+0000", 2024-09-05T16:34:58.608 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-09-05T16:34:58.608 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:58.608 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.608 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-09-05T16:34:58.608 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-09-05T16:34:58.565129+0000", 2024-09-05T16:34:58.608 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-09-05T16:34:58.609 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:58.609 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.609 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-09-05T16:34:58.609 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-09-05T16:34:58.565130+0000", 2024-09-05T16:34:58.609 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.85e-07 2024-09-05T16:34:58.609 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:58.609 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.609 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-09-05T16:34:58.609 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-09-05T16:34:58.565141+0000", 2024-09-05T16:34:58.609 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.1612e-05 2024-09-05T16:34:58.609 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:58.609 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.609 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-09-05T16:34:58.609 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-09-05T16:34:58.565147+0000", 2024-09-05T16:34:58.610 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.4430000000000002e-06 2024-09-05T16:34:58.610 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:58.610 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.610 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-09-05T16:34:58.610 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-09-05T16:34:58.565152+0000", 2024-09-05T16:34:58.610 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.3460000000000001e-06 2024-09-05T16:34:58.610 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:58.610 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.610 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-09-05T16:34:58.610 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-09-05T16:34:58.567797+0000", 2024-09-05T16:34:58.610 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.002645353 2024-09-05T16:34:58.610 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:58.610 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.610 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_started", 2024-09-05T16:34:58.611 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-09-05T16:34:58.567805+0000", 2024-09-05T16:34:58.611 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.4340000000000003e-06 2024-09-05T16:34:58.611 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-09-05T16:34:58.611 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-09-05T16:34:58.611 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-09-05T16:34:58.611 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:58.611 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.611 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "MOSDECSubOpWrite(1.5s3 33/32 ECSubWrite(tid=13133, reqid=client.14960.0:36922, at_version=33'2319, trim_to=0'0, roll_forward_to=33'2318))", 2024-09-05T16:34:58.611 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-09-05T16:34:58.565159+0000", 2024-09-05T16:34:58.611 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 0.018595237000000001, 2024-09-05T16:34:58.611 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.018992453999999999, 2024-09-05T16:34:58.611 INFO:teuthology.orchestra.run.smithi087.stdout: "continuous": false, 2024-09-05T16:34:58.611 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-09-05T16:34:58.612 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "reached pg", 2024-09-05T16:34:58.612 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-09-05T16:34:58.612 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.612 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-09-05T16:34:58.612 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-09-05T16:34:58.565159+0000", 2024-09-05T16:34:58.612 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-09-05T16:34:58.612 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:58.612 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.612 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-09-05T16:34:58.612 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-09-05T16:34:58.565159+0000", 2024-09-05T16:34:58.612 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-09-05T16:34:58.612 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:58.612 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.612 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-09-05T16:34:58.613 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-09-05T16:34:58.565160+0000", 2024-09-05T16:34:58.613 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.8799999999999999e-07 2024-09-05T16:34:58.613 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:58.613 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.613 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-09-05T16:34:58.613 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-09-05T16:34:58.565174+0000", 2024-09-05T16:34:58.613 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.4100999999999999e-05 2024-09-05T16:34:58.613 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:58.613 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.613 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-09-05T16:34:58.613 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-09-05T16:34:58.565179+0000", 2024-09-05T16:34:58.613 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.4739999999999996e-06 2024-09-05T16:34:58.613 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:58.614 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.614 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-09-05T16:34:58.614 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-09-05T16:34:58.565184+0000", 2024-09-05T16:34:58.614 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.1329999999999998e-06 2024-09-05T16:34:58.614 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:58.614 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.614 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-09-05T16:34:58.614 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-09-05T16:34:58.570172+0000", 2024-09-05T16:34:58.614 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0049874969999999996 2024-09-05T16:34:58.614 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:58.614 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.614 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_started", 2024-09-05T16:34:58.614 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-09-05T16:34:58.570179+0000", 2024-09-05T16:34:58.614 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.6429999999999995e-06 2024-09-05T16:34:58.615 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-09-05T16:34:58.615 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-09-05T16:34:58.615 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-09-05T16:34:58.615 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:58.615 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.615 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "MOSDECSubOpWrite(1.9s0 33/32 ECSubWrite(tid=22583, reqid=client.14960.0:36912, at_version=0'0, trim_to=0'0, roll_forward_to=33'2414))", 2024-09-05T16:34:58.615 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-09-05T16:34:58.565230+0000", 2024-09-05T16:34:58.615 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 0.018524684999999999, 2024-09-05T16:34:58.615 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.018963435000000001, 2024-09-05T16:34:58.615 INFO:teuthology.orchestra.run.smithi087.stdout: "continuous": false, 2024-09-05T16:34:58.615 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-09-05T16:34:58.615 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "reached pg", 2024-09-05T16:34:58.615 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-09-05T16:34:58.615 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.616 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-09-05T16:34:58.616 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-09-05T16:34:58.565230+0000", 2024-09-05T16:34:58.616 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-09-05T16:34:58.616 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:58.616 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.616 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-09-05T16:34:58.616 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-09-05T16:34:58.565230+0000", 2024-09-05T16:34:58.616 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-09-05T16:34:58.616 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:58.616 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.616 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-09-05T16:34:58.616 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-09-05T16:34:58.565230+0000", 2024-09-05T16:34:58.616 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.66e-07 2024-09-05T16:34:58.617 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:58.617 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.617 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-09-05T16:34:58.617 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-09-05T16:34:58.565236+0000", 2024-09-05T16:34:58.617 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.9259999999999997e-06 2024-09-05T16:34:58.617 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:58.617 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.617 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-09-05T16:34:58.617 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-09-05T16:34:58.565241+0000", 2024-09-05T16:34:58.617 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.7990000000000001e-06 2024-09-05T16:34:58.617 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:58.617 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.617 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-09-05T16:34:58.617 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-09-05T16:34:58.565246+0000", 2024-09-05T16:34:58.617 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.8439999999999997e-06 2024-09-05T16:34:58.618 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:58.618 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.618 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-09-05T16:34:58.618 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-09-05T16:34:58.565324+0000", 2024-09-05T16:34:58.618 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.8249000000000002e-05 2024-09-05T16:34:58.618 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:58.618 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.618 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_started", 2024-09-05T16:34:58.618 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-09-05T16:34:58.565333+0000", 2024-09-05T16:34:58.618 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.8259999999999999e-06 2024-09-05T16:34:58.618 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-09-05T16:34:58.618 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-09-05T16:34:58.618 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-09-05T16:34:58.618 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:58.619 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.619 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "MOSDECSubOpWrite(1.5s3 33/32 ECSubWrite(tid=13134, reqid=client.14960.0:36925, at_version=33'2320, trim_to=0'0, roll_forward_to=33'2318))", 2024-09-05T16:34:58.619 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-09-05T16:34:58.568150+0000", 2024-09-05T16:34:58.619 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 0.015604336, 2024-09-05T16:34:58.619 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.016085695000000001, 2024-09-05T16:34:58.619 INFO:teuthology.orchestra.run.smithi087.stdout: "continuous": false, 2024-09-05T16:34:58.619 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-09-05T16:34:58.619 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "queued for pg", 2024-09-05T16:34:58.619 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-09-05T16:34:58.619 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.619 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-09-05T16:34:58.619 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-09-05T16:34:58.568150+0000", 2024-09-05T16:34:58.619 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-09-05T16:34:58.619 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:58.620 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.620 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-09-05T16:34:58.620 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-09-05T16:34:58.568150+0000", 2024-09-05T16:34:58.620 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-09-05T16:34:58.620 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:58.620 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.620 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-09-05T16:34:58.620 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-09-05T16:34:58.568151+0000", 2024-09-05T16:34:58.620 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.6099999999999999e-07 2024-09-05T16:34:58.620 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:58.620 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.620 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-09-05T16:34:58.620 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-09-05T16:34:58.568174+0000", 2024-09-05T16:34:58.620 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.232e-05 2024-09-05T16:34:58.621 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:58.621 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.621 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-09-05T16:34:58.621 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-09-05T16:34:58.568184+0000", 2024-09-05T16:34:58.621 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.0251000000000001e-05 2024-09-05T16:34:58.621 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:58.621 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.621 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-09-05T16:34:58.621 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-09-05T16:34:58.568192+0000", 2024-09-05T16:34:58.621 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.354e-06 2024-09-05T16:34:58.621 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-09-05T16:34:58.621 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-09-05T16:34:58.621 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-09-05T16:34:58.621 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:58.622 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.622 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "MOSDECSubOpWrite(1.0s0 33/32 ECSubWrite(tid=16198, reqid=client.14960.0:36918, at_version=0'0, trim_to=0'0, roll_forward_to=33'2339))", 2024-09-05T16:34:58.622 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-09-05T16:34:58.568797+0000", 2024-09-05T16:34:58.622 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 0.014958127999999999, 2024-09-05T16:34:58.622 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.015477250999999999, 2024-09-05T16:34:58.622 INFO:teuthology.orchestra.run.smithi087.stdout: "continuous": false, 2024-09-05T16:34:58.622 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-09-05T16:34:58.622 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "reached pg", 2024-09-05T16:34:58.622 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-09-05T16:34:58.622 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.622 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-09-05T16:34:58.622 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-09-05T16:34:58.568797+0000", 2024-09-05T16:34:58.622 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-09-05T16:34:58.623 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:58.623 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.623 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-09-05T16:34:58.623 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-09-05T16:34:58.568797+0000", 2024-09-05T16:34:58.623 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-09-05T16:34:58.623 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:58.623 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.623 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-09-05T16:34:58.623 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-09-05T16:34:58.568797+0000", 2024-09-05T16:34:58.623 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.0999999999999997e-07 2024-09-05T16:34:58.623 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:58.623 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.623 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-09-05T16:34:58.623 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-09-05T16:34:58.568813+0000", 2024-09-05T16:34:58.624 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.5926e-05 2024-09-05T16:34:58.624 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:58.624 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.624 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-09-05T16:34:58.624 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-09-05T16:34:58.568825+0000", 2024-09-05T16:34:58.624 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.2038999999999999e-05 2024-09-05T16:34:58.624 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:58.624 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.624 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-09-05T16:34:58.624 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-09-05T16:34:58.568835+0000", 2024-09-05T16:34:58.624 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.3039999999999994e-06 2024-09-05T16:34:58.624 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:58.624 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.624 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-09-05T16:34:58.625 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-09-05T16:34:58.569922+0000", 2024-09-05T16:34:58.625 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.001086926 2024-09-05T16:34:58.625 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:58.625 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.625 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_started", 2024-09-05T16:34:58.625 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-09-05T16:34:58.569930+0000", 2024-09-05T16:34:58.625 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.867e-06 2024-09-05T16:34:58.625 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-09-05T16:34:58.625 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-09-05T16:34:58.625 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-09-05T16:34:58.625 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:58.625 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.625 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "MOSDECSubOpWrite(1.5s3 33/32 ECSubWrite(tid=13135, reqid=client.14960.0:36930, at_version=33'2321, trim_to=0'0, roll_forward_to=33'2318))", 2024-09-05T16:34:58.625 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-09-05T16:34:58.571268+0000", 2024-09-05T16:34:58.626 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 0.012486657, 2024-09-05T16:34:58.626 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.013052586999999999, 2024-09-05T16:34:58.626 INFO:teuthology.orchestra.run.smithi087.stdout: "continuous": false, 2024-09-05T16:34:58.626 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-09-05T16:34:58.626 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "queued for pg", 2024-09-05T16:34:58.626 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-09-05T16:34:58.626 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.626 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-09-05T16:34:58.626 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-09-05T16:34:58.571268+0000", 2024-09-05T16:34:58.626 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-09-05T16:34:58.626 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:58.626 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.626 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-09-05T16:34:58.626 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-09-05T16:34:58.571268+0000", 2024-09-05T16:34:58.627 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-09-05T16:34:58.627 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:58.627 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.627 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-09-05T16:34:58.627 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-09-05T16:34:58.571269+0000", 2024-09-05T16:34:58.627 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.263e-06 2024-09-05T16:34:58.627 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:58.627 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.627 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-09-05T16:34:58.627 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-09-05T16:34:58.571298+0000", 2024-09-05T16:34:58.627 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.8600000000000001e-05 2024-09-05T16:34:58.627 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:58.627 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.627 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-09-05T16:34:58.628 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-09-05T16:34:58.571307+0000", 2024-09-05T16:34:58.628 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.3510000000000008e-06 2024-09-05T16:34:58.628 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:58.628 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.628 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-09-05T16:34:58.628 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-09-05T16:34:58.571316+0000", 2024-09-05T16:34:58.628 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.8300000000000002e-06 2024-09-05T16:34:58.628 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-09-05T16:34:58.628 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-09-05T16:34:58.628 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-09-05T16:34:58.628 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:58.628 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.628 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "MOSDECSubOpWrite(1.ds1 33/32 ECSubWrite(tid=13136, reqid=client.14960.0:36931, at_version=33'2309, trim_to=0'0, roll_forward_to=33'2307))", 2024-09-05T16:34:58.628 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-09-05T16:34:58.574998+0000", 2024-09-05T16:34:58.629 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 0.0087561570000000005, 2024-09-05T16:34:58.629 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0093672730000000006, 2024-09-05T16:34:58.629 INFO:teuthology.orchestra.run.smithi087.stdout: "continuous": false, 2024-09-05T16:34:58.629 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-09-05T16:34:58.629 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "queued for pg", 2024-09-05T16:34:58.629 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-09-05T16:34:58.629 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.629 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-09-05T16:34:58.629 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-09-05T16:34:58.574998+0000", 2024-09-05T16:34:58.629 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-09-05T16:34:58.629 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:58.629 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.629 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-09-05T16:34:58.630 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-09-05T16:34:58.574998+0000", 2024-09-05T16:34:58.630 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-09-05T16:34:58.630 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:58.630 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.630 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-09-05T16:34:58.630 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-09-05T16:34:58.574999+0000", 2024-09-05T16:34:58.630 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.2399999999999996e-07 2024-09-05T16:34:58.630 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:58.630 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.630 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-09-05T16:34:58.630 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-09-05T16:34:58.575021+0000", 2024-09-05T16:34:58.630 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.1569999999999998e-05 2024-09-05T16:34:58.630 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:58.630 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.630 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-09-05T16:34:58.631 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-09-05T16:34:58.575031+0000", 2024-09-05T16:34:58.631 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.0249e-05 2024-09-05T16:34:58.631 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:58.631 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.631 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-09-05T16:34:58.631 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-09-05T16:34:58.575039+0000", 2024-09-05T16:34:58.631 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.7370000000000006e-06 2024-09-05T16:34:58.631 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-09-05T16:34:58.631 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-09-05T16:34:58.631 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-09-05T16:34:58.631 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:58.631 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.631 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "MOSDECSubOpWrite(1.2s2 33/32 ECSubWrite(tid=16199, reqid=client.14960.0:36932, at_version=33'2205, trim_to=0'0, roll_forward_to=33'2204))", 2024-09-05T16:34:58.631 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-09-05T16:34:58.580257+0000", 2024-09-05T16:34:58.632 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 0.0034980839999999998, 2024-09-05T16:34:58.632 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0041511309999999997, 2024-09-05T16:34:58.632 INFO:teuthology.orchestra.run.smithi087.stdout: "continuous": false, 2024-09-05T16:34:58.632 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-09-05T16:34:58.632 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "reached pg", 2024-09-05T16:34:58.632 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-09-05T16:34:58.632 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.632 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-09-05T16:34:58.632 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-09-05T16:34:58.580257+0000", 2024-09-05T16:34:58.632 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-09-05T16:34:58.632 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:58.632 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.632 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-09-05T16:34:58.632 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-09-05T16:34:58.580257+0000", 2024-09-05T16:34:58.633 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-09-05T16:34:58.633 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:58.633 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.633 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-09-05T16:34:58.633 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-09-05T16:34:58.580258+0000", 2024-09-05T16:34:58.633 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.37e-06 2024-09-05T16:34:58.633 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:58.633 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.633 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-09-05T16:34:58.633 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-09-05T16:34:58.580287+0000", 2024-09-05T16:34:58.633 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.9241e-05 2024-09-05T16:34:58.633 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:58.633 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.633 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-09-05T16:34:58.634 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-09-05T16:34:58.580303+0000", 2024-09-05T16:34:58.634 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.5824e-05 2024-09-05T16:34:58.634 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:58.634 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.634 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-09-05T16:34:58.634 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-09-05T16:34:58.580313+0000", 2024-09-05T16:34:58.634 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.7939999999999997e-06 2024-09-05T16:34:58.634 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:58.634 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.634 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-09-05T16:34:58.634 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-09-05T16:34:58.580605+0000", 2024-09-05T16:34:58.634 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00029225799999999999 2024-09-05T16:34:58.634 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:58.634 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.635 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_started", 2024-09-05T16:34:58.635 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-09-05T16:34:58.580614+0000", 2024-09-05T16:34:58.635 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.3500000000000003e-06 2024-09-05T16:34:58.635 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-09-05T16:34:58.635 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-09-05T16:34:58.635 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-09-05T16:34:58.635 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:58.635 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.635 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "MOSDECSubOpWrite(1.0s0 33/32 ECSubWrite(tid=16200, reqid=client.14960.0:36937, at_version=33'2340, trim_to=0'0, roll_forward_to=33'2339))", 2024-09-05T16:34:58.635 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-09-05T16:34:58.580338+0000", 2024-09-05T16:34:58.635 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 0.0034165710000000002, 2024-09-05T16:34:58.635 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0041193690000000003, 2024-09-05T16:34:58.635 INFO:teuthology.orchestra.run.smithi087.stdout: "continuous": false, 2024-09-05T16:34:58.635 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-09-05T16:34:58.636 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "queued for pg", 2024-09-05T16:34:58.636 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-09-05T16:34:58.636 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.636 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-09-05T16:34:58.636 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-09-05T16:34:58.580338+0000", 2024-09-05T16:34:58.636 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-09-05T16:34:58.636 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:58.636 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.636 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-09-05T16:34:58.636 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-09-05T16:34:58.580338+0000", 2024-09-05T16:34:58.636 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-09-05T16:34:58.636 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:58.636 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.636 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-09-05T16:34:58.637 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-09-05T16:34:58.580338+0000", 2024-09-05T16:34:58.637 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.79e-07 2024-09-05T16:34:58.637 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:58.637 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.637 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-09-05T16:34:58.637 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-09-05T16:34:58.580355+0000", 2024-09-05T16:34:58.637 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.7051999999999999e-05 2024-09-05T16:34:58.637 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:58.637 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.637 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-09-05T16:34:58.637 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-09-05T16:34:58.580362+0000", 2024-09-05T16:34:58.637 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.2040000000000002e-06 2024-09-05T16:34:58.637 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-09-05T16:34:58.638 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-09-05T16:34:58.638 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-09-05T16:34:58.638 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-09-05T16:34:58.580367+0000", 2024-09-05T16:34:58.638 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.4e-06 2024-09-05T16:34:58.638 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-09-05T16:34:58.638 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-09-05T16:34:58.638 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-09-05T16:34:58.638 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-09-05T16:34:58.638 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-09-05T16:34:58.638 INFO:teuthology.orchestra.run.smithi087.stdout: "num_ops": 17 2024-09-05T16:34:58.638 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-09-05T16:34:58.639 INFO:tasks.ec_lost_unfound.ceph_manager:waiting for osd.1 to be up 2024-09-05T16:34:58.640 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-05T16:34:59.192 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:34:59.192 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":33,"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","created":"2024-09-05T16:33:15.770967+0000","modified":"2024-09-05T16:34:56.551919+0000","last_up_change":"2024-09-05T16:34:54.893123+0000","last_in_change":"2024-09-05T16:33:21.020289+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":9,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"unique_pool_0","create_time":"2024-09-05T16:33:40.865276+0000","flags":1,"flags_names":"hashpspool","type":3,"size":4,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"lost_unfound","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"f76bac77-d755-450f-9fbf-fb53a6033c24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":18,"up_from":25,"up_thru":32,"down_at":19,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6804","nonce":67704509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6805","nonce":67704509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6807","nonce":67704509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6806","nonce":67704509}]},"public_addr":"172.21.15.87:6804/67704509","cluster_addr":"172.21.15.87:6805/67704509","heartbeat_back_addr":"172.21.15.87:6807/67704509","heartbeat_front_addr":"172.21.15.87:6806/67704509","state":["exists","up"]},{"osd":1,"uuid":"941684ea-0764-4182-94cc-136349df4f23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":11,"last_clean_end":21,"up_from":31,"up_thru":31,"down_at":22,"lost_at":22,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6812","nonce":3937864697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6813","nonce":3937864697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6815","nonce":3937864697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6814","nonce":3937864697}]},"public_addr":"172.21.15.87:6812/3937864697","cluster_addr":"172.21.15.87:6813/3937864697","heartbeat_back_addr":"172.21.15.87:6815/3937864697","heartbeat_front_addr":"172.21.15.87:6814/3937864697","state":["exists","up"]},{"osd":2,"uuid":"17135934-ecb7-445a-b1dc-27f55da9f8ca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6800","nonce":4234108552}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6801","nonce":4234108552}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6803","nonce":4234108552}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6802","nonce":4234108552}]},"public_addr":"172.21.15.87:6800/4234108552","cluster_addr":"172.21.15.87:6801/4234108552","heartbeat_back_addr":"172.21.15.87:6803/4234108552","heartbeat_front_addr":"172.21.15.87:6802/4234108552","state":["exists","up"]},{"osd":3,"uuid":"098f9cbd-3da5-4ee7-8439-610b8cae774e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":11,"last_clean_end":19,"up_from":28,"up_thru":32,"down_at":20,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6808","nonce":1182532508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6809","nonce":1182532508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6811","nonce":1182532508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6810","nonce":1182532508}]},"public_addr":"172.21.15.87:6808/1182532508","cluster_addr":"172.21.15.87:6809/1182532508","heartbeat_back_addr":"172.21.15.87:6811/1182532508","heartbeat_front_addr":"172.21.15.87:6810/1182532508","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-05T16:33:27.513154+0000","dead_epoch":19},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-05T16:33:27.800689+0000","dead_epoch":22},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-05T16:33:28.127983+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-05T16:33:28.046083+0000","dead_epoch":20}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[{"pgid":"1.0","osds":[2147483647,0,2,3]},{"pgid":"1.1","osds":[3,2147483647,0,2]},{"pgid":"1.2","osds":[0,3,2147483647,2]},{"pgid":"1.3","osds":[2147483647,0,2,3]},{"pgid":"1.4","osds":[3,0,2,2147483647]},{"pgid":"1.5","osds":[2,0,3,2147483647]},{"pgid":"1.6","osds":[3,0,2147483647,2]},{"pgid":"1.7","osds":[2147483647,3,2,0]},{"pgid":"1.8","osds":[2147483647,2,3,0]},{"pgid":"1.9","osds":[2147483647,3,2,0]},{"pgid":"1.a","osds":[2147483647,2,3,0]},{"pgid":"1.b","osds":[2147483647,0,2,3]},{"pgid":"1.c","osds":[3,0,2,2147483647]},{"pgid":"1.d","osds":[2,2147483647,3,0]},{"pgid":"1.e","osds":[2147483647,0,3,2]},{"pgid":"1.f","osds":[3,0,2147483647,2]}],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"lost_unfound":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-05T16:34:59.223 INFO:tasks.ec_lost_unfound.ceph_manager:osd.1 is up 2024-09-05T16:34:59.223 INFO:tasks.ec_lost_unfound.ceph_manager:waiting for clean 2024-09-05T16:34:59.223 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:34:59.494 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 37 20 37303 37283 7.87131 3.125 0.0287966 0.0198338 2024-09-05T16:34:59.791 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:34:59.791 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:34:59.817 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":75,"stamp":"2024-09-05T16:34:58.798334+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280600768,"num_objects":34258,"num_object_clones":0,"num_object_copies":137032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27594,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34258,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":34285,"num_write_kb":274162,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":34312,"ondisk_log_size":34312,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":840772,"kb_used_data":431360,"kb_used_omap":14,"kb_used_meta":409329,"kb_avail":376646588,"statfs":{"total":386547056640,"available":385686106112,"internally_reserved":0,"allocated":441712640,"data_stored":441080132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":419153463},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[9,7,11,4,9,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":23977984,"num_objects":2927,"num_object_clones":0,"num_object_copies":11708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-3737,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2927,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2927,"num_write_kb":23416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.919380"},"pg_stats":[{"pgid":"1.f","version":"30'2225","reported_seq":2256,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-09-05T16:34:53.987112+0000","last_change":"2024-09-05T16:34:22.526044+0000","last_active":"2024-09-05T16:34:53.987112+0000","last_peered":"2024-09-05T16:34:53.987112+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:15.913051+0000","last_became_peered":"2024-09-05T16:34:15.913051+0000","last_unstale":"2024-09-05T16:34:53.987112+0000","last_undegraded":"2024-09-05T16:34:22.517240+0000","last_fullsized":"2024-09-05T16:34:15.905613+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2225,"log_dups_size":0,"ondisk_log_size":2225,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T19:25:44.252388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18227200,"num_objects":2225,"num_object_clones":0,"num_object_copies":8900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2225,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2225,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2225,"num_write_kb":17800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2147483647,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2225}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"29'2000","reported_seq":2051,"reported_epoch":29,"state":"active+undersized+degraded","last_fresh":"2024-09-05T16:34:49.975250+0000","last_change":"2024-09-05T16:34:33.555748+0000","last_active":"2024-09-05T16:34:49.975250+0000","last_peered":"2024-09-05T16:34:49.975250+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:15.910788+0000","last_became_peered":"2024-09-05T16:34:15.910788+0000","last_unstale":"2024-09-05T16:34:49.975250+0000","last_undegraded":"2024-09-05T16:34:15.903622+0000","last_fullsized":"2024-09-05T16:34:15.903245+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2000,"log_dups_size":0,"ondisk_log_size":2000,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-05T16:34:53.764827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16348320,"num_objects":1996,"num_object_clones":0,"num_object_copies":7984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1996,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1996,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1998,"num_write_kb":15980,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1996}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"30'2254","reported_seq":2318,"reported_epoch":30,"state":"peering","last_fresh":"2024-09-05T16:34:54.726828+0000","last_change":"2024-09-05T16:34:54.921422+0000","last_active":"2024-09-05T16:34:54.921422+0000","last_peered":"2024-09-05T16:34:54.726828+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:15.917951+0000","last_became_peered":"2024-09-05T16:34:15.917951+0000","last_unstale":"2024-09-05T16:34:54.726828+0000","last_undegraded":"2024-09-05T16:34:15.904811+0000","last_fullsized":"2024-09-05T16:34:15.904154+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2254,"log_dups_size":0,"ondisk_log_size":2254,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18438736,"num_objects":2251,"num_object_clones":0,"num_object_copies":9004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2251,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2253,"num_write_kb":18014,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,3],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"29'2010","reported_seq":2061,"reported_epoch":29,"state":"active+undersized+degraded","last_fresh":"2024-09-05T16:34:49.965039+0000","last_change":"2024-09-05T16:34:34.682384+0000","last_active":"2024-09-05T16:34:49.965039+0000","last_peered":"2024-09-05T16:34:49.965039+0000","last_clean":"2024-09-05T16:33:53.148717+0000","last_became_active":"2024-09-05T16:34:15.912196+0000","last_became_peered":"2024-09-05T16:34:15.912196+0000","last_unstale":"2024-09-05T16:34:49.965039+0000","last_undegraded":"2024-09-05T16:34:15.905010+0000","last_fullsized":"2024-09-05T16:34:15.904610+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2010,"log_dups_size":0,"ondisk_log_size":2010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-05T16:34:50.808154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16430240,"num_objects":2006,"num_object_clones":0,"num_object_copies":8024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2006,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2006,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2008,"num_write_kb":16053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2006}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"30'2114","reported_seq":2155,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-09-05T16:34:53.988469+0000","last_change":"2024-09-05T16:34:35.192823+0000","last_active":"2024-09-05T16:34:53.988469+0000","last_peered":"2024-09-05T16:34:53.988469+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:15.910648+0000","last_became_peered":"2024-09-05T16:34:15.910648+0000","last_unstale":"2024-09-05T16:34:53.988469+0000","last_undegraded":"2024-09-05T16:34:15.904665+0000","last_fullsized":"2024-09-05T16:34:15.903277+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2114,"log_dups_size":0,"ondisk_log_size":2114,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-05T16:34:56.762265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17290400,"num_objects":2111,"num_object_clones":0,"num_object_copies":8444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2111,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2111,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2112,"num_write_kb":16893,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2147483647,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2111}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"30'2212","reported_seq":2243,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-09-05T16:34:53.996971+0000","last_change":"2024-09-05T16:34:22.533654+0000","last_active":"2024-09-05T16:34:53.996971+0000","last_peered":"2024-09-05T16:34:53.996971+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:15.913477+0000","last_became_peered":"2024-09-05T16:34:15.913477+0000","last_unstale":"2024-09-05T16:34:53.996971+0000","last_undegraded":"2024-09-05T16:34:22.533443+0000","last_fullsized":"2024-09-05T16:34:15.906631+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2212,"log_dups_size":0,"ondisk_log_size":2212,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T01:56:46.762011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18120704,"num_objects":2212,"num_object_clones":0,"num_object_copies":8848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2212,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2212,"num_write_kb":17696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,2147483647],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2212}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"30'2155","reported_seq":2209,"reported_epoch":30,"state":"peering","last_fresh":"2024-09-05T16:34:54.732728+0000","last_change":"2024-09-05T16:34:54.914099+0000","last_active":"2024-09-05T16:34:54.914099+0000","last_peered":"2024-09-05T16:34:54.732728+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:15.913479+0000","last_became_peered":"2024-09-05T16:34:15.913479+0000","last_unstale":"2024-09-05T16:34:54.732728+0000","last_undegraded":"2024-09-05T16:34:15.905955+0000","last_fullsized":"2024-09-05T16:34:15.905617+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2155,"log_dups_size":0,"ondisk_log_size":2155,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17601696,"num_objects":2149,"num_object_clones":0,"num_object_copies":8596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2149,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2153,"num_write_kb":17204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,2,3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"29'1948","reported_seq":1999,"reported_epoch":29,"state":"active+undersized+degraded","last_fresh":"2024-09-05T16:34:49.905143+0000","last_change":"2024-09-05T16:34:36.231100+0000","last_active":"2024-09-05T16:34:49.905143+0000","last_peered":"2024-09-05T16:34:49.905143+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:15.910570+0000","last_became_peered":"2024-09-05T16:34:15.910570+0000","last_unstale":"2024-09-05T16:34:49.905143+0000","last_undegraded":"2024-09-05T16:34:15.904563+0000","last_fullsized":"2024-09-05T16:34:15.903952+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":1948,"log_dups_size":0,"ondisk_log_size":1948,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-05T16:34:52.808437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15914144,"num_objects":1943,"num_object_clones":0,"num_object_copies":7772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1943,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1943,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1946,"num_write_kb":15556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1943}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"30'2236","reported_seq":2280,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-09-05T16:34:53.976776+0000","last_change":"2024-09-05T16:34:36.789538+0000","last_active":"2024-09-05T16:34:53.976776+0000","last_peered":"2024-09-05T16:34:53.976776+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:15.914297+0000","last_became_peered":"2024-09-05T16:34:15.914297+0000","last_unstale":"2024-09-05T16:34:53.976776+0000","last_undegraded":"2024-09-05T16:34:15.906625+0000","last_fullsized":"2024-09-05T16:34:15.906328+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2236,"log_dups_size":0,"ondisk_log_size":2236,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-05T16:34:58.709828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18245952,"num_objects":2228,"num_object_clones":0,"num_object_copies":8912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2228,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2233,"num_write_kb":17841,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,2147483647],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2228}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"30'2266","reported_seq":2332,"reported_epoch":30,"state":"peering","last_fresh":"2024-09-05T16:34:54.732908+0000","last_change":"2024-09-05T16:34:54.921291+0000","last_active":"2024-09-05T16:34:54.921291+0000","last_peered":"2024-09-05T16:34:54.732908+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:15.918296+0000","last_became_peered":"2024-09-05T16:34:15.918296+0000","last_unstale":"2024-09-05T16:34:54.732908+0000","last_undegraded":"2024-09-05T16:34:15.905187+0000","last_fullsized":"2024-09-05T16:34:15.904834+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2266,"log_dups_size":0,"ondisk_log_size":2266,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18543776,"num_objects":2264,"num_object_clones":0,"num_object_copies":9056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2264,"num_write_kb":18110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,3],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"30'2284","reported_seq":2328,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-09-05T16:34:53.956178+0000","last_change":"2024-09-05T16:34:37.985329+0000","last_active":"2024-09-05T16:34:53.956178+0000","last_peered":"2024-09-05T16:34:53.956178+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:15.914271+0000","last_became_peered":"2024-09-05T16:34:15.914271+0000","last_unstale":"2024-09-05T16:34:53.956178+0000","last_undegraded":"2024-09-05T16:34:15.905903+0000","last_fullsized":"2024-09-05T16:34:15.905357+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2284,"log_dups_size":0,"ondisk_log_size":2284,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18665200,"num_objects":2279,"num_object_clones":0,"num_object_copies":9116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2279,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2281,"num_write_kb":18236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2147483647,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2279}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"30'2231","reported_seq":2256,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-09-05T16:34:54.000022+0000","last_change":"2024-09-05T16:34:38.396192+0000","last_active":"2024-09-05T16:34:54.000022+0000","last_peered":"2024-09-05T16:34:54.000022+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:15.913627+0000","last_became_peered":"2024-09-05T16:34:15.913627+0000","last_unstale":"2024-09-05T16:34:54.000022+0000","last_undegraded":"2024-09-05T16:34:15.906286+0000","last_fullsized":"2024-09-05T16:34:15.906044+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2231,"log_dups_size":0,"ondisk_log_size":2231,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-05T16:34:55.808461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18266704,"num_objects":2230,"num_object_clones":0,"num_object_copies":8920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2230,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2230,"num_write_kb":17839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2230}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"29'2033","reported_seq":2094,"reported_epoch":29,"state":"active+undersized+degraded","last_fresh":"2024-09-05T16:34:50.024184+0000","last_change":"2024-09-05T16:34:38.929130+0000","last_active":"2024-09-05T16:34:50.024184+0000","last_peered":"2024-09-05T16:34:50.024184+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:15.917839+0000","last_became_peered":"2024-09-05T16:34:15.917839+0000","last_unstale":"2024-09-05T16:34:50.024184+0000","last_undegraded":"2024-09-05T16:34:15.904338+0000","last_fullsized":"2024-09-05T16:34:15.904018+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2033,"log_dups_size":0,"ondisk_log_size":2033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-05T16:34:54.794447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16644688,"num_objects":2032,"num_object_clones":0,"num_object_copies":8128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2032,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2032,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2032,"num_write_kb":16255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2032}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"30'2324","reported_seq":2365,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-09-05T16:34:53.940874+0000","last_change":"2024-09-05T16:34:39.373308+0000","last_active":"2024-09-05T16:34:53.940874+0000","last_peered":"2024-09-05T16:34:53.940874+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:15.913175+0000","last_became_peered":"2024-09-05T16:34:15.913175+0000","last_unstale":"2024-09-05T16:34:53.940874+0000","last_undegraded":"2024-09-05T16:34:15.905345+0000","last_fullsized":"2024-09-05T16:34:15.905038+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2324,"log_dups_size":0,"ondisk_log_size":2324,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-05T16:34:57.755827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19002528,"num_objects":2320,"num_object_clones":0,"num_object_copies":9280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2320,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2322,"num_write_kb":18572,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2320}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"29'2057","reported_seq":2121,"reported_epoch":29,"state":"active+undersized+degraded","last_fresh":"2024-09-05T16:34:50.029434+0000","last_change":"2024-09-05T16:34:39.868440+0000","last_active":"2024-09-05T16:34:50.029434+0000","last_peered":"2024-09-05T16:34:50.029434+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:15.917968+0000","last_became_peered":"2024-09-05T16:34:15.917968+0000","last_unstale":"2024-09-05T16:34:50.029434+0000","last_undegraded":"2024-09-05T16:34:15.904385+0000","last_fullsized":"2024-09-05T16:34:15.904040+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2057,"log_dups_size":0,"ondisk_log_size":2057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-05T16:34:51.832737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16823456,"num_objects":2054,"num_object_clones":0,"num_object_copies":8216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2054,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2055,"num_write_kb":16437,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2054}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"29'1963","reported_seq":2014,"reported_epoch":29,"state":"active+undersized+degraded","last_fresh":"2024-09-05T16:34:49.972000+0000","last_change":"2024-09-05T16:34:40.418222+0000","last_active":"2024-09-05T16:34:49.972000+0000","last_peered":"2024-09-05T16:34:49.972000+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:15.910947+0000","last_became_peered":"2024-09-05T16:34:15.910947+0000","last_unstale":"2024-09-05T16:34:49.972000+0000","last_undegraded":"2024-09-05T16:34:15.905253+0000","last_fullsized":"2024-09-05T16:34:15.904912+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":1963,"log_dups_size":0,"ondisk_log_size":1963,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16037024,"num_objects":1958,"num_object_clones":0,"num_object_copies":7832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1958,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1958,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1961,"num_write_kb":15676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1958}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":280600768,"num_objects":34258,"num_object_clones":0,"num_object_copies":137032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27594,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34258,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":34285,"num_write_kb":274162,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":441921536,"data_stored":441921536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":34312,"ondisk_log_size":34312,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084300,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":258976,"kb_used_data":141200,"kb_used_omap":4,"kb_used_meta":117755,"kb_avail":94112864,"statfs":{"total":96636764160,"available":96371572736,"internally_reserved":0,"allocated":144588800,"data_stored":144423343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":120581843},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,1,2,0,8,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":0,"seq":1,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27824,"kb_used_data":288,"kb_used_omap":4,"kb_used_meta":27515,"kb_avail":94344016,"statfs":{"total":96636764160,"available":96608272384,"internally_reserved":0,"allocated":294912,"data_stored":167519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":28176082},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949672983,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":278840,"kb_used_data":144936,"kb_used_omap":1,"kb_used_meta":133886,"kb_avail":94093000,"statfs":{"total":96636764160,"available":96351232000,"internally_reserved":0,"allocated":148414464,"data_stored":148244635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":137099717},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,4,3,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182414,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":275132,"kb_used_data":144936,"kb_used_omap":4,"kb_used_meta":130171,"kb_avail":94096708,"statfs":{"total":96636764160,"available":96355028992,"internally_reserved":0,"allocated":148414464,"data_stored":148244635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":133295821},"hb_peers":[2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,3,5,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":148160512,"data_stored":148160512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":114688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":148160512,"data_stored":148160512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":145485824,"data_stored":145485824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:34:59.818 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:35:00.379 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:35:00.380 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:35:00.410 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":75,"stamp":"2024-09-05T16:34:58.798334+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280600768,"num_objects":34258,"num_object_clones":0,"num_object_copies":137032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27594,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34258,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":34285,"num_write_kb":274162,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":34312,"ondisk_log_size":34312,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":840772,"kb_used_data":431360,"kb_used_omap":14,"kb_used_meta":409329,"kb_avail":376646588,"statfs":{"total":386547056640,"available":385686106112,"internally_reserved":0,"allocated":441712640,"data_stored":441080132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":419153463},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[9,7,11,4,9,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":23977984,"num_objects":2927,"num_object_clones":0,"num_object_copies":11708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-3737,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2927,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2927,"num_write_kb":23416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.919380"},"pg_stats":[{"pgid":"1.f","version":"30'2225","reported_seq":2256,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-09-05T16:34:53.987112+0000","last_change":"2024-09-05T16:34:22.526044+0000","last_active":"2024-09-05T16:34:53.987112+0000","last_peered":"2024-09-05T16:34:53.987112+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:15.913051+0000","last_became_peered":"2024-09-05T16:34:15.913051+0000","last_unstale":"2024-09-05T16:34:53.987112+0000","last_undegraded":"2024-09-05T16:34:22.517240+0000","last_fullsized":"2024-09-05T16:34:15.905613+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2225,"log_dups_size":0,"ondisk_log_size":2225,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T19:25:44.252388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18227200,"num_objects":2225,"num_object_clones":0,"num_object_copies":8900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2225,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2225,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2225,"num_write_kb":17800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2147483647,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2225}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"29'2000","reported_seq":2051,"reported_epoch":29,"state":"active+undersized+degraded","last_fresh":"2024-09-05T16:34:49.975250+0000","last_change":"2024-09-05T16:34:33.555748+0000","last_active":"2024-09-05T16:34:49.975250+0000","last_peered":"2024-09-05T16:34:49.975250+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:15.910788+0000","last_became_peered":"2024-09-05T16:34:15.910788+0000","last_unstale":"2024-09-05T16:34:49.975250+0000","last_undegraded":"2024-09-05T16:34:15.903622+0000","last_fullsized":"2024-09-05T16:34:15.903245+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2000,"log_dups_size":0,"ondisk_log_size":2000,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-05T16:34:53.764827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16348320,"num_objects":1996,"num_object_clones":0,"num_object_copies":7984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1996,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1996,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1998,"num_write_kb":15980,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1996}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"30'2254","reported_seq":2318,"reported_epoch":30,"state":"peering","last_fresh":"2024-09-05T16:34:54.726828+0000","last_change":"2024-09-05T16:34:54.921422+0000","last_active":"2024-09-05T16:34:54.921422+0000","last_peered":"2024-09-05T16:34:54.726828+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:15.917951+0000","last_became_peered":"2024-09-05T16:34:15.917951+0000","last_unstale":"2024-09-05T16:34:54.726828+0000","last_undegraded":"2024-09-05T16:34:15.904811+0000","last_fullsized":"2024-09-05T16:34:15.904154+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2254,"log_dups_size":0,"ondisk_log_size":2254,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18438736,"num_objects":2251,"num_object_clones":0,"num_object_copies":9004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2251,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2253,"num_write_kb":18014,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,3],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"29'2010","reported_seq":2061,"reported_epoch":29,"state":"active+undersized+degraded","last_fresh":"2024-09-05T16:34:49.965039+0000","last_change":"2024-09-05T16:34:34.682384+0000","last_active":"2024-09-05T16:34:49.965039+0000","last_peered":"2024-09-05T16:34:49.965039+0000","last_clean":"2024-09-05T16:33:53.148717+0000","last_became_active":"2024-09-05T16:34:15.912196+0000","last_became_peered":"2024-09-05T16:34:15.912196+0000","last_unstale":"2024-09-05T16:34:49.965039+0000","last_undegraded":"2024-09-05T16:34:15.905010+0000","last_fullsized":"2024-09-05T16:34:15.904610+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2010,"log_dups_size":0,"ondisk_log_size":2010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-05T16:34:50.808154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16430240,"num_objects":2006,"num_object_clones":0,"num_object_copies":8024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2006,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2006,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2008,"num_write_kb":16053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2006}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"30'2114","reported_seq":2155,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-09-05T16:34:53.988469+0000","last_change":"2024-09-05T16:34:35.192823+0000","last_active":"2024-09-05T16:34:53.988469+0000","last_peered":"2024-09-05T16:34:53.988469+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:15.910648+0000","last_became_peered":"2024-09-05T16:34:15.910648+0000","last_unstale":"2024-09-05T16:34:53.988469+0000","last_undegraded":"2024-09-05T16:34:15.904665+0000","last_fullsized":"2024-09-05T16:34:15.903277+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2114,"log_dups_size":0,"ondisk_log_size":2114,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-05T16:34:56.762265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17290400,"num_objects":2111,"num_object_clones":0,"num_object_copies":8444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2111,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2111,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2112,"num_write_kb":16893,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2147483647,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2111}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"30'2212","reported_seq":2243,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-09-05T16:34:53.996971+0000","last_change":"2024-09-05T16:34:22.533654+0000","last_active":"2024-09-05T16:34:53.996971+0000","last_peered":"2024-09-05T16:34:53.996971+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:15.913477+0000","last_became_peered":"2024-09-05T16:34:15.913477+0000","last_unstale":"2024-09-05T16:34:53.996971+0000","last_undegraded":"2024-09-05T16:34:22.533443+0000","last_fullsized":"2024-09-05T16:34:15.906631+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2212,"log_dups_size":0,"ondisk_log_size":2212,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T01:56:46.762011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18120704,"num_objects":2212,"num_object_clones":0,"num_object_copies":8848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2212,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2212,"num_write_kb":17696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,2147483647],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2212}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"30'2155","reported_seq":2209,"reported_epoch":30,"state":"peering","last_fresh":"2024-09-05T16:34:54.732728+0000","last_change":"2024-09-05T16:34:54.914099+0000","last_active":"2024-09-05T16:34:54.914099+0000","last_peered":"2024-09-05T16:34:54.732728+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:15.913479+0000","last_became_peered":"2024-09-05T16:34:15.913479+0000","last_unstale":"2024-09-05T16:34:54.732728+0000","last_undegraded":"2024-09-05T16:34:15.905955+0000","last_fullsized":"2024-09-05T16:34:15.905617+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2155,"log_dups_size":0,"ondisk_log_size":2155,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17601696,"num_objects":2149,"num_object_clones":0,"num_object_copies":8596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2149,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2153,"num_write_kb":17204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,2,3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"29'1948","reported_seq":1999,"reported_epoch":29,"state":"active+undersized+degraded","last_fresh":"2024-09-05T16:34:49.905143+0000","last_change":"2024-09-05T16:34:36.231100+0000","last_active":"2024-09-05T16:34:49.905143+0000","last_peered":"2024-09-05T16:34:49.905143+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:15.910570+0000","last_became_peered":"2024-09-05T16:34:15.910570+0000","last_unstale":"2024-09-05T16:34:49.905143+0000","last_undegraded":"2024-09-05T16:34:15.904563+0000","last_fullsized":"2024-09-05T16:34:15.903952+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":1948,"log_dups_size":0,"ondisk_log_size":1948,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-05T16:34:52.808437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15914144,"num_objects":1943,"num_object_clones":0,"num_object_copies":7772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1943,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1943,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1946,"num_write_kb":15556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1943}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"30'2236","reported_seq":2280,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-09-05T16:34:53.976776+0000","last_change":"2024-09-05T16:34:36.789538+0000","last_active":"2024-09-05T16:34:53.976776+0000","last_peered":"2024-09-05T16:34:53.976776+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:15.914297+0000","last_became_peered":"2024-09-05T16:34:15.914297+0000","last_unstale":"2024-09-05T16:34:53.976776+0000","last_undegraded":"2024-09-05T16:34:15.906625+0000","last_fullsized":"2024-09-05T16:34:15.906328+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2236,"log_dups_size":0,"ondisk_log_size":2236,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-05T16:34:58.709828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18245952,"num_objects":2228,"num_object_clones":0,"num_object_copies":8912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2228,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2233,"num_write_kb":17841,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,2147483647],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2228}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"30'2266","reported_seq":2332,"reported_epoch":30,"state":"peering","last_fresh":"2024-09-05T16:34:54.732908+0000","last_change":"2024-09-05T16:34:54.921291+0000","last_active":"2024-09-05T16:34:54.921291+0000","last_peered":"2024-09-05T16:34:54.732908+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:15.918296+0000","last_became_peered":"2024-09-05T16:34:15.918296+0000","last_unstale":"2024-09-05T16:34:54.732908+0000","last_undegraded":"2024-09-05T16:34:15.905187+0000","last_fullsized":"2024-09-05T16:34:15.904834+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2266,"log_dups_size":0,"ondisk_log_size":2266,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18543776,"num_objects":2264,"num_object_clones":0,"num_object_copies":9056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2264,"num_write_kb":18110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,3],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"30'2284","reported_seq":2328,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-09-05T16:34:53.956178+0000","last_change":"2024-09-05T16:34:37.985329+0000","last_active":"2024-09-05T16:34:53.956178+0000","last_peered":"2024-09-05T16:34:53.956178+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:15.914271+0000","last_became_peered":"2024-09-05T16:34:15.914271+0000","last_unstale":"2024-09-05T16:34:53.956178+0000","last_undegraded":"2024-09-05T16:34:15.905903+0000","last_fullsized":"2024-09-05T16:34:15.905357+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2284,"log_dups_size":0,"ondisk_log_size":2284,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18665200,"num_objects":2279,"num_object_clones":0,"num_object_copies":9116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2279,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2281,"num_write_kb":18236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2147483647,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2279}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"30'2231","reported_seq":2256,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-09-05T16:34:54.000022+0000","last_change":"2024-09-05T16:34:38.396192+0000","last_active":"2024-09-05T16:34:54.000022+0000","last_peered":"2024-09-05T16:34:54.000022+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:15.913627+0000","last_became_peered":"2024-09-05T16:34:15.913627+0000","last_unstale":"2024-09-05T16:34:54.000022+0000","last_undegraded":"2024-09-05T16:34:15.906286+0000","last_fullsized":"2024-09-05T16:34:15.906044+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2231,"log_dups_size":0,"ondisk_log_size":2231,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-05T16:34:55.808461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18266704,"num_objects":2230,"num_object_clones":0,"num_object_copies":8920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2230,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2230,"num_write_kb":17839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2230}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"29'2033","reported_seq":2094,"reported_epoch":29,"state":"active+undersized+degraded","last_fresh":"2024-09-05T16:34:50.024184+0000","last_change":"2024-09-05T16:34:38.929130+0000","last_active":"2024-09-05T16:34:50.024184+0000","last_peered":"2024-09-05T16:34:50.024184+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:15.917839+0000","last_became_peered":"2024-09-05T16:34:15.917839+0000","last_unstale":"2024-09-05T16:34:50.024184+0000","last_undegraded":"2024-09-05T16:34:15.904338+0000","last_fullsized":"2024-09-05T16:34:15.904018+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2033,"log_dups_size":0,"ondisk_log_size":2033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-05T16:34:54.794447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16644688,"num_objects":2032,"num_object_clones":0,"num_object_copies":8128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2032,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2032,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2032,"num_write_kb":16255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2032}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"30'2324","reported_seq":2365,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-09-05T16:34:53.940874+0000","last_change":"2024-09-05T16:34:39.373308+0000","last_active":"2024-09-05T16:34:53.940874+0000","last_peered":"2024-09-05T16:34:53.940874+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:15.913175+0000","last_became_peered":"2024-09-05T16:34:15.913175+0000","last_unstale":"2024-09-05T16:34:53.940874+0000","last_undegraded":"2024-09-05T16:34:15.905345+0000","last_fullsized":"2024-09-05T16:34:15.905038+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2324,"log_dups_size":0,"ondisk_log_size":2324,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-05T16:34:57.755827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19002528,"num_objects":2320,"num_object_clones":0,"num_object_copies":9280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2320,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2322,"num_write_kb":18572,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2320}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"29'2057","reported_seq":2121,"reported_epoch":29,"state":"active+undersized+degraded","last_fresh":"2024-09-05T16:34:50.029434+0000","last_change":"2024-09-05T16:34:39.868440+0000","last_active":"2024-09-05T16:34:50.029434+0000","last_peered":"2024-09-05T16:34:50.029434+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:15.917968+0000","last_became_peered":"2024-09-05T16:34:15.917968+0000","last_unstale":"2024-09-05T16:34:50.029434+0000","last_undegraded":"2024-09-05T16:34:15.904385+0000","last_fullsized":"2024-09-05T16:34:15.904040+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2057,"log_dups_size":0,"ondisk_log_size":2057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-05T16:34:51.832737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16823456,"num_objects":2054,"num_object_clones":0,"num_object_copies":8216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2054,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2055,"num_write_kb":16437,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2054}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"29'1963","reported_seq":2014,"reported_epoch":29,"state":"active+undersized+degraded","last_fresh":"2024-09-05T16:34:49.972000+0000","last_change":"2024-09-05T16:34:40.418222+0000","last_active":"2024-09-05T16:34:49.972000+0000","last_peered":"2024-09-05T16:34:49.972000+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:15.910947+0000","last_became_peered":"2024-09-05T16:34:15.910947+0000","last_unstale":"2024-09-05T16:34:49.972000+0000","last_undegraded":"2024-09-05T16:34:15.905253+0000","last_fullsized":"2024-09-05T16:34:15.904912+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":1963,"log_dups_size":0,"ondisk_log_size":1963,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16037024,"num_objects":1958,"num_object_clones":0,"num_object_copies":7832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1958,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1958,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1961,"num_write_kb":15676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1958}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":280600768,"num_objects":34258,"num_object_clones":0,"num_object_copies":137032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27594,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34258,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":34285,"num_write_kb":274162,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":441921536,"data_stored":441921536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":34312,"ondisk_log_size":34312,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084300,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":258976,"kb_used_data":141200,"kb_used_omap":4,"kb_used_meta":117755,"kb_avail":94112864,"statfs":{"total":96636764160,"available":96371572736,"internally_reserved":0,"allocated":144588800,"data_stored":144423343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":120581843},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,1,2,0,8,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":0,"seq":1,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27824,"kb_used_data":288,"kb_used_omap":4,"kb_used_meta":27515,"kb_avail":94344016,"statfs":{"total":96636764160,"available":96608272384,"internally_reserved":0,"allocated":294912,"data_stored":167519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":28176082},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949672983,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":278840,"kb_used_data":144936,"kb_used_omap":1,"kb_used_meta":133886,"kb_avail":94093000,"statfs":{"total":96636764160,"available":96351232000,"internally_reserved":0,"allocated":148414464,"data_stored":148244635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":137099717},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,4,3,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182414,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":275132,"kb_used_data":144936,"kb_used_omap":4,"kb_used_meta":130171,"kb_avail":94096708,"statfs":{"total":96636764160,"available":96355028992,"internally_reserved":0,"allocated":148414464,"data_stored":148244635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":133295821},"hb_peers":[2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,3,5,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":148160512,"data_stored":148160512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":114688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":148160512,"data_stored":148160512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":145485824,"data_stored":145485824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:35:00.411 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:35:00.495 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 38 20 37681 37661 7.74188 2.95312 0.0504153 0.020166 2024-09-05T16:35:00.994 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:35:00.994 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:35:01.022 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":76,"stamp":"2024-09-05T16:35:00.798708+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":306036928,"num_objects":37363,"num_object_clones":0,"num_object_copies":149452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36150,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37363,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":37390,"num_write_kb":299002,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":376832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":37417,"ondisk_log_size":37417,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":916576,"kb_used_data":451608,"kb_used_omap":14,"kb_used_meta":464881,"kb_avail":376570784,"statfs":{"total":386547056640,"available":385608482816,"internally_reserved":0,"allocated":462446592,"data_stored":461715064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":476038711},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,7,7,10,12,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":32505856,"num_objects":3968,"num_object_clones":0,"num_object_copies":15872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2755,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3968,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3968,"num_write_kb":31744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":376832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001388"},"pg_stats":[{"pgid":"1.f","version":"33'2306","reported_seq":2347,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:34:58.939236+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:34:58.939236+0000","last_peered":"2024-09-05T16:34:58.939236+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:34:58.939236+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2306,"log_dups_size":0,"ondisk_log_size":2306,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18890752,"num_objects":2306,"num_object_clones":0,"num_object_copies":9224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2306,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2306,"num_write_kb":18448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":44}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'2345","reported_seq":2407,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:34:59.997335+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:34:59.997335+0000","last_peered":"2024-09-05T16:34:59.997335+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:34:59.997335+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2345,"log_dups_size":0,"ondisk_log_size":2345,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19174560,"num_objects":2341,"num_object_clones":0,"num_object_copies":9364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2341,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2343,"num_write_kb":18740,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":74}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'2348","reported_seq":2422,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:00.024763+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:35:00.024763+0000","last_peered":"2024-09-05T16:35:00.024763+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:35:00.024763+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2348,"log_dups_size":0,"ondisk_log_size":2348,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19208784,"num_objects":2345,"num_object_clones":0,"num_object_copies":9380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2345,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2347,"num_write_kb":18766,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":95}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'2364","reported_seq":2520,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:34:59.948794+0000","last_change":"2024-09-05T16:34:56.800371+0000","last_active":"2024-09-05T16:34:59.948794+0000","last_peered":"2024-09-05T16:34:59.948794+0000","last_clean":"2024-09-05T16:33:53.148717+0000","last_became_active":"2024-09-05T16:34:56.791755+0000","last_became_peered":"2024-09-05T16:34:56.791755+0000","last_unstale":"2024-09-05T16:34:59.948794+0000","last_undegraded":"2024-09-05T16:34:56.597522+0000","last_fullsized":"2024-09-05T16:34:56.568150+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2364,"log_dups_size":0,"ondisk_log_size":2364,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19330208,"num_objects":2360,"num_object_clones":0,"num_object_copies":9440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2242,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2360,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2362,"num_write_kb":18885,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":376832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":118}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'2229","reported_seq":2282,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:34:58.999015+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:34:58.999015+0000","last_peered":"2024-09-05T16:34:58.999015+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:34:58.999015+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2229,"log_dups_size":0,"ondisk_log_size":2229,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18232480,"num_objects":2226,"num_object_clones":0,"num_object_copies":8904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2226,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2227,"num_write_kb":17813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":64}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'2319","reported_seq":2360,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:34:59.029565+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:34:59.029565+0000","last_peered":"2024-09-05T16:34:59.029565+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:34:59.029565+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2319,"log_dups_size":0,"ondisk_log_size":2319,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18997248,"num_objects":2319,"num_object_clones":0,"num_object_copies":9276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2319,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2319,"num_write_kb":18552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":69}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'2244","reported_seq":2306,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:34:59.957086+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:34:59.957086+0000","last_peered":"2024-09-05T16:34:59.957086+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:34:59.957086+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2244,"log_dups_size":0,"ondisk_log_size":2244,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18330784,"num_objects":2238,"num_object_clones":0,"num_object_copies":8952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2238,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2242,"num_write_kb":17916,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":91}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'2339","reported_seq":2402,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:34:59.791695+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:34:59.791695+0000","last_peered":"2024-09-05T16:34:59.791695+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:34:59.791695+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2339,"log_dups_size":0,"ondisk_log_size":2339,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19117216,"num_objects":2334,"num_object_clones":0,"num_object_copies":9336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2334,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2337,"num_write_kb":18684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":95}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'2316","reported_seq":2371,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:34:58.890394+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:34:58.890394+0000","last_peered":"2024-09-05T16:34:58.890394+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:34:58.890394+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2316,"log_dups_size":0,"ondisk_log_size":2316,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18901312,"num_objects":2308,"num_object_clones":0,"num_object_copies":9232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2308,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2313,"num_write_kb":18481,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":47}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'2358","reported_seq":2433,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:00.065067+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:35:00.065067+0000","last_peered":"2024-09-05T16:35:00.065067+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:35:00.065067+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2358,"log_dups_size":0,"ondisk_log_size":2358,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19297440,"num_objects":2356,"num_object_clones":0,"num_object_copies":9424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2356,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2356,"num_write_kb":18846,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":94}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'2384","reported_seq":2439,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:34:58.980479+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:34:58.980479+0000","last_peered":"2024-09-05T16:34:58.980479+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:34:58.980479+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2384,"log_dups_size":0,"ondisk_log_size":2384,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19484400,"num_objects":2379,"num_object_clones":0,"num_object_copies":9516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2379,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2381,"num_write_kb":19036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":68}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'2338","reported_seq":2373,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:34:58.976094+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:34:58.976094+0000","last_peered":"2024-09-05T16:34:58.976094+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:34:58.976094+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2338,"log_dups_size":0,"ondisk_log_size":2338,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19143248,"num_objects":2337,"num_object_clones":0,"num_object_copies":9348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2337,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2337,"num_write_kb":18695,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":73}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'2386","reported_seq":2459,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:00.063610+0000","last_change":"2024-09-05T16:34:56.798081+0000","last_active":"2024-09-05T16:35:00.063610+0000","last_peered":"2024-09-05T16:35:00.063610+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:35:00.063610+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2386,"log_dups_size":0,"ondisk_log_size":2386,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19536464,"num_objects":2385,"num_object_clones":0,"num_object_copies":9540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2385,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2385,"num_write_kb":19079,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":91}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'2427","reported_seq":2478,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:34:58.941919+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:34:58.941919+0000","last_peered":"2024-09-05T16:34:58.941919+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:34:58.941919+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2427,"log_dups_size":0,"ondisk_log_size":2427,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19846304,"num_objects":2423,"num_object_clones":0,"num_object_copies":9692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2423,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2425,"num_write_kb":19396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":66}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'2394","reported_seq":2471,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:00.006919+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:35:00.006919+0000","last_peered":"2024-09-05T16:35:00.006919+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:35:00.006919+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2394,"log_dups_size":0,"ondisk_log_size":2394,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19584160,"num_objects":2391,"num_object_clones":0,"num_object_copies":9564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2392,"num_write_kb":19133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":71}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'2320","reported_seq":2382,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:34:59.968863+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:34:59.968863+0000","last_peered":"2024-09-05T16:34:59.968863+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:34:59.968863+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2320,"log_dups_size":0,"ondisk_log_size":2320,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18961568,"num_objects":2315,"num_object_clones":0,"num_object_copies":9260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2315,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2318,"num_write_kb":18532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":78}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":306036928,"num_objects":37363,"num_object_clones":0,"num_object_copies":149452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36150,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37363,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":37390,"num_write_kb":299002,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":376832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":463192064,"data_stored":463192064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":37417,"ondisk_log_size":37417,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084301,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":292528,"kb_used_data":148380,"kb_used_omap":4,"kb_used_meta":144123,"kb_avail":94079312,"statfs":{"total":96636764160,"available":96337215488,"internally_reserved":0,"allocated":151941120,"data_stored":151758238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":147582675},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,3,1,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986178,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53536,"kb_used_data":3788,"kb_used_omap":4,"kb_used_meta":49723,"kb_avail":94318304,"statfs":{"total":96636764160,"available":96581943296,"internally_reserved":0,"allocated":3878912,"data_stored":3696030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":50917074},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949672984,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":285072,"kb_used_data":149760,"kb_used_omap":1,"kb_used_meta":135294,"kb_avail":94086768,"statfs":{"total":96636764160,"available":96344850432,"internally_reserved":0,"allocated":153354240,"data_stored":153171358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":138541509},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,4,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182415,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":285440,"kb_used_data":149680,"kb_used_omap":4,"kb_used_meta":135739,"kb_avail":94086400,"statfs":{"total":96636764160,"available":96344473600,"internally_reserved":0,"allocated":153272320,"data_stored":153089438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":138997453},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,8,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":153546752,"data_stored":153546752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3932160,"data_stored":3932160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":153681920,"data_stored":153681920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":152031232,"data_stored":152031232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:35:01.023 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-05T16:35:01.023 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'2306", 'reported_seq': 2347, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:34:58.939236+0000', 'last_change': '2024-09-05T16:34:56.833497+0000', 'last_active': '2024-09-05T16:34:58.939236+0000', 'last_peered': '2024-09-05T16:34:58.939236+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.828289+0000', 'last_became_peered': '2024-09-05T16:34:56.828289+0000', 'last_unstale': '2024-09-05T16:34:58.939236+0000', 'last_undegraded': '2024-09-05T16:34:56.612160+0000', 'last_fullsized': '2024-09-05T16:34:56.557632+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 2306, 'log_dups_size': 0, 'ondisk_log_size': 2306, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18890752, 'num_objects': 2306, 'num_object_clones': 0, 'num_object_copies': 9224, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2262, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2306, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2306, 'num_write_kb': 18448, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 44}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:35:01.023 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-05T16:35:01.023 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'2345", 'reported_seq': 2407, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:34:59.997335+0000', 'last_change': '2024-09-05T16:34:56.825031+0000', 'last_active': '2024-09-05T16:34:59.997335+0000', 'last_peered': '2024-09-05T16:34:59.997335+0000', 'last_clean': '2024-09-05T16:33:53.200762+0000', 'last_became_active': '2024-09-05T16:34:56.820292+0000', 'last_became_peered': '2024-09-05T16:34:56.820292+0000', 'last_unstale': '2024-09-05T16:34:59.997335+0000', 'last_undegraded': '2024-09-05T16:34:56.605776+0000', 'last_fullsized': '2024-09-05T16:34:56.559383+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 2345, 'log_dups_size': 0, 'ondisk_log_size': 2345, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19174560, 'num_objects': 2341, 'num_object_clones': 0, 'num_object_copies': 9364, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2269, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2341, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2343, 'num_write_kb': 18740, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 74}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:35:01.024 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:35:01.024 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'2348", 'reported_seq': 2422, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:00.024763+0000', 'last_change': '2024-09-05T16:34:56.867900+0000', 'last_active': '2024-09-05T16:35:00.024763+0000', 'last_peered': '2024-09-05T16:35:00.024763+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:35:00.024763+0000', 'last_undegraded': '2024-09-05T16:34:56.590972+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 2348, 'log_dups_size': 0, 'ondisk_log_size': 2348, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19208784, 'num_objects': 2345, 'num_object_clones': 0, 'num_object_copies': 9380, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2251, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2345, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2347, 'num_write_kb': 18766, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 95}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:35:01.024 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-05T16:35:01.024 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'2364", 'reported_seq': 2520, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:34:59.948794+0000', 'last_change': '2024-09-05T16:34:56.800371+0000', 'last_active': '2024-09-05T16:34:59.948794+0000', 'last_peered': '2024-09-05T16:34:59.948794+0000', 'last_clean': '2024-09-05T16:33:53.148717+0000', 'last_became_active': '2024-09-05T16:34:56.791755+0000', 'last_became_peered': '2024-09-05T16:34:56.791755+0000', 'last_unstale': '2024-09-05T16:34:59.948794+0000', 'last_undegraded': '2024-09-05T16:34:56.597522+0000', 'last_fullsized': '2024-09-05T16:34:56.568150+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 2364, 'log_dups_size': 0, 'ondisk_log_size': 2364, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19330208, 'num_objects': 2360, 'num_object_clones': 0, 'num_object_copies': 9440, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2242, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2360, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2362, 'num_write_kb': 18885, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 48, 'num_bytes_recovered': 376832, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 118}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:35:01.024 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:35:01.024 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'2229", 'reported_seq': 2282, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:34:58.999015+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:34:58.999015+0000', 'last_peered': '2024-09-05T16:34:58.999015+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:34:58.999015+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 2229, 'log_dups_size': 0, 'ondisk_log_size': 2229, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18232480, 'num_objects': 2226, 'num_object_clones': 0, 'num_object_copies': 8904, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2164, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2226, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2227, 'num_write_kb': 17813, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 64}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:35:01.024 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:35:01.024 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'2319", 'reported_seq': 2360, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:34:59.029565+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:34:59.029565+0000', 'last_peered': '2024-09-05T16:34:59.029565+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:34:59.029565+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 2319, 'log_dups_size': 0, 'ondisk_log_size': 2319, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18997248, 'num_objects': 2319, 'num_object_clones': 0, 'num_object_copies': 9276, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2319, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2319, 'num_write_kb': 18552, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 69}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:35:01.024 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:35:01.024 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'2244", 'reported_seq': 2306, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:34:59.957086+0000', 'last_change': '2024-09-05T16:34:56.903436+0000', 'last_active': '2024-09-05T16:34:59.957086+0000', 'last_peered': '2024-09-05T16:34:59.957086+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:34:59.957086+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 2244, 'log_dups_size': 0, 'ondisk_log_size': 2244, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18330784, 'num_objects': 2238, 'num_object_clones': 0, 'num_object_copies': 8952, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2149, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2238, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2242, 'num_write_kb': 17916, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 91}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:35:01.025 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-05T16:35:01.025 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'2339", 'reported_seq': 2402, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:34:59.791695+0000', 'last_change': '2024-09-05T16:34:56.883319+0000', 'last_active': '2024-09-05T16:34:59.791695+0000', 'last_peered': '2024-09-05T16:34:59.791695+0000', 'last_clean': '2024-09-05T16:33:53.505305+0000', 'last_became_active': '2024-09-05T16:34:56.873389+0000', 'last_became_peered': '2024-09-05T16:34:56.873389+0000', 'last_unstale': '2024-09-05T16:34:59.791695+0000', 'last_undegraded': '2024-09-05T16:34:56.602567+0000', 'last_fullsized': '2024-09-05T16:34:56.559958+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 2339, 'log_dups_size': 0, 'ondisk_log_size': 2339, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19117216, 'num_objects': 2334, 'num_object_clones': 0, 'num_object_copies': 9336, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2241, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2334, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2337, 'num_write_kb': 18684, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 95}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:35:01.025 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:35:01.025 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'2316", 'reported_seq': 2371, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:34:58.890394+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:34:58.890394+0000', 'last_peered': '2024-09-05T16:34:58.890394+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:34:58.890394+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 2316, 'log_dups_size': 0, 'ondisk_log_size': 2316, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18901312, 'num_objects': 2308, 'num_object_clones': 0, 'num_object_copies': 9232, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2264, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2308, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2313, 'num_write_kb': 18481, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 47}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:35:01.025 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:35:01.025 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'2358", 'reported_seq': 2433, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:00.065067+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:35:00.065067+0000', 'last_peered': '2024-09-05T16:35:00.065067+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:35:00.065067+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 2358, 'log_dups_size': 0, 'ondisk_log_size': 2358, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19297440, 'num_objects': 2356, 'num_object_clones': 0, 'num_object_copies': 9424, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2264, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2356, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2356, 'num_write_kb': 18846, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 94}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:35:01.025 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:35:01.025 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'2384", 'reported_seq': 2439, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:34:58.980479+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:34:58.980479+0000', 'last_peered': '2024-09-05T16:34:58.980479+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:34:58.980479+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 2384, 'log_dups_size': 0, 'ondisk_log_size': 2384, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19484400, 'num_objects': 2379, 'num_object_clones': 0, 'num_object_copies': 9516, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2314, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2379, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2381, 'num_write_kb': 19036, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 68}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:35:01.025 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:35:01.025 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'2338", 'reported_seq': 2373, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:34:58.976094+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:34:58.976094+0000', 'last_peered': '2024-09-05T16:34:58.976094+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:34:58.976094+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 2338, 'log_dups_size': 0, 'ondisk_log_size': 2338, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19143248, 'num_objects': 2337, 'num_object_clones': 0, 'num_object_copies': 9348, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2265, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2337, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2337, 'num_write_kb': 18695, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 73}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:35:01.025 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-05T16:35:01.026 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'2386", 'reported_seq': 2459, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:00.063610+0000', 'last_change': '2024-09-05T16:34:56.798081+0000', 'last_active': '2024-09-05T16:35:00.063610+0000', 'last_peered': '2024-09-05T16:35:00.063610+0000', 'last_clean': '2024-09-05T16:33:45.082039+0000', 'last_became_active': '2024-09-05T16:34:56.792386+0000', 'last_became_peered': '2024-09-05T16:34:56.792386+0000', 'last_unstale': '2024-09-05T16:35:00.063610+0000', 'last_undegraded': '2024-09-05T16:34:56.600595+0000', 'last_fullsized': '2024-09-05T16:34:56.558203+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 2386, 'log_dups_size': 0, 'ondisk_log_size': 2386, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19536464, 'num_objects': 2385, 'num_object_clones': 0, 'num_object_copies': 9540, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2295, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2385, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2385, 'num_write_kb': 19079, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 91}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:35:01.026 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:35:01.026 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'2427", 'reported_seq': 2478, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:34:58.941919+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:34:58.941919+0000', 'last_peered': '2024-09-05T16:34:58.941919+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:34:58.941919+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 2427, 'log_dups_size': 0, 'ondisk_log_size': 2427, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19846304, 'num_objects': 2423, 'num_object_clones': 0, 'num_object_copies': 9692, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2359, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2423, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2425, 'num_write_kb': 19396, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 66}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:35:01.026 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:35:01.026 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'2394", 'reported_seq': 2471, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:00.006919+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:35:00.006919+0000', 'last_peered': '2024-09-05T16:35:00.006919+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:35:00.006919+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 2394, 'log_dups_size': 0, 'ondisk_log_size': 2394, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19584160, 'num_objects': 2391, 'num_object_clones': 0, 'num_object_copies': 9564, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2322, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2391, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2392, 'num_write_kb': 19133, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 71}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:35:01.026 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:35:01.026 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'2320", 'reported_seq': 2382, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:34:59.968863+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:34:59.968863+0000', 'last_peered': '2024-09-05T16:34:59.968863+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:34:59.968863+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 2320, 'log_dups_size': 0, 'ondisk_log_size': 2320, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18961568, 'num_objects': 2315, 'num_object_clones': 0, 'num_object_copies': 9260, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2315, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2318, 'num_write_kb': 18532, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 78}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:35:01.026 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:35:01.494 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 39 20 38076 38056 7.6225 3.08594 0.0342194 0.0204799 2024-09-05T16:35:01.794 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:35:01.794 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 36150/149452 objects degraded (24.188%), 16 pgs degraded","count":16},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":105,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":37363,"data_bytes":306036928,"bytes_used":938573824,"bytes_avail":385608482816,"bytes_total":386547056640,"degraded_objects":36150,"degraded_total":149452,"degraded_ratio":0.24188368171720687,"recovering_objects_per_sec":7,"recovering_bytes_per_sec":62790,"recovering_keys_per_sec":0,"num_objects_recovered":46,"num_bytes_recovered":376832,"num_keys_recovered":0,"write_bytes_sec":5416389,"read_op_per_sec":0,"write_op_per_sec":661},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-05T16:34:32.790751+0000","services":{"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"3":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-05T16:35:01.823 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:35:01.824 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:35:02.410 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:35:02.411 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:35:02.433 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":76,"stamp":"2024-09-05T16:35:00.798708+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":306036928,"num_objects":37363,"num_object_clones":0,"num_object_copies":149452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36150,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37363,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":37390,"num_write_kb":299002,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":376832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":37417,"ondisk_log_size":37417,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":916576,"kb_used_data":451608,"kb_used_omap":14,"kb_used_meta":464881,"kb_avail":376570784,"statfs":{"total":386547056640,"available":385608482816,"internally_reserved":0,"allocated":462446592,"data_stored":461715064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":476038711},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,7,7,10,12,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":32505856,"num_objects":3968,"num_object_clones":0,"num_object_copies":15872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2755,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3968,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3968,"num_write_kb":31744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":376832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001388"},"pg_stats":[{"pgid":"1.f","version":"33'2306","reported_seq":2347,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:34:58.939236+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:34:58.939236+0000","last_peered":"2024-09-05T16:34:58.939236+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:34:58.939236+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2306,"log_dups_size":0,"ondisk_log_size":2306,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18890752,"num_objects":2306,"num_object_clones":0,"num_object_copies":9224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2306,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2306,"num_write_kb":18448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":44}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'2345","reported_seq":2407,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:34:59.997335+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:34:59.997335+0000","last_peered":"2024-09-05T16:34:59.997335+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:34:59.997335+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2345,"log_dups_size":0,"ondisk_log_size":2345,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19174560,"num_objects":2341,"num_object_clones":0,"num_object_copies":9364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2341,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2343,"num_write_kb":18740,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":74}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'2348","reported_seq":2422,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:00.024763+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:35:00.024763+0000","last_peered":"2024-09-05T16:35:00.024763+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:35:00.024763+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2348,"log_dups_size":0,"ondisk_log_size":2348,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19208784,"num_objects":2345,"num_object_clones":0,"num_object_copies":9380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2345,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2347,"num_write_kb":18766,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":95}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'2364","reported_seq":2520,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:34:59.948794+0000","last_change":"2024-09-05T16:34:56.800371+0000","last_active":"2024-09-05T16:34:59.948794+0000","last_peered":"2024-09-05T16:34:59.948794+0000","last_clean":"2024-09-05T16:33:53.148717+0000","last_became_active":"2024-09-05T16:34:56.791755+0000","last_became_peered":"2024-09-05T16:34:56.791755+0000","last_unstale":"2024-09-05T16:34:59.948794+0000","last_undegraded":"2024-09-05T16:34:56.597522+0000","last_fullsized":"2024-09-05T16:34:56.568150+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2364,"log_dups_size":0,"ondisk_log_size":2364,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19330208,"num_objects":2360,"num_object_clones":0,"num_object_copies":9440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2242,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2360,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2362,"num_write_kb":18885,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":376832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":118}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'2229","reported_seq":2282,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:34:58.999015+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:34:58.999015+0000","last_peered":"2024-09-05T16:34:58.999015+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:34:58.999015+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2229,"log_dups_size":0,"ondisk_log_size":2229,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18232480,"num_objects":2226,"num_object_clones":0,"num_object_copies":8904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2226,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2227,"num_write_kb":17813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":64}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'2319","reported_seq":2360,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:34:59.029565+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:34:59.029565+0000","last_peered":"2024-09-05T16:34:59.029565+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:34:59.029565+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2319,"log_dups_size":0,"ondisk_log_size":2319,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18997248,"num_objects":2319,"num_object_clones":0,"num_object_copies":9276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2319,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2319,"num_write_kb":18552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":69}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'2244","reported_seq":2306,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:34:59.957086+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:34:59.957086+0000","last_peered":"2024-09-05T16:34:59.957086+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:34:59.957086+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2244,"log_dups_size":0,"ondisk_log_size":2244,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18330784,"num_objects":2238,"num_object_clones":0,"num_object_copies":8952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2238,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2242,"num_write_kb":17916,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":91}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'2339","reported_seq":2402,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:34:59.791695+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:34:59.791695+0000","last_peered":"2024-09-05T16:34:59.791695+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:34:59.791695+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2339,"log_dups_size":0,"ondisk_log_size":2339,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19117216,"num_objects":2334,"num_object_clones":0,"num_object_copies":9336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2334,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2337,"num_write_kb":18684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":95}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'2316","reported_seq":2371,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:34:58.890394+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:34:58.890394+0000","last_peered":"2024-09-05T16:34:58.890394+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:34:58.890394+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2316,"log_dups_size":0,"ondisk_log_size":2316,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18901312,"num_objects":2308,"num_object_clones":0,"num_object_copies":9232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2308,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2313,"num_write_kb":18481,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":47}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'2358","reported_seq":2433,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:00.065067+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:35:00.065067+0000","last_peered":"2024-09-05T16:35:00.065067+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:35:00.065067+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2358,"log_dups_size":0,"ondisk_log_size":2358,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19297440,"num_objects":2356,"num_object_clones":0,"num_object_copies":9424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2356,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2356,"num_write_kb":18846,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":94}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'2384","reported_seq":2439,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:34:58.980479+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:34:58.980479+0000","last_peered":"2024-09-05T16:34:58.980479+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:34:58.980479+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2384,"log_dups_size":0,"ondisk_log_size":2384,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19484400,"num_objects":2379,"num_object_clones":0,"num_object_copies":9516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2379,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2381,"num_write_kb":19036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":68}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'2338","reported_seq":2373,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:34:58.976094+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:34:58.976094+0000","last_peered":"2024-09-05T16:34:58.976094+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:34:58.976094+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2338,"log_dups_size":0,"ondisk_log_size":2338,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19143248,"num_objects":2337,"num_object_clones":0,"num_object_copies":9348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2337,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2337,"num_write_kb":18695,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":73}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'2386","reported_seq":2459,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:00.063610+0000","last_change":"2024-09-05T16:34:56.798081+0000","last_active":"2024-09-05T16:35:00.063610+0000","last_peered":"2024-09-05T16:35:00.063610+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:35:00.063610+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2386,"log_dups_size":0,"ondisk_log_size":2386,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19536464,"num_objects":2385,"num_object_clones":0,"num_object_copies":9540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2385,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2385,"num_write_kb":19079,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":91}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'2427","reported_seq":2478,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:34:58.941919+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:34:58.941919+0000","last_peered":"2024-09-05T16:34:58.941919+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:34:58.941919+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2427,"log_dups_size":0,"ondisk_log_size":2427,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19846304,"num_objects":2423,"num_object_clones":0,"num_object_copies":9692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2423,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2425,"num_write_kb":19396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":66}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'2394","reported_seq":2471,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:00.006919+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:35:00.006919+0000","last_peered":"2024-09-05T16:35:00.006919+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:35:00.006919+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2394,"log_dups_size":0,"ondisk_log_size":2394,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19584160,"num_objects":2391,"num_object_clones":0,"num_object_copies":9564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2392,"num_write_kb":19133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":71}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'2320","reported_seq":2382,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:34:59.968863+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:34:59.968863+0000","last_peered":"2024-09-05T16:34:59.968863+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:34:59.968863+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2320,"log_dups_size":0,"ondisk_log_size":2320,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18961568,"num_objects":2315,"num_object_clones":0,"num_object_copies":9260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2315,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2318,"num_write_kb":18532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":78}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":306036928,"num_objects":37363,"num_object_clones":0,"num_object_copies":149452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36150,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37363,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":37390,"num_write_kb":299002,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":376832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":463192064,"data_stored":463192064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":37417,"ondisk_log_size":37417,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084301,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":292528,"kb_used_data":148380,"kb_used_omap":4,"kb_used_meta":144123,"kb_avail":94079312,"statfs":{"total":96636764160,"available":96337215488,"internally_reserved":0,"allocated":151941120,"data_stored":151758238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":147582675},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,3,1,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986178,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53536,"kb_used_data":3788,"kb_used_omap":4,"kb_used_meta":49723,"kb_avail":94318304,"statfs":{"total":96636764160,"available":96581943296,"internally_reserved":0,"allocated":3878912,"data_stored":3696030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":50917074},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949672984,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":285072,"kb_used_data":149760,"kb_used_omap":1,"kb_used_meta":135294,"kb_avail":94086768,"statfs":{"total":96636764160,"available":96344850432,"internally_reserved":0,"allocated":153354240,"data_stored":153171358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":138541509},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,4,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182415,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":285440,"kb_used_data":149680,"kb_used_omap":4,"kb_used_meta":135739,"kb_avail":94086400,"statfs":{"total":96636764160,"available":96344473600,"internally_reserved":0,"allocated":153272320,"data_stored":153089438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":138997453},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,8,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":153546752,"data_stored":153546752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3932160,"data_stored":3932160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":153681920,"data_stored":153681920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":152031232,"data_stored":152031232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:35:02.494 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout:2024-09-05T16:35:02.501840+0000 min lat: 0.00357551 max lat: 2.29705 avg lat: 0.0207945 2024-09-05T16:35:02.494 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-05T16:35:02.494 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 40 20 38472 38452 7.50928 3.09375 0.0323625 0.0207945 2024-09-05T16:35:03.494 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 41 20 38911 38891 7.40977 3.42969 0.0649233 0.0210682 2024-09-05T16:35:04.494 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 42 20 39352 39332 7.31537 3.44531 0.0529813 0.0213435 2024-09-05T16:35:05.435 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:35:05.494 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 43 20 39794 39774 7.22555 3.45312 0.0299138 0.0216091 2024-09-05T16:35:06.006 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:35:06.006 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:35:06.041 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":78,"stamp":"2024-09-05T16:35:04.799639+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":313188544,"num_objects":38236,"num_object_clones":0,"num_object_copies":152944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36150,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":38263,"num_write_kb":305986,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":376832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38290,"ondisk_log_size":38290,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":941292,"kb_used_data":468008,"kb_used_omap":14,"kb_used_meta":473201,"kb_avail":376546068,"statfs":{"total":386547056640,"available":385583173632,"internally_reserved":0,"allocated":479240192,"data_stored":478508664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":484558391},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,2,6,15,22,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":32587776,"num_objects":3978,"num_object_clones":0,"num_object_copies":15912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8556,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3978,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3978,"num_write_kb":31824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":376832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.245972"},"pg_stats":[{"pgid":"1.f","version":"33'2415","reported_seq":2456,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:03.821071+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:35:03.821071+0000","last_peered":"2024-09-05T16:35:03.821071+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:35:03.821071+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2415,"log_dups_size":0,"ondisk_log_size":2415,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19783680,"num_objects":2415,"num_object_clones":0,"num_object_copies":9660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2415,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2415,"num_write_kb":19320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":153}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'2345","reported_seq":2407,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:34:59.997335+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:34:59.997335+0000","last_peered":"2024-09-05T16:34:59.997335+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:34:59.997335+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2345,"log_dups_size":0,"ondisk_log_size":2345,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19174560,"num_objects":2341,"num_object_clones":0,"num_object_copies":9364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2341,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2343,"num_write_kb":18740,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":74}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'2348","reported_seq":2422,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:00.024763+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:35:00.024763+0000","last_peered":"2024-09-05T16:35:00.024763+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:35:00.024763+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2348,"log_dups_size":0,"ondisk_log_size":2348,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19208784,"num_objects":2345,"num_object_clones":0,"num_object_copies":9380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2345,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2347,"num_write_kb":18766,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":95}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'2364","reported_seq":2520,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:34:59.948794+0000","last_change":"2024-09-05T16:34:56.800371+0000","last_active":"2024-09-05T16:34:59.948794+0000","last_peered":"2024-09-05T16:34:59.948794+0000","last_clean":"2024-09-05T16:33:53.148717+0000","last_became_active":"2024-09-05T16:34:56.791755+0000","last_became_peered":"2024-09-05T16:34:56.791755+0000","last_unstale":"2024-09-05T16:34:59.948794+0000","last_undegraded":"2024-09-05T16:34:56.597522+0000","last_fullsized":"2024-09-05T16:34:56.568150+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2364,"log_dups_size":0,"ondisk_log_size":2364,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19330208,"num_objects":2360,"num_object_clones":0,"num_object_copies":9440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2242,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2360,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2362,"num_write_kb":18885,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":376832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":118}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'2346","reported_seq":2399,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:03.967181+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:35:03.967181+0000","last_peered":"2024-09-05T16:35:03.967181+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:35:03.967181+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2346,"log_dups_size":0,"ondisk_log_size":2346,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19190944,"num_objects":2343,"num_object_clones":0,"num_object_copies":9372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2343,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2344,"num_write_kb":18749,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":181}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'2443","reported_seq":2484,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:04.012497+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:35:04.012497+0000","last_peered":"2024-09-05T16:35:04.012497+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:35:04.012497+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2443,"log_dups_size":0,"ondisk_log_size":2443,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20013056,"num_objects":2443,"num_object_clones":0,"num_object_copies":9772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2443,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2443,"num_write_kb":19544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":193}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'2244","reported_seq":2306,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:34:59.957086+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:34:59.957086+0000","last_peered":"2024-09-05T16:34:59.957086+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:34:59.957086+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2244,"log_dups_size":0,"ondisk_log_size":2244,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18330784,"num_objects":2238,"num_object_clones":0,"num_object_copies":8952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2238,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2242,"num_write_kb":17916,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":91}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'2339","reported_seq":2402,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:34:59.791695+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:34:59.791695+0000","last_peered":"2024-09-05T16:34:59.791695+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:34:59.791695+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2339,"log_dups_size":0,"ondisk_log_size":2339,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19117216,"num_objects":2334,"num_object_clones":0,"num_object_copies":9336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2334,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2337,"num_write_kb":18684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":95}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'2452","reported_seq":2505,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:04.021376+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:35:04.021376+0000","last_peered":"2024-09-05T16:35:04.021376+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:35:04.021376+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2452,"log_dups_size":0,"ondisk_log_size":2452,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20015424,"num_objects":2444,"num_object_clones":0,"num_object_copies":9776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2444,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2449,"num_write_kb":19569,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":183}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'2358","reported_seq":2433,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:00.065067+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:35:00.065067+0000","last_peered":"2024-09-05T16:35:00.065067+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:35:00.065067+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2358,"log_dups_size":0,"ondisk_log_size":2358,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19297440,"num_objects":2356,"num_object_clones":0,"num_object_copies":9424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2356,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2356,"num_write_kb":18846,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":94}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'2510","reported_seq":2565,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:04.007092+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:35:04.007092+0000","last_peered":"2024-09-05T16:35:04.007092+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:35:04.007092+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2510,"log_dups_size":0,"ondisk_log_size":2510,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20516592,"num_objects":2505,"num_object_clones":0,"num_object_copies":10020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2505,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2507,"num_write_kb":20044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":194}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'2465","reported_seq":2499,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:04.011824+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:35:04.011824+0000","last_peered":"2024-09-05T16:35:04.011824+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:35:04.011824+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2465,"log_dups_size":0,"ondisk_log_size":2465,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20183632,"num_objects":2464,"num_object_clones":0,"num_object_copies":9856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2464,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2464,"num_write_kb":19711,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":200}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'2386","reported_seq":2459,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:00.063610+0000","last_change":"2024-09-05T16:34:56.798081+0000","last_active":"2024-09-05T16:35:00.063610+0000","last_peered":"2024-09-05T16:35:00.063610+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:35:00.063610+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2386,"log_dups_size":0,"ondisk_log_size":2386,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19536464,"num_objects":2385,"num_object_clones":0,"num_object_copies":9540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2385,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2385,"num_write_kb":19079,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":91}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'2561","reported_seq":2612,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:03.914196+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:35:03.914196+0000","last_peered":"2024-09-05T16:35:03.914196+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:35:03.914196+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2561,"log_dups_size":0,"ondisk_log_size":2561,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20944032,"num_objects":2557,"num_object_clones":0,"num_object_copies":10228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2557,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2559,"num_write_kb":20468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":200}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'2394","reported_seq":2471,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:00.006919+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:35:00.006919+0000","last_peered":"2024-09-05T16:35:00.006919+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:35:00.006919+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2394,"log_dups_size":0,"ondisk_log_size":2394,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19584160,"num_objects":2391,"num_object_clones":0,"num_object_copies":9564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2392,"num_write_kb":19133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":71}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'2320","reported_seq":2382,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:34:59.968863+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:34:59.968863+0000","last_peered":"2024-09-05T16:34:59.968863+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:34:59.968863+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2320,"log_dups_size":0,"ondisk_log_size":2320,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18961568,"num_objects":2315,"num_object_clones":0,"num_object_copies":9260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2315,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2318,"num_write_kb":18532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":78}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":313188544,"num_objects":38236,"num_object_clones":0,"num_object_copies":152944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36150,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":38263,"num_write_kb":305986,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":376832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":480169984,"data_stored":480169984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38290,"ondisk_log_size":38290,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084302,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":304748,"kb_used_data":156444,"kb_used_omap":4,"kb_used_meta":148283,"kb_avail":94067092,"statfs":{"total":96636764160,"available":96324702208,"internally_reserved":0,"allocated":160198656,"data_stored":160015774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":151842515},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,3,6,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986179,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66032,"kb_used_data":12124,"kb_used_omap":4,"kb_used_meta":53883,"kb_avail":94305808,"statfs":{"total":96636764160,"available":96569147392,"internally_reserved":0,"allocated":12414976,"data_stored":12232094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":55176914},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,1,1,4,7],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949672984,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":285072,"kb_used_data":149760,"kb_used_omap":1,"kb_used_meta":135294,"kb_avail":94086768,"statfs":{"total":96636764160,"available":96344850432,"internally_reserved":0,"allocated":153354240,"data_stored":153171358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":138541509},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,4,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182415,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":285440,"kb_used_data":149680,"kb_used_omap":4,"kb_used_meta":135739,"kb_avail":94086400,"statfs":{"total":96636764160,"available":96344473600,"internally_reserved":0,"allocated":153272320,"data_stored":153089438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":138997453},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,8,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":153546752,"data_stored":153546752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12550144,"data_stored":12550144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":153681920,"data_stored":153681920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":160391168,"data_stored":160391168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:35:06.042 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:35:06.494 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 44 20 40162 40142 7.12667 2.875 0.0534686 0.0219066 2024-09-05T16:35:06.625 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:35:06.626 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:35:06.647 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":78,"stamp":"2024-09-05T16:35:04.799639+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":313188544,"num_objects":38236,"num_object_clones":0,"num_object_copies":152944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36150,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":38263,"num_write_kb":305986,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":376832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38290,"ondisk_log_size":38290,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":941292,"kb_used_data":468008,"kb_used_omap":14,"kb_used_meta":473201,"kb_avail":376546068,"statfs":{"total":386547056640,"available":385583173632,"internally_reserved":0,"allocated":479240192,"data_stored":478508664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":484558391},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,2,6,15,22,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":32587776,"num_objects":3978,"num_object_clones":0,"num_object_copies":15912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8556,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3978,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3978,"num_write_kb":31824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":376832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.245972"},"pg_stats":[{"pgid":"1.f","version":"33'2415","reported_seq":2456,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:03.821071+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:35:03.821071+0000","last_peered":"2024-09-05T16:35:03.821071+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:35:03.821071+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2415,"log_dups_size":0,"ondisk_log_size":2415,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19783680,"num_objects":2415,"num_object_clones":0,"num_object_copies":9660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2415,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2415,"num_write_kb":19320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":153}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'2345","reported_seq":2407,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:34:59.997335+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:34:59.997335+0000","last_peered":"2024-09-05T16:34:59.997335+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:34:59.997335+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2345,"log_dups_size":0,"ondisk_log_size":2345,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19174560,"num_objects":2341,"num_object_clones":0,"num_object_copies":9364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2341,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2343,"num_write_kb":18740,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":74}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'2348","reported_seq":2422,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:00.024763+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:35:00.024763+0000","last_peered":"2024-09-05T16:35:00.024763+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:35:00.024763+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2348,"log_dups_size":0,"ondisk_log_size":2348,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19208784,"num_objects":2345,"num_object_clones":0,"num_object_copies":9380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2345,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2347,"num_write_kb":18766,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":95}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'2364","reported_seq":2520,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:34:59.948794+0000","last_change":"2024-09-05T16:34:56.800371+0000","last_active":"2024-09-05T16:34:59.948794+0000","last_peered":"2024-09-05T16:34:59.948794+0000","last_clean":"2024-09-05T16:33:53.148717+0000","last_became_active":"2024-09-05T16:34:56.791755+0000","last_became_peered":"2024-09-05T16:34:56.791755+0000","last_unstale":"2024-09-05T16:34:59.948794+0000","last_undegraded":"2024-09-05T16:34:56.597522+0000","last_fullsized":"2024-09-05T16:34:56.568150+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2364,"log_dups_size":0,"ondisk_log_size":2364,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19330208,"num_objects":2360,"num_object_clones":0,"num_object_copies":9440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2242,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2360,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2362,"num_write_kb":18885,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":376832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":118}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'2346","reported_seq":2399,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:03.967181+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:35:03.967181+0000","last_peered":"2024-09-05T16:35:03.967181+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:35:03.967181+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2346,"log_dups_size":0,"ondisk_log_size":2346,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19190944,"num_objects":2343,"num_object_clones":0,"num_object_copies":9372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2343,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2344,"num_write_kb":18749,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":181}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'2443","reported_seq":2484,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:04.012497+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:35:04.012497+0000","last_peered":"2024-09-05T16:35:04.012497+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:35:04.012497+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2443,"log_dups_size":0,"ondisk_log_size":2443,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20013056,"num_objects":2443,"num_object_clones":0,"num_object_copies":9772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2443,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2443,"num_write_kb":19544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":193}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'2244","reported_seq":2306,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:34:59.957086+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:34:59.957086+0000","last_peered":"2024-09-05T16:34:59.957086+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:34:59.957086+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2244,"log_dups_size":0,"ondisk_log_size":2244,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18330784,"num_objects":2238,"num_object_clones":0,"num_object_copies":8952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2238,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2242,"num_write_kb":17916,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":91}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'2339","reported_seq":2402,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:34:59.791695+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:34:59.791695+0000","last_peered":"2024-09-05T16:34:59.791695+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:34:59.791695+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2339,"log_dups_size":0,"ondisk_log_size":2339,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19117216,"num_objects":2334,"num_object_clones":0,"num_object_copies":9336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2334,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2337,"num_write_kb":18684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":95}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'2452","reported_seq":2505,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:04.021376+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:35:04.021376+0000","last_peered":"2024-09-05T16:35:04.021376+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:35:04.021376+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2452,"log_dups_size":0,"ondisk_log_size":2452,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20015424,"num_objects":2444,"num_object_clones":0,"num_object_copies":9776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2444,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2449,"num_write_kb":19569,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":183}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'2358","reported_seq":2433,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:00.065067+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:35:00.065067+0000","last_peered":"2024-09-05T16:35:00.065067+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:35:00.065067+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2358,"log_dups_size":0,"ondisk_log_size":2358,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19297440,"num_objects":2356,"num_object_clones":0,"num_object_copies":9424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2356,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2356,"num_write_kb":18846,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":94}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'2510","reported_seq":2565,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:04.007092+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:35:04.007092+0000","last_peered":"2024-09-05T16:35:04.007092+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:35:04.007092+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2510,"log_dups_size":0,"ondisk_log_size":2510,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20516592,"num_objects":2505,"num_object_clones":0,"num_object_copies":10020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2505,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2507,"num_write_kb":20044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":194}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'2465","reported_seq":2499,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:04.011824+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:35:04.011824+0000","last_peered":"2024-09-05T16:35:04.011824+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:35:04.011824+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2465,"log_dups_size":0,"ondisk_log_size":2465,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20183632,"num_objects":2464,"num_object_clones":0,"num_object_copies":9856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2464,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2464,"num_write_kb":19711,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":200}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'2386","reported_seq":2459,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:00.063610+0000","last_change":"2024-09-05T16:34:56.798081+0000","last_active":"2024-09-05T16:35:00.063610+0000","last_peered":"2024-09-05T16:35:00.063610+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:35:00.063610+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2386,"log_dups_size":0,"ondisk_log_size":2386,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19536464,"num_objects":2385,"num_object_clones":0,"num_object_copies":9540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2385,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2385,"num_write_kb":19079,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":91}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'2561","reported_seq":2612,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:03.914196+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:35:03.914196+0000","last_peered":"2024-09-05T16:35:03.914196+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:35:03.914196+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2561,"log_dups_size":0,"ondisk_log_size":2561,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20944032,"num_objects":2557,"num_object_clones":0,"num_object_copies":10228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2557,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2559,"num_write_kb":20468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":200}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'2394","reported_seq":2471,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:00.006919+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:35:00.006919+0000","last_peered":"2024-09-05T16:35:00.006919+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:35:00.006919+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2394,"log_dups_size":0,"ondisk_log_size":2394,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19584160,"num_objects":2391,"num_object_clones":0,"num_object_copies":9564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2392,"num_write_kb":19133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":71}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'2320","reported_seq":2382,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:34:59.968863+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:34:59.968863+0000","last_peered":"2024-09-05T16:34:59.968863+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:34:59.968863+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2320,"log_dups_size":0,"ondisk_log_size":2320,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18961568,"num_objects":2315,"num_object_clones":0,"num_object_copies":9260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2315,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2318,"num_write_kb":18532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":78}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":313188544,"num_objects":38236,"num_object_clones":0,"num_object_copies":152944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36150,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":38263,"num_write_kb":305986,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":376832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":480169984,"data_stored":480169984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38290,"ondisk_log_size":38290,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084302,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":304748,"kb_used_data":156444,"kb_used_omap":4,"kb_used_meta":148283,"kb_avail":94067092,"statfs":{"total":96636764160,"available":96324702208,"internally_reserved":0,"allocated":160198656,"data_stored":160015774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":151842515},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,3,6,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986179,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66032,"kb_used_data":12124,"kb_used_omap":4,"kb_used_meta":53883,"kb_avail":94305808,"statfs":{"total":96636764160,"available":96569147392,"internally_reserved":0,"allocated":12414976,"data_stored":12232094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":55176914},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,1,1,4,7],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949672984,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":285072,"kb_used_data":149760,"kb_used_omap":1,"kb_used_meta":135294,"kb_avail":94086768,"statfs":{"total":96636764160,"available":96344850432,"internally_reserved":0,"allocated":153354240,"data_stored":153171358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":138541509},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,4,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182415,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":285440,"kb_used_data":149680,"kb_used_omap":4,"kb_used_meta":135739,"kb_avail":94086400,"statfs":{"total":96636764160,"available":96344473600,"internally_reserved":0,"allocated":153272320,"data_stored":153089438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":138997453},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,8,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":153546752,"data_stored":153546752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12550144,"data_stored":12550144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":153681920,"data_stored":153681920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":160391168,"data_stored":160391168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:35:06.648 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-05T16:35:06.649 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'2415", 'reported_seq': 2456, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:03.821071+0000', 'last_change': '2024-09-05T16:34:56.833497+0000', 'last_active': '2024-09-05T16:35:03.821071+0000', 'last_peered': '2024-09-05T16:35:03.821071+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.828289+0000', 'last_became_peered': '2024-09-05T16:34:56.828289+0000', 'last_unstale': '2024-09-05T16:35:03.821071+0000', 'last_undegraded': '2024-09-05T16:34:56.612160+0000', 'last_fullsized': '2024-09-05T16:34:56.557632+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 2415, 'log_dups_size': 0, 'ondisk_log_size': 2415, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19783680, 'num_objects': 2415, 'num_object_clones': 0, 'num_object_copies': 9660, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2262, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2415, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2415, 'num_write_kb': 19320, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 153}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:35:06.649 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-05T16:35:06.649 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'2345", 'reported_seq': 2407, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:34:59.997335+0000', 'last_change': '2024-09-05T16:34:56.825031+0000', 'last_active': '2024-09-05T16:34:59.997335+0000', 'last_peered': '2024-09-05T16:34:59.997335+0000', 'last_clean': '2024-09-05T16:33:53.200762+0000', 'last_became_active': '2024-09-05T16:34:56.820292+0000', 'last_became_peered': '2024-09-05T16:34:56.820292+0000', 'last_unstale': '2024-09-05T16:34:59.997335+0000', 'last_undegraded': '2024-09-05T16:34:56.605776+0000', 'last_fullsized': '2024-09-05T16:34:56.559383+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 2345, 'log_dups_size': 0, 'ondisk_log_size': 2345, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19174560, 'num_objects': 2341, 'num_object_clones': 0, 'num_object_copies': 9364, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2269, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2341, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2343, 'num_write_kb': 18740, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 74}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:35:06.649 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:35:06.649 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'2348", 'reported_seq': 2422, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:00.024763+0000', 'last_change': '2024-09-05T16:34:56.867900+0000', 'last_active': '2024-09-05T16:35:00.024763+0000', 'last_peered': '2024-09-05T16:35:00.024763+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:35:00.024763+0000', 'last_undegraded': '2024-09-05T16:34:56.590972+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 2348, 'log_dups_size': 0, 'ondisk_log_size': 2348, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19208784, 'num_objects': 2345, 'num_object_clones': 0, 'num_object_copies': 9380, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2251, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2345, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2347, 'num_write_kb': 18766, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 95}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:35:06.649 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-05T16:35:06.649 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'2364", 'reported_seq': 2520, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:34:59.948794+0000', 'last_change': '2024-09-05T16:34:56.800371+0000', 'last_active': '2024-09-05T16:34:59.948794+0000', 'last_peered': '2024-09-05T16:34:59.948794+0000', 'last_clean': '2024-09-05T16:33:53.148717+0000', 'last_became_active': '2024-09-05T16:34:56.791755+0000', 'last_became_peered': '2024-09-05T16:34:56.791755+0000', 'last_unstale': '2024-09-05T16:34:59.948794+0000', 'last_undegraded': '2024-09-05T16:34:56.597522+0000', 'last_fullsized': '2024-09-05T16:34:56.568150+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 2364, 'log_dups_size': 0, 'ondisk_log_size': 2364, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19330208, 'num_objects': 2360, 'num_object_clones': 0, 'num_object_copies': 9440, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2242, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2360, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2362, 'num_write_kb': 18885, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 48, 'num_bytes_recovered': 376832, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 118}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:35:06.649 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:35:06.650 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'2346", 'reported_seq': 2399, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:03.967181+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:35:03.967181+0000', 'last_peered': '2024-09-05T16:35:03.967181+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:35:03.967181+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 2346, 'log_dups_size': 0, 'ondisk_log_size': 2346, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19190944, 'num_objects': 2343, 'num_object_clones': 0, 'num_object_copies': 9372, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2164, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2343, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2344, 'num_write_kb': 18749, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 181}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:35:06.650 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:35:06.650 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'2443", 'reported_seq': 2484, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:04.012497+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:35:04.012497+0000', 'last_peered': '2024-09-05T16:35:04.012497+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:35:04.012497+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 2443, 'log_dups_size': 0, 'ondisk_log_size': 2443, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20013056, 'num_objects': 2443, 'num_object_clones': 0, 'num_object_copies': 9772, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2443, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2443, 'num_write_kb': 19544, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 193}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:35:06.650 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:35:06.650 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'2244", 'reported_seq': 2306, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:34:59.957086+0000', 'last_change': '2024-09-05T16:34:56.903436+0000', 'last_active': '2024-09-05T16:34:59.957086+0000', 'last_peered': '2024-09-05T16:34:59.957086+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:34:59.957086+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 2244, 'log_dups_size': 0, 'ondisk_log_size': 2244, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18330784, 'num_objects': 2238, 'num_object_clones': 0, 'num_object_copies': 8952, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2149, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2238, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2242, 'num_write_kb': 17916, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 91}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:35:06.650 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-05T16:35:06.650 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'2339", 'reported_seq': 2402, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:34:59.791695+0000', 'last_change': '2024-09-05T16:34:56.883319+0000', 'last_active': '2024-09-05T16:34:59.791695+0000', 'last_peered': '2024-09-05T16:34:59.791695+0000', 'last_clean': '2024-09-05T16:33:53.505305+0000', 'last_became_active': '2024-09-05T16:34:56.873389+0000', 'last_became_peered': '2024-09-05T16:34:56.873389+0000', 'last_unstale': '2024-09-05T16:34:59.791695+0000', 'last_undegraded': '2024-09-05T16:34:56.602567+0000', 'last_fullsized': '2024-09-05T16:34:56.559958+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 2339, 'log_dups_size': 0, 'ondisk_log_size': 2339, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19117216, 'num_objects': 2334, 'num_object_clones': 0, 'num_object_copies': 9336, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2241, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2334, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2337, 'num_write_kb': 18684, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 95}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:35:06.650 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:35:06.650 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'2452", 'reported_seq': 2505, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:04.021376+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:35:04.021376+0000', 'last_peered': '2024-09-05T16:35:04.021376+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:35:04.021376+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 2452, 'log_dups_size': 0, 'ondisk_log_size': 2452, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20015424, 'num_objects': 2444, 'num_object_clones': 0, 'num_object_copies': 9776, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2264, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2444, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2449, 'num_write_kb': 19569, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 183}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:35:06.650 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:35:06.650 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'2358", 'reported_seq': 2433, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:00.065067+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:35:00.065067+0000', 'last_peered': '2024-09-05T16:35:00.065067+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:35:00.065067+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 2358, 'log_dups_size': 0, 'ondisk_log_size': 2358, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19297440, 'num_objects': 2356, 'num_object_clones': 0, 'num_object_copies': 9424, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2264, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2356, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2356, 'num_write_kb': 18846, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 94}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:35:06.651 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:35:06.651 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'2510", 'reported_seq': 2565, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:04.007092+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:35:04.007092+0000', 'last_peered': '2024-09-05T16:35:04.007092+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:35:04.007092+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 2510, 'log_dups_size': 0, 'ondisk_log_size': 2510, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20516592, 'num_objects': 2505, 'num_object_clones': 0, 'num_object_copies': 10020, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2314, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2505, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2507, 'num_write_kb': 20044, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 194}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:35:06.651 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:35:06.651 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'2465", 'reported_seq': 2499, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:04.011824+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:35:04.011824+0000', 'last_peered': '2024-09-05T16:35:04.011824+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:35:04.011824+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 2465, 'log_dups_size': 0, 'ondisk_log_size': 2465, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20183632, 'num_objects': 2464, 'num_object_clones': 0, 'num_object_copies': 9856, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2265, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2464, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2464, 'num_write_kb': 19711, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 200}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:35:06.651 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-05T16:35:06.651 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'2386", 'reported_seq': 2459, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:00.063610+0000', 'last_change': '2024-09-05T16:34:56.798081+0000', 'last_active': '2024-09-05T16:35:00.063610+0000', 'last_peered': '2024-09-05T16:35:00.063610+0000', 'last_clean': '2024-09-05T16:33:45.082039+0000', 'last_became_active': '2024-09-05T16:34:56.792386+0000', 'last_became_peered': '2024-09-05T16:34:56.792386+0000', 'last_unstale': '2024-09-05T16:35:00.063610+0000', 'last_undegraded': '2024-09-05T16:34:56.600595+0000', 'last_fullsized': '2024-09-05T16:34:56.558203+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 2386, 'log_dups_size': 0, 'ondisk_log_size': 2386, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19536464, 'num_objects': 2385, 'num_object_clones': 0, 'num_object_copies': 9540, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2295, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2385, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2385, 'num_write_kb': 19079, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 91}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:35:06.651 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:35:06.651 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'2561", 'reported_seq': 2612, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:03.914196+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:35:03.914196+0000', 'last_peered': '2024-09-05T16:35:03.914196+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:35:03.914196+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 2561, 'log_dups_size': 0, 'ondisk_log_size': 2561, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20944032, 'num_objects': 2557, 'num_object_clones': 0, 'num_object_copies': 10228, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2359, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2557, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2559, 'num_write_kb': 20468, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 200}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:35:06.651 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:35:06.651 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'2394", 'reported_seq': 2471, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:00.006919+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:35:00.006919+0000', 'last_peered': '2024-09-05T16:35:00.006919+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:35:00.006919+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 2394, 'log_dups_size': 0, 'ondisk_log_size': 2394, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19584160, 'num_objects': 2391, 'num_object_clones': 0, 'num_object_copies': 9564, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2322, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2391, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2392, 'num_write_kb': 19133, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 71}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:35:06.651 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:35:06.652 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'2320", 'reported_seq': 2382, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:34:59.968863+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:34:59.968863+0000', 'last_peered': '2024-09-05T16:34:59.968863+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:34:59.968863+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 2320, 'log_dups_size': 0, 'ondisk_log_size': 2320, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18961568, 'num_objects': 2315, 'num_object_clones': 0, 'num_object_copies': 9260, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2315, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2318, 'num_write_kb': 18532, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 78}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:35:06.652 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:35:07.313 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:35:07.313 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 36080/157704 objects degraded (22.878%), 16 pgs degraded","count":16},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":110,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":39426,"data_bytes":322937024,"bytes_used":993538048,"bytes_avail":385553518592,"bytes_total":386547056640,"degraded_objects":36080,"degraded_total":157704,"degraded_ratio":0.22878303657484905,"recovering_objects_per_sec":11,"recovering_bytes_per_sec":92756,"recovering_keys_per_sec":0,"num_objects_recovered":116,"num_bytes_recovered":950272,"num_keys_recovered":0,"write_bytes_sec":4132454,"read_op_per_sec":0,"write_op_per_sec":504},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-09-05T16:35:06.800151+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"3":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-05T16:35:07.341 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:35:07.341 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:35:07.495 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 45 20 40513 40493 7.02924 2.74219 0.0517915 0.0222103 2024-09-05T16:35:07.962 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:35:07.963 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:35:07.994 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":79,"stamp":"2024-09-05T16:35:06.800061+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":322937024,"num_objects":39426,"num_object_clones":0,"num_object_copies":157704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36080,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39426,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":39453,"num_write_kb":315506,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":143,"num_bytes_recovered":950272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39480,"ondisk_log_size":39480,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":970252,"kb_used_data":484488,"kb_used_omap":14,"kb_used_meta":485681,"kb_avail":376517108,"statfs":{"total":386547056640,"available":385553518592,"internally_reserved":0,"allocated":496115712,"data_stored":495384184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":497337911},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[7,2,2,8,10,19,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":42336256,"num_objects":5168,"num_object_clones":0,"num_object_copies":20672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8486,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5168,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5168,"num_write_kb":41344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":950272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.244820"},"pg_stats":[{"pgid":"1.f","version":"33'2415","reported_seq":2456,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:03.821071+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:35:03.821071+0000","last_peered":"2024-09-05T16:35:03.821071+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:35:03.821071+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2415,"log_dups_size":0,"ondisk_log_size":2415,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19783680,"num_objects":2415,"num_object_clones":0,"num_object_copies":9660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2415,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2415,"num_write_kb":19320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":153}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'2459","reported_seq":2521,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:04.962284+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:35:04.962284+0000","last_peered":"2024-09-05T16:35:04.962284+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:35:04.962284+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2459,"log_dups_size":0,"ondisk_log_size":2459,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20108448,"num_objects":2455,"num_object_clones":0,"num_object_copies":9820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2455,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2457,"num_write_kb":19652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":188}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'2490","reported_seq":2563,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:05.044287+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:35:05.044287+0000","last_peered":"2024-09-05T16:35:05.044287+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:35:05.044287+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2490,"log_dups_size":0,"ondisk_log_size":2490,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20372048,"num_objects":2487,"num_object_clones":0,"num_object_copies":9948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2487,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2489,"num_write_kb":19902,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":237}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'2477","reported_seq":2773,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:04.985788+0000","last_change":"2024-09-05T16:34:56.800371+0000","last_active":"2024-09-05T16:35:04.985788+0000","last_peered":"2024-09-05T16:35:04.985788+0000","last_clean":"2024-09-05T16:33:53.148717+0000","last_became_active":"2024-09-05T16:34:56.791755+0000","last_became_peered":"2024-09-05T16:34:56.791755+0000","last_unstale":"2024-09-05T16:35:04.985788+0000","last_undegraded":"2024-09-05T16:34:56.597522+0000","last_fullsized":"2024-09-05T16:34:56.568150+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2477,"log_dups_size":0,"ondisk_log_size":2477,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20255904,"num_objects":2473,"num_object_clones":0,"num_object_copies":9892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2172,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2473,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2475,"num_write_kb":19789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":118,"num_bytes_recovered":950272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":301}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'2346","reported_seq":2399,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:03.967181+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:35:03.967181+0000","last_peered":"2024-09-05T16:35:03.967181+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:35:03.967181+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2346,"log_dups_size":0,"ondisk_log_size":2346,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19190944,"num_objects":2343,"num_object_clones":0,"num_object_copies":9372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2343,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2344,"num_write_kb":18749,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":181}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'2443","reported_seq":2484,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:04.012497+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:35:04.012497+0000","last_peered":"2024-09-05T16:35:04.012497+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:35:04.012497+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2443,"log_dups_size":0,"ondisk_log_size":2443,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20013056,"num_objects":2443,"num_object_clones":0,"num_object_copies":9772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2443,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2443,"num_write_kb":19544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":193}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'2377","reported_seq":2441,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:04.986146+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:35:04.986146+0000","last_peered":"2024-09-05T16:35:04.986146+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:35:04.986146+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2377,"log_dups_size":0,"ondisk_log_size":2377,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19420320,"num_objects":2371,"num_object_clones":0,"num_object_copies":9484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2371,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2375,"num_write_kb":18980,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":224}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'2477","reported_seq":2540,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:04.919663+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:35:04.919663+0000","last_peered":"2024-09-05T16:35:04.919663+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:35:04.919663+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2477,"log_dups_size":0,"ondisk_log_size":2477,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20247712,"num_objects":2472,"num_object_clones":0,"num_object_copies":9888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2472,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2475,"num_write_kb":19788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":233}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'2452","reported_seq":2505,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:04.021376+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:35:04.021376+0000","last_peered":"2024-09-05T16:35:04.021376+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:35:04.021376+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2452,"log_dups_size":0,"ondisk_log_size":2452,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20015424,"num_objects":2444,"num_object_clones":0,"num_object_copies":9776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2444,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2449,"num_write_kb":19569,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":183}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'2483","reported_seq":2559,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:05.005679+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:35:05.005679+0000","last_peered":"2024-09-05T16:35:05.005679+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:35:05.005679+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2483,"log_dups_size":0,"ondisk_log_size":2483,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20321440,"num_objects":2481,"num_object_clones":0,"num_object_copies":9924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2481,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2481,"num_write_kb":19846,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":219}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'2510","reported_seq":2565,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:04.007092+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:35:04.007092+0000","last_peered":"2024-09-05T16:35:04.007092+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:35:04.007092+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2510,"log_dups_size":0,"ondisk_log_size":2510,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20516592,"num_objects":2505,"num_object_clones":0,"num_object_copies":10020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2505,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2507,"num_write_kb":20044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":194}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'2465","reported_seq":2499,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:04.011824+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:35:04.011824+0000","last_peered":"2024-09-05T16:35:04.011824+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:35:04.011824+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2465,"log_dups_size":0,"ondisk_log_size":2465,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20183632,"num_objects":2464,"num_object_clones":0,"num_object_copies":9856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2464,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2464,"num_write_kb":19711,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":200}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'2520","reported_seq":2592,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:05.089008+0000","last_change":"2024-09-05T16:34:56.798081+0000","last_active":"2024-09-05T16:35:05.089008+0000","last_peered":"2024-09-05T16:35:05.089008+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:35:05.089008+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2520,"log_dups_size":0,"ondisk_log_size":2520,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20634192,"num_objects":2519,"num_object_clones":0,"num_object_copies":10076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2519,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2519,"num_write_kb":20151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":225}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'2561","reported_seq":2612,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:03.914196+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:35:03.914196+0000","last_peered":"2024-09-05T16:35:03.914196+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:35:03.914196+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2561,"log_dups_size":0,"ondisk_log_size":2561,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20944032,"num_objects":2557,"num_object_clones":0,"num_object_copies":10228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2557,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2559,"num_write_kb":20468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":200}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'2533","reported_seq":2609,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:05.066014+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:35:05.066014+0000","last_peered":"2024-09-05T16:35:05.066014+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:35:05.066014+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2533,"log_dups_size":0,"ondisk_log_size":2533,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20722848,"num_objects":2530,"num_object_clones":0,"num_object_copies":10120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2530,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2531,"num_write_kb":20245,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":210}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'2472","reported_seq":2535,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:04.989304+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:35:04.989304+0000","last_peered":"2024-09-05T16:35:04.989304+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:35:04.989304+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2472,"log_dups_size":0,"ondisk_log_size":2472,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20206752,"num_objects":2467,"num_object_clones":0,"num_object_copies":9868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2467,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2470,"num_write_kb":19748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":230}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":322937024,"num_objects":39426,"num_object_clones":0,"num_object_copies":157704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36080,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39426,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":39453,"num_write_kb":315506,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":143,"num_bytes_recovered":950272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":497311744,"data_stored":497311744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39480,"ondisk_log_size":39480,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084302,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":304748,"kb_used_data":156444,"kb_used_omap":4,"kb_used_meta":148283,"kb_avail":94067092,"statfs":{"total":96636764160,"available":96324702208,"internally_reserved":0,"allocated":160198656,"data_stored":160015774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":151842515},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,3,6,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986179,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66032,"kb_used_data":12124,"kb_used_omap":4,"kb_used_meta":53883,"kb_avail":94305808,"statfs":{"total":96636764160,"available":96569147392,"internally_reserved":0,"allocated":12414976,"data_stored":12232094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":55176914},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,1,1,4,7],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949672985,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301684,"kb_used_data":158052,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":94070156,"statfs":{"total":96636764160,"available":96327839744,"internally_reserved":0,"allocated":161845248,"data_stored":161662366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":147061189},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2,1,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182416,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297788,"kb_used_data":157868,"kb_used_omap":4,"kb_used_meta":139899,"kb_avail":94074052,"statfs":{"total":96636764160,"available":96331829248,"internally_reserved":0,"allocated":161656832,"data_stored":161473950,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":143257293},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,0,2,2,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":162119680,"data_stored":162119680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12550144,"data_stored":12550144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":162250752,"data_stored":162250752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":160391168,"data_stored":160391168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:35:08.495 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 46 20 40920 40900 6.94555 3.17969 0.0312436 0.0224797 2024-09-05T16:35:09.495 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 47 20 41365 41345 6.87174 3.47656 0.0448204 0.0227234 2024-09-05T16:35:10.497 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 48 20 41803 41783 6.79957 3.42188 0.0287625 0.0229606 2024-09-05T16:35:10.997 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:35:11.498 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 49 20 42219 42199 6.72713 3.25 0.0454123 0.0232067 2024-09-05T16:35:11.659 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:35:11.660 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:35:11.686 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":81,"stamp":"2024-09-05T16:35:10.800966+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":339312832,"num_objects":41425,"num_object_clones":0,"num_object_copies":165700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36009,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41425,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":41452,"num_write_kb":331498,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":214,"num_bytes_recovered":1531904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41479,"ondisk_log_size":41479,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":998208,"kb_used_data":516608,"kb_used_omap":14,"kb_used_meta":481521,"kb_avail":376489152,"statfs":{"total":386547056640,"available":385524891648,"internally_reserved":0,"allocated":529006592,"data_stored":528275064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":493078071},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,2,12,12,7,6,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":58712064,"num_objects":7167,"num_object_clones":0,"num_object_copies":28668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8415,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7167,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7167,"num_write_kb":57336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":187,"num_bytes_recovered":1531904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002632"},"pg_stats":[{"pgid":"1.f","version":"33'2542","reported_seq":2582,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:09.023046+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:35:09.023046+0000","last_peered":"2024-09-05T16:35:09.023046+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:35:09.023046+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2542,"log_dups_size":0,"ondisk_log_size":2542,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20824064,"num_objects":2542,"num_object_clones":0,"num_object_copies":10168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2542,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2542,"num_write_kb":20336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":280}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'2581","reported_seq":2644,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:09.941023+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:35:09.941023+0000","last_peered":"2024-09-05T16:35:09.941023+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:35:09.941023+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2581,"log_dups_size":0,"ondisk_log_size":2581,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21107872,"num_objects":2577,"num_object_clones":0,"num_object_copies":10308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2577,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2579,"num_write_kb":20628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":310}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'2607","reported_seq":2681,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:10.079292+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:35:10.079292+0000","last_peered":"2024-09-05T16:35:10.079292+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:35:10.079292+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2607,"log_dups_size":0,"ondisk_log_size":2607,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21330512,"num_objects":2604,"num_object_clones":0,"num_object_copies":10416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2604,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2606,"num_write_kb":20838,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":354}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'2592","reported_seq":3030,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:10.068793+0000","last_change":"2024-09-05T16:34:56.800371+0000","last_active":"2024-09-05T16:35:10.068793+0000","last_peered":"2024-09-05T16:35:10.068793+0000","last_clean":"2024-09-05T16:33:53.148717+0000","last_became_active":"2024-09-05T16:34:56.791755+0000","last_became_peered":"2024-09-05T16:34:56.791755+0000","last_unstale":"2024-09-05T16:35:10.068793+0000","last_undegraded":"2024-09-05T16:34:56.597522+0000","last_fullsized":"2024-09-05T16:34:56.568150+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2592,"log_dups_size":0,"ondisk_log_size":2592,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21197984,"num_objects":2588,"num_object_clones":0,"num_object_copies":10352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2101,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2588,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2590,"num_write_kb":20709,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":189,"num_bytes_recovered":1531904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":487}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'2473","reported_seq":2526,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:08.965340+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:35:08.965340+0000","last_peered":"2024-09-05T16:35:08.965340+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:35:08.965340+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2473,"log_dups_size":0,"ondisk_log_size":2473,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20231328,"num_objects":2470,"num_object_clones":0,"num_object_copies":9880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2470,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2471,"num_write_kb":19765,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":308}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'2559","reported_seq":2600,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:08.965700+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:35:08.965700+0000","last_peered":"2024-09-05T16:35:08.965700+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:35:08.965700+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2559,"log_dups_size":0,"ondisk_log_size":2559,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20963328,"num_objects":2559,"num_object_clones":0,"num_object_copies":10236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2559,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2559,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":309}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'2487","reported_seq":2551,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:09.997914+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:35:09.997914+0000","last_peered":"2024-09-05T16:35:09.997914+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:35:09.997914+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2487,"log_dups_size":0,"ondisk_log_size":2487,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20321440,"num_objects":2481,"num_object_clones":0,"num_object_copies":9924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2481,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2485,"num_write_kb":19860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":334}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'2606","reported_seq":2669,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:10.020712+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:35:10.020712+0000","last_peered":"2024-09-05T16:35:10.020712+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:35:10.020712+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2606,"log_dups_size":0,"ondisk_log_size":2606,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21304480,"num_objects":2601,"num_object_clones":0,"num_object_copies":10404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2601,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2604,"num_write_kb":20820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":362}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'2589","reported_seq":2643,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:09.053121+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:35:09.053121+0000","last_peered":"2024-09-05T16:35:09.053121+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:35:09.053121+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2589,"log_dups_size":0,"ondisk_log_size":2589,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21137728,"num_objects":2581,"num_object_clones":0,"num_object_copies":10324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2581,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2586,"num_write_kb":20665,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":320}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'2634","reported_seq":2710,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:10.045949+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:35:10.045949+0000","last_peered":"2024-09-05T16:35:10.045949+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:35:10.045949+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2634,"log_dups_size":0,"ondisk_log_size":2634,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21558432,"num_objects":2632,"num_object_clones":0,"num_object_copies":10528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2632,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2632,"num_write_kb":21054,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":370}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'2636","reported_seq":2691,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:09.050725+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:35:09.050725+0000","last_peered":"2024-09-05T16:35:09.050725+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:35:09.050725+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2636,"log_dups_size":0,"ondisk_log_size":2636,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21548784,"num_objects":2631,"num_object_clones":0,"num_object_copies":10524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2631,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2633,"num_write_kb":21052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":320}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'2579","reported_seq":2614,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:09.036684+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:35:09.036684+0000","last_peered":"2024-09-05T16:35:09.036684+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:35:09.036684+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2579,"log_dups_size":0,"ondisk_log_size":2579,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21117520,"num_objects":2578,"num_object_clones":0,"num_object_copies":10312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2578,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2578,"num_write_kb":20623,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":314}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'2635","reported_seq":2708,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:10.070303+0000","last_change":"2024-09-05T16:34:56.798081+0000","last_active":"2024-09-05T16:35:10.070303+0000","last_peered":"2024-09-05T16:35:10.070303+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:35:10.070303+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2635,"log_dups_size":0,"ondisk_log_size":2635,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21576272,"num_objects":2634,"num_object_clones":0,"num_object_copies":10536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2634,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2634,"num_write_kb":21071,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":340}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'2695","reported_seq":2746,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:09.017968+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:35:09.017968+0000","last_peered":"2024-09-05T16:35:09.017968+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:35:09.017968+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2695,"log_dups_size":0,"ondisk_log_size":2695,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22041760,"num_objects":2691,"num_object_clones":0,"num_object_copies":10764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2693,"num_write_kb":21540,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":334}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'2663","reported_seq":2740,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:10.062549+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:35:10.062549+0000","last_peered":"2024-09-05T16:35:10.062549+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:35:10.062549+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2663,"log_dups_size":0,"ondisk_log_size":2663,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21787808,"num_objects":2660,"num_object_clones":0,"num_object_copies":10640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2660,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2661,"num_write_kb":21285,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":340}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'2601","reported_seq":2662,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:10.025419+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:35:10.025419+0000","last_peered":"2024-09-05T16:35:10.025419+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:35:10.025419+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2601,"log_dups_size":0,"ondisk_log_size":2601,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21263520,"num_objects":2596,"num_object_clones":0,"num_object_copies":10384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2596,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2599,"num_write_kb":20780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":359}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":339312832,"num_objects":41425,"num_object_clones":0,"num_object_copies":165700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36009,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41425,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":41452,"num_write_kb":331498,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":214,"num_bytes_recovered":1531904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":530702336,"data_stored":530702336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41479,"ondisk_log_size":41479,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084303,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":304256,"kb_used_data":164272,"kb_used_omap":4,"kb_used_meta":139963,"kb_avail":94067584,"statfs":{"total":96636764160,"available":96325206016,"internally_reserved":0,"allocated":168214528,"data_stored":168031646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":143322835},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,6,1,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986180,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65896,"kb_used_data":20312,"kb_used_omap":4,"kb_used_meta":45563,"kb_avail":94305944,"statfs":{"total":96636764160,"available":96569286656,"internally_reserved":0,"allocated":20799488,"data_stored":20616606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":46657234},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,0,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949672986,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":313984,"kb_used_data":166192,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":94057856,"statfs":{"total":96636764160,"available":96315244544,"internally_reserved":0,"allocated":170180608,"data_stored":169997726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":151321029},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,5,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182417,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":314072,"kb_used_data":165832,"kb_used_omap":4,"kb_used_meta":148219,"kb_avail":94057768,"statfs":{"total":96636764160,"available":96315154432,"internally_reserved":0,"allocated":169811968,"data_stored":169629086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":151776973},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,5,2,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":170479616,"data_stored":170479616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":21053440,"data_stored":21053440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":170545152,"data_stored":170545152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":168624128,"data_stored":168624128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:35:11.687 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:35:12.306 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:35:12.307 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:35:12.327 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":81,"stamp":"2024-09-05T16:35:10.800966+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":339312832,"num_objects":41425,"num_object_clones":0,"num_object_copies":165700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36009,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41425,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":41452,"num_write_kb":331498,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":214,"num_bytes_recovered":1531904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41479,"ondisk_log_size":41479,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":998208,"kb_used_data":516608,"kb_used_omap":14,"kb_used_meta":481521,"kb_avail":376489152,"statfs":{"total":386547056640,"available":385524891648,"internally_reserved":0,"allocated":529006592,"data_stored":528275064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":493078071},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,2,12,12,7,6,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":58712064,"num_objects":7167,"num_object_clones":0,"num_object_copies":28668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8415,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7167,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7167,"num_write_kb":57336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":187,"num_bytes_recovered":1531904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002632"},"pg_stats":[{"pgid":"1.f","version":"33'2542","reported_seq":2582,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:09.023046+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:35:09.023046+0000","last_peered":"2024-09-05T16:35:09.023046+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:35:09.023046+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2542,"log_dups_size":0,"ondisk_log_size":2542,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20824064,"num_objects":2542,"num_object_clones":0,"num_object_copies":10168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2542,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2542,"num_write_kb":20336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":280}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'2581","reported_seq":2644,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:09.941023+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:35:09.941023+0000","last_peered":"2024-09-05T16:35:09.941023+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:35:09.941023+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2581,"log_dups_size":0,"ondisk_log_size":2581,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21107872,"num_objects":2577,"num_object_clones":0,"num_object_copies":10308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2577,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2579,"num_write_kb":20628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":310}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'2607","reported_seq":2681,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:10.079292+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:35:10.079292+0000","last_peered":"2024-09-05T16:35:10.079292+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:35:10.079292+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2607,"log_dups_size":0,"ondisk_log_size":2607,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21330512,"num_objects":2604,"num_object_clones":0,"num_object_copies":10416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2604,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2606,"num_write_kb":20838,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":354}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'2592","reported_seq":3030,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:10.068793+0000","last_change":"2024-09-05T16:34:56.800371+0000","last_active":"2024-09-05T16:35:10.068793+0000","last_peered":"2024-09-05T16:35:10.068793+0000","last_clean":"2024-09-05T16:33:53.148717+0000","last_became_active":"2024-09-05T16:34:56.791755+0000","last_became_peered":"2024-09-05T16:34:56.791755+0000","last_unstale":"2024-09-05T16:35:10.068793+0000","last_undegraded":"2024-09-05T16:34:56.597522+0000","last_fullsized":"2024-09-05T16:34:56.568150+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2592,"log_dups_size":0,"ondisk_log_size":2592,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21197984,"num_objects":2588,"num_object_clones":0,"num_object_copies":10352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2101,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2588,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2590,"num_write_kb":20709,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":189,"num_bytes_recovered":1531904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":487}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'2473","reported_seq":2526,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:08.965340+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:35:08.965340+0000","last_peered":"2024-09-05T16:35:08.965340+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:35:08.965340+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2473,"log_dups_size":0,"ondisk_log_size":2473,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20231328,"num_objects":2470,"num_object_clones":0,"num_object_copies":9880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2470,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2471,"num_write_kb":19765,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":308}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'2559","reported_seq":2600,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:08.965700+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:35:08.965700+0000","last_peered":"2024-09-05T16:35:08.965700+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:35:08.965700+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2559,"log_dups_size":0,"ondisk_log_size":2559,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20963328,"num_objects":2559,"num_object_clones":0,"num_object_copies":10236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2559,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2559,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":309}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'2487","reported_seq":2551,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:09.997914+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:35:09.997914+0000","last_peered":"2024-09-05T16:35:09.997914+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:35:09.997914+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2487,"log_dups_size":0,"ondisk_log_size":2487,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20321440,"num_objects":2481,"num_object_clones":0,"num_object_copies":9924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2481,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2485,"num_write_kb":19860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":334}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'2606","reported_seq":2669,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:10.020712+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:35:10.020712+0000","last_peered":"2024-09-05T16:35:10.020712+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:35:10.020712+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2606,"log_dups_size":0,"ondisk_log_size":2606,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21304480,"num_objects":2601,"num_object_clones":0,"num_object_copies":10404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2601,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2604,"num_write_kb":20820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":362}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'2589","reported_seq":2643,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:09.053121+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:35:09.053121+0000","last_peered":"2024-09-05T16:35:09.053121+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:35:09.053121+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2589,"log_dups_size":0,"ondisk_log_size":2589,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21137728,"num_objects":2581,"num_object_clones":0,"num_object_copies":10324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2581,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2586,"num_write_kb":20665,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":320}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'2634","reported_seq":2710,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:10.045949+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:35:10.045949+0000","last_peered":"2024-09-05T16:35:10.045949+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:35:10.045949+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2634,"log_dups_size":0,"ondisk_log_size":2634,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21558432,"num_objects":2632,"num_object_clones":0,"num_object_copies":10528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2632,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2632,"num_write_kb":21054,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":370}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'2636","reported_seq":2691,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:09.050725+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:35:09.050725+0000","last_peered":"2024-09-05T16:35:09.050725+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:35:09.050725+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2636,"log_dups_size":0,"ondisk_log_size":2636,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21548784,"num_objects":2631,"num_object_clones":0,"num_object_copies":10524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2631,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2633,"num_write_kb":21052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":320}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'2579","reported_seq":2614,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:09.036684+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:35:09.036684+0000","last_peered":"2024-09-05T16:35:09.036684+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:35:09.036684+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2579,"log_dups_size":0,"ondisk_log_size":2579,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21117520,"num_objects":2578,"num_object_clones":0,"num_object_copies":10312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2578,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2578,"num_write_kb":20623,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":314}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'2635","reported_seq":2708,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:10.070303+0000","last_change":"2024-09-05T16:34:56.798081+0000","last_active":"2024-09-05T16:35:10.070303+0000","last_peered":"2024-09-05T16:35:10.070303+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:35:10.070303+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2635,"log_dups_size":0,"ondisk_log_size":2635,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21576272,"num_objects":2634,"num_object_clones":0,"num_object_copies":10536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2634,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2634,"num_write_kb":21071,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":340}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'2695","reported_seq":2746,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:09.017968+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:35:09.017968+0000","last_peered":"2024-09-05T16:35:09.017968+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:35:09.017968+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2695,"log_dups_size":0,"ondisk_log_size":2695,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22041760,"num_objects":2691,"num_object_clones":0,"num_object_copies":10764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2693,"num_write_kb":21540,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":334}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'2663","reported_seq":2740,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:10.062549+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:35:10.062549+0000","last_peered":"2024-09-05T16:35:10.062549+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:35:10.062549+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2663,"log_dups_size":0,"ondisk_log_size":2663,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21787808,"num_objects":2660,"num_object_clones":0,"num_object_copies":10640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2660,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2661,"num_write_kb":21285,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":340}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'2601","reported_seq":2662,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:10.025419+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:35:10.025419+0000","last_peered":"2024-09-05T16:35:10.025419+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:35:10.025419+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2601,"log_dups_size":0,"ondisk_log_size":2601,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21263520,"num_objects":2596,"num_object_clones":0,"num_object_copies":10384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2596,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2599,"num_write_kb":20780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":359}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":339312832,"num_objects":41425,"num_object_clones":0,"num_object_copies":165700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36009,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41425,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":41452,"num_write_kb":331498,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":214,"num_bytes_recovered":1531904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":530702336,"data_stored":530702336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41479,"ondisk_log_size":41479,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084303,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":304256,"kb_used_data":164272,"kb_used_omap":4,"kb_used_meta":139963,"kb_avail":94067584,"statfs":{"total":96636764160,"available":96325206016,"internally_reserved":0,"allocated":168214528,"data_stored":168031646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":143322835},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,6,1,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986180,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65896,"kb_used_data":20312,"kb_used_omap":4,"kb_used_meta":45563,"kb_avail":94305944,"statfs":{"total":96636764160,"available":96569286656,"internally_reserved":0,"allocated":20799488,"data_stored":20616606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":46657234},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,0,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949672986,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":313984,"kb_used_data":166192,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":94057856,"statfs":{"total":96636764160,"available":96315244544,"internally_reserved":0,"allocated":170180608,"data_stored":169997726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":151321029},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,5,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182417,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":314072,"kb_used_data":165832,"kb_used_omap":4,"kb_used_meta":148219,"kb_avail":94057768,"statfs":{"total":96636764160,"available":96315154432,"internally_reserved":0,"allocated":169811968,"data_stored":169629086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":151776973},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,5,2,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":170479616,"data_stored":170479616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":21053440,"data_stored":21053440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":170545152,"data_stored":170545152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":168624128,"data_stored":168624128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:35:12.328 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-05T16:35:12.328 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'2542", 'reported_seq': 2582, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:09.023046+0000', 'last_change': '2024-09-05T16:34:56.833497+0000', 'last_active': '2024-09-05T16:35:09.023046+0000', 'last_peered': '2024-09-05T16:35:09.023046+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.828289+0000', 'last_became_peered': '2024-09-05T16:34:56.828289+0000', 'last_unstale': '2024-09-05T16:35:09.023046+0000', 'last_undegraded': '2024-09-05T16:34:56.612160+0000', 'last_fullsized': '2024-09-05T16:34:56.557632+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 2542, 'log_dups_size': 0, 'ondisk_log_size': 2542, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20824064, 'num_objects': 2542, 'num_object_clones': 0, 'num_object_copies': 10168, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2262, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2542, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2542, 'num_write_kb': 20336, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 280}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:35:12.328 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-05T16:35:12.328 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'2581", 'reported_seq': 2644, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:09.941023+0000', 'last_change': '2024-09-05T16:34:56.825031+0000', 'last_active': '2024-09-05T16:35:09.941023+0000', 'last_peered': '2024-09-05T16:35:09.941023+0000', 'last_clean': '2024-09-05T16:33:53.200762+0000', 'last_became_active': '2024-09-05T16:34:56.820292+0000', 'last_became_peered': '2024-09-05T16:34:56.820292+0000', 'last_unstale': '2024-09-05T16:35:09.941023+0000', 'last_undegraded': '2024-09-05T16:34:56.605776+0000', 'last_fullsized': '2024-09-05T16:34:56.559383+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 2581, 'log_dups_size': 0, 'ondisk_log_size': 2581, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21107872, 'num_objects': 2577, 'num_object_clones': 0, 'num_object_copies': 10308, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2269, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2577, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2579, 'num_write_kb': 20628, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 310}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:35:12.329 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:35:12.329 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'2607", 'reported_seq': 2681, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:10.079292+0000', 'last_change': '2024-09-05T16:34:56.867900+0000', 'last_active': '2024-09-05T16:35:10.079292+0000', 'last_peered': '2024-09-05T16:35:10.079292+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:35:10.079292+0000', 'last_undegraded': '2024-09-05T16:34:56.590972+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 2607, 'log_dups_size': 0, 'ondisk_log_size': 2607, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21330512, 'num_objects': 2604, 'num_object_clones': 0, 'num_object_copies': 10416, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2251, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2604, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2606, 'num_write_kb': 20838, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 354}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:35:12.329 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-05T16:35:12.329 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'2592", 'reported_seq': 3030, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:10.068793+0000', 'last_change': '2024-09-05T16:34:56.800371+0000', 'last_active': '2024-09-05T16:35:10.068793+0000', 'last_peered': '2024-09-05T16:35:10.068793+0000', 'last_clean': '2024-09-05T16:33:53.148717+0000', 'last_became_active': '2024-09-05T16:34:56.791755+0000', 'last_became_peered': '2024-09-05T16:34:56.791755+0000', 'last_unstale': '2024-09-05T16:35:10.068793+0000', 'last_undegraded': '2024-09-05T16:34:56.597522+0000', 'last_fullsized': '2024-09-05T16:34:56.568150+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 2592, 'log_dups_size': 0, 'ondisk_log_size': 2592, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21197984, 'num_objects': 2588, 'num_object_clones': 0, 'num_object_copies': 10352, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2101, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2588, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2590, 'num_write_kb': 20709, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 189, 'num_bytes_recovered': 1531904, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 487}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:35:12.329 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:35:12.329 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'2473", 'reported_seq': 2526, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:08.965340+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:35:08.965340+0000', 'last_peered': '2024-09-05T16:35:08.965340+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:35:08.965340+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 2473, 'log_dups_size': 0, 'ondisk_log_size': 2473, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20231328, 'num_objects': 2470, 'num_object_clones': 0, 'num_object_copies': 9880, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2164, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2470, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2471, 'num_write_kb': 19765, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 308}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:35:12.329 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:35:12.329 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'2559", 'reported_seq': 2600, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:08.965700+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:35:08.965700+0000', 'last_peered': '2024-09-05T16:35:08.965700+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:35:08.965700+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 2559, 'log_dups_size': 0, 'ondisk_log_size': 2559, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20963328, 'num_objects': 2559, 'num_object_clones': 0, 'num_object_copies': 10236, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2559, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2559, 'num_write_kb': 20472, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 309}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:35:12.329 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:35:12.329 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'2487", 'reported_seq': 2551, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:09.997914+0000', 'last_change': '2024-09-05T16:34:56.903436+0000', 'last_active': '2024-09-05T16:35:09.997914+0000', 'last_peered': '2024-09-05T16:35:09.997914+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:35:09.997914+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 2487, 'log_dups_size': 0, 'ondisk_log_size': 2487, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20321440, 'num_objects': 2481, 'num_object_clones': 0, 'num_object_copies': 9924, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2149, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2481, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2485, 'num_write_kb': 19860, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 334}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:35:12.330 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-05T16:35:12.330 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'2606", 'reported_seq': 2669, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:10.020712+0000', 'last_change': '2024-09-05T16:34:56.883319+0000', 'last_active': '2024-09-05T16:35:10.020712+0000', 'last_peered': '2024-09-05T16:35:10.020712+0000', 'last_clean': '2024-09-05T16:33:53.505305+0000', 'last_became_active': '2024-09-05T16:34:56.873389+0000', 'last_became_peered': '2024-09-05T16:34:56.873389+0000', 'last_unstale': '2024-09-05T16:35:10.020712+0000', 'last_undegraded': '2024-09-05T16:34:56.602567+0000', 'last_fullsized': '2024-09-05T16:34:56.559958+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 2606, 'log_dups_size': 0, 'ondisk_log_size': 2606, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21304480, 'num_objects': 2601, 'num_object_clones': 0, 'num_object_copies': 10404, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2241, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2601, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2604, 'num_write_kb': 20820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 362}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:35:12.330 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:35:12.330 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'2589", 'reported_seq': 2643, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:09.053121+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:35:09.053121+0000', 'last_peered': '2024-09-05T16:35:09.053121+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:35:09.053121+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 2589, 'log_dups_size': 0, 'ondisk_log_size': 2589, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21137728, 'num_objects': 2581, 'num_object_clones': 0, 'num_object_copies': 10324, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2264, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2581, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2586, 'num_write_kb': 20665, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 320}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:35:12.330 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:35:12.330 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'2634", 'reported_seq': 2710, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:10.045949+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:35:10.045949+0000', 'last_peered': '2024-09-05T16:35:10.045949+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:35:10.045949+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 2634, 'log_dups_size': 0, 'ondisk_log_size': 2634, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21558432, 'num_objects': 2632, 'num_object_clones': 0, 'num_object_copies': 10528, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2264, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2632, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2632, 'num_write_kb': 21054, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 370}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:35:12.330 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:35:12.330 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'2636", 'reported_seq': 2691, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:09.050725+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:35:09.050725+0000', 'last_peered': '2024-09-05T16:35:09.050725+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:35:09.050725+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 2636, 'log_dups_size': 0, 'ondisk_log_size': 2636, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21548784, 'num_objects': 2631, 'num_object_clones': 0, 'num_object_copies': 10524, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2314, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2631, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2633, 'num_write_kb': 21052, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 320}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:35:12.330 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:35:12.330 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'2579", 'reported_seq': 2614, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:09.036684+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:35:09.036684+0000', 'last_peered': '2024-09-05T16:35:09.036684+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:35:09.036684+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 2579, 'log_dups_size': 0, 'ondisk_log_size': 2579, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21117520, 'num_objects': 2578, 'num_object_clones': 0, 'num_object_copies': 10312, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2265, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2578, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2578, 'num_write_kb': 20623, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 314}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:35:12.330 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-05T16:35:12.331 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'2635", 'reported_seq': 2708, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:10.070303+0000', 'last_change': '2024-09-05T16:34:56.798081+0000', 'last_active': '2024-09-05T16:35:10.070303+0000', 'last_peered': '2024-09-05T16:35:10.070303+0000', 'last_clean': '2024-09-05T16:33:45.082039+0000', 'last_became_active': '2024-09-05T16:34:56.792386+0000', 'last_became_peered': '2024-09-05T16:34:56.792386+0000', 'last_unstale': '2024-09-05T16:35:10.070303+0000', 'last_undegraded': '2024-09-05T16:34:56.600595+0000', 'last_fullsized': '2024-09-05T16:34:56.558203+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 2635, 'log_dups_size': 0, 'ondisk_log_size': 2635, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21576272, 'num_objects': 2634, 'num_object_clones': 0, 'num_object_copies': 10536, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2295, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2634, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2634, 'num_write_kb': 21071, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 340}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:35:12.332 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:35:12.332 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'2695", 'reported_seq': 2746, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:09.017968+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:35:09.017968+0000', 'last_peered': '2024-09-05T16:35:09.017968+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:35:09.017968+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 2695, 'log_dups_size': 0, 'ondisk_log_size': 2695, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22041760, 'num_objects': 2691, 'num_object_clones': 0, 'num_object_copies': 10764, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2359, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2691, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2693, 'num_write_kb': 21540, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 334}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:35:12.332 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:35:12.332 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'2663", 'reported_seq': 2740, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:10.062549+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:35:10.062549+0000', 'last_peered': '2024-09-05T16:35:10.062549+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:35:10.062549+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 2663, 'log_dups_size': 0, 'ondisk_log_size': 2663, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21787808, 'num_objects': 2660, 'num_object_clones': 0, 'num_object_copies': 10640, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2322, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2660, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2661, 'num_write_kb': 21285, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 340}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:35:12.332 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:35:12.332 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'2601", 'reported_seq': 2662, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:10.025419+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:35:10.025419+0000', 'last_peered': '2024-09-05T16:35:10.025419+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:35:10.025419+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 2601, 'log_dups_size': 0, 'ondisk_log_size': 2601, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21263520, 'num_objects': 2596, 'num_object_clones': 0, 'num_object_copies': 10384, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2596, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2599, 'num_write_kb': 20780, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 359}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:35:12.332 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:35:12.497 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 50 20 42604 42584 6.65274 3.00781 0.0407084 0.0234696 2024-09-05T16:35:13.049 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:35:13.049 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 36009/165700 objects degraded (21.731%), 16 pgs degraded","count":16},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":116,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":41425,"data_bytes":339312832,"bytes_used":1022164992,"bytes_avail":385524891648,"bytes_total":386547056640,"degraded_objects":36009,"degraded_total":165700,"degraded_ratio":0.21731442365721182,"recovering_objects_per_sec":11,"recovering_bytes_per_sec":96233,"recovering_keys_per_sec":0,"num_objects_recovered":141,"num_bytes_recovered":1155072,"num_keys_recovered":0,"write_bytes_sec":2772351,"read_op_per_sec":0,"write_op_per_sec":338},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-09-05T16:35:06.800151+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"3":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-05T16:35:13.078 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:35:13.079 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:35:13.497 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 51 20 42984 42964 6.58051 2.96875 0.0362441 0.0237272 2024-09-05T16:35:13.639 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:35:13.639 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:35:13.665 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":82,"stamp":"2024-09-05T16:35:12.801479+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":339312832,"num_objects":41425,"num_object_clones":0,"num_object_copies":165700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36009,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41425,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":41452,"num_write_kb":331498,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":214,"num_bytes_recovered":1531904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41479,"ondisk_log_size":41479,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":998208,"kb_used_data":516608,"kb_used_omap":14,"kb_used_meta":481521,"kb_avail":376489152,"statfs":{"total":386547056640,"available":385524891648,"internally_reserved":0,"allocated":529006592,"data_stored":528275064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":493078071},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,2,12,12,7,6,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":33275904,"num_objects":4062,"num_object_clones":0,"num_object_copies":16248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-141,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4062,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4062,"num_write_kb":32496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":141,"num_bytes_recovered":1155072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002771"},"pg_stats":[{"pgid":"1.f","version":"33'2542","reported_seq":2582,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:09.023046+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:35:09.023046+0000","last_peered":"2024-09-05T16:35:09.023046+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:35:09.023046+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2542,"log_dups_size":0,"ondisk_log_size":2542,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20824064,"num_objects":2542,"num_object_clones":0,"num_object_copies":10168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2542,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2542,"num_write_kb":20336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":280}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'2581","reported_seq":2644,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:09.941023+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:35:09.941023+0000","last_peered":"2024-09-05T16:35:09.941023+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:35:09.941023+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2581,"log_dups_size":0,"ondisk_log_size":2581,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21107872,"num_objects":2577,"num_object_clones":0,"num_object_copies":10308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2577,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2579,"num_write_kb":20628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":310}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'2607","reported_seq":2681,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:10.079292+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:35:10.079292+0000","last_peered":"2024-09-05T16:35:10.079292+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:35:10.079292+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2607,"log_dups_size":0,"ondisk_log_size":2607,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21330512,"num_objects":2604,"num_object_clones":0,"num_object_copies":10416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2604,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2606,"num_write_kb":20838,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":354}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'2592","reported_seq":3030,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:10.068793+0000","last_change":"2024-09-05T16:34:56.800371+0000","last_active":"2024-09-05T16:35:10.068793+0000","last_peered":"2024-09-05T16:35:10.068793+0000","last_clean":"2024-09-05T16:33:53.148717+0000","last_became_active":"2024-09-05T16:34:56.791755+0000","last_became_peered":"2024-09-05T16:34:56.791755+0000","last_unstale":"2024-09-05T16:35:10.068793+0000","last_undegraded":"2024-09-05T16:34:56.597522+0000","last_fullsized":"2024-09-05T16:34:56.568150+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2592,"log_dups_size":0,"ondisk_log_size":2592,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21197984,"num_objects":2588,"num_object_clones":0,"num_object_copies":10352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2101,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2588,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2590,"num_write_kb":20709,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":189,"num_bytes_recovered":1531904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":487}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'2473","reported_seq":2526,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:08.965340+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:35:08.965340+0000","last_peered":"2024-09-05T16:35:08.965340+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:35:08.965340+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2473,"log_dups_size":0,"ondisk_log_size":2473,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20231328,"num_objects":2470,"num_object_clones":0,"num_object_copies":9880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2470,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2471,"num_write_kb":19765,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":308}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'2559","reported_seq":2600,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:08.965700+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:35:08.965700+0000","last_peered":"2024-09-05T16:35:08.965700+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:35:08.965700+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2559,"log_dups_size":0,"ondisk_log_size":2559,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20963328,"num_objects":2559,"num_object_clones":0,"num_object_copies":10236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2559,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2559,"num_write_kb":20472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":309}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'2487","reported_seq":2551,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:09.997914+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:35:09.997914+0000","last_peered":"2024-09-05T16:35:09.997914+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:35:09.997914+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2487,"log_dups_size":0,"ondisk_log_size":2487,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20321440,"num_objects":2481,"num_object_clones":0,"num_object_copies":9924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2481,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2485,"num_write_kb":19860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":334}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'2606","reported_seq":2669,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:10.020712+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:35:10.020712+0000","last_peered":"2024-09-05T16:35:10.020712+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:35:10.020712+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2606,"log_dups_size":0,"ondisk_log_size":2606,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21304480,"num_objects":2601,"num_object_clones":0,"num_object_copies":10404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2601,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2604,"num_write_kb":20820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":362}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'2589","reported_seq":2643,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:09.053121+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:35:09.053121+0000","last_peered":"2024-09-05T16:35:09.053121+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:35:09.053121+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2589,"log_dups_size":0,"ondisk_log_size":2589,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21137728,"num_objects":2581,"num_object_clones":0,"num_object_copies":10324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2581,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2586,"num_write_kb":20665,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":320}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'2634","reported_seq":2710,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:10.045949+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:35:10.045949+0000","last_peered":"2024-09-05T16:35:10.045949+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:35:10.045949+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2634,"log_dups_size":0,"ondisk_log_size":2634,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21558432,"num_objects":2632,"num_object_clones":0,"num_object_copies":10528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2632,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2632,"num_write_kb":21054,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":370}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'2636","reported_seq":2691,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:09.050725+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:35:09.050725+0000","last_peered":"2024-09-05T16:35:09.050725+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:35:09.050725+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2636,"log_dups_size":0,"ondisk_log_size":2636,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21548784,"num_objects":2631,"num_object_clones":0,"num_object_copies":10524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2631,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2633,"num_write_kb":21052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":320}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'2579","reported_seq":2614,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:09.036684+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:35:09.036684+0000","last_peered":"2024-09-05T16:35:09.036684+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:35:09.036684+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2579,"log_dups_size":0,"ondisk_log_size":2579,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21117520,"num_objects":2578,"num_object_clones":0,"num_object_copies":10312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2578,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2578,"num_write_kb":20623,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":314}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'2635","reported_seq":2708,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:10.070303+0000","last_change":"2024-09-05T16:34:56.798081+0000","last_active":"2024-09-05T16:35:10.070303+0000","last_peered":"2024-09-05T16:35:10.070303+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:35:10.070303+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2635,"log_dups_size":0,"ondisk_log_size":2635,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21576272,"num_objects":2634,"num_object_clones":0,"num_object_copies":10536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2634,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2634,"num_write_kb":21071,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":340}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'2695","reported_seq":2746,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:09.017968+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:35:09.017968+0000","last_peered":"2024-09-05T16:35:09.017968+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:35:09.017968+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2695,"log_dups_size":0,"ondisk_log_size":2695,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22041760,"num_objects":2691,"num_object_clones":0,"num_object_copies":10764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2693,"num_write_kb":21540,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":334}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'2663","reported_seq":2740,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:10.062549+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:35:10.062549+0000","last_peered":"2024-09-05T16:35:10.062549+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:35:10.062549+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2663,"log_dups_size":0,"ondisk_log_size":2663,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21787808,"num_objects":2660,"num_object_clones":0,"num_object_copies":10640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2660,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2661,"num_write_kb":21285,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":340}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'2601","reported_seq":2662,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:10.025419+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:35:10.025419+0000","last_peered":"2024-09-05T16:35:10.025419+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:35:10.025419+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2601,"log_dups_size":0,"ondisk_log_size":2601,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21263520,"num_objects":2596,"num_object_clones":0,"num_object_copies":10384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2596,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2599,"num_write_kb":20780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":359}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":339312832,"num_objects":41425,"num_object_clones":0,"num_object_copies":165700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36009,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41425,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":41452,"num_write_kb":331498,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":214,"num_bytes_recovered":1531904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":530702336,"data_stored":530702336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41479,"ondisk_log_size":41479,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084303,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":304256,"kb_used_data":164272,"kb_used_omap":4,"kb_used_meta":139963,"kb_avail":94067584,"statfs":{"total":96636764160,"available":96325206016,"internally_reserved":0,"allocated":168214528,"data_stored":168031646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":143322835},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,6,1,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986180,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65896,"kb_used_data":20312,"kb_used_omap":4,"kb_used_meta":45563,"kb_avail":94305944,"statfs":{"total":96636764160,"available":96569286656,"internally_reserved":0,"allocated":20799488,"data_stored":20616606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":46657234},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,0,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949672986,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":313984,"kb_used_data":166192,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":94057856,"statfs":{"total":96636764160,"available":96315244544,"internally_reserved":0,"allocated":170180608,"data_stored":169997726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":151321029},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,5,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182417,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":314072,"kb_used_data":165832,"kb_used_omap":4,"kb_used_meta":148219,"kb_avail":94057768,"statfs":{"total":96636764160,"available":96315154432,"internally_reserved":0,"allocated":169811968,"data_stored":169629086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":151776973},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,5,2,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":170479616,"data_stored":170479616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":21053440,"data_stored":21053440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":170545152,"data_stored":170545152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":168624128,"data_stored":168624128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:35:14.497 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 52 20 43405 43385 6.51721 3.28906 0.0373564 0.0239623 2024-09-05T16:35:15.497 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 53 20 43834 43814 6.45748 3.35156 0.0540769 0.0241845 2024-09-05T16:35:16.497 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 54 20 44254 44234 6.39866 3.28125 0.00452938 0.0243862 2024-09-05T16:35:16.667 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:35:17.224 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:35:17.225 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:35:17.247 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":84,"stamp":"2024-09-05T16:35:16.802424+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":356229312,"num_objects":43490,"num_object_clones":0,"num_object_copies":173960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35941,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":43517,"num_write_kb":348018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":282,"num_bytes_recovered":2088960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43544,"ondisk_log_size":43544,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1023092,"kb_used_data":549812,"kb_used_omap":14,"kb_used_meta":473201,"kb_avail":376464268,"statfs":{"total":386547056640,"available":385499410432,"internally_reserved":0,"allocated":563007488,"data_stored":562275960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":484558391},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,7,5,10,13,12,6,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":43040768,"num_objects":5254,"num_object_clones":0,"num_object_copies":21016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-209,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5254,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5254,"num_write_kb":42032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":209,"num_bytes_recovered":1712128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002784"},"pg_stats":[{"pgid":"1.f","version":"33'2687","reported_seq":2728,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:14.082702+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:35:14.082702+0000","last_peered":"2024-09-05T16:35:14.082702+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:35:14.082702+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2687,"log_dups_size":0,"ondisk_log_size":2687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22011904,"num_objects":2687,"num_object_clones":0,"num_object_copies":10748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2687,"num_write_kb":21496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":425}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'2705","reported_seq":2767,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:15.081849+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:35:15.081849+0000","last_peered":"2024-09-05T16:35:15.081849+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:35:15.081849+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2705,"log_dups_size":0,"ondisk_log_size":2705,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22123680,"num_objects":2701,"num_object_clones":0,"num_object_copies":10804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2701,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2703,"num_write_kb":21620,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":434}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'2744","reported_seq":2818,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:15.016640+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:35:15.016640+0000","last_peered":"2024-09-05T16:35:15.016640+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:35:15.016640+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2744,"log_dups_size":0,"ondisk_log_size":2744,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22452816,"num_objects":2741,"num_object_clones":0,"num_object_copies":10964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2741,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2743,"num_write_kb":21934,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":491}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'2714","reported_seq":3288,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:15.096382+0000","last_change":"2024-09-05T16:34:56.800371+0000","last_active":"2024-09-05T16:35:15.096382+0000","last_peered":"2024-09-05T16:35:15.096382+0000","last_clean":"2024-09-05T16:33:53.148717+0000","last_became_active":"2024-09-05T16:34:56.791755+0000","last_became_peered":"2024-09-05T16:34:56.791755+0000","last_unstale":"2024-09-05T16:35:15.096382+0000","last_undegraded":"2024-09-05T16:34:56.597522+0000","last_fullsized":"2024-09-05T16:34:56.568150+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2714,"log_dups_size":0,"ondisk_log_size":2714,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22197408,"num_objects":2710,"num_object_clones":0,"num_object_copies":10840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2033,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2710,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2712,"num_write_kb":21685,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":257,"num_bytes_recovered":2088960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":677}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'2616","reported_seq":2669,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:14.024127+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:35:14.024127+0000","last_peered":"2024-09-05T16:35:14.024127+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:35:14.024127+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2616,"log_dups_size":0,"ondisk_log_size":2616,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21402784,"num_objects":2613,"num_object_clones":0,"num_object_copies":10452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2613,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2614,"num_write_kb":20909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":451}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'2689","reported_seq":2730,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:14.059572+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:35:14.059572+0000","last_peered":"2024-09-05T16:35:14.059572+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:35:14.059572+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2689,"log_dups_size":0,"ondisk_log_size":2689,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22028288,"num_objects":2689,"num_object_clones":0,"num_object_copies":10756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2689,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2689,"num_write_kb":21512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":439}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'2618","reported_seq":2682,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:15.114723+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:35:15.114723+0000","last_peered":"2024-09-05T16:35:15.114723+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:35:15.114723+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2618,"log_dups_size":0,"ondisk_log_size":2618,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21394592,"num_objects":2612,"num_object_clones":0,"num_object_copies":10448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2612,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2616,"num_write_kb":20908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":465}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'2729","reported_seq":2792,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:15.085588+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:35:15.085588+0000","last_peered":"2024-09-05T16:35:15.085588+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:35:15.085588+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2729,"log_dups_size":0,"ondisk_log_size":2729,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22312096,"num_objects":2724,"num_object_clones":0,"num_object_copies":10896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2727,"num_write_kb":21804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":485}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'2722","reported_seq":2777,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:14.062207+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:35:14.062207+0000","last_peered":"2024-09-05T16:35:14.062207+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:35:14.062207+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2722,"log_dups_size":0,"ondisk_log_size":2722,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22227264,"num_objects":2714,"num_object_clones":0,"num_object_copies":10856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2714,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2719,"num_write_kb":21729,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":453}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'2767","reported_seq":2843,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:15.081563+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:35:15.081563+0000","last_peered":"2024-09-05T16:35:15.081563+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:35:15.081563+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2767,"log_dups_size":0,"ondisk_log_size":2767,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22647968,"num_objects":2765,"num_object_clones":0,"num_object_copies":11060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2765,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2765,"num_write_kb":22118,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":503}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'2763","reported_seq":2818,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:14.071459+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:35:14.071459+0000","last_peered":"2024-09-05T16:35:14.071459+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:35:14.071459+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2763,"log_dups_size":0,"ondisk_log_size":2763,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22589168,"num_objects":2758,"num_object_clones":0,"num_object_copies":11032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2758,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2760,"num_write_kb":22068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":447}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'2697","reported_seq":2732,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:14.002409+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:35:14.002409+0000","last_peered":"2024-09-05T16:35:14.002409+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:35:14.002409+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2697,"log_dups_size":0,"ondisk_log_size":2697,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22084176,"num_objects":2696,"num_object_clones":0,"num_object_copies":10784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2696,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2696,"num_write_kb":21567,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":432}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'2751","reported_seq":2824,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:15.127922+0000","last_change":"2024-09-05T16:34:56.798081+0000","last_active":"2024-09-05T16:35:15.127922+0000","last_peered":"2024-09-05T16:35:15.127922+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:35:15.127922+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2751,"log_dups_size":0,"ondisk_log_size":2751,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22526544,"num_objects":2750,"num_object_clones":0,"num_object_copies":11000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2750,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2750,"num_write_kb":21999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":456}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'2825","reported_seq":2876,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:14.033299+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:35:14.033299+0000","last_peered":"2024-09-05T16:35:14.033299+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:35:14.033299+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2825,"log_dups_size":0,"ondisk_log_size":2825,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23106720,"num_objects":2821,"num_object_clones":0,"num_object_copies":11284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2821,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2823,"num_write_kb":22580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":464}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'2785","reported_seq":2862,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:15.067106+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:35:15.067106+0000","last_peered":"2024-09-05T16:35:15.067106+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:35:15.067106+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2785,"log_dups_size":0,"ondisk_log_size":2785,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22787232,"num_objects":2782,"num_object_clones":0,"num_object_copies":11128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2782,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2783,"num_write_kb":22261,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":462}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'2732","reported_seq":2795,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:15.097378+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:35:15.097378+0000","last_peered":"2024-09-05T16:35:15.097378+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:35:15.097378+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2732,"log_dups_size":0,"ondisk_log_size":2732,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22336672,"num_objects":2727,"num_object_clones":0,"num_object_copies":10908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2727,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2730,"num_write_kb":21828,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":490}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":356229312,"num_objects":43490,"num_object_clones":0,"num_object_copies":173960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35941,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":43517,"num_write_kb":348018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":282,"num_bytes_recovered":2088960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":564518912,"data_stored":564518912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43544,"ondisk_log_size":43544,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084304,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":320684,"kb_used_data":172380,"kb_used_omap":4,"kb_used_meta":148283,"kb_avail":94051156,"statfs":{"total":96636764160,"available":96308383744,"internally_reserved":0,"allocated":176517120,"data_stored":176334238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":151842515},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,4,2,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986181,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":82944,"kb_used_data":29040,"kb_used_omap":4,"kb_used_meta":53883,"kb_avail":94288896,"statfs":{"total":96636764160,"available":96551829504,"internally_reserved":0,"allocated":29736960,"data_stored":29554078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":55176914},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949672987,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":309604,"kb_used_data":174292,"kb_used_omap":1,"kb_used_meta":135294,"kb_avail":94062236,"statfs":{"total":96636764160,"available":96319729664,"internally_reserved":0,"allocated":178475008,"data_stored":178292126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":138541509},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,7,0,2,6,8],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182418,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":309860,"kb_used_data":174100,"kb_used_omap":4,"kb_used_meta":135739,"kb_avail":94061980,"statfs":{"total":96636764160,"available":96319467520,"internally_reserved":0,"allocated":178278400,"data_stored":178095518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":138997453},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,5,2,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":178831360,"data_stored":178831360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":29753344,"data_stored":29753344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":178884608,"data_stored":178884608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":177049600,"data_stored":177049600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:35:17.249 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:35:17.497 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 55 20 44567 44547 6.32678 2.44531 0.0876675 0.0246834 2024-09-05T16:35:17.878 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:35:17.879 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:35:17.910 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":84,"stamp":"2024-09-05T16:35:16.802424+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":356229312,"num_objects":43490,"num_object_clones":0,"num_object_copies":173960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35941,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":43517,"num_write_kb":348018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":282,"num_bytes_recovered":2088960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43544,"ondisk_log_size":43544,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1023092,"kb_used_data":549812,"kb_used_omap":14,"kb_used_meta":473201,"kb_avail":376464268,"statfs":{"total":386547056640,"available":385499410432,"internally_reserved":0,"allocated":563007488,"data_stored":562275960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":484558391},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,7,5,10,13,12,6,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":43040768,"num_objects":5254,"num_object_clones":0,"num_object_copies":21016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-209,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5254,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5254,"num_write_kb":42032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":209,"num_bytes_recovered":1712128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002784"},"pg_stats":[{"pgid":"1.f","version":"33'2687","reported_seq":2728,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:14.082702+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:35:14.082702+0000","last_peered":"2024-09-05T16:35:14.082702+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:35:14.082702+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2687,"log_dups_size":0,"ondisk_log_size":2687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22011904,"num_objects":2687,"num_object_clones":0,"num_object_copies":10748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2687,"num_write_kb":21496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":425}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'2705","reported_seq":2767,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:15.081849+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:35:15.081849+0000","last_peered":"2024-09-05T16:35:15.081849+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:35:15.081849+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2705,"log_dups_size":0,"ondisk_log_size":2705,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22123680,"num_objects":2701,"num_object_clones":0,"num_object_copies":10804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2701,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2703,"num_write_kb":21620,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":434}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'2744","reported_seq":2818,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:15.016640+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:35:15.016640+0000","last_peered":"2024-09-05T16:35:15.016640+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:35:15.016640+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2744,"log_dups_size":0,"ondisk_log_size":2744,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22452816,"num_objects":2741,"num_object_clones":0,"num_object_copies":10964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2741,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2743,"num_write_kb":21934,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":491}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'2714","reported_seq":3288,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:15.096382+0000","last_change":"2024-09-05T16:34:56.800371+0000","last_active":"2024-09-05T16:35:15.096382+0000","last_peered":"2024-09-05T16:35:15.096382+0000","last_clean":"2024-09-05T16:33:53.148717+0000","last_became_active":"2024-09-05T16:34:56.791755+0000","last_became_peered":"2024-09-05T16:34:56.791755+0000","last_unstale":"2024-09-05T16:35:15.096382+0000","last_undegraded":"2024-09-05T16:34:56.597522+0000","last_fullsized":"2024-09-05T16:34:56.568150+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2714,"log_dups_size":0,"ondisk_log_size":2714,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22197408,"num_objects":2710,"num_object_clones":0,"num_object_copies":10840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2033,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2710,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2712,"num_write_kb":21685,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":257,"num_bytes_recovered":2088960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":677}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'2616","reported_seq":2669,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:14.024127+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:35:14.024127+0000","last_peered":"2024-09-05T16:35:14.024127+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:35:14.024127+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2616,"log_dups_size":0,"ondisk_log_size":2616,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21402784,"num_objects":2613,"num_object_clones":0,"num_object_copies":10452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2613,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2614,"num_write_kb":20909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":451}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'2689","reported_seq":2730,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:14.059572+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:35:14.059572+0000","last_peered":"2024-09-05T16:35:14.059572+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:35:14.059572+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2689,"log_dups_size":0,"ondisk_log_size":2689,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22028288,"num_objects":2689,"num_object_clones":0,"num_object_copies":10756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2689,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2689,"num_write_kb":21512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":439}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'2618","reported_seq":2682,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:15.114723+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:35:15.114723+0000","last_peered":"2024-09-05T16:35:15.114723+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:35:15.114723+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2618,"log_dups_size":0,"ondisk_log_size":2618,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21394592,"num_objects":2612,"num_object_clones":0,"num_object_copies":10448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2612,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2616,"num_write_kb":20908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":465}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'2729","reported_seq":2792,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:15.085588+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:35:15.085588+0000","last_peered":"2024-09-05T16:35:15.085588+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:35:15.085588+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2729,"log_dups_size":0,"ondisk_log_size":2729,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22312096,"num_objects":2724,"num_object_clones":0,"num_object_copies":10896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2727,"num_write_kb":21804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":485}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'2722","reported_seq":2777,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:14.062207+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:35:14.062207+0000","last_peered":"2024-09-05T16:35:14.062207+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:35:14.062207+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2722,"log_dups_size":0,"ondisk_log_size":2722,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22227264,"num_objects":2714,"num_object_clones":0,"num_object_copies":10856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2714,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2719,"num_write_kb":21729,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":453}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'2767","reported_seq":2843,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:15.081563+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:35:15.081563+0000","last_peered":"2024-09-05T16:35:15.081563+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:35:15.081563+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2767,"log_dups_size":0,"ondisk_log_size":2767,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22647968,"num_objects":2765,"num_object_clones":0,"num_object_copies":11060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2765,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2765,"num_write_kb":22118,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":503}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'2763","reported_seq":2818,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:14.071459+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:35:14.071459+0000","last_peered":"2024-09-05T16:35:14.071459+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:35:14.071459+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2763,"log_dups_size":0,"ondisk_log_size":2763,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22589168,"num_objects":2758,"num_object_clones":0,"num_object_copies":11032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2758,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2760,"num_write_kb":22068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":447}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'2697","reported_seq":2732,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:14.002409+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:35:14.002409+0000","last_peered":"2024-09-05T16:35:14.002409+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:35:14.002409+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2697,"log_dups_size":0,"ondisk_log_size":2697,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22084176,"num_objects":2696,"num_object_clones":0,"num_object_copies":10784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2696,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2696,"num_write_kb":21567,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":432}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'2751","reported_seq":2824,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:15.127922+0000","last_change":"2024-09-05T16:34:56.798081+0000","last_active":"2024-09-05T16:35:15.127922+0000","last_peered":"2024-09-05T16:35:15.127922+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:35:15.127922+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2751,"log_dups_size":0,"ondisk_log_size":2751,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22526544,"num_objects":2750,"num_object_clones":0,"num_object_copies":11000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2750,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2750,"num_write_kb":21999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":456}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'2825","reported_seq":2876,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:14.033299+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:35:14.033299+0000","last_peered":"2024-09-05T16:35:14.033299+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:35:14.033299+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2825,"log_dups_size":0,"ondisk_log_size":2825,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23106720,"num_objects":2821,"num_object_clones":0,"num_object_copies":11284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2821,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2823,"num_write_kb":22580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":464}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'2785","reported_seq":2862,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:15.067106+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:35:15.067106+0000","last_peered":"2024-09-05T16:35:15.067106+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:35:15.067106+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2785,"log_dups_size":0,"ondisk_log_size":2785,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22787232,"num_objects":2782,"num_object_clones":0,"num_object_copies":11128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2782,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2783,"num_write_kb":22261,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":462}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'2732","reported_seq":2795,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:15.097378+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:35:15.097378+0000","last_peered":"2024-09-05T16:35:15.097378+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:35:15.097378+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2732,"log_dups_size":0,"ondisk_log_size":2732,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22336672,"num_objects":2727,"num_object_clones":0,"num_object_copies":10908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2727,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2730,"num_write_kb":21828,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":490}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":356229312,"num_objects":43490,"num_object_clones":0,"num_object_copies":173960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35941,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":43517,"num_write_kb":348018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":282,"num_bytes_recovered":2088960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":564518912,"data_stored":564518912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43544,"ondisk_log_size":43544,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084304,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":320684,"kb_used_data":172380,"kb_used_omap":4,"kb_used_meta":148283,"kb_avail":94051156,"statfs":{"total":96636764160,"available":96308383744,"internally_reserved":0,"allocated":176517120,"data_stored":176334238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":151842515},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,4,2,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986181,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":82944,"kb_used_data":29040,"kb_used_omap":4,"kb_used_meta":53883,"kb_avail":94288896,"statfs":{"total":96636764160,"available":96551829504,"internally_reserved":0,"allocated":29736960,"data_stored":29554078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":55176914},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949672987,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":309604,"kb_used_data":174292,"kb_used_omap":1,"kb_used_meta":135294,"kb_avail":94062236,"statfs":{"total":96636764160,"available":96319729664,"internally_reserved":0,"allocated":178475008,"data_stored":178292126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":138541509},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,7,0,2,6,8],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182418,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":309860,"kb_used_data":174100,"kb_used_omap":4,"kb_used_meta":135739,"kb_avail":94061980,"statfs":{"total":96636764160,"available":96319467520,"internally_reserved":0,"allocated":178278400,"data_stored":178095518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":138997453},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,5,2,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":178831360,"data_stored":178831360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":29753344,"data_stored":29753344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":178884608,"data_stored":178884608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":177049600,"data_stored":177049600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:35:17.912 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-05T16:35:17.912 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'2687", 'reported_seq': 2728, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:14.082702+0000', 'last_change': '2024-09-05T16:34:56.833497+0000', 'last_active': '2024-09-05T16:35:14.082702+0000', 'last_peered': '2024-09-05T16:35:14.082702+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.828289+0000', 'last_became_peered': '2024-09-05T16:34:56.828289+0000', 'last_unstale': '2024-09-05T16:35:14.082702+0000', 'last_undegraded': '2024-09-05T16:34:56.612160+0000', 'last_fullsized': '2024-09-05T16:34:56.557632+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 2687, 'log_dups_size': 0, 'ondisk_log_size': 2687, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22011904, 'num_objects': 2687, 'num_object_clones': 0, 'num_object_copies': 10748, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2262, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2687, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2687, 'num_write_kb': 21496, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 425}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:35:17.912 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-05T16:35:17.912 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'2705", 'reported_seq': 2767, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:15.081849+0000', 'last_change': '2024-09-05T16:34:56.825031+0000', 'last_active': '2024-09-05T16:35:15.081849+0000', 'last_peered': '2024-09-05T16:35:15.081849+0000', 'last_clean': '2024-09-05T16:33:53.200762+0000', 'last_became_active': '2024-09-05T16:34:56.820292+0000', 'last_became_peered': '2024-09-05T16:34:56.820292+0000', 'last_unstale': '2024-09-05T16:35:15.081849+0000', 'last_undegraded': '2024-09-05T16:34:56.605776+0000', 'last_fullsized': '2024-09-05T16:34:56.559383+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 2705, 'log_dups_size': 0, 'ondisk_log_size': 2705, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22123680, 'num_objects': 2701, 'num_object_clones': 0, 'num_object_copies': 10804, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2269, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2701, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2703, 'num_write_kb': 21620, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 434}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:35:17.912 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:35:17.912 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'2744", 'reported_seq': 2818, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:15.016640+0000', 'last_change': '2024-09-05T16:34:56.867900+0000', 'last_active': '2024-09-05T16:35:15.016640+0000', 'last_peered': '2024-09-05T16:35:15.016640+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:35:15.016640+0000', 'last_undegraded': '2024-09-05T16:34:56.590972+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 2744, 'log_dups_size': 0, 'ondisk_log_size': 2744, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22452816, 'num_objects': 2741, 'num_object_clones': 0, 'num_object_copies': 10964, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2251, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2741, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2743, 'num_write_kb': 21934, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 491}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:35:17.912 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-05T16:35:17.913 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'2714", 'reported_seq': 3288, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:15.096382+0000', 'last_change': '2024-09-05T16:34:56.800371+0000', 'last_active': '2024-09-05T16:35:15.096382+0000', 'last_peered': '2024-09-05T16:35:15.096382+0000', 'last_clean': '2024-09-05T16:33:53.148717+0000', 'last_became_active': '2024-09-05T16:34:56.791755+0000', 'last_became_peered': '2024-09-05T16:34:56.791755+0000', 'last_unstale': '2024-09-05T16:35:15.096382+0000', 'last_undegraded': '2024-09-05T16:34:56.597522+0000', 'last_fullsized': '2024-09-05T16:34:56.568150+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 2714, 'log_dups_size': 0, 'ondisk_log_size': 2714, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22197408, 'num_objects': 2710, 'num_object_clones': 0, 'num_object_copies': 10840, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2033, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2710, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2712, 'num_write_kb': 21685, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 257, 'num_bytes_recovered': 2088960, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 677}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:35:17.913 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:35:17.913 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'2616", 'reported_seq': 2669, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:14.024127+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:35:14.024127+0000', 'last_peered': '2024-09-05T16:35:14.024127+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:35:14.024127+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 2616, 'log_dups_size': 0, 'ondisk_log_size': 2616, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21402784, 'num_objects': 2613, 'num_object_clones': 0, 'num_object_copies': 10452, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2164, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2613, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2614, 'num_write_kb': 20909, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 451}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:35:17.913 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:35:17.913 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'2689", 'reported_seq': 2730, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:14.059572+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:35:14.059572+0000', 'last_peered': '2024-09-05T16:35:14.059572+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:35:14.059572+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 2689, 'log_dups_size': 0, 'ondisk_log_size': 2689, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22028288, 'num_objects': 2689, 'num_object_clones': 0, 'num_object_copies': 10756, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2689, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2689, 'num_write_kb': 21512, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 439}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:35:17.913 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:35:17.913 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'2618", 'reported_seq': 2682, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:15.114723+0000', 'last_change': '2024-09-05T16:34:56.903436+0000', 'last_active': '2024-09-05T16:35:15.114723+0000', 'last_peered': '2024-09-05T16:35:15.114723+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:35:15.114723+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 2618, 'log_dups_size': 0, 'ondisk_log_size': 2618, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21394592, 'num_objects': 2612, 'num_object_clones': 0, 'num_object_copies': 10448, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2149, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2612, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2616, 'num_write_kb': 20908, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 465}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:35:17.913 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-05T16:35:17.913 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'2729", 'reported_seq': 2792, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:15.085588+0000', 'last_change': '2024-09-05T16:34:56.883319+0000', 'last_active': '2024-09-05T16:35:15.085588+0000', 'last_peered': '2024-09-05T16:35:15.085588+0000', 'last_clean': '2024-09-05T16:33:53.505305+0000', 'last_became_active': '2024-09-05T16:34:56.873389+0000', 'last_became_peered': '2024-09-05T16:34:56.873389+0000', 'last_unstale': '2024-09-05T16:35:15.085588+0000', 'last_undegraded': '2024-09-05T16:34:56.602567+0000', 'last_fullsized': '2024-09-05T16:34:56.559958+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 2729, 'log_dups_size': 0, 'ondisk_log_size': 2729, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22312096, 'num_objects': 2724, 'num_object_clones': 0, 'num_object_copies': 10896, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2241, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2724, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2727, 'num_write_kb': 21804, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 485}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:35:17.913 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:35:17.914 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'2722", 'reported_seq': 2777, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:14.062207+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:35:14.062207+0000', 'last_peered': '2024-09-05T16:35:14.062207+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:35:14.062207+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 2722, 'log_dups_size': 0, 'ondisk_log_size': 2722, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22227264, 'num_objects': 2714, 'num_object_clones': 0, 'num_object_copies': 10856, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2264, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2714, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2719, 'num_write_kb': 21729, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 453}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:35:17.914 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:35:17.914 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'2767", 'reported_seq': 2843, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:15.081563+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:35:15.081563+0000', 'last_peered': '2024-09-05T16:35:15.081563+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:35:15.081563+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 2767, 'log_dups_size': 0, 'ondisk_log_size': 2767, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22647968, 'num_objects': 2765, 'num_object_clones': 0, 'num_object_copies': 11060, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2264, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2765, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2765, 'num_write_kb': 22118, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 503}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:35:17.914 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:35:17.914 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'2763", 'reported_seq': 2818, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:14.071459+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:35:14.071459+0000', 'last_peered': '2024-09-05T16:35:14.071459+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:35:14.071459+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 2763, 'log_dups_size': 0, 'ondisk_log_size': 2763, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22589168, 'num_objects': 2758, 'num_object_clones': 0, 'num_object_copies': 11032, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2314, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2758, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2760, 'num_write_kb': 22068, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 447}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:35:17.914 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:35:17.914 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'2697", 'reported_seq': 2732, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:14.002409+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:35:14.002409+0000', 'last_peered': '2024-09-05T16:35:14.002409+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:35:14.002409+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 2697, 'log_dups_size': 0, 'ondisk_log_size': 2697, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22084176, 'num_objects': 2696, 'num_object_clones': 0, 'num_object_copies': 10784, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2265, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2696, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2696, 'num_write_kb': 21567, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 432}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:35:17.914 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-05T16:35:17.914 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'2751", 'reported_seq': 2824, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:15.127922+0000', 'last_change': '2024-09-05T16:34:56.798081+0000', 'last_active': '2024-09-05T16:35:15.127922+0000', 'last_peered': '2024-09-05T16:35:15.127922+0000', 'last_clean': '2024-09-05T16:33:45.082039+0000', 'last_became_active': '2024-09-05T16:34:56.792386+0000', 'last_became_peered': '2024-09-05T16:34:56.792386+0000', 'last_unstale': '2024-09-05T16:35:15.127922+0000', 'last_undegraded': '2024-09-05T16:34:56.600595+0000', 'last_fullsized': '2024-09-05T16:34:56.558203+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 2751, 'log_dups_size': 0, 'ondisk_log_size': 2751, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22526544, 'num_objects': 2750, 'num_object_clones': 0, 'num_object_copies': 11000, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2295, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2750, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2750, 'num_write_kb': 21999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 456}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:35:17.914 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:35:17.915 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'2825", 'reported_seq': 2876, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:14.033299+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:35:14.033299+0000', 'last_peered': '2024-09-05T16:35:14.033299+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:35:14.033299+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 2825, 'log_dups_size': 0, 'ondisk_log_size': 2825, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23106720, 'num_objects': 2821, 'num_object_clones': 0, 'num_object_copies': 11284, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2359, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2821, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2823, 'num_write_kb': 22580, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 464}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:35:17.915 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:35:17.915 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'2785", 'reported_seq': 2862, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:15.067106+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:35:15.067106+0000', 'last_peered': '2024-09-05T16:35:15.067106+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:35:15.067106+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 2785, 'log_dups_size': 0, 'ondisk_log_size': 2785, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22787232, 'num_objects': 2782, 'num_object_clones': 0, 'num_object_copies': 11128, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2322, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2782, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2783, 'num_write_kb': 22261, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 462}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:35:17.915 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:35:17.915 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'2732", 'reported_seq': 2795, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:15.097378+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:35:15.097378+0000', 'last_peered': '2024-09-05T16:35:15.097378+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:35:15.097378+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 2732, 'log_dups_size': 0, 'ondisk_log_size': 2732, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22336672, 'num_objects': 2727, 'num_object_clones': 0, 'num_object_copies': 10908, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2727, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2730, 'num_write_kb': 21828, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 490}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:35:17.915 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:35:18.497 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 56 19 44940 44921 6.26598 2.92188 0.0688893 0.0249211 2024-09-05T16:35:18.654 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:35:18.654 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 35941/173960 objects degraded (20.660%), 16 pgs degraded","count":16},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":121,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":43490,"data_bytes":356229312,"bytes_used":1047646208,"bytes_avail":385499410432,"bytes_total":386547056640,"degraded_objects":35941,"degraded_total":173960,"degraded_ratio":0.20660496665900208,"recovering_objects_per_sec":17,"recovering_bytes_per_sec":142644,"recovering_keys_per_sec":0,"num_objects_recovered":209,"num_bytes_recovered":1712128,"num_keys_recovered":0,"write_bytes_sec":3585898,"read_op_per_sec":0,"write_op_per_sec":437},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-05T16:35:14.802011+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-05T16:35:18.682 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:35:18.682 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:35:19.280 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:35:19.313 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:35:19.315 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":85,"stamp":"2024-09-05T16:35:18.802793+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":356229312,"num_objects":43490,"num_object_clones":0,"num_object_copies":173960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35941,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":43517,"num_write_kb":348018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":282,"num_bytes_recovered":2088960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43544,"ondisk_log_size":43544,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1023092,"kb_used_data":549812,"kb_used_omap":14,"kb_used_meta":473201,"kb_avail":376464268,"statfs":{"total":386547056640,"available":385499410432,"internally_reserved":0,"allocated":563007488,"data_stored":562275960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":484558391},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,7,5,10,13,12,6,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":33292288,"num_objects":4064,"num_object_clones":0,"num_object_copies":16256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-139,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4064,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4064,"num_write_kb":32512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":139,"num_bytes_recovered":1138688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002732"},"pg_stats":[{"pgid":"1.f","version":"33'2687","reported_seq":2728,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:14.082702+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:35:14.082702+0000","last_peered":"2024-09-05T16:35:14.082702+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:35:14.082702+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2687,"log_dups_size":0,"ondisk_log_size":2687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22011904,"num_objects":2687,"num_object_clones":0,"num_object_copies":10748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2687,"num_write_kb":21496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":425}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'2705","reported_seq":2767,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:15.081849+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:35:15.081849+0000","last_peered":"2024-09-05T16:35:15.081849+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:35:15.081849+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2705,"log_dups_size":0,"ondisk_log_size":2705,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22123680,"num_objects":2701,"num_object_clones":0,"num_object_copies":10804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2701,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2703,"num_write_kb":21620,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":434}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'2744","reported_seq":2818,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:15.016640+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:35:15.016640+0000","last_peered":"2024-09-05T16:35:15.016640+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:35:15.016640+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2744,"log_dups_size":0,"ondisk_log_size":2744,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22452816,"num_objects":2741,"num_object_clones":0,"num_object_copies":10964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2741,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2743,"num_write_kb":21934,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":491}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'2714","reported_seq":3288,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:15.096382+0000","last_change":"2024-09-05T16:34:56.800371+0000","last_active":"2024-09-05T16:35:15.096382+0000","last_peered":"2024-09-05T16:35:15.096382+0000","last_clean":"2024-09-05T16:33:53.148717+0000","last_became_active":"2024-09-05T16:34:56.791755+0000","last_became_peered":"2024-09-05T16:34:56.791755+0000","last_unstale":"2024-09-05T16:35:15.096382+0000","last_undegraded":"2024-09-05T16:34:56.597522+0000","last_fullsized":"2024-09-05T16:34:56.568150+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2714,"log_dups_size":0,"ondisk_log_size":2714,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22197408,"num_objects":2710,"num_object_clones":0,"num_object_copies":10840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2033,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2710,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2712,"num_write_kb":21685,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":257,"num_bytes_recovered":2088960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":677}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'2616","reported_seq":2669,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:14.024127+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:35:14.024127+0000","last_peered":"2024-09-05T16:35:14.024127+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:35:14.024127+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2616,"log_dups_size":0,"ondisk_log_size":2616,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21402784,"num_objects":2613,"num_object_clones":0,"num_object_copies":10452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2613,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2614,"num_write_kb":20909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":451}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'2689","reported_seq":2730,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:14.059572+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:35:14.059572+0000","last_peered":"2024-09-05T16:35:14.059572+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:35:14.059572+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2689,"log_dups_size":0,"ondisk_log_size":2689,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22028288,"num_objects":2689,"num_object_clones":0,"num_object_copies":10756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2689,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2689,"num_write_kb":21512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":439}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'2618","reported_seq":2682,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:15.114723+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:35:15.114723+0000","last_peered":"2024-09-05T16:35:15.114723+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:35:15.114723+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2618,"log_dups_size":0,"ondisk_log_size":2618,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21394592,"num_objects":2612,"num_object_clones":0,"num_object_copies":10448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2612,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2616,"num_write_kb":20908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":465}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'2729","reported_seq":2792,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:15.085588+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:35:15.085588+0000","last_peered":"2024-09-05T16:35:15.085588+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:35:15.085588+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2729,"log_dups_size":0,"ondisk_log_size":2729,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22312096,"num_objects":2724,"num_object_clones":0,"num_object_copies":10896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2727,"num_write_kb":21804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":485}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'2722","reported_seq":2777,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:14.062207+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:35:14.062207+0000","last_peered":"2024-09-05T16:35:14.062207+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:35:14.062207+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2722,"log_dups_size":0,"ondisk_log_size":2722,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22227264,"num_objects":2714,"num_object_clones":0,"num_object_copies":10856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2714,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2719,"num_write_kb":21729,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":453}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'2767","reported_seq":2843,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:15.081563+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:35:15.081563+0000","last_peered":"2024-09-05T16:35:15.081563+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:35:15.081563+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2767,"log_dups_size":0,"ondisk_log_size":2767,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22647968,"num_objects":2765,"num_object_clones":0,"num_object_copies":11060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2765,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2765,"num_write_kb":22118,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":503}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'2763","reported_seq":2818,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:14.071459+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:35:14.071459+0000","last_peered":"2024-09-05T16:35:14.071459+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:35:14.071459+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2763,"log_dups_size":0,"ondisk_log_size":2763,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22589168,"num_objects":2758,"num_object_clones":0,"num_object_copies":11032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2758,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2760,"num_write_kb":22068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":447}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'2697","reported_seq":2732,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:14.002409+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:35:14.002409+0000","last_peered":"2024-09-05T16:35:14.002409+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:35:14.002409+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2697,"log_dups_size":0,"ondisk_log_size":2697,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22084176,"num_objects":2696,"num_object_clones":0,"num_object_copies":10784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2696,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2696,"num_write_kb":21567,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":432}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'2751","reported_seq":2824,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:15.127922+0000","last_change":"2024-09-05T16:34:56.798081+0000","last_active":"2024-09-05T16:35:15.127922+0000","last_peered":"2024-09-05T16:35:15.127922+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:35:15.127922+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2751,"log_dups_size":0,"ondisk_log_size":2751,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22526544,"num_objects":2750,"num_object_clones":0,"num_object_copies":11000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2750,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2750,"num_write_kb":21999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":456}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'2825","reported_seq":2876,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:14.033299+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:35:14.033299+0000","last_peered":"2024-09-05T16:35:14.033299+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:35:14.033299+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2825,"log_dups_size":0,"ondisk_log_size":2825,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23106720,"num_objects":2821,"num_object_clones":0,"num_object_copies":11284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2821,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2823,"num_write_kb":22580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":464}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'2785","reported_seq":2862,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:15.067106+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:35:15.067106+0000","last_peered":"2024-09-05T16:35:15.067106+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:35:15.067106+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2785,"log_dups_size":0,"ondisk_log_size":2785,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22787232,"num_objects":2782,"num_object_clones":0,"num_object_copies":11128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2782,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2783,"num_write_kb":22261,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":462}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'2732","reported_seq":2795,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:15.097378+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:35:15.097378+0000","last_peered":"2024-09-05T16:35:15.097378+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:35:15.097378+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2732,"log_dups_size":0,"ondisk_log_size":2732,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22336672,"num_objects":2727,"num_object_clones":0,"num_object_copies":10908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2727,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2730,"num_write_kb":21828,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":490}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":356229312,"num_objects":43490,"num_object_clones":0,"num_object_copies":173960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35941,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":43517,"num_write_kb":348018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":282,"num_bytes_recovered":2088960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":564518912,"data_stored":564518912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43544,"ondisk_log_size":43544,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084304,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":320684,"kb_used_data":172380,"kb_used_omap":4,"kb_used_meta":148283,"kb_avail":94051156,"statfs":{"total":96636764160,"available":96308383744,"internally_reserved":0,"allocated":176517120,"data_stored":176334238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":151842515},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,4,2,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986181,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":82944,"kb_used_data":29040,"kb_used_omap":4,"kb_used_meta":53883,"kb_avail":94288896,"statfs":{"total":96636764160,"available":96551829504,"internally_reserved":0,"allocated":29736960,"data_stored":29554078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":55176914},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949672987,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":309604,"kb_used_data":174292,"kb_used_omap":1,"kb_used_meta":135294,"kb_avail":94062236,"statfs":{"total":96636764160,"available":96319729664,"internally_reserved":0,"allocated":178475008,"data_stored":178292126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":138541509},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,7,0,2,6,8],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182418,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":309860,"kb_used_data":174100,"kb_used_omap":4,"kb_used_meta":135739,"kb_avail":94061980,"statfs":{"total":96636764160,"available":96319467520,"internally_reserved":0,"allocated":178278400,"data_stored":178095518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":138997453},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,5,2,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":178831360,"data_stored":178831360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":29753344,"data_stored":29753344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":178884608,"data_stored":178884608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":177049600,"data_stored":177049600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:35:19.500 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 57 20 45315 45295 6.20707 2.92188 0.0338936 0.0251563 2024-09-05T16:35:20.501 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 58 20 45736 45716 6.15667 3.28906 0.0541479 0.0253607 2024-09-05T16:35:21.501 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 59 20 46151 46131 6.10727 3.24219 0.0336382 0.0255648 2024-09-05T16:35:22.316 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:35:22.501 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout:2024-09-05T16:35:22.508625+0000 min lat: 0.00357551 max lat: 2.29705 avg lat: 0.0257662 2024-09-05T16:35:22.501 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-05T16:35:22.501 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 60 20 46571 46551 6.06017 3.28125 0.0400537 0.0257662 2024-09-05T16:35:22.926 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:35:22.927 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:35:22.954 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":87,"stamp":"2024-09-05T16:35:22.803803+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":371957952,"num_objects":45410,"num_object_clones":0,"num_object_copies":181640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35843,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45410,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45437,"num_write_kb":363378,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":380,"num_bytes_recovered":2891776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":45464,"ondisk_log_size":45464,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1082252,"kb_used_data":581196,"kb_used_omap":14,"kb_used_meta":500977,"kb_avail":376405108,"statfs":{"total":386547056640,"available":385438830592,"internally_reserved":0,"allocated":595144704,"data_stored":594413176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":513001015},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,1,2,8,12,5,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":32645120,"num_objects":3985,"num_object_clones":0,"num_object_copies":15940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-166,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":31880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":166,"num_bytes_recovered":1359872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002836"},"pg_stats":[{"pgid":"1.f","version":"33'2811","reported_seq":2852,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:19.067571+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:35:19.067571+0000","last_peered":"2024-09-05T16:35:19.067571+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:35:19.067571+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2811,"log_dups_size":0,"ondisk_log_size":2811,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23027712,"num_objects":2811,"num_object_clones":0,"num_object_copies":11244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2811,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2811,"num_write_kb":22488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":549}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'2827","reported_seq":2887,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:20.085937+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:35:20.085937+0000","last_peered":"2024-09-05T16:35:20.085937+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:35:20.085937+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2827,"log_dups_size":0,"ondisk_log_size":2827,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23123104,"num_objects":2823,"num_object_clones":0,"num_object_copies":11292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2823,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2825,"num_write_kb":22596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":556}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'2869","reported_seq":2943,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:20.078358+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:35:20.078358+0000","last_peered":"2024-09-05T16:35:20.078358+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:35:20.078358+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2869,"log_dups_size":0,"ondisk_log_size":2869,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23476816,"num_objects":2866,"num_object_clones":0,"num_object_copies":11464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2866,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2868,"num_write_kb":22934,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":616}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'2829","reported_seq":3599,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:20.125085+0000","last_change":"2024-09-05T16:34:56.800371+0000","last_active":"2024-09-05T16:35:20.125085+0000","last_peered":"2024-09-05T16:35:20.125085+0000","last_clean":"2024-09-05T16:33:53.148717+0000","last_became_active":"2024-09-05T16:34:56.791755+0000","last_became_peered":"2024-09-05T16:34:56.791755+0000","last_unstale":"2024-09-05T16:35:20.125085+0000","last_undegraded":"2024-09-05T16:34:56.597522+0000","last_fullsized":"2024-09-05T16:34:56.568150+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2829,"log_dups_size":0,"ondisk_log_size":2829,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23139488,"num_objects":2825,"num_object_clones":0,"num_object_copies":11300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1935,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2825,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2827,"num_write_kb":22605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":355,"num_bytes_recovered":2891776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":890}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'2738","reported_seq":2791,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:19.073649+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:35:19.073649+0000","last_peered":"2024-09-05T16:35:19.073649+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:35:19.073649+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2738,"log_dups_size":0,"ondisk_log_size":2738,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22402208,"num_objects":2735,"num_object_clones":0,"num_object_copies":10940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2735,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2736,"num_write_kb":21885,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":573}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'2805","reported_seq":2846,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:19.056470+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:35:19.056470+0000","last_peered":"2024-09-05T16:35:19.056470+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:35:19.056470+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2805,"log_dups_size":0,"ondisk_log_size":2805,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22978560,"num_objects":2805,"num_object_clones":0,"num_object_copies":11220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2805,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2805,"num_write_kb":22440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":555}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'2721","reported_seq":2785,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:20.091315+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:35:20.091315+0000","last_peered":"2024-09-05T16:35:20.091315+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:35:20.091315+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2721,"log_dups_size":0,"ondisk_log_size":2721,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22238368,"num_objects":2715,"num_object_clones":0,"num_object_copies":10860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2715,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2719,"num_write_kb":21732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":568}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'2850","reported_seq":2912,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:20.156112+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:35:20.156112+0000","last_peered":"2024-09-05T16:35:20.156112+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:35:20.156112+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2850,"log_dups_size":0,"ondisk_log_size":2850,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23303328,"num_objects":2845,"num_object_clones":0,"num_object_copies":11380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2845,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2848,"num_write_kb":22772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":606}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'2844","reported_seq":2899,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:18.932541+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:35:18.932541+0000","last_peered":"2024-09-05T16:35:18.932541+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:35:18.932541+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2844,"log_dups_size":0,"ondisk_log_size":2844,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23226688,"num_objects":2836,"num_object_clones":0,"num_object_copies":11344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2836,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2841,"num_write_kb":22705,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":575}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'2922","reported_seq":2998,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:20.122703+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:35:20.122703+0000","last_peered":"2024-09-05T16:35:20.122703+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:35:20.122703+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2922,"log_dups_size":0,"ondisk_log_size":2922,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23917728,"num_objects":2920,"num_object_clones":0,"num_object_copies":11680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2920,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2920,"num_write_kb":23358,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":658}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'2882","reported_seq":2937,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:18.990219+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:35:18.990219+0000","last_peered":"2024-09-05T16:35:18.990219+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:35:18.990219+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2882,"log_dups_size":0,"ondisk_log_size":2882,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23564016,"num_objects":2877,"num_object_clones":0,"num_object_copies":11508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2877,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2879,"num_write_kb":23020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":566}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'2830","reported_seq":2865,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:19.016300+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:35:19.016300+0000","last_peered":"2024-09-05T16:35:19.016300+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:35:19.016300+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2830,"log_dups_size":0,"ondisk_log_size":2830,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23173712,"num_objects":2829,"num_object_clones":0,"num_object_copies":11316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2829,"num_write_kb":22631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":565}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'2848","reported_seq":2921,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:20.013228+0000","last_change":"2024-09-05T16:34:56.798081+0000","last_active":"2024-09-05T16:35:20.013228+0000","last_peered":"2024-09-05T16:35:20.013228+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:35:20.013228+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2848,"log_dups_size":0,"ondisk_log_size":2848,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23321168,"num_objects":2847,"num_object_clones":0,"num_object_copies":11388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2847,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2847,"num_write_kb":22775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":553}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'2930","reported_seq":2981,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:19.071712+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:35:19.071712+0000","last_peered":"2024-09-05T16:35:19.071712+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:35:19.071712+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2930,"log_dups_size":0,"ondisk_log_size":2930,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23966880,"num_objects":2926,"num_object_clones":0,"num_object_copies":11704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2928,"num_write_kb":23420,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":569}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'2904","reported_seq":2981,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:20.138843+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:35:20.138843+0000","last_peered":"2024-09-05T16:35:20.138843+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:35:20.138843+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2904,"log_dups_size":0,"ondisk_log_size":2904,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23762080,"num_objects":2901,"num_object_clones":0,"num_object_copies":11604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2901,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2902,"num_write_kb":23213,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":581}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'2854","reported_seq":2917,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:20.107757+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:35:20.107757+0000","last_peered":"2024-09-05T16:35:20.107757+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:35:20.107757+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2854,"log_dups_size":0,"ondisk_log_size":2854,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23336096,"num_objects":2849,"num_object_clones":0,"num_object_copies":11396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2849,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2852,"num_write_kb":22804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":612}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":371957952,"num_objects":45410,"num_object_clones":0,"num_object_copies":181640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35843,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45410,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45437,"num_write_kb":363378,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":380,"num_bytes_recovered":2891776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":596566016,"data_stored":596566016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":45464,"ondisk_log_size":45464,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084305,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":320124,"kb_used_data":180140,"kb_used_omap":4,"kb_used_meta":139963,"kb_avail":94051716,"statfs":{"total":96636764160,"available":96308957184,"internally_reserved":0,"allocated":184463360,"data_stored":184280478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":143322835},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,7,2,0,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986182,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":118928,"kb_used_data":37248,"kb_used_omap":4,"kb_used_meta":81659,"kb_avail":94252912,"statfs":{"total":96636764160,"available":96514981888,"internally_reserved":0,"allocated":38141952,"data_stored":37959070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":83619538},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949672988,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":321400,"kb_used_data":181928,"kb_used_omap":1,"kb_used_meta":139454,"kb_avail":94050440,"statfs":{"total":96636764160,"available":96307650560,"internally_reserved":0,"allocated":186294272,"data_stored":186111390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":142801349},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,1,1,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182419,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":321800,"kb_used_data":181880,"kb_used_omap":4,"kb_used_meta":139899,"kb_avail":94050040,"statfs":{"total":96636764160,"available":96307240960,"internally_reserved":0,"allocated":186245120,"data_stored":186062238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":143257293},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,0,2,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":186732544,"data_stored":186732544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":38117376,"data_stored":38117376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":186724352,"data_stored":186724352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":184991744,"data_stored":184991744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:35:22.955 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:35:23.503 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 61 20 46932 46912 6.00683 2.82031 0.0642645 0.025998 2024-09-05T16:35:23.528 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:35:23.528 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:35:23.551 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":87,"stamp":"2024-09-05T16:35:22.803803+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":371957952,"num_objects":45410,"num_object_clones":0,"num_object_copies":181640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35843,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45410,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45437,"num_write_kb":363378,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":380,"num_bytes_recovered":2891776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":45464,"ondisk_log_size":45464,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1082252,"kb_used_data":581196,"kb_used_omap":14,"kb_used_meta":500977,"kb_avail":376405108,"statfs":{"total":386547056640,"available":385438830592,"internally_reserved":0,"allocated":595144704,"data_stored":594413176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":513001015},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,1,2,8,12,5,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":32645120,"num_objects":3985,"num_object_clones":0,"num_object_copies":15940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-166,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":31880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":166,"num_bytes_recovered":1359872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002836"},"pg_stats":[{"pgid":"1.f","version":"33'2811","reported_seq":2852,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:19.067571+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:35:19.067571+0000","last_peered":"2024-09-05T16:35:19.067571+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:35:19.067571+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2811,"log_dups_size":0,"ondisk_log_size":2811,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23027712,"num_objects":2811,"num_object_clones":0,"num_object_copies":11244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2811,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2811,"num_write_kb":22488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":549}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'2827","reported_seq":2887,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:20.085937+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:35:20.085937+0000","last_peered":"2024-09-05T16:35:20.085937+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:35:20.085937+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2827,"log_dups_size":0,"ondisk_log_size":2827,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23123104,"num_objects":2823,"num_object_clones":0,"num_object_copies":11292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2823,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2825,"num_write_kb":22596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":556}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'2869","reported_seq":2943,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:20.078358+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:35:20.078358+0000","last_peered":"2024-09-05T16:35:20.078358+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:35:20.078358+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2869,"log_dups_size":0,"ondisk_log_size":2869,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23476816,"num_objects":2866,"num_object_clones":0,"num_object_copies":11464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2866,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2868,"num_write_kb":22934,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":616}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'2829","reported_seq":3599,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:20.125085+0000","last_change":"2024-09-05T16:34:56.800371+0000","last_active":"2024-09-05T16:35:20.125085+0000","last_peered":"2024-09-05T16:35:20.125085+0000","last_clean":"2024-09-05T16:33:53.148717+0000","last_became_active":"2024-09-05T16:34:56.791755+0000","last_became_peered":"2024-09-05T16:34:56.791755+0000","last_unstale":"2024-09-05T16:35:20.125085+0000","last_undegraded":"2024-09-05T16:34:56.597522+0000","last_fullsized":"2024-09-05T16:34:56.568150+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2829,"log_dups_size":0,"ondisk_log_size":2829,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23139488,"num_objects":2825,"num_object_clones":0,"num_object_copies":11300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1935,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2825,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2827,"num_write_kb":22605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":355,"num_bytes_recovered":2891776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":890}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'2738","reported_seq":2791,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:19.073649+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:35:19.073649+0000","last_peered":"2024-09-05T16:35:19.073649+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:35:19.073649+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2738,"log_dups_size":0,"ondisk_log_size":2738,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22402208,"num_objects":2735,"num_object_clones":0,"num_object_copies":10940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2735,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2736,"num_write_kb":21885,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":573}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'2805","reported_seq":2846,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:19.056470+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:35:19.056470+0000","last_peered":"2024-09-05T16:35:19.056470+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:35:19.056470+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2805,"log_dups_size":0,"ondisk_log_size":2805,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22978560,"num_objects":2805,"num_object_clones":0,"num_object_copies":11220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2805,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2805,"num_write_kb":22440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":555}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'2721","reported_seq":2785,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:20.091315+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:35:20.091315+0000","last_peered":"2024-09-05T16:35:20.091315+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:35:20.091315+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2721,"log_dups_size":0,"ondisk_log_size":2721,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22238368,"num_objects":2715,"num_object_clones":0,"num_object_copies":10860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2715,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2719,"num_write_kb":21732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":568}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'2850","reported_seq":2912,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:20.156112+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:35:20.156112+0000","last_peered":"2024-09-05T16:35:20.156112+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:35:20.156112+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2850,"log_dups_size":0,"ondisk_log_size":2850,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23303328,"num_objects":2845,"num_object_clones":0,"num_object_copies":11380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2845,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2848,"num_write_kb":22772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":606}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'2844","reported_seq":2899,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:18.932541+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:35:18.932541+0000","last_peered":"2024-09-05T16:35:18.932541+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:35:18.932541+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2844,"log_dups_size":0,"ondisk_log_size":2844,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23226688,"num_objects":2836,"num_object_clones":0,"num_object_copies":11344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2836,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2841,"num_write_kb":22705,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":575}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'2922","reported_seq":2998,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:20.122703+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:35:20.122703+0000","last_peered":"2024-09-05T16:35:20.122703+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:35:20.122703+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2922,"log_dups_size":0,"ondisk_log_size":2922,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23917728,"num_objects":2920,"num_object_clones":0,"num_object_copies":11680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2920,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2920,"num_write_kb":23358,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":658}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'2882","reported_seq":2937,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:18.990219+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:35:18.990219+0000","last_peered":"2024-09-05T16:35:18.990219+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:35:18.990219+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2882,"log_dups_size":0,"ondisk_log_size":2882,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23564016,"num_objects":2877,"num_object_clones":0,"num_object_copies":11508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2877,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2879,"num_write_kb":23020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":566}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'2830","reported_seq":2865,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:19.016300+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:35:19.016300+0000","last_peered":"2024-09-05T16:35:19.016300+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:35:19.016300+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2830,"log_dups_size":0,"ondisk_log_size":2830,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23173712,"num_objects":2829,"num_object_clones":0,"num_object_copies":11316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2829,"num_write_kb":22631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":565}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'2848","reported_seq":2921,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:20.013228+0000","last_change":"2024-09-05T16:34:56.798081+0000","last_active":"2024-09-05T16:35:20.013228+0000","last_peered":"2024-09-05T16:35:20.013228+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:35:20.013228+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2848,"log_dups_size":0,"ondisk_log_size":2848,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23321168,"num_objects":2847,"num_object_clones":0,"num_object_copies":11388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2847,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2847,"num_write_kb":22775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":553}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'2930","reported_seq":2981,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:19.071712+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:35:19.071712+0000","last_peered":"2024-09-05T16:35:19.071712+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:35:19.071712+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2930,"log_dups_size":0,"ondisk_log_size":2930,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23966880,"num_objects":2926,"num_object_clones":0,"num_object_copies":11704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2928,"num_write_kb":23420,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":569}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'2904","reported_seq":2981,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:20.138843+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:35:20.138843+0000","last_peered":"2024-09-05T16:35:20.138843+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:35:20.138843+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2904,"log_dups_size":0,"ondisk_log_size":2904,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23762080,"num_objects":2901,"num_object_clones":0,"num_object_copies":11604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2901,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2902,"num_write_kb":23213,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":581}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'2854","reported_seq":2917,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:20.107757+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:35:20.107757+0000","last_peered":"2024-09-05T16:35:20.107757+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:35:20.107757+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2854,"log_dups_size":0,"ondisk_log_size":2854,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23336096,"num_objects":2849,"num_object_clones":0,"num_object_copies":11396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2849,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2852,"num_write_kb":22804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":612}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":371957952,"num_objects":45410,"num_object_clones":0,"num_object_copies":181640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35843,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45410,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45437,"num_write_kb":363378,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":380,"num_bytes_recovered":2891776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":596566016,"data_stored":596566016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":45464,"ondisk_log_size":45464,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084305,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":320124,"kb_used_data":180140,"kb_used_omap":4,"kb_used_meta":139963,"kb_avail":94051716,"statfs":{"total":96636764160,"available":96308957184,"internally_reserved":0,"allocated":184463360,"data_stored":184280478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":143322835},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,7,2,0,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986182,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":118928,"kb_used_data":37248,"kb_used_omap":4,"kb_used_meta":81659,"kb_avail":94252912,"statfs":{"total":96636764160,"available":96514981888,"internally_reserved":0,"allocated":38141952,"data_stored":37959070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":83619538},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949672988,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":321400,"kb_used_data":181928,"kb_used_omap":1,"kb_used_meta":139454,"kb_avail":94050440,"statfs":{"total":96636764160,"available":96307650560,"internally_reserved":0,"allocated":186294272,"data_stored":186111390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":142801349},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,1,1,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182419,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":321800,"kb_used_data":181880,"kb_used_omap":4,"kb_used_meta":139899,"kb_avail":94050040,"statfs":{"total":96636764160,"available":96307240960,"internally_reserved":0,"allocated":186245120,"data_stored":186062238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":143257293},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,0,2,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":186732544,"data_stored":186732544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":38117376,"data_stored":38117376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":186724352,"data_stored":186724352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":184991744,"data_stored":184991744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:35:23.553 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-05T16:35:23.553 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'2811", 'reported_seq': 2852, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:19.067571+0000', 'last_change': '2024-09-05T16:34:56.833497+0000', 'last_active': '2024-09-05T16:35:19.067571+0000', 'last_peered': '2024-09-05T16:35:19.067571+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.828289+0000', 'last_became_peered': '2024-09-05T16:34:56.828289+0000', 'last_unstale': '2024-09-05T16:35:19.067571+0000', 'last_undegraded': '2024-09-05T16:34:56.612160+0000', 'last_fullsized': '2024-09-05T16:34:56.557632+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 2811, 'log_dups_size': 0, 'ondisk_log_size': 2811, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23027712, 'num_objects': 2811, 'num_object_clones': 0, 'num_object_copies': 11244, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2262, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2811, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2811, 'num_write_kb': 22488, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 549}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:35:23.553 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-05T16:35:23.553 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'2827", 'reported_seq': 2887, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:20.085937+0000', 'last_change': '2024-09-05T16:34:56.825031+0000', 'last_active': '2024-09-05T16:35:20.085937+0000', 'last_peered': '2024-09-05T16:35:20.085937+0000', 'last_clean': '2024-09-05T16:33:53.200762+0000', 'last_became_active': '2024-09-05T16:34:56.820292+0000', 'last_became_peered': '2024-09-05T16:34:56.820292+0000', 'last_unstale': '2024-09-05T16:35:20.085937+0000', 'last_undegraded': '2024-09-05T16:34:56.605776+0000', 'last_fullsized': '2024-09-05T16:34:56.559383+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 2827, 'log_dups_size': 0, 'ondisk_log_size': 2827, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23123104, 'num_objects': 2823, 'num_object_clones': 0, 'num_object_copies': 11292, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2269, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2823, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2825, 'num_write_kb': 22596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 556}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:35:23.553 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:35:23.553 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'2869", 'reported_seq': 2943, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:20.078358+0000', 'last_change': '2024-09-05T16:34:56.867900+0000', 'last_active': '2024-09-05T16:35:20.078358+0000', 'last_peered': '2024-09-05T16:35:20.078358+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:35:20.078358+0000', 'last_undegraded': '2024-09-05T16:34:56.590972+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 2869, 'log_dups_size': 0, 'ondisk_log_size': 2869, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23476816, 'num_objects': 2866, 'num_object_clones': 0, 'num_object_copies': 11464, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2251, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2866, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2868, 'num_write_kb': 22934, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 616}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:35:23.553 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-05T16:35:23.553 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'2829", 'reported_seq': 3599, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:20.125085+0000', 'last_change': '2024-09-05T16:34:56.800371+0000', 'last_active': '2024-09-05T16:35:20.125085+0000', 'last_peered': '2024-09-05T16:35:20.125085+0000', 'last_clean': '2024-09-05T16:33:53.148717+0000', 'last_became_active': '2024-09-05T16:34:56.791755+0000', 'last_became_peered': '2024-09-05T16:34:56.791755+0000', 'last_unstale': '2024-09-05T16:35:20.125085+0000', 'last_undegraded': '2024-09-05T16:34:56.597522+0000', 'last_fullsized': '2024-09-05T16:34:56.568150+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 2829, 'log_dups_size': 0, 'ondisk_log_size': 2829, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23139488, 'num_objects': 2825, 'num_object_clones': 0, 'num_object_copies': 11300, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1935, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2825, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2827, 'num_write_kb': 22605, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 355, 'num_bytes_recovered': 2891776, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 890}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:35:23.554 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:35:23.554 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'2738", 'reported_seq': 2791, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:19.073649+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:35:19.073649+0000', 'last_peered': '2024-09-05T16:35:19.073649+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:35:19.073649+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 2738, 'log_dups_size': 0, 'ondisk_log_size': 2738, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22402208, 'num_objects': 2735, 'num_object_clones': 0, 'num_object_copies': 10940, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2164, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2735, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2736, 'num_write_kb': 21885, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 573}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:35:23.554 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:35:23.554 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'2805", 'reported_seq': 2846, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:19.056470+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:35:19.056470+0000', 'last_peered': '2024-09-05T16:35:19.056470+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:35:19.056470+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 2805, 'log_dups_size': 0, 'ondisk_log_size': 2805, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22978560, 'num_objects': 2805, 'num_object_clones': 0, 'num_object_copies': 11220, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2805, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2805, 'num_write_kb': 22440, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 555}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:35:23.554 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:35:23.554 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'2721", 'reported_seq': 2785, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:20.091315+0000', 'last_change': '2024-09-05T16:34:56.903436+0000', 'last_active': '2024-09-05T16:35:20.091315+0000', 'last_peered': '2024-09-05T16:35:20.091315+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:35:20.091315+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 2721, 'log_dups_size': 0, 'ondisk_log_size': 2721, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22238368, 'num_objects': 2715, 'num_object_clones': 0, 'num_object_copies': 10860, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2149, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2715, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2719, 'num_write_kb': 21732, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 568}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:35:23.554 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-05T16:35:23.554 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'2850", 'reported_seq': 2912, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:20.156112+0000', 'last_change': '2024-09-05T16:34:56.883319+0000', 'last_active': '2024-09-05T16:35:20.156112+0000', 'last_peered': '2024-09-05T16:35:20.156112+0000', 'last_clean': '2024-09-05T16:33:53.505305+0000', 'last_became_active': '2024-09-05T16:34:56.873389+0000', 'last_became_peered': '2024-09-05T16:34:56.873389+0000', 'last_unstale': '2024-09-05T16:35:20.156112+0000', 'last_undegraded': '2024-09-05T16:34:56.602567+0000', 'last_fullsized': '2024-09-05T16:34:56.559958+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 2850, 'log_dups_size': 0, 'ondisk_log_size': 2850, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23303328, 'num_objects': 2845, 'num_object_clones': 0, 'num_object_copies': 11380, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2241, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2845, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2848, 'num_write_kb': 22772, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 606}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:35:23.555 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:35:23.555 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'2844", 'reported_seq': 2899, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:18.932541+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:35:18.932541+0000', 'last_peered': '2024-09-05T16:35:18.932541+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:35:18.932541+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 2844, 'log_dups_size': 0, 'ondisk_log_size': 2844, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23226688, 'num_objects': 2836, 'num_object_clones': 0, 'num_object_copies': 11344, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2264, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2836, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2841, 'num_write_kb': 22705, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 575}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:35:23.555 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:35:23.555 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'2922", 'reported_seq': 2998, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:20.122703+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:35:20.122703+0000', 'last_peered': '2024-09-05T16:35:20.122703+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:35:20.122703+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 2922, 'log_dups_size': 0, 'ondisk_log_size': 2922, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23917728, 'num_objects': 2920, 'num_object_clones': 0, 'num_object_copies': 11680, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2264, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2920, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2920, 'num_write_kb': 23358, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 658}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:35:23.555 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:35:23.555 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'2882", 'reported_seq': 2937, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:18.990219+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:35:18.990219+0000', 'last_peered': '2024-09-05T16:35:18.990219+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:35:18.990219+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 2882, 'log_dups_size': 0, 'ondisk_log_size': 2882, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23564016, 'num_objects': 2877, 'num_object_clones': 0, 'num_object_copies': 11508, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2314, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2877, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2879, 'num_write_kb': 23020, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 566}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:35:23.555 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:35:23.555 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'2830", 'reported_seq': 2865, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:19.016300+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:35:19.016300+0000', 'last_peered': '2024-09-05T16:35:19.016300+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:35:19.016300+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 2830, 'log_dups_size': 0, 'ondisk_log_size': 2830, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23173712, 'num_objects': 2829, 'num_object_clones': 0, 'num_object_copies': 11316, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2265, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2829, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2829, 'num_write_kb': 22631, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 565}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:35:23.555 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-05T16:35:23.555 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'2848", 'reported_seq': 2921, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:20.013228+0000', 'last_change': '2024-09-05T16:34:56.798081+0000', 'last_active': '2024-09-05T16:35:20.013228+0000', 'last_peered': '2024-09-05T16:35:20.013228+0000', 'last_clean': '2024-09-05T16:33:45.082039+0000', 'last_became_active': '2024-09-05T16:34:56.792386+0000', 'last_became_peered': '2024-09-05T16:34:56.792386+0000', 'last_unstale': '2024-09-05T16:35:20.013228+0000', 'last_undegraded': '2024-09-05T16:34:56.600595+0000', 'last_fullsized': '2024-09-05T16:34:56.558203+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 2848, 'log_dups_size': 0, 'ondisk_log_size': 2848, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23321168, 'num_objects': 2847, 'num_object_clones': 0, 'num_object_copies': 11388, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2295, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2847, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2847, 'num_write_kb': 22775, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 553}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:35:23.556 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:35:23.556 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'2930", 'reported_seq': 2981, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:19.071712+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:35:19.071712+0000', 'last_peered': '2024-09-05T16:35:19.071712+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:35:19.071712+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 2930, 'log_dups_size': 0, 'ondisk_log_size': 2930, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23966880, 'num_objects': 2926, 'num_object_clones': 0, 'num_object_copies': 11704, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2359, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2926, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2928, 'num_write_kb': 23420, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 569}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:35:23.556 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:35:23.556 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'2904", 'reported_seq': 2981, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:20.138843+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:35:20.138843+0000', 'last_peered': '2024-09-05T16:35:20.138843+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:35:20.138843+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 2904, 'log_dups_size': 0, 'ondisk_log_size': 2904, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23762080, 'num_objects': 2901, 'num_object_clones': 0, 'num_object_copies': 11604, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2322, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2901, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2902, 'num_write_kb': 23213, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 581}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:35:23.556 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:35:23.556 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'2854", 'reported_seq': 2917, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:20.107757+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:35:20.107757+0000', 'last_peered': '2024-09-05T16:35:20.107757+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:35:20.107757+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 2854, 'log_dups_size': 0, 'ondisk_log_size': 2854, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23336096, 'num_objects': 2849, 'num_object_clones': 0, 'num_object_copies': 11396, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2849, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2852, 'num_write_kb': 22804, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 612}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:35:23.557 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:35:24.300 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:35:24.300 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 35843/181640 objects degraded (19.733%), 16 pgs degraded","count":16},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":127,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":45410,"data_bytes":371957952,"bytes_used":1108226048,"bytes_avail":385438830592,"bytes_total":386547056640,"degraded_objects":35843,"degraded_total":181640,"degraded_ratio":0.1973298832856199,"recovering_objects_per_sec":13,"recovering_bytes_per_sec":113295,"recovering_keys_per_sec":0,"num_objects_recovered":166,"num_bytes_recovered":1359872,"num_keys_recovered":0,"write_bytes_sec":2719783,"read_op_per_sec":0,"write_op_per_sec":332},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-05T16:35:14.802011+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-05T16:35:24.327 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:35:24.327 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:35:24.503 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 62 20 47289 47269 5.95494 2.78906 0.0593219 0.0262198 2024-09-05T16:35:24.913 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:35:24.913 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:35:24.946 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":88,"stamp":"2024-09-05T16:35:24.804259+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":379314368,"num_objects":46308,"num_object_clones":0,"num_object_copies":185232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35843,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46308,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":46335,"num_write_kb":370562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":380,"num_bytes_recovered":2891776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":46362,"ondisk_log_size":46362,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1129108,"kb_used_data":597524,"kb_used_omap":14,"kb_used_meta":531505,"kb_avail":376358252,"statfs":{"total":386547056640,"available":385390850048,"internally_reserved":0,"allocated":611864576,"data_stored":611133048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":544261687},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,0,8,3,10,11,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":40001536,"num_objects":4883,"num_object_clones":0,"num_object_copies":19532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-166,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4883,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4883,"num_write_kb":39064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":166,"num_bytes_recovered":1359872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002779"},"pg_stats":[{"pgid":"1.f","version":"33'2945","reported_seq":2986,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:24.059355+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:35:24.059355+0000","last_peered":"2024-09-05T16:35:24.059355+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:35:24.059355+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2945,"log_dups_size":0,"ondisk_log_size":2945,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24125440,"num_objects":2945,"num_object_clones":0,"num_object_copies":11780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2945,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2945,"num_write_kb":23560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":683}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'2827","reported_seq":2887,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:20.085937+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:35:20.085937+0000","last_peered":"2024-09-05T16:35:20.085937+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:35:20.085937+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2827,"log_dups_size":0,"ondisk_log_size":2827,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23123104,"num_objects":2823,"num_object_clones":0,"num_object_copies":11292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2823,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2825,"num_write_kb":22596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":556}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'2869","reported_seq":2943,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:20.078358+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:35:20.078358+0000","last_peered":"2024-09-05T16:35:20.078358+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:35:20.078358+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2869,"log_dups_size":0,"ondisk_log_size":2869,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23476816,"num_objects":2866,"num_object_clones":0,"num_object_copies":11464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2866,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2868,"num_write_kb":22934,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":616}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'2829","reported_seq":3599,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:20.125085+0000","last_change":"2024-09-05T16:34:56.800371+0000","last_active":"2024-09-05T16:35:20.125085+0000","last_peered":"2024-09-05T16:35:20.125085+0000","last_clean":"2024-09-05T16:33:53.148717+0000","last_became_active":"2024-09-05T16:34:56.791755+0000","last_became_peered":"2024-09-05T16:34:56.791755+0000","last_unstale":"2024-09-05T16:35:20.125085+0000","last_undegraded":"2024-09-05T16:34:56.597522+0000","last_fullsized":"2024-09-05T16:34:56.568150+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2829,"log_dups_size":0,"ondisk_log_size":2829,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23139488,"num_objects":2825,"num_object_clones":0,"num_object_copies":11300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1935,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2825,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2827,"num_write_kb":22605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":355,"num_bytes_recovered":2891776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":890}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'2861","reported_seq":2914,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:24.054310+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:35:24.054310+0000","last_peered":"2024-09-05T16:35:24.054310+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:35:24.054310+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2861,"log_dups_size":0,"ondisk_log_size":2861,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23409824,"num_objects":2858,"num_object_clones":0,"num_object_copies":11432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2858,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2859,"num_write_kb":22869,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":696}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'2950","reported_seq":2990,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:24.094079+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:35:24.094079+0000","last_peered":"2024-09-05T16:35:24.094079+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:35:24.094079+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2950,"log_dups_size":0,"ondisk_log_size":2950,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24166400,"num_objects":2950,"num_object_clones":0,"num_object_copies":11800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2950,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2950,"num_write_kb":23600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":700}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'2721","reported_seq":2785,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:20.091315+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:35:20.091315+0000","last_peered":"2024-09-05T16:35:20.091315+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:35:20.091315+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2721,"log_dups_size":0,"ondisk_log_size":2721,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22238368,"num_objects":2715,"num_object_clones":0,"num_object_copies":10860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2715,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2719,"num_write_kb":21732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":568}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'2850","reported_seq":2912,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:20.156112+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:35:20.156112+0000","last_peered":"2024-09-05T16:35:20.156112+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:35:20.156112+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2850,"log_dups_size":0,"ondisk_log_size":2850,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23303328,"num_objects":2845,"num_object_clones":0,"num_object_copies":11380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2845,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2848,"num_write_kb":22772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":606}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'2968","reported_seq":3023,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:24.101078+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:35:24.101078+0000","last_peered":"2024-09-05T16:35:24.101078+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:35:24.101078+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2968,"log_dups_size":0,"ondisk_log_size":2968,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24242496,"num_objects":2960,"num_object_clones":0,"num_object_copies":11840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2960,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2965,"num_write_kb":23697,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":699}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'2922","reported_seq":2998,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:20.122703+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:35:20.122703+0000","last_peered":"2024-09-05T16:35:20.122703+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:35:20.122703+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2922,"log_dups_size":0,"ondisk_log_size":2922,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23917728,"num_objects":2920,"num_object_clones":0,"num_object_copies":11680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2920,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2920,"num_write_kb":23358,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":658}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'3005","reported_seq":3060,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:24.044136+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:35:24.044136+0000","last_peered":"2024-09-05T16:35:24.044136+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:35:24.044136+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3005,"log_dups_size":0,"ondisk_log_size":3005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24571632,"num_objects":3000,"num_object_clones":0,"num_object_copies":12000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3000,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3002,"num_write_kb":24004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":689}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'2946","reported_seq":2981,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:23.938095+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:35:23.938095+0000","last_peered":"2024-09-05T16:35:23.938095+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:35:23.938095+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2946,"log_dups_size":0,"ondisk_log_size":2946,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24123984,"num_objects":2945,"num_object_clones":0,"num_object_copies":11780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2945,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2945,"num_write_kb":23559,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":681}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'2848","reported_seq":2921,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:20.013228+0000","last_change":"2024-09-05T16:34:56.798081+0000","last_active":"2024-09-05T16:35:20.013228+0000","last_peered":"2024-09-05T16:35:20.013228+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:35:20.013228+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2848,"log_dups_size":0,"ondisk_log_size":2848,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23321168,"num_objects":2847,"num_object_clones":0,"num_object_copies":11388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2847,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2847,"num_write_kb":22775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":553}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'3063","reported_seq":3111,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:24.078328+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:35:24.078328+0000","last_peered":"2024-09-05T16:35:24.078328+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:35:24.078328+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3063,"log_dups_size":0,"ondisk_log_size":3063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25056416,"num_objects":3059,"num_object_clones":0,"num_object_copies":12236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3059,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3061,"num_write_kb":24484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":702}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'2904","reported_seq":2981,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:20.138843+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:35:20.138843+0000","last_peered":"2024-09-05T16:35:20.138843+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:35:20.138843+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2904,"log_dups_size":0,"ondisk_log_size":2904,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23762080,"num_objects":2901,"num_object_clones":0,"num_object_copies":11604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2901,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2902,"num_write_kb":23213,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":581}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'2854","reported_seq":2917,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:20.107757+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:35:20.107757+0000","last_peered":"2024-09-05T16:35:20.107757+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:35:20.107757+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2854,"log_dups_size":0,"ondisk_log_size":2854,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23336096,"num_objects":2849,"num_object_clones":0,"num_object_copies":11396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2849,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2852,"num_write_kb":22804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":612}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":379314368,"num_objects":46308,"num_object_clones":0,"num_object_copies":185232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35843,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46308,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":46335,"num_write_kb":370562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":380,"num_bytes_recovered":2891776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":613203968,"data_stored":613203968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":46362,"ondisk_log_size":46362,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084306,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":350276,"kb_used_data":188084,"kb_used_omap":4,"kb_used_meta":162171,"kb_avail":94021564,"statfs":{"total":96636764160,"available":96278081536,"internally_reserved":0,"allocated":192598016,"data_stored":192415134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":166063827},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,1,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986183,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":135632,"kb_used_data":45632,"kb_used_omap":4,"kb_used_meta":89979,"kb_avail":94236208,"statfs":{"total":96636764160,"available":96497876992,"internally_reserved":0,"allocated":46727168,"data_stored":46544286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":92139218},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,5,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949672988,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":321400,"kb_used_data":181928,"kb_used_omap":1,"kb_used_meta":139454,"kb_avail":94050440,"statfs":{"total":96636764160,"available":96307650560,"internally_reserved":0,"allocated":186294272,"data_stored":186111390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":142801349},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,1,1,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182419,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":321800,"kb_used_data":181880,"kb_used_omap":4,"kb_used_meta":139899,"kb_avail":94050040,"statfs":{"total":96636764160,"available":96307240960,"internally_reserved":0,"allocated":186245120,"data_stored":186062238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":143257293},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,0,2,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":186732544,"data_stored":186732544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":46587904,"data_stored":46587904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":186724352,"data_stored":186724352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":193159168,"data_stored":193159168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:35:25.503 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 63 20 47684 47664 5.9094 3.08594 0.0423041 0.0264261 2024-09-05T16:35:26.505 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 64 20 48104 48084 5.86824 3.28125 0.0301921 0.0266116 2024-09-05T16:35:27.505 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 65 20 48527 48507 5.82878 3.30469 0.0394315 0.0267941 2024-09-05T16:35:27.948 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:35:28.505 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 66 20 48907 48887 5.78545 2.96875 0.0625979 0.0269917 2024-09-05T16:35:28.551 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:35:28.551 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:35:28.580 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":89,"stamp":"2024-09-05T16:35:26.804718+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":388300992,"num_objects":47405,"num_object_clones":0,"num_object_copies":189620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35772,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47405,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47432,"num_write_kb":379338,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":451,"num_bytes_recovered":3473408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":47459,"ondisk_log_size":47459,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1193256,"kb_used_data":613096,"kb_used_omap":14,"kb_used_meta":580081,"kb_avail":376294104,"statfs":{"total":386547056640,"available":385325162496,"internally_reserved":0,"allocated":627810304,"data_stored":627078776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":594003511},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,7,2,8,15,7,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":41402368,"num_objects":5054,"num_object_clones":0,"num_object_copies":20216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-237,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5054,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5054,"num_write_kb":40432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":237,"num_bytes_recovered":1941504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002790"},"pg_stats":[{"pgid":"1.f","version":"33'2945","reported_seq":2986,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:24.059355+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:35:24.059355+0000","last_peered":"2024-09-05T16:35:24.059355+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:35:24.059355+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2945,"log_dups_size":0,"ondisk_log_size":2945,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24125440,"num_objects":2945,"num_object_clones":0,"num_object_copies":11780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2945,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2945,"num_write_kb":23560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":683}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'2938","reported_seq":3001,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:25.121601+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:35:25.121601+0000","last_peered":"2024-09-05T16:35:25.121601+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:35:25.121601+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2938,"log_dups_size":0,"ondisk_log_size":2938,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24032416,"num_objects":2934,"num_object_clones":0,"num_object_copies":11736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2934,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2936,"num_write_kb":23484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":667}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'2984","reported_seq":3058,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:25.110337+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:35:25.110337+0000","last_peered":"2024-09-05T16:35:25.110337+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:35:25.110337+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2984,"log_dups_size":0,"ondisk_log_size":2984,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24418896,"num_objects":2981,"num_object_clones":0,"num_object_copies":11924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2981,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2983,"num_write_kb":23854,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":731}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'2948","reported_seq":3860,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:25.162705+0000","last_change":"2024-09-05T16:34:56.800371+0000","last_active":"2024-09-05T16:35:25.162705+0000","last_peered":"2024-09-05T16:35:25.162705+0000","last_clean":"2024-09-05T16:33:53.148717+0000","last_became_active":"2024-09-05T16:34:56.791755+0000","last_became_peered":"2024-09-05T16:34:56.791755+0000","last_unstale":"2024-09-05T16:35:25.162705+0000","last_undegraded":"2024-09-05T16:34:56.597522+0000","last_fullsized":"2024-09-05T16:34:56.568150+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2948,"log_dups_size":0,"ondisk_log_size":2948,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24114336,"num_objects":2944,"num_object_clones":0,"num_object_copies":11776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1864,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2944,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2946,"num_write_kb":23557,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":426,"num_bytes_recovered":3473408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1080}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'2861","reported_seq":2914,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:24.054310+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:35:24.054310+0000","last_peered":"2024-09-05T16:35:24.054310+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:35:24.054310+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2861,"log_dups_size":0,"ondisk_log_size":2861,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23409824,"num_objects":2858,"num_object_clones":0,"num_object_copies":11432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2858,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2859,"num_write_kb":22869,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":696}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'2950","reported_seq":2990,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:24.094079+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:35:24.094079+0000","last_peered":"2024-09-05T16:35:24.094079+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:35:24.094079+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2950,"log_dups_size":0,"ondisk_log_size":2950,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24166400,"num_objects":2950,"num_object_clones":0,"num_object_copies":11800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2950,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2950,"num_write_kb":23600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":700}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'2856","reported_seq":2919,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:25.163986+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:35:25.163986+0000","last_peered":"2024-09-05T16:35:25.163986+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:35:25.163986+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2856,"log_dups_size":0,"ondisk_log_size":2856,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23344288,"num_objects":2850,"num_object_clones":0,"num_object_copies":11400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2850,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2854,"num_write_kb":22812,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":703}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'2978","reported_seq":3040,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:25.122138+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:35:25.122138+0000","last_peered":"2024-09-05T16:35:25.122138+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:35:25.122138+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2978,"log_dups_size":0,"ondisk_log_size":2978,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24351904,"num_objects":2973,"num_object_clones":0,"num_object_copies":11892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2973,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2976,"num_write_kb":23796,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":734}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'2968","reported_seq":3023,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:24.101078+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:35:24.101078+0000","last_peered":"2024-09-05T16:35:24.101078+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:35:24.101078+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2968,"log_dups_size":0,"ondisk_log_size":2968,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24242496,"num_objects":2960,"num_object_clones":0,"num_object_copies":11840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2960,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2965,"num_write_kb":23697,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":699}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'3031","reported_seq":3106,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:25.052126+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:35:25.052126+0000","last_peered":"2024-09-05T16:35:25.052126+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:35:25.052126+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3031,"log_dups_size":0,"ondisk_log_size":3031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24810656,"num_objects":3029,"num_object_clones":0,"num_object_copies":12116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3029,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3029,"num_write_kb":24230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":767}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'3005","reported_seq":3060,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:24.044136+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:35:24.044136+0000","last_peered":"2024-09-05T16:35:24.044136+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:35:24.044136+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3005,"log_dups_size":0,"ondisk_log_size":3005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24571632,"num_objects":3000,"num_object_clones":0,"num_object_copies":12000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3000,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3002,"num_write_kb":24004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":689}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'2946","reported_seq":2981,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:23.938095+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:35:23.938095+0000","last_peered":"2024-09-05T16:35:23.938095+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:35:23.938095+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2946,"log_dups_size":0,"ondisk_log_size":2946,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24123984,"num_objects":2945,"num_object_clones":0,"num_object_copies":11780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2945,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2945,"num_write_kb":23559,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":681}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'2984","reported_seq":3057,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:25.123445+0000","last_change":"2024-09-05T16:34:56.798081+0000","last_active":"2024-09-05T16:35:25.123445+0000","last_peered":"2024-09-05T16:35:25.123445+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:35:25.123445+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2984,"log_dups_size":0,"ondisk_log_size":2984,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24435280,"num_objects":2983,"num_object_clones":0,"num_object_copies":11932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2983,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2983,"num_write_kb":23863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":689}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'3063","reported_seq":3111,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:24.078328+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:35:24.078328+0000","last_peered":"2024-09-05T16:35:24.078328+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:35:24.078328+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3063,"log_dups_size":0,"ondisk_log_size":3063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25056416,"num_objects":3059,"num_object_clones":0,"num_object_copies":12236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3059,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3061,"num_write_kb":24484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":702}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'3032","reported_seq":3109,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:25.091854+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:35:25.091854+0000","last_peered":"2024-09-05T16:35:25.091854+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:35:25.091854+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3032,"log_dups_size":0,"ondisk_log_size":3032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24810656,"num_objects":3029,"num_object_clones":0,"num_object_copies":12116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3029,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3030,"num_write_kb":24237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":709}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'2970","reported_seq":3033,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:25.077592+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:35:25.077592+0000","last_peered":"2024-09-05T16:35:25.077592+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:35:25.077592+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2970,"log_dups_size":0,"ondisk_log_size":2970,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24286368,"num_objects":2965,"num_object_clones":0,"num_object_copies":11860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2965,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2968,"num_write_kb":23732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":728}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":388300992,"num_objects":47405,"num_object_clones":0,"num_object_copies":189620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35772,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47405,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47432,"num_write_kb":379338,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":451,"num_bytes_recovered":3473408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":629182464,"data_stored":629182464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":47459,"ondisk_log_size":47459,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084306,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":350276,"kb_used_data":188084,"kb_used_omap":4,"kb_used_meta":162171,"kb_avail":94021564,"statfs":{"total":96636764160,"available":96278081536,"internally_reserved":0,"allocated":192598016,"data_stored":192415134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":166063827},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,1,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986183,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":135632,"kb_used_data":45632,"kb_used_omap":4,"kb_used_meta":89979,"kb_avail":94236208,"statfs":{"total":96636764160,"available":96497876992,"internally_reserved":0,"allocated":46727168,"data_stored":46544286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":92139218},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,5,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949672989,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":355584,"kb_used_data":189744,"kb_used_omap":1,"kb_used_meta":165822,"kb_avail":94016256,"statfs":{"total":96636764160,"available":96272646144,"internally_reserved":0,"allocated":194297856,"data_stored":194114974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":169802181},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,2,2,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182420,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":351764,"kb_used_data":189636,"kb_used_omap":4,"kb_used_meta":162107,"kb_avail":94020076,"statfs":{"total":96636764160,"available":96276557824,"internally_reserved":0,"allocated":194187264,"data_stored":194004382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":165998285},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2,0,5,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":194727936,"data_stored":194727936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":46587904,"data_stored":46587904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":194707456,"data_stored":194707456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":193159168,"data_stored":193159168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:35:28.581 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:35:29.189 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:35:29.189 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:35:29.218 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":90,"stamp":"2024-09-05T16:35:28.805344+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":388300992,"num_objects":47405,"num_object_clones":0,"num_object_copies":189620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35772,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47405,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47432,"num_write_kb":379338,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":451,"num_bytes_recovered":3473408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":47459,"ondisk_log_size":47459,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1193256,"kb_used_data":613096,"kb_used_omap":14,"kb_used_meta":580081,"kb_avail":376294104,"statfs":{"total":386547056640,"available":385325162496,"internally_reserved":0,"allocated":627810304,"data_stored":627078776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":594003511},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,7,2,8,15,7,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":32071680,"num_objects":3915,"num_object_clones":0,"num_object_copies":15660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3915,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3915,"num_write_kb":31320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":169,"num_bytes_recovered":1384448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002920"},"pg_stats":[{"pgid":"1.f","version":"33'2945","reported_seq":2986,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:24.059355+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:35:24.059355+0000","last_peered":"2024-09-05T16:35:24.059355+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:35:24.059355+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2945,"log_dups_size":0,"ondisk_log_size":2945,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24125440,"num_objects":2945,"num_object_clones":0,"num_object_copies":11780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2945,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2945,"num_write_kb":23560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":683}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'2938","reported_seq":3001,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:25.121601+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:35:25.121601+0000","last_peered":"2024-09-05T16:35:25.121601+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:35:25.121601+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2938,"log_dups_size":0,"ondisk_log_size":2938,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24032416,"num_objects":2934,"num_object_clones":0,"num_object_copies":11736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2934,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2936,"num_write_kb":23484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":667}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'2984","reported_seq":3058,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:25.110337+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:35:25.110337+0000","last_peered":"2024-09-05T16:35:25.110337+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:35:25.110337+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2984,"log_dups_size":0,"ondisk_log_size":2984,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24418896,"num_objects":2981,"num_object_clones":0,"num_object_copies":11924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2981,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2983,"num_write_kb":23854,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":731}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'2948","reported_seq":3860,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:25.162705+0000","last_change":"2024-09-05T16:34:56.800371+0000","last_active":"2024-09-05T16:35:25.162705+0000","last_peered":"2024-09-05T16:35:25.162705+0000","last_clean":"2024-09-05T16:33:53.148717+0000","last_became_active":"2024-09-05T16:34:56.791755+0000","last_became_peered":"2024-09-05T16:34:56.791755+0000","last_unstale":"2024-09-05T16:35:25.162705+0000","last_undegraded":"2024-09-05T16:34:56.597522+0000","last_fullsized":"2024-09-05T16:34:56.568150+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2948,"log_dups_size":0,"ondisk_log_size":2948,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24114336,"num_objects":2944,"num_object_clones":0,"num_object_copies":11776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1864,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2944,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2946,"num_write_kb":23557,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":426,"num_bytes_recovered":3473408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1080}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'2861","reported_seq":2914,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:24.054310+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:35:24.054310+0000","last_peered":"2024-09-05T16:35:24.054310+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:35:24.054310+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2861,"log_dups_size":0,"ondisk_log_size":2861,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23409824,"num_objects":2858,"num_object_clones":0,"num_object_copies":11432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2858,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2859,"num_write_kb":22869,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":696}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'2950","reported_seq":2990,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:24.094079+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:35:24.094079+0000","last_peered":"2024-09-05T16:35:24.094079+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:35:24.094079+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2950,"log_dups_size":0,"ondisk_log_size":2950,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24166400,"num_objects":2950,"num_object_clones":0,"num_object_copies":11800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2950,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2950,"num_write_kb":23600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":700}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'2856","reported_seq":2919,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:25.163986+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:35:25.163986+0000","last_peered":"2024-09-05T16:35:25.163986+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:35:25.163986+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2856,"log_dups_size":0,"ondisk_log_size":2856,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23344288,"num_objects":2850,"num_object_clones":0,"num_object_copies":11400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2850,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2854,"num_write_kb":22812,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":703}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'2978","reported_seq":3040,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:25.122138+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:35:25.122138+0000","last_peered":"2024-09-05T16:35:25.122138+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:35:25.122138+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2978,"log_dups_size":0,"ondisk_log_size":2978,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24351904,"num_objects":2973,"num_object_clones":0,"num_object_copies":11892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2973,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2976,"num_write_kb":23796,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":734}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'2968","reported_seq":3023,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:24.101078+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:35:24.101078+0000","last_peered":"2024-09-05T16:35:24.101078+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:35:24.101078+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2968,"log_dups_size":0,"ondisk_log_size":2968,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24242496,"num_objects":2960,"num_object_clones":0,"num_object_copies":11840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2960,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2965,"num_write_kb":23697,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":699}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'3031","reported_seq":3106,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:25.052126+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:35:25.052126+0000","last_peered":"2024-09-05T16:35:25.052126+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:35:25.052126+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3031,"log_dups_size":0,"ondisk_log_size":3031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24810656,"num_objects":3029,"num_object_clones":0,"num_object_copies":12116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3029,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3029,"num_write_kb":24230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":767}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'3005","reported_seq":3060,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:24.044136+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:35:24.044136+0000","last_peered":"2024-09-05T16:35:24.044136+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:35:24.044136+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3005,"log_dups_size":0,"ondisk_log_size":3005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24571632,"num_objects":3000,"num_object_clones":0,"num_object_copies":12000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3000,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3002,"num_write_kb":24004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":689}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'2946","reported_seq":2981,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:23.938095+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:35:23.938095+0000","last_peered":"2024-09-05T16:35:23.938095+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:35:23.938095+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2946,"log_dups_size":0,"ondisk_log_size":2946,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24123984,"num_objects":2945,"num_object_clones":0,"num_object_copies":11780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2945,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2945,"num_write_kb":23559,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":681}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'2984","reported_seq":3057,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:25.123445+0000","last_change":"2024-09-05T16:34:56.798081+0000","last_active":"2024-09-05T16:35:25.123445+0000","last_peered":"2024-09-05T16:35:25.123445+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:35:25.123445+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2984,"log_dups_size":0,"ondisk_log_size":2984,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24435280,"num_objects":2983,"num_object_clones":0,"num_object_copies":11932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2983,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2983,"num_write_kb":23863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":689}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'3063","reported_seq":3111,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:24.078328+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:35:24.078328+0000","last_peered":"2024-09-05T16:35:24.078328+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:35:24.078328+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3063,"log_dups_size":0,"ondisk_log_size":3063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25056416,"num_objects":3059,"num_object_clones":0,"num_object_copies":12236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3059,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3061,"num_write_kb":24484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":702}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'3032","reported_seq":3109,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:25.091854+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:35:25.091854+0000","last_peered":"2024-09-05T16:35:25.091854+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:35:25.091854+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3032,"log_dups_size":0,"ondisk_log_size":3032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24810656,"num_objects":3029,"num_object_clones":0,"num_object_copies":12116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3029,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3030,"num_write_kb":24237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":709}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'2970","reported_seq":3033,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:25.077592+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:35:25.077592+0000","last_peered":"2024-09-05T16:35:25.077592+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:35:25.077592+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2970,"log_dups_size":0,"ondisk_log_size":2970,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24286368,"num_objects":2965,"num_object_clones":0,"num_object_copies":11860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2965,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2968,"num_write_kb":23732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":728}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":388300992,"num_objects":47405,"num_object_clones":0,"num_object_copies":189620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35772,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47405,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47432,"num_write_kb":379338,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":451,"num_bytes_recovered":3473408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":629182464,"data_stored":629182464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":47459,"ondisk_log_size":47459,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084306,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":350276,"kb_used_data":188084,"kb_used_omap":4,"kb_used_meta":162171,"kb_avail":94021564,"statfs":{"total":96636764160,"available":96278081536,"internally_reserved":0,"allocated":192598016,"data_stored":192415134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":166063827},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,1,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986183,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":135632,"kb_used_data":45632,"kb_used_omap":4,"kb_used_meta":89979,"kb_avail":94236208,"statfs":{"total":96636764160,"available":96497876992,"internally_reserved":0,"allocated":46727168,"data_stored":46544286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":92139218},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,5,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949672989,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":355584,"kb_used_data":189744,"kb_used_omap":1,"kb_used_meta":165822,"kb_avail":94016256,"statfs":{"total":96636764160,"available":96272646144,"internally_reserved":0,"allocated":194297856,"data_stored":194114974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":169802181},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,2,2,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182420,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":351764,"kb_used_data":189636,"kb_used_omap":4,"kb_used_meta":162107,"kb_avail":94020076,"statfs":{"total":96636764160,"available":96276557824,"internally_reserved":0,"allocated":194187264,"data_stored":194004382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":165998285},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2,0,5,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":194727936,"data_stored":194727936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":46587904,"data_stored":46587904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":194707456,"data_stored":194707456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":193159168,"data_stored":193159168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:35:29.219 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-05T16:35:29.219 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'2945", 'reported_seq': 2986, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:24.059355+0000', 'last_change': '2024-09-05T16:34:56.833497+0000', 'last_active': '2024-09-05T16:35:24.059355+0000', 'last_peered': '2024-09-05T16:35:24.059355+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.828289+0000', 'last_became_peered': '2024-09-05T16:34:56.828289+0000', 'last_unstale': '2024-09-05T16:35:24.059355+0000', 'last_undegraded': '2024-09-05T16:34:56.612160+0000', 'last_fullsized': '2024-09-05T16:34:56.557632+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 2945, 'log_dups_size': 0, 'ondisk_log_size': 2945, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24125440, 'num_objects': 2945, 'num_object_clones': 0, 'num_object_copies': 11780, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2262, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2945, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2945, 'num_write_kb': 23560, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 683}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:35:29.219 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-05T16:35:29.219 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'2938", 'reported_seq': 3001, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:25.121601+0000', 'last_change': '2024-09-05T16:34:56.825031+0000', 'last_active': '2024-09-05T16:35:25.121601+0000', 'last_peered': '2024-09-05T16:35:25.121601+0000', 'last_clean': '2024-09-05T16:33:53.200762+0000', 'last_became_active': '2024-09-05T16:34:56.820292+0000', 'last_became_peered': '2024-09-05T16:34:56.820292+0000', 'last_unstale': '2024-09-05T16:35:25.121601+0000', 'last_undegraded': '2024-09-05T16:34:56.605776+0000', 'last_fullsized': '2024-09-05T16:34:56.559383+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 2938, 'log_dups_size': 0, 'ondisk_log_size': 2938, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24032416, 'num_objects': 2934, 'num_object_clones': 0, 'num_object_copies': 11736, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2269, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2934, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2936, 'num_write_kb': 23484, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 667}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:35:29.220 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:35:29.220 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'2984", 'reported_seq': 3058, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:25.110337+0000', 'last_change': '2024-09-05T16:34:56.867900+0000', 'last_active': '2024-09-05T16:35:25.110337+0000', 'last_peered': '2024-09-05T16:35:25.110337+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:35:25.110337+0000', 'last_undegraded': '2024-09-05T16:34:56.590972+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 2984, 'log_dups_size': 0, 'ondisk_log_size': 2984, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24418896, 'num_objects': 2981, 'num_object_clones': 0, 'num_object_copies': 11924, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2251, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2981, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2983, 'num_write_kb': 23854, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 731}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:35:29.220 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-05T16:35:29.220 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'2948", 'reported_seq': 3860, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:25.162705+0000', 'last_change': '2024-09-05T16:34:56.800371+0000', 'last_active': '2024-09-05T16:35:25.162705+0000', 'last_peered': '2024-09-05T16:35:25.162705+0000', 'last_clean': '2024-09-05T16:33:53.148717+0000', 'last_became_active': '2024-09-05T16:34:56.791755+0000', 'last_became_peered': '2024-09-05T16:34:56.791755+0000', 'last_unstale': '2024-09-05T16:35:25.162705+0000', 'last_undegraded': '2024-09-05T16:34:56.597522+0000', 'last_fullsized': '2024-09-05T16:34:56.568150+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 2948, 'log_dups_size': 0, 'ondisk_log_size': 2948, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24114336, 'num_objects': 2944, 'num_object_clones': 0, 'num_object_copies': 11776, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1864, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2944, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2946, 'num_write_kb': 23557, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 426, 'num_bytes_recovered': 3473408, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1080}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:35:29.220 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:35:29.220 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'2861", 'reported_seq': 2914, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:24.054310+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:35:24.054310+0000', 'last_peered': '2024-09-05T16:35:24.054310+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:35:24.054310+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 2861, 'log_dups_size': 0, 'ondisk_log_size': 2861, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23409824, 'num_objects': 2858, 'num_object_clones': 0, 'num_object_copies': 11432, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2164, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2858, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2859, 'num_write_kb': 22869, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 696}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:35:29.220 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:35:29.220 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'2950", 'reported_seq': 2990, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:24.094079+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:35:24.094079+0000', 'last_peered': '2024-09-05T16:35:24.094079+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:35:24.094079+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 2950, 'log_dups_size': 0, 'ondisk_log_size': 2950, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24166400, 'num_objects': 2950, 'num_object_clones': 0, 'num_object_copies': 11800, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2950, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2950, 'num_write_kb': 23600, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 700}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:35:29.220 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:35:29.220 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'2856", 'reported_seq': 2919, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:25.163986+0000', 'last_change': '2024-09-05T16:34:56.903436+0000', 'last_active': '2024-09-05T16:35:25.163986+0000', 'last_peered': '2024-09-05T16:35:25.163986+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:35:25.163986+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 2856, 'log_dups_size': 0, 'ondisk_log_size': 2856, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23344288, 'num_objects': 2850, 'num_object_clones': 0, 'num_object_copies': 11400, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2149, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2850, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2854, 'num_write_kb': 22812, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 703}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:35:29.221 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-05T16:35:29.221 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'2978", 'reported_seq': 3040, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:25.122138+0000', 'last_change': '2024-09-05T16:34:56.883319+0000', 'last_active': '2024-09-05T16:35:25.122138+0000', 'last_peered': '2024-09-05T16:35:25.122138+0000', 'last_clean': '2024-09-05T16:33:53.505305+0000', 'last_became_active': '2024-09-05T16:34:56.873389+0000', 'last_became_peered': '2024-09-05T16:34:56.873389+0000', 'last_unstale': '2024-09-05T16:35:25.122138+0000', 'last_undegraded': '2024-09-05T16:34:56.602567+0000', 'last_fullsized': '2024-09-05T16:34:56.559958+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 2978, 'log_dups_size': 0, 'ondisk_log_size': 2978, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24351904, 'num_objects': 2973, 'num_object_clones': 0, 'num_object_copies': 11892, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2241, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2973, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2976, 'num_write_kb': 23796, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 734}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:35:29.221 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:35:29.221 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'2968", 'reported_seq': 3023, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:24.101078+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:35:24.101078+0000', 'last_peered': '2024-09-05T16:35:24.101078+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:35:24.101078+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 2968, 'log_dups_size': 0, 'ondisk_log_size': 2968, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24242496, 'num_objects': 2960, 'num_object_clones': 0, 'num_object_copies': 11840, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2264, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2960, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2965, 'num_write_kb': 23697, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 699}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:35:29.221 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:35:29.221 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'3031", 'reported_seq': 3106, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:25.052126+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:35:25.052126+0000', 'last_peered': '2024-09-05T16:35:25.052126+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:35:25.052126+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 3031, 'log_dups_size': 0, 'ondisk_log_size': 3031, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24810656, 'num_objects': 3029, 'num_object_clones': 0, 'num_object_copies': 12116, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2264, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3029, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3029, 'num_write_kb': 24230, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 767}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:35:29.221 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:35:29.221 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'3005", 'reported_seq': 3060, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:24.044136+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:35:24.044136+0000', 'last_peered': '2024-09-05T16:35:24.044136+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:35:24.044136+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 3005, 'log_dups_size': 0, 'ondisk_log_size': 3005, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24571632, 'num_objects': 3000, 'num_object_clones': 0, 'num_object_copies': 12000, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2314, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3000, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3002, 'num_write_kb': 24004, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 689}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:35:29.221 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:35:29.221 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'2946", 'reported_seq': 2981, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:23.938095+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:35:23.938095+0000', 'last_peered': '2024-09-05T16:35:23.938095+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:35:23.938095+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 2946, 'log_dups_size': 0, 'ondisk_log_size': 2946, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24123984, 'num_objects': 2945, 'num_object_clones': 0, 'num_object_copies': 11780, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2265, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2945, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2945, 'num_write_kb': 23559, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 681}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:35:29.222 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-05T16:35:29.222 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'2984", 'reported_seq': 3057, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:25.123445+0000', 'last_change': '2024-09-05T16:34:56.798081+0000', 'last_active': '2024-09-05T16:35:25.123445+0000', 'last_peered': '2024-09-05T16:35:25.123445+0000', 'last_clean': '2024-09-05T16:33:45.082039+0000', 'last_became_active': '2024-09-05T16:34:56.792386+0000', 'last_became_peered': '2024-09-05T16:34:56.792386+0000', 'last_unstale': '2024-09-05T16:35:25.123445+0000', 'last_undegraded': '2024-09-05T16:34:56.600595+0000', 'last_fullsized': '2024-09-05T16:34:56.558203+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 2984, 'log_dups_size': 0, 'ondisk_log_size': 2984, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24435280, 'num_objects': 2983, 'num_object_clones': 0, 'num_object_copies': 11932, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2295, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2983, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2983, 'num_write_kb': 23863, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 689}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:35:29.222 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:35:29.222 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'3063", 'reported_seq': 3111, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:24.078328+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:35:24.078328+0000', 'last_peered': '2024-09-05T16:35:24.078328+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:35:24.078328+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 3063, 'log_dups_size': 0, 'ondisk_log_size': 3063, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25056416, 'num_objects': 3059, 'num_object_clones': 0, 'num_object_copies': 12236, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2359, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3059, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3061, 'num_write_kb': 24484, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 702}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:35:29.222 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:35:29.222 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'3032", 'reported_seq': 3109, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:25.091854+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:35:25.091854+0000', 'last_peered': '2024-09-05T16:35:25.091854+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:35:25.091854+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 3032, 'log_dups_size': 0, 'ondisk_log_size': 3032, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24810656, 'num_objects': 3029, 'num_object_clones': 0, 'num_object_copies': 12116, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2322, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3029, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3030, 'num_write_kb': 24237, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 709}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:35:29.222 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:35:29.222 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'2970", 'reported_seq': 3033, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:25.077592+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:35:25.077592+0000', 'last_peered': '2024-09-05T16:35:25.077592+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:35:25.077592+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 2970, 'log_dups_size': 0, 'ondisk_log_size': 2970, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24286368, 'num_objects': 2965, 'num_object_clones': 0, 'num_object_copies': 11860, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2965, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2968, 'num_write_kb': 23732, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 728}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:35:29.223 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:35:29.506 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 67 20 49276 49256 5.74213 2.88281 0.0731191 0.0271955 2024-09-05T16:35:29.952 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:35:29.952 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 35772/189620 objects degraded (18.865%), 16 pgs degraded","count":16},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":133,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":47405,"data_bytes":388300992,"bytes_used":1221894144,"bytes_avail":385325162496,"bytes_total":386547056640,"degraded_objects":35772,"degraded_total":189620,"degraded_ratio":0.18865098618289211,"recovering_objects_per_sec":14,"recovering_bytes_per_sec":115342,"recovering_keys_per_sec":0,"num_objects_recovered":169,"num_bytes_recovered":1384448,"num_keys_recovered":0,"write_bytes_sec":2671989,"read_op_per_sec":0,"write_op_per_sec":326},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-05T16:35:14.802011+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-05T16:35:29.971 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:35:29.972 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:35:30.506 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 68 20 49634 49614 5.69871 2.79688 0.0300857 0.0273994 2024-09-05T16:35:30.555 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:35:30.555 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:35:30.581 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":90,"stamp":"2024-09-05T16:35:28.805344+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":388300992,"num_objects":47405,"num_object_clones":0,"num_object_copies":189620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35772,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47405,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47432,"num_write_kb":379338,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":451,"num_bytes_recovered":3473408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":47459,"ondisk_log_size":47459,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1193256,"kb_used_data":613096,"kb_used_omap":14,"kb_used_meta":580081,"kb_avail":376294104,"statfs":{"total":386547056640,"available":385325162496,"internally_reserved":0,"allocated":627810304,"data_stored":627078776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":594003511},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,7,2,8,15,7,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":32071680,"num_objects":3915,"num_object_clones":0,"num_object_copies":15660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3915,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3915,"num_write_kb":31320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":169,"num_bytes_recovered":1384448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002920"},"pg_stats":[{"pgid":"1.f","version":"33'2945","reported_seq":2986,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:24.059355+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:35:24.059355+0000","last_peered":"2024-09-05T16:35:24.059355+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:35:24.059355+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2945,"log_dups_size":0,"ondisk_log_size":2945,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24125440,"num_objects":2945,"num_object_clones":0,"num_object_copies":11780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2945,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2945,"num_write_kb":23560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":683}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'2938","reported_seq":3001,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:25.121601+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:35:25.121601+0000","last_peered":"2024-09-05T16:35:25.121601+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:35:25.121601+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2938,"log_dups_size":0,"ondisk_log_size":2938,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24032416,"num_objects":2934,"num_object_clones":0,"num_object_copies":11736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2934,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2936,"num_write_kb":23484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":667}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'2984","reported_seq":3058,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:25.110337+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:35:25.110337+0000","last_peered":"2024-09-05T16:35:25.110337+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:35:25.110337+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2984,"log_dups_size":0,"ondisk_log_size":2984,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24418896,"num_objects":2981,"num_object_clones":0,"num_object_copies":11924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2981,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2983,"num_write_kb":23854,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":731}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'2948","reported_seq":3860,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:25.162705+0000","last_change":"2024-09-05T16:34:56.800371+0000","last_active":"2024-09-05T16:35:25.162705+0000","last_peered":"2024-09-05T16:35:25.162705+0000","last_clean":"2024-09-05T16:33:53.148717+0000","last_became_active":"2024-09-05T16:34:56.791755+0000","last_became_peered":"2024-09-05T16:34:56.791755+0000","last_unstale":"2024-09-05T16:35:25.162705+0000","last_undegraded":"2024-09-05T16:34:56.597522+0000","last_fullsized":"2024-09-05T16:34:56.568150+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2948,"log_dups_size":0,"ondisk_log_size":2948,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24114336,"num_objects":2944,"num_object_clones":0,"num_object_copies":11776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1864,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2944,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2946,"num_write_kb":23557,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":426,"num_bytes_recovered":3473408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1080}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'2861","reported_seq":2914,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:24.054310+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:35:24.054310+0000","last_peered":"2024-09-05T16:35:24.054310+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:35:24.054310+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2861,"log_dups_size":0,"ondisk_log_size":2861,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23409824,"num_objects":2858,"num_object_clones":0,"num_object_copies":11432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2858,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2859,"num_write_kb":22869,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":696}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'2950","reported_seq":2990,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:24.094079+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:35:24.094079+0000","last_peered":"2024-09-05T16:35:24.094079+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:35:24.094079+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2950,"log_dups_size":0,"ondisk_log_size":2950,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24166400,"num_objects":2950,"num_object_clones":0,"num_object_copies":11800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2950,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2950,"num_write_kb":23600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":700}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'2856","reported_seq":2919,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:25.163986+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:35:25.163986+0000","last_peered":"2024-09-05T16:35:25.163986+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:35:25.163986+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2856,"log_dups_size":0,"ondisk_log_size":2856,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23344288,"num_objects":2850,"num_object_clones":0,"num_object_copies":11400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2850,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2854,"num_write_kb":22812,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":703}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'2978","reported_seq":3040,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:25.122138+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:35:25.122138+0000","last_peered":"2024-09-05T16:35:25.122138+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:35:25.122138+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2978,"log_dups_size":0,"ondisk_log_size":2978,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24351904,"num_objects":2973,"num_object_clones":0,"num_object_copies":11892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2973,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2976,"num_write_kb":23796,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":734}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'2968","reported_seq":3023,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:24.101078+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:35:24.101078+0000","last_peered":"2024-09-05T16:35:24.101078+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:35:24.101078+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2968,"log_dups_size":0,"ondisk_log_size":2968,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24242496,"num_objects":2960,"num_object_clones":0,"num_object_copies":11840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2960,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2965,"num_write_kb":23697,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":699}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'3031","reported_seq":3106,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:25.052126+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:35:25.052126+0000","last_peered":"2024-09-05T16:35:25.052126+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:35:25.052126+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3031,"log_dups_size":0,"ondisk_log_size":3031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24810656,"num_objects":3029,"num_object_clones":0,"num_object_copies":12116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3029,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3029,"num_write_kb":24230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":767}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'3005","reported_seq":3060,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:24.044136+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:35:24.044136+0000","last_peered":"2024-09-05T16:35:24.044136+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:35:24.044136+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3005,"log_dups_size":0,"ondisk_log_size":3005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24571632,"num_objects":3000,"num_object_clones":0,"num_object_copies":12000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3000,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3002,"num_write_kb":24004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":689}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'2946","reported_seq":2981,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:23.938095+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:35:23.938095+0000","last_peered":"2024-09-05T16:35:23.938095+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:35:23.938095+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2946,"log_dups_size":0,"ondisk_log_size":2946,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24123984,"num_objects":2945,"num_object_clones":0,"num_object_copies":11780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2945,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2945,"num_write_kb":23559,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":681}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'2984","reported_seq":3057,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:25.123445+0000","last_change":"2024-09-05T16:34:56.798081+0000","last_active":"2024-09-05T16:35:25.123445+0000","last_peered":"2024-09-05T16:35:25.123445+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:35:25.123445+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2984,"log_dups_size":0,"ondisk_log_size":2984,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24435280,"num_objects":2983,"num_object_clones":0,"num_object_copies":11932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2983,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2983,"num_write_kb":23863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":689}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'3063","reported_seq":3111,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:24.078328+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:35:24.078328+0000","last_peered":"2024-09-05T16:35:24.078328+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:35:24.078328+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3063,"log_dups_size":0,"ondisk_log_size":3063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25056416,"num_objects":3059,"num_object_clones":0,"num_object_copies":12236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3059,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3061,"num_write_kb":24484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":702}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'3032","reported_seq":3109,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:25.091854+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:35:25.091854+0000","last_peered":"2024-09-05T16:35:25.091854+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:35:25.091854+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3032,"log_dups_size":0,"ondisk_log_size":3032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24810656,"num_objects":3029,"num_object_clones":0,"num_object_copies":12116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3029,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3030,"num_write_kb":24237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":709}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'2970","reported_seq":3033,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:25.077592+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:35:25.077592+0000","last_peered":"2024-09-05T16:35:25.077592+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:35:25.077592+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2970,"log_dups_size":0,"ondisk_log_size":2970,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24286368,"num_objects":2965,"num_object_clones":0,"num_object_copies":11860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2965,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2968,"num_write_kb":23732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":728}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":388300992,"num_objects":47405,"num_object_clones":0,"num_object_copies":189620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35772,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47405,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47432,"num_write_kb":379338,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":451,"num_bytes_recovered":3473408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":629182464,"data_stored":629182464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":47459,"ondisk_log_size":47459,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084306,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":350276,"kb_used_data":188084,"kb_used_omap":4,"kb_used_meta":162171,"kb_avail":94021564,"statfs":{"total":96636764160,"available":96278081536,"internally_reserved":0,"allocated":192598016,"data_stored":192415134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":166063827},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,1,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986183,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":135632,"kb_used_data":45632,"kb_used_omap":4,"kb_used_meta":89979,"kb_avail":94236208,"statfs":{"total":96636764160,"available":96497876992,"internally_reserved":0,"allocated":46727168,"data_stored":46544286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":92139218},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,5,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949672989,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":355584,"kb_used_data":189744,"kb_used_omap":1,"kb_used_meta":165822,"kb_avail":94016256,"statfs":{"total":96636764160,"available":96272646144,"internally_reserved":0,"allocated":194297856,"data_stored":194114974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":169802181},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,2,2,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182420,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":351764,"kb_used_data":189636,"kb_used_omap":4,"kb_used_meta":162107,"kb_avail":94020076,"statfs":{"total":96636764160,"available":96276557824,"internally_reserved":0,"allocated":194187264,"data_stored":194004382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":165998285},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2,0,5,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":194727936,"data_stored":194727936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":46587904,"data_stored":46587904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":194707456,"data_stored":194707456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":193159168,"data_stored":193159168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:35:31.507 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 69 20 50038 50018 5.66187 3.15625 0.0431689 0.0275824 2024-09-05T16:35:32.507 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 70 20 50434 50414 5.62518 3.09375 0.0923028 0.0277572 2024-09-05T16:35:33.507 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 71 20 50833 50813 5.58986 3.11719 0.0556818 0.0279362 2024-09-05T16:35:33.583 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:35:34.167 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:35:34.167 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:35:34.195 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":92,"stamp":"2024-09-05T16:35:32.806899+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":404529344,"num_objects":49386,"num_object_clones":0,"num_object_copies":197544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35705,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49386,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":49413,"num_write_kb":395186,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":518,"num_bytes_recovered":4022272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":49440,"ondisk_log_size":49440,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1284104,"kb_used_data":645640,"kb_used_omap":14,"kb_used_meta":638385,"kb_avail":376203256,"statfs":{"total":386547056640,"available":385232134144,"internally_reserved":0,"allocated":661135360,"data_stored":660403832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":653706807},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,7,8,25,9,10,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":32571392,"num_objects":3976,"num_object_clones":0,"num_object_copies":15904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3976,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3976,"num_write_kb":31808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":138,"num_bytes_recovered":1130496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003654"},"pg_stats":[{"pgid":"1.f","version":"33'3057","reported_seq":3098,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:28.825576+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:35:28.825576+0000","last_peered":"2024-09-05T16:35:28.825576+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:35:28.825576+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3057,"log_dups_size":0,"ondisk_log_size":3057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25042944,"num_objects":3057,"num_object_clones":0,"num_object_copies":12228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3057,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3057,"num_write_kb":24456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":795}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'3067","reported_seq":3127,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:30.160252+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:35:30.160252+0000","last_peered":"2024-09-05T16:35:30.160252+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:35:30.160252+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3067,"log_dups_size":0,"ondisk_log_size":3067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25089184,"num_objects":3063,"num_object_clones":0,"num_object_copies":12252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3063,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3065,"num_write_kb":24516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":796}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'3083","reported_seq":3157,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:30.028168+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:35:30.028168+0000","last_peered":"2024-09-05T16:35:30.028168+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:35:30.028168+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3083,"log_dups_size":0,"ondisk_log_size":3083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25229904,"num_objects":3080,"num_object_clones":0,"num_object_copies":12320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3080,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3082,"num_write_kb":24646,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":830}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'3094","reported_seq":4139,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:30.185339+0000","last_change":"2024-09-05T16:34:56.800371+0000","last_active":"2024-09-05T16:35:30.185339+0000","last_peered":"2024-09-05T16:35:30.185339+0000","last_clean":"2024-09-05T16:33:53.148717+0000","last_became_active":"2024-09-05T16:34:56.791755+0000","last_became_peered":"2024-09-05T16:34:56.791755+0000","last_unstale":"2024-09-05T16:35:30.185339+0000","last_undegraded":"2024-09-05T16:34:56.597522+0000","last_fullsized":"2024-09-05T16:34:56.568150+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3094,"log_dups_size":0,"ondisk_log_size":3094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25310368,"num_objects":3090,"num_object_clones":0,"num_object_copies":12360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1797,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3090,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3092,"num_write_kb":24725,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":493,"num_bytes_recovered":4022272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1293}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'2983","reported_seq":3032,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:29.147075+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:35:29.147075+0000","last_peered":"2024-09-05T16:35:29.147075+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:35:29.147075+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2983,"log_dups_size":0,"ondisk_log_size":2983,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24409248,"num_objects":2980,"num_object_clones":0,"num_object_copies":11920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2980,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2981,"num_write_kb":23845,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":818}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'3052","reported_seq":3093,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:29.118732+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:35:29.118732+0000","last_peered":"2024-09-05T16:35:29.118732+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:35:29.118732+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3052,"log_dups_size":0,"ondisk_log_size":3052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25001984,"num_objects":3052,"num_object_clones":0,"num_object_copies":12208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3052,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3052,"num_write_kb":24416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":802}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'2996","reported_seq":3060,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:30.063101+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:35:30.063101+0000","last_peered":"2024-09-05T16:35:30.063101+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:35:30.063101+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2996,"log_dups_size":0,"ondisk_log_size":2996,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24491168,"num_objects":2990,"num_object_clones":0,"num_object_copies":11960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2990,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2994,"num_write_kb":23932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":843}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'3099","reported_seq":3162,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:30.139393+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:35:30.139393+0000","last_peered":"2024-09-05T16:35:30.139393+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:35:30.139393+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3099,"log_dups_size":0,"ondisk_log_size":3099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25343136,"num_objects":3094,"num_object_clones":0,"num_object_copies":12376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3094,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3097,"num_write_kb":24764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":855}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'3090","reported_seq":3145,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:29.157789+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:35:29.157789+0000","last_peered":"2024-09-05T16:35:29.157789+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:35:29.157789+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3090,"log_dups_size":0,"ondisk_log_size":3090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25241920,"num_objects":3082,"num_object_clones":0,"num_object_copies":12328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3082,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3087,"num_write_kb":24673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":821}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'3138","reported_seq":3213,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:30.141880+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:35:30.141880+0000","last_peered":"2024-09-05T16:35:30.141880+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:35:30.141880+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3138,"log_dups_size":0,"ondisk_log_size":3138,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25687200,"num_objects":3136,"num_object_clones":0,"num_object_copies":12544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3136,"num_write_kb":25086,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":874}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'3155","reported_seq":3210,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:29.118948+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:35:29.118948+0000","last_peered":"2024-09-05T16:35:29.118948+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:35:29.118948+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3155,"log_dups_size":0,"ondisk_log_size":3155,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25800432,"num_objects":3150,"num_object_clones":0,"num_object_copies":12600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3150,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3152,"num_write_kb":25204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":839}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'3079","reported_seq":3114,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:29.134048+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:35:29.134048+0000","last_peered":"2024-09-05T16:35:29.134048+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:35:29.134048+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3079,"log_dups_size":0,"ondisk_log_size":3079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25213520,"num_objects":3078,"num_object_clones":0,"num_object_copies":12312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3078,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3078,"num_write_kb":24623,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":814}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'3112","reported_seq":3185,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:30.166584+0000","last_change":"2024-09-05T16:34:56.798081+0000","last_active":"2024-09-05T16:35:30.166584+0000","last_peered":"2024-09-05T16:35:30.166584+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:35:30.166584+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3112,"log_dups_size":0,"ondisk_log_size":3112,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25483856,"num_objects":3111,"num_object_clones":0,"num_object_copies":12444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3111,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3111,"num_write_kb":24887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":817}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'3197","reported_seq":3248,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:29.044464+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:35:29.044464+0000","last_peered":"2024-09-05T16:35:29.044464+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:35:29.044464+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3197,"log_dups_size":0,"ondisk_log_size":3197,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26154144,"num_objects":3193,"num_object_clones":0,"num_object_copies":12772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3195,"num_write_kb":25556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":836}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'3155","reported_seq":3232,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:30.141795+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:35:30.141795+0000","last_peered":"2024-09-05T16:35:30.141795+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:35:30.141795+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3155,"log_dups_size":0,"ondisk_log_size":3155,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25818272,"num_objects":3152,"num_object_clones":0,"num_object_copies":12608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3152,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3153,"num_write_kb":25221,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":832}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'3083","reported_seq":3146,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:30.059492+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:35:30.059492+0000","last_peered":"2024-09-05T16:35:30.059492+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:35:30.059492+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3083,"log_dups_size":0,"ondisk_log_size":3083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25212064,"num_objects":3078,"num_object_clones":0,"num_object_copies":12312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3078,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3081,"num_write_kb":24636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":841}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":404529344,"num_objects":49386,"num_object_clones":0,"num_object_copies":197544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35705,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49386,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":49413,"num_write_kb":395186,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":518,"num_bytes_recovered":4022272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":661999616,"data_stored":661999616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":49440,"ondisk_log_size":49440,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084307,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":380584,"kb_used_data":196184,"kb_used_omap":4,"kb_used_meta":184379,"kb_avail":93991256,"statfs":{"total":96636764160,"available":96247046144,"internally_reserved":0,"allocated":200892416,"data_stored":200709534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":188804819},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,8,6,4,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986184,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":147996,"kb_used_data":53836,"kb_used_omap":4,"kb_used_meta":94139,"kb_avail":94223844,"statfs":{"total":96636764160,"available":96485216256,"internally_reserved":0,"allocated":55128064,"data_stored":54945182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":96399058},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,3,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949672990,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":369228,"kb_used_data":197820,"kb_used_omap":1,"kb_used_meta":171390,"kb_avail":94002612,"statfs":{"total":96636764160,"available":96258674688,"internally_reserved":0,"allocated":202567680,"data_stored":202384798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":175503813},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,10,1,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182421,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":386296,"kb_used_data":197800,"kb_used_omap":4,"kb_used_meta":188475,"kb_avail":93985544,"statfs":{"total":96636764160,"available":96241197056,"internally_reserved":0,"allocated":202547200,"data_stored":202364318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":192999117},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,3,0,6,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":202821632,"data_stored":202821632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":55013376,"data_stored":55013376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":202825728,"data_stored":202825728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":201338880,"data_stored":201338880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:35:34.196 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:35:34.507 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 72 20 51192 51172 5.55118 2.80469 0.0255779 0.0281304 2024-09-05T16:35:34.797 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:35:34.798 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:35:34.822 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":92,"stamp":"2024-09-05T16:35:32.806899+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":404529344,"num_objects":49386,"num_object_clones":0,"num_object_copies":197544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35705,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49386,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":49413,"num_write_kb":395186,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":518,"num_bytes_recovered":4022272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":49440,"ondisk_log_size":49440,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1284104,"kb_used_data":645640,"kb_used_omap":14,"kb_used_meta":638385,"kb_avail":376203256,"statfs":{"total":386547056640,"available":385232134144,"internally_reserved":0,"allocated":661135360,"data_stored":660403832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":653706807},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,7,8,25,9,10,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":32571392,"num_objects":3976,"num_object_clones":0,"num_object_copies":15904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3976,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3976,"num_write_kb":31808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":138,"num_bytes_recovered":1130496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003654"},"pg_stats":[{"pgid":"1.f","version":"33'3057","reported_seq":3098,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:28.825576+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:35:28.825576+0000","last_peered":"2024-09-05T16:35:28.825576+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:35:28.825576+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3057,"log_dups_size":0,"ondisk_log_size":3057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25042944,"num_objects":3057,"num_object_clones":0,"num_object_copies":12228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3057,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3057,"num_write_kb":24456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":795}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'3067","reported_seq":3127,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:30.160252+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:35:30.160252+0000","last_peered":"2024-09-05T16:35:30.160252+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:35:30.160252+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3067,"log_dups_size":0,"ondisk_log_size":3067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25089184,"num_objects":3063,"num_object_clones":0,"num_object_copies":12252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3063,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3065,"num_write_kb":24516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":796}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'3083","reported_seq":3157,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:30.028168+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:35:30.028168+0000","last_peered":"2024-09-05T16:35:30.028168+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:35:30.028168+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3083,"log_dups_size":0,"ondisk_log_size":3083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25229904,"num_objects":3080,"num_object_clones":0,"num_object_copies":12320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3080,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3082,"num_write_kb":24646,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":830}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'3094","reported_seq":4139,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:30.185339+0000","last_change":"2024-09-05T16:34:56.800371+0000","last_active":"2024-09-05T16:35:30.185339+0000","last_peered":"2024-09-05T16:35:30.185339+0000","last_clean":"2024-09-05T16:33:53.148717+0000","last_became_active":"2024-09-05T16:34:56.791755+0000","last_became_peered":"2024-09-05T16:34:56.791755+0000","last_unstale":"2024-09-05T16:35:30.185339+0000","last_undegraded":"2024-09-05T16:34:56.597522+0000","last_fullsized":"2024-09-05T16:34:56.568150+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3094,"log_dups_size":0,"ondisk_log_size":3094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25310368,"num_objects":3090,"num_object_clones":0,"num_object_copies":12360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1797,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3090,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3092,"num_write_kb":24725,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":493,"num_bytes_recovered":4022272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1293}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'2983","reported_seq":3032,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:29.147075+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:35:29.147075+0000","last_peered":"2024-09-05T16:35:29.147075+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:35:29.147075+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2983,"log_dups_size":0,"ondisk_log_size":2983,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24409248,"num_objects":2980,"num_object_clones":0,"num_object_copies":11920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2980,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2981,"num_write_kb":23845,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":818}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'3052","reported_seq":3093,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:29.118732+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:35:29.118732+0000","last_peered":"2024-09-05T16:35:29.118732+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:35:29.118732+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3052,"log_dups_size":0,"ondisk_log_size":3052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25001984,"num_objects":3052,"num_object_clones":0,"num_object_copies":12208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3052,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3052,"num_write_kb":24416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":802}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'2996","reported_seq":3060,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:30.063101+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:35:30.063101+0000","last_peered":"2024-09-05T16:35:30.063101+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:35:30.063101+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2996,"log_dups_size":0,"ondisk_log_size":2996,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24491168,"num_objects":2990,"num_object_clones":0,"num_object_copies":11960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2990,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2994,"num_write_kb":23932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":843}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'3099","reported_seq":3162,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:30.139393+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:35:30.139393+0000","last_peered":"2024-09-05T16:35:30.139393+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:35:30.139393+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3099,"log_dups_size":0,"ondisk_log_size":3099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25343136,"num_objects":3094,"num_object_clones":0,"num_object_copies":12376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3094,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3097,"num_write_kb":24764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":855}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'3090","reported_seq":3145,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:29.157789+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:35:29.157789+0000","last_peered":"2024-09-05T16:35:29.157789+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:35:29.157789+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3090,"log_dups_size":0,"ondisk_log_size":3090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25241920,"num_objects":3082,"num_object_clones":0,"num_object_copies":12328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3082,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3087,"num_write_kb":24673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":821}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'3138","reported_seq":3213,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:30.141880+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:35:30.141880+0000","last_peered":"2024-09-05T16:35:30.141880+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:35:30.141880+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3138,"log_dups_size":0,"ondisk_log_size":3138,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25687200,"num_objects":3136,"num_object_clones":0,"num_object_copies":12544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3136,"num_write_kb":25086,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":874}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'3155","reported_seq":3210,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:29.118948+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:35:29.118948+0000","last_peered":"2024-09-05T16:35:29.118948+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:35:29.118948+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3155,"log_dups_size":0,"ondisk_log_size":3155,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25800432,"num_objects":3150,"num_object_clones":0,"num_object_copies":12600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3150,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3152,"num_write_kb":25204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":839}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'3079","reported_seq":3114,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:29.134048+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:35:29.134048+0000","last_peered":"2024-09-05T16:35:29.134048+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:35:29.134048+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3079,"log_dups_size":0,"ondisk_log_size":3079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25213520,"num_objects":3078,"num_object_clones":0,"num_object_copies":12312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3078,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3078,"num_write_kb":24623,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":814}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'3112","reported_seq":3185,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:30.166584+0000","last_change":"2024-09-05T16:34:56.798081+0000","last_active":"2024-09-05T16:35:30.166584+0000","last_peered":"2024-09-05T16:35:30.166584+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:35:30.166584+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3112,"log_dups_size":0,"ondisk_log_size":3112,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25483856,"num_objects":3111,"num_object_clones":0,"num_object_copies":12444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3111,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3111,"num_write_kb":24887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":817}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'3197","reported_seq":3248,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:29.044464+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:35:29.044464+0000","last_peered":"2024-09-05T16:35:29.044464+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:35:29.044464+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3197,"log_dups_size":0,"ondisk_log_size":3197,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26154144,"num_objects":3193,"num_object_clones":0,"num_object_copies":12772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3195,"num_write_kb":25556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":836}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'3155","reported_seq":3232,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:30.141795+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:35:30.141795+0000","last_peered":"2024-09-05T16:35:30.141795+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:35:30.141795+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3155,"log_dups_size":0,"ondisk_log_size":3155,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25818272,"num_objects":3152,"num_object_clones":0,"num_object_copies":12608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3152,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3153,"num_write_kb":25221,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":832}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'3083","reported_seq":3146,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:30.059492+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:35:30.059492+0000","last_peered":"2024-09-05T16:35:30.059492+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:35:30.059492+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3083,"log_dups_size":0,"ondisk_log_size":3083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25212064,"num_objects":3078,"num_object_clones":0,"num_object_copies":12312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3078,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3081,"num_write_kb":24636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":841}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":404529344,"num_objects":49386,"num_object_clones":0,"num_object_copies":197544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35705,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49386,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":49413,"num_write_kb":395186,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":518,"num_bytes_recovered":4022272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":661999616,"data_stored":661999616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":49440,"ondisk_log_size":49440,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084307,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":380584,"kb_used_data":196184,"kb_used_omap":4,"kb_used_meta":184379,"kb_avail":93991256,"statfs":{"total":96636764160,"available":96247046144,"internally_reserved":0,"allocated":200892416,"data_stored":200709534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":188804819},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,8,6,4,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986184,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":147996,"kb_used_data":53836,"kb_used_omap":4,"kb_used_meta":94139,"kb_avail":94223844,"statfs":{"total":96636764160,"available":96485216256,"internally_reserved":0,"allocated":55128064,"data_stored":54945182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":96399058},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,3,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949672990,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":369228,"kb_used_data":197820,"kb_used_omap":1,"kb_used_meta":171390,"kb_avail":94002612,"statfs":{"total":96636764160,"available":96258674688,"internally_reserved":0,"allocated":202567680,"data_stored":202384798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":175503813},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,10,1,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182421,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":386296,"kb_used_data":197800,"kb_used_omap":4,"kb_used_meta":188475,"kb_avail":93985544,"statfs":{"total":96636764160,"available":96241197056,"internally_reserved":0,"allocated":202547200,"data_stored":202364318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":192999117},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,3,0,6,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":202821632,"data_stored":202821632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":55013376,"data_stored":55013376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":202825728,"data_stored":202825728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":201338880,"data_stored":201338880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:35:34.823 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-05T16:35:34.823 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'3057", 'reported_seq': 3098, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:28.825576+0000', 'last_change': '2024-09-05T16:34:56.833497+0000', 'last_active': '2024-09-05T16:35:28.825576+0000', 'last_peered': '2024-09-05T16:35:28.825576+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.828289+0000', 'last_became_peered': '2024-09-05T16:34:56.828289+0000', 'last_unstale': '2024-09-05T16:35:28.825576+0000', 'last_undegraded': '2024-09-05T16:34:56.612160+0000', 'last_fullsized': '2024-09-05T16:34:56.557632+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 3057, 'log_dups_size': 0, 'ondisk_log_size': 3057, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25042944, 'num_objects': 3057, 'num_object_clones': 0, 'num_object_copies': 12228, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2262, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3057, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3057, 'num_write_kb': 24456, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 795}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:35:34.824 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-05T16:35:34.824 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'3067", 'reported_seq': 3127, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:30.160252+0000', 'last_change': '2024-09-05T16:34:56.825031+0000', 'last_active': '2024-09-05T16:35:30.160252+0000', 'last_peered': '2024-09-05T16:35:30.160252+0000', 'last_clean': '2024-09-05T16:33:53.200762+0000', 'last_became_active': '2024-09-05T16:34:56.820292+0000', 'last_became_peered': '2024-09-05T16:34:56.820292+0000', 'last_unstale': '2024-09-05T16:35:30.160252+0000', 'last_undegraded': '2024-09-05T16:34:56.605776+0000', 'last_fullsized': '2024-09-05T16:34:56.559383+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 3067, 'log_dups_size': 0, 'ondisk_log_size': 3067, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25089184, 'num_objects': 3063, 'num_object_clones': 0, 'num_object_copies': 12252, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2269, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3063, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3065, 'num_write_kb': 24516, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 796}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:35:34.824 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:35:34.824 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'3083", 'reported_seq': 3157, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:30.028168+0000', 'last_change': '2024-09-05T16:34:56.867900+0000', 'last_active': '2024-09-05T16:35:30.028168+0000', 'last_peered': '2024-09-05T16:35:30.028168+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:35:30.028168+0000', 'last_undegraded': '2024-09-05T16:34:56.590972+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 3083, 'log_dups_size': 0, 'ondisk_log_size': 3083, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25229904, 'num_objects': 3080, 'num_object_clones': 0, 'num_object_copies': 12320, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2251, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3080, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3082, 'num_write_kb': 24646, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 830}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:35:34.824 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-05T16:35:34.824 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'3094", 'reported_seq': 4139, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:30.185339+0000', 'last_change': '2024-09-05T16:34:56.800371+0000', 'last_active': '2024-09-05T16:35:30.185339+0000', 'last_peered': '2024-09-05T16:35:30.185339+0000', 'last_clean': '2024-09-05T16:33:53.148717+0000', 'last_became_active': '2024-09-05T16:34:56.791755+0000', 'last_became_peered': '2024-09-05T16:34:56.791755+0000', 'last_unstale': '2024-09-05T16:35:30.185339+0000', 'last_undegraded': '2024-09-05T16:34:56.597522+0000', 'last_fullsized': '2024-09-05T16:34:56.568150+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 3094, 'log_dups_size': 0, 'ondisk_log_size': 3094, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25310368, 'num_objects': 3090, 'num_object_clones': 0, 'num_object_copies': 12360, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1797, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3090, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3092, 'num_write_kb': 24725, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 493, 'num_bytes_recovered': 4022272, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1293}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:35:34.825 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:35:34.825 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'2983", 'reported_seq': 3032, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:29.147075+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:35:29.147075+0000', 'last_peered': '2024-09-05T16:35:29.147075+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:35:29.147075+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 2983, 'log_dups_size': 0, 'ondisk_log_size': 2983, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24409248, 'num_objects': 2980, 'num_object_clones': 0, 'num_object_copies': 11920, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2164, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2980, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2981, 'num_write_kb': 23845, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 818}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:35:34.825 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:35:34.825 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'3052", 'reported_seq': 3093, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:29.118732+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:35:29.118732+0000', 'last_peered': '2024-09-05T16:35:29.118732+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:35:29.118732+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 3052, 'log_dups_size': 0, 'ondisk_log_size': 3052, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25001984, 'num_objects': 3052, 'num_object_clones': 0, 'num_object_copies': 12208, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3052, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3052, 'num_write_kb': 24416, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 802}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:35:34.825 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:35:34.826 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'2996", 'reported_seq': 3060, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:30.063101+0000', 'last_change': '2024-09-05T16:34:56.903436+0000', 'last_active': '2024-09-05T16:35:30.063101+0000', 'last_peered': '2024-09-05T16:35:30.063101+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:35:30.063101+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 2996, 'log_dups_size': 0, 'ondisk_log_size': 2996, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24491168, 'num_objects': 2990, 'num_object_clones': 0, 'num_object_copies': 11960, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2149, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2990, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2994, 'num_write_kb': 23932, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 843}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:35:34.826 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-05T16:35:34.826 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'3099", 'reported_seq': 3162, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:30.139393+0000', 'last_change': '2024-09-05T16:34:56.883319+0000', 'last_active': '2024-09-05T16:35:30.139393+0000', 'last_peered': '2024-09-05T16:35:30.139393+0000', 'last_clean': '2024-09-05T16:33:53.505305+0000', 'last_became_active': '2024-09-05T16:34:56.873389+0000', 'last_became_peered': '2024-09-05T16:34:56.873389+0000', 'last_unstale': '2024-09-05T16:35:30.139393+0000', 'last_undegraded': '2024-09-05T16:34:56.602567+0000', 'last_fullsized': '2024-09-05T16:34:56.559958+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 3099, 'log_dups_size': 0, 'ondisk_log_size': 3099, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25343136, 'num_objects': 3094, 'num_object_clones': 0, 'num_object_copies': 12376, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2241, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3094, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3097, 'num_write_kb': 24764, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 855}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:35:34.826 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:35:34.826 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'3090", 'reported_seq': 3145, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:29.157789+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:35:29.157789+0000', 'last_peered': '2024-09-05T16:35:29.157789+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:35:29.157789+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 3090, 'log_dups_size': 0, 'ondisk_log_size': 3090, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25241920, 'num_objects': 3082, 'num_object_clones': 0, 'num_object_copies': 12328, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2264, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3082, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3087, 'num_write_kb': 24673, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 821}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:35:34.826 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:35:34.826 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'3138", 'reported_seq': 3213, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:30.141880+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:35:30.141880+0000', 'last_peered': '2024-09-05T16:35:30.141880+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:35:30.141880+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 3138, 'log_dups_size': 0, 'ondisk_log_size': 3138, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25687200, 'num_objects': 3136, 'num_object_clones': 0, 'num_object_copies': 12544, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2264, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3136, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3136, 'num_write_kb': 25086, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 874}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:35:34.827 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:35:34.827 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'3155", 'reported_seq': 3210, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:29.118948+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:35:29.118948+0000', 'last_peered': '2024-09-05T16:35:29.118948+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:35:29.118948+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 3155, 'log_dups_size': 0, 'ondisk_log_size': 3155, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25800432, 'num_objects': 3150, 'num_object_clones': 0, 'num_object_copies': 12600, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2314, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3150, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3152, 'num_write_kb': 25204, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 839}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:35:34.827 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:35:34.827 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'3079", 'reported_seq': 3114, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:29.134048+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:35:29.134048+0000', 'last_peered': '2024-09-05T16:35:29.134048+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:35:29.134048+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 3079, 'log_dups_size': 0, 'ondisk_log_size': 3079, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25213520, 'num_objects': 3078, 'num_object_clones': 0, 'num_object_copies': 12312, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2265, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3078, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3078, 'num_write_kb': 24623, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 814}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:35:34.827 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-05T16:35:34.827 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'3112", 'reported_seq': 3185, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:30.166584+0000', 'last_change': '2024-09-05T16:34:56.798081+0000', 'last_active': '2024-09-05T16:35:30.166584+0000', 'last_peered': '2024-09-05T16:35:30.166584+0000', 'last_clean': '2024-09-05T16:33:45.082039+0000', 'last_became_active': '2024-09-05T16:34:56.792386+0000', 'last_became_peered': '2024-09-05T16:34:56.792386+0000', 'last_unstale': '2024-09-05T16:35:30.166584+0000', 'last_undegraded': '2024-09-05T16:34:56.600595+0000', 'last_fullsized': '2024-09-05T16:34:56.558203+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 3112, 'log_dups_size': 0, 'ondisk_log_size': 3112, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25483856, 'num_objects': 3111, 'num_object_clones': 0, 'num_object_copies': 12444, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2295, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3111, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3111, 'num_write_kb': 24887, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 817}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:35:34.827 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:35:34.827 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'3197", 'reported_seq': 3248, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:29.044464+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:35:29.044464+0000', 'last_peered': '2024-09-05T16:35:29.044464+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:35:29.044464+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 3197, 'log_dups_size': 0, 'ondisk_log_size': 3197, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26154144, 'num_objects': 3193, 'num_object_clones': 0, 'num_object_copies': 12772, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2359, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3193, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3195, 'num_write_kb': 25556, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 836}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:35:34.827 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:35:34.827 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'3155", 'reported_seq': 3232, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:30.141795+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:35:30.141795+0000', 'last_peered': '2024-09-05T16:35:30.141795+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:35:30.141795+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 3155, 'log_dups_size': 0, 'ondisk_log_size': 3155, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25818272, 'num_objects': 3152, 'num_object_clones': 0, 'num_object_copies': 12608, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2322, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3152, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3153, 'num_write_kb': 25221, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 832}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:35:34.827 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:35:34.828 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'3083", 'reported_seq': 3146, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:30.059492+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:35:30.059492+0000', 'last_peered': '2024-09-05T16:35:30.059492+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:35:30.059492+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 3083, 'log_dups_size': 0, 'ondisk_log_size': 3083, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25212064, 'num_objects': 3078, 'num_object_clones': 0, 'num_object_copies': 12312, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3078, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3081, 'num_write_kb': 24636, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 841}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:35:34.828 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:35:35.507 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 73 19 51545 51526 5.51302 2.76562 0.0625475 0.0283296 2024-09-05T16:35:35.582 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:35:35.582 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 35705/200880 objects degraded (17.774%), 16 pgs degraded","count":16},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":138,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":50220,"data_bytes":411361472,"bytes_used":1309765632,"bytes_avail":385237291008,"bytes_total":386547056640,"degraded_objects":35705,"degraded_total":200880,"degraded_ratio":0.17774293110314615,"recovering_objects_per_sec":11,"recovering_bytes_per_sec":94177,"recovering_keys_per_sec":0,"num_objects_recovered":138,"num_bytes_recovered":1130496,"num_keys_recovered":0,"write_bytes_sec":3282567,"read_op_per_sec":0,"write_op_per_sec":400},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-05T16:35:14.802011+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-05T16:35:35.614 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:35:35.615 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:35:36.236 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:35:36.237 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:35:36.267 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":93,"stamp":"2024-09-05T16:35:34.807676+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":411361472,"num_objects":50220,"num_object_clones":0,"num_object_copies":200880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35705,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50220,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":50247,"num_write_kb":401858,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":518,"num_bytes_recovered":4022272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":50274,"ondisk_log_size":50274,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1279068,"kb_used_data":661404,"kb_used_omap":14,"kb_used_meta":617585,"kb_avail":376208292,"statfs":{"total":386547056640,"available":385237291008,"internally_reserved":0,"allocated":677277696,"data_stored":676546168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":632407607},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,4,9,28,7,10,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":39403520,"num_objects":4810,"num_object_clones":0,"num_object_copies":19240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4810,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4810,"num_write_kb":38480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":138,"num_bytes_recovered":1130496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003872"},"pg_stats":[{"pgid":"1.f","version":"33'3180","reported_seq":3221,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:34.212269+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:35:34.212269+0000","last_peered":"2024-09-05T16:35:34.212269+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:35:34.212269+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3180,"log_dups_size":0,"ondisk_log_size":3180,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26050560,"num_objects":3180,"num_object_clones":0,"num_object_copies":12720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3180,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3180,"num_write_kb":25440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":918}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'3067","reported_seq":3127,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:30.160252+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:35:30.160252+0000","last_peered":"2024-09-05T16:35:30.160252+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:35:30.160252+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3067,"log_dups_size":0,"ondisk_log_size":3067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25089184,"num_objects":3063,"num_object_clones":0,"num_object_copies":12252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3063,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3065,"num_write_kb":24516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":796}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'3083","reported_seq":3157,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:30.028168+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:35:30.028168+0000","last_peered":"2024-09-05T16:35:30.028168+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:35:30.028168+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3083,"log_dups_size":0,"ondisk_log_size":3083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25229904,"num_objects":3080,"num_object_clones":0,"num_object_copies":12320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3080,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3082,"num_write_kb":24646,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":830}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'3094","reported_seq":4139,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:30.185339+0000","last_change":"2024-09-05T16:34:56.800371+0000","last_active":"2024-09-05T16:35:30.185339+0000","last_peered":"2024-09-05T16:35:30.185339+0000","last_clean":"2024-09-05T16:33:53.148717+0000","last_became_active":"2024-09-05T16:34:56.791755+0000","last_became_peered":"2024-09-05T16:34:56.791755+0000","last_unstale":"2024-09-05T16:35:30.185339+0000","last_undegraded":"2024-09-05T16:34:56.597522+0000","last_fullsized":"2024-09-05T16:34:56.568150+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3094,"log_dups_size":0,"ondisk_log_size":3094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25310368,"num_objects":3090,"num_object_clones":0,"num_object_copies":12360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1797,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3090,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3092,"num_write_kb":24725,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":493,"num_bytes_recovered":4022272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1293}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'3095","reported_seq":3148,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:34.209570+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:35:34.209570+0000","last_peered":"2024-09-05T16:35:34.209570+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:35:34.209570+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3095,"log_dups_size":0,"ondisk_log_size":3095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25326752,"num_objects":3092,"num_object_clones":0,"num_object_copies":12368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3092,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3093,"num_write_kb":24741,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":930}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'3188","reported_seq":3229,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:34.108212+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:35:34.108212+0000","last_peered":"2024-09-05T16:35:34.108212+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:35:34.108212+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3188,"log_dups_size":0,"ondisk_log_size":3188,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26116096,"num_objects":3188,"num_object_clones":0,"num_object_copies":12752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3188,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3188,"num_write_kb":25504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":938}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'2996","reported_seq":3060,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:30.063101+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:35:30.063101+0000","last_peered":"2024-09-05T16:35:30.063101+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:35:30.063101+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":2996,"log_dups_size":0,"ondisk_log_size":2996,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24491168,"num_objects":2990,"num_object_clones":0,"num_object_copies":11960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2990,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2994,"num_write_kb":23932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":843}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'3099","reported_seq":3162,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:30.139393+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:35:30.139393+0000","last_peered":"2024-09-05T16:35:30.139393+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:35:30.139393+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3099,"log_dups_size":0,"ondisk_log_size":3099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25343136,"num_objects":3094,"num_object_clones":0,"num_object_copies":12376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3094,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3097,"num_write_kb":24764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":855}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'3208","reported_seq":3263,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:34.105076+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:35:34.105076+0000","last_peered":"2024-09-05T16:35:34.105076+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:35:34.105076+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3208,"log_dups_size":0,"ondisk_log_size":3208,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26208576,"num_objects":3200,"num_object_clones":0,"num_object_copies":12800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3200,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3205,"num_write_kb":25617,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":939}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'3138","reported_seq":3213,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:30.141880+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:35:30.141880+0000","last_peered":"2024-09-05T16:35:30.141880+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:35:30.141880+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3138,"log_dups_size":0,"ondisk_log_size":3138,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25687200,"num_objects":3136,"num_object_clones":0,"num_object_copies":12544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3136,"num_write_kb":25086,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":874}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'3277","reported_seq":3332,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:34.181346+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:35:34.181346+0000","last_peered":"2024-09-05T16:35:34.181346+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:35:34.181346+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3277,"log_dups_size":0,"ondisk_log_size":3277,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26799856,"num_objects":3272,"num_object_clones":0,"num_object_copies":13088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3272,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3274,"num_write_kb":26180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":961}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'3194","reported_seq":3229,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:34.119786+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:35:34.119786+0000","last_peered":"2024-09-05T16:35:34.119786+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:35:34.119786+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3194,"log_dups_size":0,"ondisk_log_size":3194,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26155600,"num_objects":3193,"num_object_clones":0,"num_object_copies":12772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3193,"num_write_kb":25543,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":929}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'3112","reported_seq":3185,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:30.166584+0000","last_change":"2024-09-05T16:34:56.798081+0000","last_active":"2024-09-05T16:35:30.166584+0000","last_peered":"2024-09-05T16:35:30.166584+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:35:30.166584+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3112,"log_dups_size":0,"ondisk_log_size":3112,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25483856,"num_objects":3111,"num_object_clones":0,"num_object_copies":12444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3111,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3111,"num_write_kb":24887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":817}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'3305","reported_seq":3356,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:34.232066+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:35:34.232066+0000","last_peered":"2024-09-05T16:35:34.232066+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:35:34.232066+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3305,"log_dups_size":0,"ondisk_log_size":3305,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27038880,"num_objects":3301,"num_object_clones":0,"num_object_copies":13204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3301,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3303,"num_write_kb":26420,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":944}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'3155","reported_seq":3232,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:30.141795+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:35:30.141795+0000","last_peered":"2024-09-05T16:35:30.141795+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:35:30.141795+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3155,"log_dups_size":0,"ondisk_log_size":3155,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25818272,"num_objects":3152,"num_object_clones":0,"num_object_copies":12608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3152,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3153,"num_write_kb":25221,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":832}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'3083","reported_seq":3146,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:30.059492+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:35:30.059492+0000","last_peered":"2024-09-05T16:35:30.059492+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:35:30.059492+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3083,"log_dups_size":0,"ondisk_log_size":3083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25212064,"num_objects":3078,"num_object_clones":0,"num_object_copies":12312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3078,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3081,"num_write_kb":24636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":841}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":411361472,"num_objects":50220,"num_object_clones":0,"num_object_copies":200880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35705,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50220,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":50247,"num_write_kb":401858,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":518,"num_bytes_recovered":4022272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":678162432,"data_stored":678162432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":50274,"ondisk_log_size":50274,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084308,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":375908,"kb_used_data":203988,"kb_used_omap":4,"kb_used_meta":171899,"kb_avail":93995932,"statfs":{"total":96636764160,"available":96251834368,"internally_reserved":0,"allocated":208883712,"data_stored":208700830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":176025299},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,8,6,4,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986185,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":147636,"kb_used_data":61796,"kb_used_omap":4,"kb_used_meta":85819,"kb_avail":94224204,"statfs":{"total":96636764160,"available":96485584896,"internally_reserved":0,"allocated":63279104,"data_stored":63096222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":87879378},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,6],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949672990,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":369228,"kb_used_data":197820,"kb_used_omap":1,"kb_used_meta":171390,"kb_avail":94002612,"statfs":{"total":96636764160,"available":96258674688,"internally_reserved":0,"allocated":202567680,"data_stored":202384798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":175503813},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,10,1,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182421,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":386296,"kb_used_data":197800,"kb_used_omap":4,"kb_used_meta":188475,"kb_avail":93985544,"statfs":{"total":96636764160,"available":96241197056,"internally_reserved":0,"allocated":202547200,"data_stored":202364318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":192999117},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,3,0,6,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":202821632,"data_stored":202821632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63238144,"data_stored":63238144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":202825728,"data_stored":202825728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":209276928,"data_stored":209276928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:35:36.507 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 74 20 51906 51886 5.47653 2.8125 0.0776973 0.0285177 2024-09-05T16:35:37.507 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 75 19 52300 52281 5.44466 3.08594 0.0557498 0.0286814 2024-09-05T16:35:38.507 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 76 20 52692 52672 5.41322 3.05469 0.0602263 0.0288452 2024-09-05T16:35:39.269 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:35:39.507 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 77 20 53082 53062 5.38248 3.04688 0.102245 0.0290132 2024-09-05T16:35:39.906 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:35:39.906 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:35:39.937 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":95,"stamp":"2024-09-05T16:35:38.809194+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":420372672,"num_objects":51320,"num_object_clones":0,"num_object_copies":205280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35631,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51320,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":51347,"num_write_kb":410658,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":592,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":51374,"ondisk_log_size":51374,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1285928,"kb_used_data":676584,"kb_used_omap":14,"kb_used_meta":609265,"kb_avail":376201432,"statfs":{"total":386547056640,"available":385230266368,"internally_reserved":0,"allocated":692822016,"data_stored":692090488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":623887927},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,4,18,13,9,11,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":32071680,"num_objects":3915,"num_object_clones":0,"num_object_copies":15660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-141,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3915,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3915,"num_write_kb":31320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":141,"num_bytes_recovered":1155072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004476"},"pg_stats":[{"pgid":"1.f","version":"33'3180","reported_seq":3221,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:34.212269+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:35:34.212269+0000","last_peered":"2024-09-05T16:35:34.212269+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:35:34.212269+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3180,"log_dups_size":0,"ondisk_log_size":3180,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26050560,"num_objects":3180,"num_object_clones":0,"num_object_copies":12720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3180,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3180,"num_write_kb":25440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":918}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'3186","reported_seq":3249,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:35.111834+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:35:35.111834+0000","last_peered":"2024-09-05T16:35:35.111834+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:35:35.111834+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3186,"log_dups_size":0,"ondisk_log_size":3186,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26064032,"num_objects":3182,"num_object_clones":0,"num_object_copies":12728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3182,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3184,"num_write_kb":25468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":915}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'3217","reported_seq":3291,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:35.178263+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:35:35.178263+0000","last_peered":"2024-09-05T16:35:35.178263+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:35:35.178263+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3217,"log_dups_size":0,"ondisk_log_size":3217,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26327632,"num_objects":3214,"num_object_clones":0,"num_object_copies":12856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3214,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3216,"num_write_kb":25718,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":964}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'3220","reported_seq":4414,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:35.253583+0000","last_change":"2024-09-05T16:34:56.800371+0000","last_active":"2024-09-05T16:35:35.253583+0000","last_peered":"2024-09-05T16:35:35.253583+0000","last_clean":"2024-09-05T16:33:53.148717+0000","last_became_active":"2024-09-05T16:34:56.791755+0000","last_became_peered":"2024-09-05T16:34:56.791755+0000","last_unstale":"2024-09-05T16:35:35.253583+0000","last_undegraded":"2024-09-05T16:34:56.597522+0000","last_fullsized":"2024-09-05T16:34:56.568150+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3220,"log_dups_size":0,"ondisk_log_size":3220,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26342560,"num_objects":3216,"num_object_clones":0,"num_object_copies":12864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1723,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3218,"num_write_kb":25733,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":567,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1493}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'3095","reported_seq":3148,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:34.209570+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:35:34.209570+0000","last_peered":"2024-09-05T16:35:34.209570+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:35:34.209570+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3095,"log_dups_size":0,"ondisk_log_size":3095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25326752,"num_objects":3092,"num_object_clones":0,"num_object_copies":12368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3092,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3093,"num_write_kb":24741,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":930}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'3188","reported_seq":3229,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:34.108212+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:35:34.108212+0000","last_peered":"2024-09-05T16:35:34.108212+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:35:34.108212+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3188,"log_dups_size":0,"ondisk_log_size":3188,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26116096,"num_objects":3188,"num_object_clones":0,"num_object_copies":12752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3188,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3188,"num_write_kb":25504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":938}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'3118","reported_seq":3182,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:35.205906+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:35:35.205906+0000","last_peered":"2024-09-05T16:35:35.205906+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:35:35.205906+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3118,"log_dups_size":0,"ondisk_log_size":3118,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25490592,"num_objects":3112,"num_object_clones":0,"num_object_copies":12448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3112,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3116,"num_write_kb":24908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":965}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'3237","reported_seq":3300,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:35.157549+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:35:35.157549+0000","last_peered":"2024-09-05T16:35:35.157549+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:35:35.157549+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3237,"log_dups_size":0,"ondisk_log_size":3237,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26473632,"num_objects":3232,"num_object_clones":0,"num_object_copies":12928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3232,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3235,"num_write_kb":25868,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":993}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'3208","reported_seq":3263,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:34.105076+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:35:34.105076+0000","last_peered":"2024-09-05T16:35:34.105076+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:35:34.105076+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3208,"log_dups_size":0,"ondisk_log_size":3208,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26208576,"num_objects":3200,"num_object_clones":0,"num_object_copies":12800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3200,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3205,"num_write_kb":25617,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":939}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'3267","reported_seq":3343,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:35.108833+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:35:35.108833+0000","last_peered":"2024-09-05T16:35:35.108833+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:35:35.108833+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3267,"log_dups_size":0,"ondisk_log_size":3267,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26743968,"num_objects":3265,"num_object_clones":0,"num_object_copies":13060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3265,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3265,"num_write_kb":26118,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1003}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'3277","reported_seq":3332,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:34.181346+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:35:34.181346+0000","last_peered":"2024-09-05T16:35:34.181346+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:35:34.181346+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3277,"log_dups_size":0,"ondisk_log_size":3277,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26799856,"num_objects":3272,"num_object_clones":0,"num_object_copies":13088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3272,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3274,"num_write_kb":26180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":961}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'3194","reported_seq":3229,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:34.119786+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:35:34.119786+0000","last_peered":"2024-09-05T16:35:34.119786+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:35:34.119786+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3194,"log_dups_size":0,"ondisk_log_size":3194,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26155600,"num_objects":3193,"num_object_clones":0,"num_object_copies":12772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3193,"num_write_kb":25543,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":929}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'3215","reported_seq":3288,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:35.172524+0000","last_change":"2024-09-05T16:34:56.798081+0000","last_active":"2024-09-05T16:35:35.172524+0000","last_peered":"2024-09-05T16:35:35.172524+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:35:35.172524+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3215,"log_dups_size":0,"ondisk_log_size":3215,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26327632,"num_objects":3214,"num_object_clones":0,"num_object_copies":12856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3214,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3214,"num_write_kb":25711,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":920}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'3305","reported_seq":3356,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:34.232066+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:35:34.232066+0000","last_peered":"2024-09-05T16:35:34.232066+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:35:34.232066+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3305,"log_dups_size":0,"ondisk_log_size":3305,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27038880,"num_objects":3301,"num_object_clones":0,"num_object_copies":13204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3301,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3303,"num_write_kb":26420,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":944}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'3272","reported_seq":3349,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:35.056112+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:35:35.056112+0000","last_peered":"2024-09-05T16:35:35.056112+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:35:35.056112+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3272,"log_dups_size":0,"ondisk_log_size":3272,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26776736,"num_objects":3269,"num_object_clones":0,"num_object_copies":13076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3269,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3270,"num_write_kb":26157,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":949}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'3195","reported_seq":3258,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:35.257174+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:35:35.257174+0000","last_peered":"2024-09-05T16:35:35.257174+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:35:35.257174+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3195,"log_dups_size":0,"ondisk_log_size":3195,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26129568,"num_objects":3190,"num_object_clones":0,"num_object_copies":12760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3190,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3193,"num_write_kb":25532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":953}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":420372672,"num_objects":51320,"num_object_clones":0,"num_object_copies":205280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35631,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51320,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":51347,"num_write_kb":410658,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":592,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":693972992,"data_stored":693972992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":51374,"ondisk_log_size":51374,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084308,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":375908,"kb_used_data":203988,"kb_used_omap":4,"kb_used_meta":171899,"kb_avail":93995932,"statfs":{"total":96636764160,"available":96251834368,"internally_reserved":0,"allocated":208883712,"data_stored":208700830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":176025299},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,8,6,4,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986185,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":147636,"kb_used_data":61796,"kb_used_omap":4,"kb_used_meta":85819,"kb_avail":94224204,"statfs":{"total":96636764160,"available":96485584896,"internally_reserved":0,"allocated":63279104,"data_stored":63096222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":87879378},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,6],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949672991,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":380908,"kb_used_data":205340,"kb_used_omap":1,"kb_used_meta":175550,"kb_avail":93990932,"statfs":{"total":96636764160,"available":96246714368,"internally_reserved":0,"allocated":210268160,"data_stored":210085278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":179763653},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,1,0,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182422,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":381476,"kb_used_data":205460,"kb_used_omap":4,"kb_used_meta":175995,"kb_avail":93990364,"statfs":{"total":96636764160,"available":96246132736,"internally_reserved":0,"allocated":210391040,"data_stored":210208158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":180219597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,3,0,5,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":210751488,"data_stored":210751488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63238144,"data_stored":63238144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":210706432,"data_stored":210706432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":209276928,"data_stored":209276928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:35:39.938 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:35:40.507 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 78 20 53432 53412 5.34854 2.73438 0.0536371 0.0291992 2024-09-05T16:35:40.535 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:35:40.536 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:35:40.561 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":95,"stamp":"2024-09-05T16:35:38.809194+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":420372672,"num_objects":51320,"num_object_clones":0,"num_object_copies":205280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35631,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51320,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":51347,"num_write_kb":410658,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":592,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":51374,"ondisk_log_size":51374,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1285928,"kb_used_data":676584,"kb_used_omap":14,"kb_used_meta":609265,"kb_avail":376201432,"statfs":{"total":386547056640,"available":385230266368,"internally_reserved":0,"allocated":692822016,"data_stored":692090488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":623887927},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,4,18,13,9,11,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":32071680,"num_objects":3915,"num_object_clones":0,"num_object_copies":15660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-141,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3915,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3915,"num_write_kb":31320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":141,"num_bytes_recovered":1155072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004476"},"pg_stats":[{"pgid":"1.f","version":"33'3180","reported_seq":3221,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:34.212269+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:35:34.212269+0000","last_peered":"2024-09-05T16:35:34.212269+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:35:34.212269+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3180,"log_dups_size":0,"ondisk_log_size":3180,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26050560,"num_objects":3180,"num_object_clones":0,"num_object_copies":12720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3180,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3180,"num_write_kb":25440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":918}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'3186","reported_seq":3249,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:35.111834+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:35:35.111834+0000","last_peered":"2024-09-05T16:35:35.111834+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:35:35.111834+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3186,"log_dups_size":0,"ondisk_log_size":3186,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26064032,"num_objects":3182,"num_object_clones":0,"num_object_copies":12728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3182,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3184,"num_write_kb":25468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":915}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'3217","reported_seq":3291,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:35.178263+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:35:35.178263+0000","last_peered":"2024-09-05T16:35:35.178263+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:35:35.178263+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3217,"log_dups_size":0,"ondisk_log_size":3217,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26327632,"num_objects":3214,"num_object_clones":0,"num_object_copies":12856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3214,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3216,"num_write_kb":25718,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":964}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'3220","reported_seq":4414,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:35.253583+0000","last_change":"2024-09-05T16:34:56.800371+0000","last_active":"2024-09-05T16:35:35.253583+0000","last_peered":"2024-09-05T16:35:35.253583+0000","last_clean":"2024-09-05T16:33:53.148717+0000","last_became_active":"2024-09-05T16:34:56.791755+0000","last_became_peered":"2024-09-05T16:34:56.791755+0000","last_unstale":"2024-09-05T16:35:35.253583+0000","last_undegraded":"2024-09-05T16:34:56.597522+0000","last_fullsized":"2024-09-05T16:34:56.568150+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3220,"log_dups_size":0,"ondisk_log_size":3220,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26342560,"num_objects":3216,"num_object_clones":0,"num_object_copies":12864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1723,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3218,"num_write_kb":25733,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":567,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1493}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'3095","reported_seq":3148,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:34.209570+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:35:34.209570+0000","last_peered":"2024-09-05T16:35:34.209570+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:35:34.209570+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3095,"log_dups_size":0,"ondisk_log_size":3095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25326752,"num_objects":3092,"num_object_clones":0,"num_object_copies":12368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3092,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3093,"num_write_kb":24741,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":930}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'3188","reported_seq":3229,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:34.108212+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:35:34.108212+0000","last_peered":"2024-09-05T16:35:34.108212+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:35:34.108212+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3188,"log_dups_size":0,"ondisk_log_size":3188,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26116096,"num_objects":3188,"num_object_clones":0,"num_object_copies":12752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3188,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3188,"num_write_kb":25504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":938}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'3118","reported_seq":3182,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:35.205906+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:35:35.205906+0000","last_peered":"2024-09-05T16:35:35.205906+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:35:35.205906+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3118,"log_dups_size":0,"ondisk_log_size":3118,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25490592,"num_objects":3112,"num_object_clones":0,"num_object_copies":12448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3112,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3116,"num_write_kb":24908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":965}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'3237","reported_seq":3300,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:35.157549+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:35:35.157549+0000","last_peered":"2024-09-05T16:35:35.157549+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:35:35.157549+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3237,"log_dups_size":0,"ondisk_log_size":3237,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26473632,"num_objects":3232,"num_object_clones":0,"num_object_copies":12928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3232,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3235,"num_write_kb":25868,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":993}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'3208","reported_seq":3263,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:34.105076+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:35:34.105076+0000","last_peered":"2024-09-05T16:35:34.105076+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:35:34.105076+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3208,"log_dups_size":0,"ondisk_log_size":3208,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26208576,"num_objects":3200,"num_object_clones":0,"num_object_copies":12800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3200,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3205,"num_write_kb":25617,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":939}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'3267","reported_seq":3343,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:35.108833+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:35:35.108833+0000","last_peered":"2024-09-05T16:35:35.108833+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:35:35.108833+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3267,"log_dups_size":0,"ondisk_log_size":3267,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26743968,"num_objects":3265,"num_object_clones":0,"num_object_copies":13060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3265,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3265,"num_write_kb":26118,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1003}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'3277","reported_seq":3332,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:34.181346+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:35:34.181346+0000","last_peered":"2024-09-05T16:35:34.181346+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:35:34.181346+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3277,"log_dups_size":0,"ondisk_log_size":3277,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26799856,"num_objects":3272,"num_object_clones":0,"num_object_copies":13088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3272,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3274,"num_write_kb":26180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":961}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'3194","reported_seq":3229,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:34.119786+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:35:34.119786+0000","last_peered":"2024-09-05T16:35:34.119786+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:35:34.119786+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3194,"log_dups_size":0,"ondisk_log_size":3194,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26155600,"num_objects":3193,"num_object_clones":0,"num_object_copies":12772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3193,"num_write_kb":25543,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":929}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'3215","reported_seq":3288,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:35.172524+0000","last_change":"2024-09-05T16:34:56.798081+0000","last_active":"2024-09-05T16:35:35.172524+0000","last_peered":"2024-09-05T16:35:35.172524+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:35:35.172524+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3215,"log_dups_size":0,"ondisk_log_size":3215,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26327632,"num_objects":3214,"num_object_clones":0,"num_object_copies":12856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3214,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3214,"num_write_kb":25711,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":920}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'3305","reported_seq":3356,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:34.232066+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:35:34.232066+0000","last_peered":"2024-09-05T16:35:34.232066+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:35:34.232066+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3305,"log_dups_size":0,"ondisk_log_size":3305,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27038880,"num_objects":3301,"num_object_clones":0,"num_object_copies":13204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3301,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3303,"num_write_kb":26420,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":944}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'3272","reported_seq":3349,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:35.056112+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:35:35.056112+0000","last_peered":"2024-09-05T16:35:35.056112+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:35:35.056112+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3272,"log_dups_size":0,"ondisk_log_size":3272,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26776736,"num_objects":3269,"num_object_clones":0,"num_object_copies":13076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3269,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3270,"num_write_kb":26157,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":949}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'3195","reported_seq":3258,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:35.257174+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:35:35.257174+0000","last_peered":"2024-09-05T16:35:35.257174+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:35:35.257174+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3195,"log_dups_size":0,"ondisk_log_size":3195,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26129568,"num_objects":3190,"num_object_clones":0,"num_object_copies":12760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3190,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3193,"num_write_kb":25532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":953}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":420372672,"num_objects":51320,"num_object_clones":0,"num_object_copies":205280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35631,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51320,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":51347,"num_write_kb":410658,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":592,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":693972992,"data_stored":693972992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":51374,"ondisk_log_size":51374,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084308,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":375908,"kb_used_data":203988,"kb_used_omap":4,"kb_used_meta":171899,"kb_avail":93995932,"statfs":{"total":96636764160,"available":96251834368,"internally_reserved":0,"allocated":208883712,"data_stored":208700830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":176025299},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,8,6,4,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986185,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":147636,"kb_used_data":61796,"kb_used_omap":4,"kb_used_meta":85819,"kb_avail":94224204,"statfs":{"total":96636764160,"available":96485584896,"internally_reserved":0,"allocated":63279104,"data_stored":63096222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":87879378},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,6],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949672991,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":380908,"kb_used_data":205340,"kb_used_omap":1,"kb_used_meta":175550,"kb_avail":93990932,"statfs":{"total":96636764160,"available":96246714368,"internally_reserved":0,"allocated":210268160,"data_stored":210085278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":179763653},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,1,0,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182422,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":381476,"kb_used_data":205460,"kb_used_omap":4,"kb_used_meta":175995,"kb_avail":93990364,"statfs":{"total":96636764160,"available":96246132736,"internally_reserved":0,"allocated":210391040,"data_stored":210208158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":180219597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,3,0,5,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":210751488,"data_stored":210751488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63238144,"data_stored":63238144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":210706432,"data_stored":210706432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":209276928,"data_stored":209276928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:35:40.562 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-05T16:35:40.563 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'3180", 'reported_seq': 3221, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:34.212269+0000', 'last_change': '2024-09-05T16:34:56.833497+0000', 'last_active': '2024-09-05T16:35:34.212269+0000', 'last_peered': '2024-09-05T16:35:34.212269+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.828289+0000', 'last_became_peered': '2024-09-05T16:34:56.828289+0000', 'last_unstale': '2024-09-05T16:35:34.212269+0000', 'last_undegraded': '2024-09-05T16:34:56.612160+0000', 'last_fullsized': '2024-09-05T16:34:56.557632+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 3180, 'log_dups_size': 0, 'ondisk_log_size': 3180, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26050560, 'num_objects': 3180, 'num_object_clones': 0, 'num_object_copies': 12720, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2262, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3180, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3180, 'num_write_kb': 25440, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 918}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:35:40.563 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-05T16:35:40.563 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'3186", 'reported_seq': 3249, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:35.111834+0000', 'last_change': '2024-09-05T16:34:56.825031+0000', 'last_active': '2024-09-05T16:35:35.111834+0000', 'last_peered': '2024-09-05T16:35:35.111834+0000', 'last_clean': '2024-09-05T16:33:53.200762+0000', 'last_became_active': '2024-09-05T16:34:56.820292+0000', 'last_became_peered': '2024-09-05T16:34:56.820292+0000', 'last_unstale': '2024-09-05T16:35:35.111834+0000', 'last_undegraded': '2024-09-05T16:34:56.605776+0000', 'last_fullsized': '2024-09-05T16:34:56.559383+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 3186, 'log_dups_size': 0, 'ondisk_log_size': 3186, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26064032, 'num_objects': 3182, 'num_object_clones': 0, 'num_object_copies': 12728, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2269, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3182, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3184, 'num_write_kb': 25468, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 915}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:35:40.563 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:35:40.563 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'3217", 'reported_seq': 3291, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:35.178263+0000', 'last_change': '2024-09-05T16:34:56.867900+0000', 'last_active': '2024-09-05T16:35:35.178263+0000', 'last_peered': '2024-09-05T16:35:35.178263+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:35:35.178263+0000', 'last_undegraded': '2024-09-05T16:34:56.590972+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 3217, 'log_dups_size': 0, 'ondisk_log_size': 3217, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26327632, 'num_objects': 3214, 'num_object_clones': 0, 'num_object_copies': 12856, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2251, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3214, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3216, 'num_write_kb': 25718, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 964}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:35:40.563 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-05T16:35:40.563 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'3220", 'reported_seq': 4414, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:35.253583+0000', 'last_change': '2024-09-05T16:34:56.800371+0000', 'last_active': '2024-09-05T16:35:35.253583+0000', 'last_peered': '2024-09-05T16:35:35.253583+0000', 'last_clean': '2024-09-05T16:33:53.148717+0000', 'last_became_active': '2024-09-05T16:34:56.791755+0000', 'last_became_peered': '2024-09-05T16:34:56.791755+0000', 'last_unstale': '2024-09-05T16:35:35.253583+0000', 'last_undegraded': '2024-09-05T16:34:56.597522+0000', 'last_fullsized': '2024-09-05T16:34:56.568150+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 3220, 'log_dups_size': 0, 'ondisk_log_size': 3220, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26342560, 'num_objects': 3216, 'num_object_clones': 0, 'num_object_copies': 12864, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1723, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3216, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3218, 'num_write_kb': 25733, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 567, 'num_bytes_recovered': 4628480, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1493}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:35:40.563 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:35:40.564 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'3095", 'reported_seq': 3148, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:34.209570+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:35:34.209570+0000', 'last_peered': '2024-09-05T16:35:34.209570+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:35:34.209570+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 3095, 'log_dups_size': 0, 'ondisk_log_size': 3095, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25326752, 'num_objects': 3092, 'num_object_clones': 0, 'num_object_copies': 12368, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2164, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3092, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3093, 'num_write_kb': 24741, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 930}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:35:40.564 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:35:40.564 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'3188", 'reported_seq': 3229, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:34.108212+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:35:34.108212+0000', 'last_peered': '2024-09-05T16:35:34.108212+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:35:34.108212+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 3188, 'log_dups_size': 0, 'ondisk_log_size': 3188, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26116096, 'num_objects': 3188, 'num_object_clones': 0, 'num_object_copies': 12752, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3188, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3188, 'num_write_kb': 25504, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 938}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:35:40.564 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:35:40.564 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'3118", 'reported_seq': 3182, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:35.205906+0000', 'last_change': '2024-09-05T16:34:56.903436+0000', 'last_active': '2024-09-05T16:35:35.205906+0000', 'last_peered': '2024-09-05T16:35:35.205906+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:35:35.205906+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 3118, 'log_dups_size': 0, 'ondisk_log_size': 3118, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25490592, 'num_objects': 3112, 'num_object_clones': 0, 'num_object_copies': 12448, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2149, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3112, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3116, 'num_write_kb': 24908, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 965}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:35:40.564 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-05T16:35:40.564 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'3237", 'reported_seq': 3300, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:35.157549+0000', 'last_change': '2024-09-05T16:34:56.883319+0000', 'last_active': '2024-09-05T16:35:35.157549+0000', 'last_peered': '2024-09-05T16:35:35.157549+0000', 'last_clean': '2024-09-05T16:33:53.505305+0000', 'last_became_active': '2024-09-05T16:34:56.873389+0000', 'last_became_peered': '2024-09-05T16:34:56.873389+0000', 'last_unstale': '2024-09-05T16:35:35.157549+0000', 'last_undegraded': '2024-09-05T16:34:56.602567+0000', 'last_fullsized': '2024-09-05T16:34:56.559958+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 3237, 'log_dups_size': 0, 'ondisk_log_size': 3237, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26473632, 'num_objects': 3232, 'num_object_clones': 0, 'num_object_copies': 12928, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2241, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3232, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3235, 'num_write_kb': 25868, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 993}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:35:40.564 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:35:40.564 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'3208", 'reported_seq': 3263, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:34.105076+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:35:34.105076+0000', 'last_peered': '2024-09-05T16:35:34.105076+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:35:34.105076+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 3208, 'log_dups_size': 0, 'ondisk_log_size': 3208, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26208576, 'num_objects': 3200, 'num_object_clones': 0, 'num_object_copies': 12800, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2264, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3200, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3205, 'num_write_kb': 25617, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 939}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:35:40.564 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:35:40.564 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'3267", 'reported_seq': 3343, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:35.108833+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:35:35.108833+0000', 'last_peered': '2024-09-05T16:35:35.108833+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:35:35.108833+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 3267, 'log_dups_size': 0, 'ondisk_log_size': 3267, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26743968, 'num_objects': 3265, 'num_object_clones': 0, 'num_object_copies': 13060, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2264, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3265, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3265, 'num_write_kb': 26118, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1003}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:35:40.565 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:35:40.565 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'3277", 'reported_seq': 3332, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:34.181346+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:35:34.181346+0000', 'last_peered': '2024-09-05T16:35:34.181346+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:35:34.181346+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 3277, 'log_dups_size': 0, 'ondisk_log_size': 3277, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26799856, 'num_objects': 3272, 'num_object_clones': 0, 'num_object_copies': 13088, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2314, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3272, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3274, 'num_write_kb': 26180, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 961}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:35:40.565 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:35:40.565 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'3194", 'reported_seq': 3229, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:34.119786+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:35:34.119786+0000', 'last_peered': '2024-09-05T16:35:34.119786+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:35:34.119786+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 3194, 'log_dups_size': 0, 'ondisk_log_size': 3194, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26155600, 'num_objects': 3193, 'num_object_clones': 0, 'num_object_copies': 12772, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2265, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3193, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3193, 'num_write_kb': 25543, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 929}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:35:40.565 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-05T16:35:40.565 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'3215", 'reported_seq': 3288, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:35.172524+0000', 'last_change': '2024-09-05T16:34:56.798081+0000', 'last_active': '2024-09-05T16:35:35.172524+0000', 'last_peered': '2024-09-05T16:35:35.172524+0000', 'last_clean': '2024-09-05T16:33:45.082039+0000', 'last_became_active': '2024-09-05T16:34:56.792386+0000', 'last_became_peered': '2024-09-05T16:34:56.792386+0000', 'last_unstale': '2024-09-05T16:35:35.172524+0000', 'last_undegraded': '2024-09-05T16:34:56.600595+0000', 'last_fullsized': '2024-09-05T16:34:56.558203+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 3215, 'log_dups_size': 0, 'ondisk_log_size': 3215, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26327632, 'num_objects': 3214, 'num_object_clones': 0, 'num_object_copies': 12856, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2295, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3214, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3214, 'num_write_kb': 25711, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 920}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:35:40.566 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:35:40.566 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'3305", 'reported_seq': 3356, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:34.232066+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:35:34.232066+0000', 'last_peered': '2024-09-05T16:35:34.232066+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:35:34.232066+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 3305, 'log_dups_size': 0, 'ondisk_log_size': 3305, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27038880, 'num_objects': 3301, 'num_object_clones': 0, 'num_object_copies': 13204, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2359, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3301, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3303, 'num_write_kb': 26420, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 944}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:35:40.566 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:35:40.566 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'3272", 'reported_seq': 3349, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:35.056112+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:35:35.056112+0000', 'last_peered': '2024-09-05T16:35:35.056112+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:35:35.056112+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 3272, 'log_dups_size': 0, 'ondisk_log_size': 3272, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26776736, 'num_objects': 3269, 'num_object_clones': 0, 'num_object_copies': 13076, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2322, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3269, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3270, 'num_write_kb': 26157, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 949}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:35:40.566 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:35:40.567 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'3195", 'reported_seq': 3258, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:35.257174+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:35:35.257174+0000', 'last_peered': '2024-09-05T16:35:35.257174+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:35:35.257174+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 3195, 'log_dups_size': 0, 'ondisk_log_size': 3195, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26129568, 'num_objects': 3190, 'num_object_clones': 0, 'num_object_copies': 12760, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3190, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3193, 'num_write_kb': 25532, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 953}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:35:40.567 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:35:41.360 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:35:41.360 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 35566/212888 objects degraded (16.706%), 16 pgs degraded","count":16},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":144,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":53222,"data_bytes":435953856,"bytes_used":1373351936,"bytes_avail":385173704704,"bytes_total":386547056640,"degraded_objects":35566,"degraded_total":212888,"degraded_ratio":0.16706437187629175,"recovering_objects_per_sec":17,"recovering_bytes_per_sec":140577,"recovering_keys_per_sec":0,"num_objects_recovered":206,"num_bytes_recovered":1687552,"num_keys_recovered":0,"write_bytes_sec":3969605,"read_op_per_sec":0,"write_op_per_sec":484},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-05T16:35:14.802011+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-05T16:35:41.400 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:35:41.401 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:35:41.507 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 79 20 53777 53757 5.31495 2.69531 0.0684324 0.0293761 2024-09-05T16:35:41.976 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:35:41.976 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:35:42.004 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":96,"stamp":"2024-09-05T16:35:40.809778+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":435953856,"num_objects":53222,"num_object_clones":0,"num_object_copies":212888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35566,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53222,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53249,"num_write_kb":425874,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":657,"num_bytes_recovered":5160960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":53276,"ondisk_log_size":53276,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1341164,"kb_used_data":706860,"kb_used_omap":14,"kb_used_meta":634225,"kb_avail":376146196,"statfs":{"total":386547056640,"available":385173704704,"internally_reserved":0,"allocated":723824640,"data_stored":723093112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":649446967},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,6,4,2,8,13,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":47652864,"num_objects":5817,"num_object_clones":0,"num_object_copies":23268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-206,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5817,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5817,"num_write_kb":46536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":206,"num_bytes_recovered":1687552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004433"},"pg_stats":[{"pgid":"1.f","version":"33'3302","reported_seq":3343,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:39.230499+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:35:39.230499+0000","last_peered":"2024-09-05T16:35:39.230499+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:35:39.230499+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3302,"log_dups_size":0,"ondisk_log_size":3302,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27049984,"num_objects":3302,"num_object_clones":0,"num_object_copies":13208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3302,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3302,"num_write_kb":26416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1040}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'3297","reported_seq":3359,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:40.232574+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:35:40.232574+0000","last_peered":"2024-09-05T16:35:40.232574+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:35:40.232574+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3297,"log_dups_size":0,"ondisk_log_size":3297,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26973344,"num_objects":3293,"num_object_clones":0,"num_object_copies":13172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3293,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3295,"num_write_kb":26356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1026}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'3346","reported_seq":3420,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:40.091001+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:35:40.091001+0000","last_peered":"2024-09-05T16:35:40.091001+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:35:40.091001+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3346,"log_dups_size":0,"ondisk_log_size":3346,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27384400,"num_objects":3343,"num_object_clones":0,"num_object_copies":13372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3343,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3345,"num_write_kb":26750,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1093}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'3356","reported_seq":4680,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:40.244287+0000","last_change":"2024-09-05T16:34:56.800371+0000","last_active":"2024-09-05T16:35:40.244287+0000","last_peered":"2024-09-05T16:35:40.244287+0000","last_clean":"2024-09-05T16:33:53.148717+0000","last_became_active":"2024-09-05T16:34:56.791755+0000","last_became_peered":"2024-09-05T16:34:56.791755+0000","last_unstale":"2024-09-05T16:35:40.244287+0000","last_undegraded":"2024-09-05T16:34:56.597522+0000","last_fullsized":"2024-09-05T16:34:56.568150+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3356,"log_dups_size":0,"ondisk_log_size":3356,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27456672,"num_objects":3352,"num_object_clones":0,"num_object_copies":13408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1658,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3352,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3354,"num_write_kb":26821,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":632,"num_bytes_recovered":5160960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1694}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'3214","reported_seq":3267,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:39.245953+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:35:39.245953+0000","last_peered":"2024-09-05T16:35:39.245953+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:35:39.245953+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3214,"log_dups_size":0,"ondisk_log_size":3214,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26301600,"num_objects":3211,"num_object_clones":0,"num_object_copies":12844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3211,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3212,"num_write_kb":25693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1049}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'3308","reported_seq":3349,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:39.193385+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:35:39.193385+0000","last_peered":"2024-09-05T16:35:39.193385+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:35:39.193385+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3308,"log_dups_size":0,"ondisk_log_size":3308,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27099136,"num_objects":3308,"num_object_clones":0,"num_object_copies":13232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3308,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3308,"num_write_kb":26464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1058}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'3249","reported_seq":3312,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:40.273591+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:35:40.273591+0000","last_peered":"2024-09-05T16:35:40.273591+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:35:40.273591+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3249,"log_dups_size":0,"ondisk_log_size":3249,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26563744,"num_objects":3243,"num_object_clones":0,"num_object_copies":12972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3243,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3247,"num_write_kb":25956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1096}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'3350","reported_seq":3410,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:40.266305+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:35:40.266305+0000","last_peered":"2024-09-05T16:35:40.266305+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:35:40.266305+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3350,"log_dups_size":0,"ondisk_log_size":3350,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27399328,"num_objects":3345,"num_object_clones":0,"num_object_copies":13380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3345,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3348,"num_write_kb":26772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1106}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'3320","reported_seq":3374,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:39.185856+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:35:39.185856+0000","last_peered":"2024-09-05T16:35:39.185856+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:35:39.185856+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3320,"log_dups_size":0,"ondisk_log_size":3320,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27126080,"num_objects":3312,"num_object_clones":0,"num_object_copies":13248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3312,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3317,"num_write_kb":26513,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1051}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'3386","reported_seq":3462,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:40.184100+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:35:40.184100+0000","last_peered":"2024-09-05T16:35:40.184100+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:35:40.184100+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3386,"log_dups_size":0,"ondisk_log_size":3386,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27718816,"num_objects":3384,"num_object_clones":0,"num_object_copies":13536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3384,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3384,"num_write_kb":27070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1122}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'3382","reported_seq":3437,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:39.147393+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:35:39.147393+0000","last_peered":"2024-09-05T16:35:39.147393+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:35:39.147393+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3382,"log_dups_size":0,"ondisk_log_size":3382,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27660016,"num_objects":3377,"num_object_clones":0,"num_object_copies":13508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3377,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3379,"num_write_kb":27020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1066}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'3310","reported_seq":3345,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:39.196800+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:35:39.196800+0000","last_peered":"2024-09-05T16:35:39.196800+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:35:39.196800+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3310,"log_dups_size":0,"ondisk_log_size":3310,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27105872,"num_objects":3309,"num_object_clones":0,"num_object_copies":13236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3309,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3309,"num_write_kb":26471,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1045}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'3340","reported_seq":3413,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:40.233827+0000","last_change":"2024-09-05T16:34:56.798081+0000","last_active":"2024-09-05T16:35:40.233827+0000","last_peered":"2024-09-05T16:35:40.233827+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:35:40.233827+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3340,"log_dups_size":0,"ondisk_log_size":3340,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27351632,"num_objects":3339,"num_object_clones":0,"num_object_copies":13356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3339,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3339,"num_write_kb":26711,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1045}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'3419","reported_seq":3470,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:39.251935+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:35:39.251935+0000","last_peered":"2024-09-05T16:35:39.251935+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:35:39.251935+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3419,"log_dups_size":0,"ondisk_log_size":3419,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27972768,"num_objects":3415,"num_object_clones":0,"num_object_copies":13660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3415,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3417,"num_write_kb":27332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1058}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'3384","reported_seq":3461,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:40.110991+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:35:40.110991+0000","last_peered":"2024-09-05T16:35:40.110991+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:35:40.110991+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3384,"log_dups_size":0,"ondisk_log_size":3384,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27694240,"num_objects":3381,"num_object_clones":0,"num_object_copies":13524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3381,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3382,"num_write_kb":27053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1061}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'3313","reported_seq":3376,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:40.287461+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:35:40.287461+0000","last_peered":"2024-09-05T16:35:40.287461+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:35:40.287461+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3313,"log_dups_size":0,"ondisk_log_size":3313,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27096224,"num_objects":3308,"num_object_clones":0,"num_object_copies":13232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3308,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3311,"num_write_kb":26476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1071}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":435953856,"num_objects":53222,"num_object_clones":0,"num_object_copies":212888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35566,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53222,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53249,"num_write_kb":425874,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":657,"num_bytes_recovered":5160960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":725360640,"data_stored":725360640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":53276,"ondisk_log_size":53276,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084309,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":391664,"kb_used_data":211424,"kb_used_omap":4,"kb_used_meta":180219,"kb_avail":93980176,"statfs":{"total":96636764160,"available":96235700224,"internally_reserved":0,"allocated":216498176,"data_stored":216315294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":184544979},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1,0,2,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986186,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":163680,"kb_used_data":69520,"kb_used_omap":4,"kb_used_meta":94139,"kb_avail":94208160,"statfs":{"total":96636764160,"available":96469155840,"internally_reserved":0,"allocated":71188480,"data_stored":71005598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":96399058},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,0,3,6],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949672992,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":392576,"kb_used_data":212848,"kb_used_omap":1,"kb_used_meta":179710,"kb_avail":93979264,"statfs":{"total":96636764160,"available":96234766336,"internally_reserved":0,"allocated":217956352,"data_stored":217773470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":184023493},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,1,5,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182423,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":393244,"kb_used_data":213068,"kb_used_omap":4,"kb_used_meta":180155,"kb_avail":93978596,"statfs":{"total":96636764160,"available":96234082304,"internally_reserved":0,"allocated":218181632,"data_stored":217998750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":184479437},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,0,0,0,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":218558464,"data_stored":218558464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":71299072,"data_stored":71299072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":218447872,"data_stored":218447872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":217055232,"data_stored":217055232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:35:42.507 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout:2024-09-05T16:35:42.515322+0000 min lat: 0.00357551 max lat: 2.29705 avg lat: 0.0295587 2024-09-05T16:35:42.507 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-05T16:35:42.508 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 80 20 54139 54119 5.28386 2.82812 0.0770538 0.0295587 2024-09-05T16:35:43.507 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 81 20 54520 54500 5.25537 2.97656 0.0339064 0.0297171 2024-09-05T16:35:44.508 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 82 19 54921 54902 5.22959 3.14062 0.038671 0.0298687 2024-09-05T16:35:45.006 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:35:45.508 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 83 20 55285 55265 5.20075 2.83594 0.0904812 0.0300299 2024-09-05T16:35:45.598 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:35:45.598 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:35:45.631 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":98,"stamp":"2024-09-05T16:35:44.811206+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":442613952,"num_objects":54035,"num_object_clones":0,"num_object_copies":216140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35566,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54035,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54062,"num_write_kb":432378,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":657,"num_bytes_recovered":5160960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54089,"ondisk_log_size":54089,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1370220,"kb_used_data":722028,"kb_used_omap":14,"kb_used_meta":648113,"kb_avail":376117140,"statfs":{"total":386547056640,"available":385143951360,"internally_reserved":0,"allocated":739356672,"data_stored":738625144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":663668279},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[7,7,0,7,8,10,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":38084608,"num_objects":4649,"num_object_clones":0,"num_object_copies":18596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-139,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4649,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4649,"num_write_kb":37192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":139,"num_bytes_recovered":1138688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004307"},"pg_stats":[{"pgid":"1.f","version":"33'3431","reported_seq":3472,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:44.176268+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:35:44.176268+0000","last_peered":"2024-09-05T16:35:44.176268+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:35:44.176268+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3431,"log_dups_size":0,"ondisk_log_size":3431,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28106752,"num_objects":3431,"num_object_clones":0,"num_object_copies":13724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3431,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3431,"num_write_kb":27448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1169}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'3297","reported_seq":3359,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:40.232574+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:35:40.232574+0000","last_peered":"2024-09-05T16:35:40.232574+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:35:40.232574+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3297,"log_dups_size":0,"ondisk_log_size":3297,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26973344,"num_objects":3293,"num_object_clones":0,"num_object_copies":13172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3293,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3295,"num_write_kb":26356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1026}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'3346","reported_seq":3420,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:40.091001+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:35:40.091001+0000","last_peered":"2024-09-05T16:35:40.091001+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:35:40.091001+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3346,"log_dups_size":0,"ondisk_log_size":3346,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27384400,"num_objects":3343,"num_object_clones":0,"num_object_copies":13372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3343,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3345,"num_write_kb":26750,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1093}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'3356","reported_seq":4680,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:40.244287+0000","last_change":"2024-09-05T16:34:56.800371+0000","last_active":"2024-09-05T16:35:40.244287+0000","last_peered":"2024-09-05T16:35:40.244287+0000","last_clean":"2024-09-05T16:33:53.148717+0000","last_became_active":"2024-09-05T16:34:56.791755+0000","last_became_peered":"2024-09-05T16:34:56.791755+0000","last_unstale":"2024-09-05T16:35:40.244287+0000","last_undegraded":"2024-09-05T16:34:56.597522+0000","last_fullsized":"2024-09-05T16:34:56.568150+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3356,"log_dups_size":0,"ondisk_log_size":3356,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27456672,"num_objects":3352,"num_object_clones":0,"num_object_copies":13408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1658,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3352,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3354,"num_write_kb":26821,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":632,"num_bytes_recovered":5160960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1694}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'3323","reported_seq":3376,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:44.148364+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:35:44.148364+0000","last_peered":"2024-09-05T16:35:44.148364+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:35:44.148364+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3323,"log_dups_size":0,"ondisk_log_size":3323,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27194528,"num_objects":3320,"num_object_clones":0,"num_object_copies":13280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3320,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3321,"num_write_kb":26565,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1158}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'3429","reported_seq":3470,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:44.205571+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:35:44.205571+0000","last_peered":"2024-09-05T16:35:44.205571+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:35:44.205571+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3429,"log_dups_size":0,"ondisk_log_size":3429,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28090368,"num_objects":3429,"num_object_clones":0,"num_object_copies":13716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3429,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3429,"num_write_kb":27432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1179}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'3249","reported_seq":3312,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:40.273591+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:35:40.273591+0000","last_peered":"2024-09-05T16:35:40.273591+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:35:40.273591+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3249,"log_dups_size":0,"ondisk_log_size":3249,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26563744,"num_objects":3243,"num_object_clones":0,"num_object_copies":12972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3243,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3247,"num_write_kb":25956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1096}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'3350","reported_seq":3410,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:40.266305+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:35:40.266305+0000","last_peered":"2024-09-05T16:35:40.266305+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:35:40.266305+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3350,"log_dups_size":0,"ondisk_log_size":3350,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27399328,"num_objects":3345,"num_object_clones":0,"num_object_copies":13380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3345,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3348,"num_write_kb":26772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1106}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'3438","reported_seq":3492,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:44.232187+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:35:44.232187+0000","last_peered":"2024-09-05T16:35:44.232187+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:35:44.232187+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3438,"log_dups_size":0,"ondisk_log_size":3438,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28092736,"num_objects":3430,"num_object_clones":0,"num_object_copies":13720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3430,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3435,"num_write_kb":27457,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1169}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'3386","reported_seq":3462,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:40.184100+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:35:40.184100+0000","last_peered":"2024-09-05T16:35:40.184100+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:35:40.184100+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3386,"log_dups_size":0,"ondisk_log_size":3386,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27718816,"num_objects":3384,"num_object_clones":0,"num_object_copies":13536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3384,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3384,"num_write_kb":27070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1122}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'3499","reported_seq":3552,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:44.257621+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:35:44.257621+0000","last_peered":"2024-09-05T16:35:44.257621+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:35:44.257621+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3499,"log_dups_size":0,"ondisk_log_size":3499,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28618480,"num_objects":3494,"num_object_clones":0,"num_object_copies":13976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3494,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3496,"num_write_kb":27956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1183}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'3412","reported_seq":3446,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:44.252530+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:35:44.252530+0000","last_peered":"2024-09-05T16:35:44.252530+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:35:44.252530+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3412,"log_dups_size":0,"ondisk_log_size":3412,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27941456,"num_objects":3411,"num_object_clones":0,"num_object_copies":13644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3411,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3411,"num_write_kb":27287,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1147}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'3340","reported_seq":3413,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:40.233827+0000","last_change":"2024-09-05T16:34:56.798081+0000","last_active":"2024-09-05T16:35:40.233827+0000","last_peered":"2024-09-05T16:35:40.233827+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:35:40.233827+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3340,"log_dups_size":0,"ondisk_log_size":3340,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27351632,"num_objects":3339,"num_object_clones":0,"num_object_copies":13356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3339,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3339,"num_write_kb":26711,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1045}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'3536","reported_seq":3587,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:44.289355+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:35:44.289355+0000","last_peered":"2024-09-05T16:35:44.289355+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:35:44.289355+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3536,"log_dups_size":0,"ondisk_log_size":3536,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28931232,"num_objects":3532,"num_object_clones":0,"num_object_copies":14128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3532,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3534,"num_write_kb":28268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1175}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'3384","reported_seq":3461,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:40.110991+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:35:40.110991+0000","last_peered":"2024-09-05T16:35:40.110991+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:35:40.110991+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3384,"log_dups_size":0,"ondisk_log_size":3384,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27694240,"num_objects":3381,"num_object_clones":0,"num_object_copies":13524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3381,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3382,"num_write_kb":27053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1061}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'3313","reported_seq":3376,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:40.287461+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:35:40.287461+0000","last_peered":"2024-09-05T16:35:40.287461+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:35:40.287461+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3313,"log_dups_size":0,"ondisk_log_size":3313,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27096224,"num_objects":3308,"num_object_clones":0,"num_object_copies":13232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3308,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3311,"num_write_kb":26476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1071}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":442613952,"num_objects":54035,"num_object_clones":0,"num_object_copies":216140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35566,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54035,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54062,"num_write_kb":432378,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":657,"num_bytes_recovered":5160960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":740728832,"data_stored":740728832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54089,"ondisk_log_size":54089,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084310,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":403252,"kb_used_data":218852,"kb_used_omap":4,"kb_used_meta":184379,"kb_avail":93968588,"statfs":{"total":96636764160,"available":96223834112,"internally_reserved":0,"allocated":224104448,"data_stored":223921566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":188804819},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,3,3,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986187,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":181148,"kb_used_data":77260,"kb_used_omap":4,"kb_used_meta":103867,"kb_avail":94190692,"statfs":{"total":96636764160,"available":96451268608,"internally_reserved":0,"allocated":79114240,"data_stored":78931358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":106360530},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,0,3,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949672992,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":392576,"kb_used_data":212848,"kb_used_omap":1,"kb_used_meta":179710,"kb_avail":93979264,"statfs":{"total":96636764160,"available":96234766336,"internally_reserved":0,"allocated":217956352,"data_stored":217773470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":184023493},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,1,5,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182423,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":393244,"kb_used_data":213068,"kb_used_omap":4,"kb_used_meta":180155,"kb_avail":93978596,"statfs":{"total":96636764160,"available":96234082304,"internally_reserved":0,"allocated":218181632,"data_stored":217998750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":184479437},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,0,0,0,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":218558464,"data_stored":218558464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":79138816,"data_stored":79138816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":218447872,"data_stored":218447872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":224583680,"data_stored":224583680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:35:45.632 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:35:46.253 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:35:46.255 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:35:46.292 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":98,"stamp":"2024-09-05T16:35:44.811206+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":442613952,"num_objects":54035,"num_object_clones":0,"num_object_copies":216140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35566,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54035,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54062,"num_write_kb":432378,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":657,"num_bytes_recovered":5160960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54089,"ondisk_log_size":54089,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1370220,"kb_used_data":722028,"kb_used_omap":14,"kb_used_meta":648113,"kb_avail":376117140,"statfs":{"total":386547056640,"available":385143951360,"internally_reserved":0,"allocated":739356672,"data_stored":738625144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":663668279},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[7,7,0,7,8,10,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":38084608,"num_objects":4649,"num_object_clones":0,"num_object_copies":18596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-139,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4649,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4649,"num_write_kb":37192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":139,"num_bytes_recovered":1138688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004307"},"pg_stats":[{"pgid":"1.f","version":"33'3431","reported_seq":3472,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:44.176268+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:35:44.176268+0000","last_peered":"2024-09-05T16:35:44.176268+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:35:44.176268+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3431,"log_dups_size":0,"ondisk_log_size":3431,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28106752,"num_objects":3431,"num_object_clones":0,"num_object_copies":13724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3431,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3431,"num_write_kb":27448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1169}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'3297","reported_seq":3359,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:40.232574+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:35:40.232574+0000","last_peered":"2024-09-05T16:35:40.232574+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:35:40.232574+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3297,"log_dups_size":0,"ondisk_log_size":3297,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26973344,"num_objects":3293,"num_object_clones":0,"num_object_copies":13172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3293,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3295,"num_write_kb":26356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1026}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'3346","reported_seq":3420,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:40.091001+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:35:40.091001+0000","last_peered":"2024-09-05T16:35:40.091001+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:35:40.091001+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3346,"log_dups_size":0,"ondisk_log_size":3346,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27384400,"num_objects":3343,"num_object_clones":0,"num_object_copies":13372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3343,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3345,"num_write_kb":26750,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1093}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'3356","reported_seq":4680,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:40.244287+0000","last_change":"2024-09-05T16:34:56.800371+0000","last_active":"2024-09-05T16:35:40.244287+0000","last_peered":"2024-09-05T16:35:40.244287+0000","last_clean":"2024-09-05T16:33:53.148717+0000","last_became_active":"2024-09-05T16:34:56.791755+0000","last_became_peered":"2024-09-05T16:34:56.791755+0000","last_unstale":"2024-09-05T16:35:40.244287+0000","last_undegraded":"2024-09-05T16:34:56.597522+0000","last_fullsized":"2024-09-05T16:34:56.568150+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3356,"log_dups_size":0,"ondisk_log_size":3356,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27456672,"num_objects":3352,"num_object_clones":0,"num_object_copies":13408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1658,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3352,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3354,"num_write_kb":26821,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":632,"num_bytes_recovered":5160960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1694}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'3323","reported_seq":3376,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:44.148364+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:35:44.148364+0000","last_peered":"2024-09-05T16:35:44.148364+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:35:44.148364+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3323,"log_dups_size":0,"ondisk_log_size":3323,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27194528,"num_objects":3320,"num_object_clones":0,"num_object_copies":13280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3320,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3321,"num_write_kb":26565,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1158}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'3429","reported_seq":3470,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:44.205571+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:35:44.205571+0000","last_peered":"2024-09-05T16:35:44.205571+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:35:44.205571+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3429,"log_dups_size":0,"ondisk_log_size":3429,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28090368,"num_objects":3429,"num_object_clones":0,"num_object_copies":13716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3429,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3429,"num_write_kb":27432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1179}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'3249","reported_seq":3312,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:40.273591+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:35:40.273591+0000","last_peered":"2024-09-05T16:35:40.273591+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:35:40.273591+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3249,"log_dups_size":0,"ondisk_log_size":3249,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26563744,"num_objects":3243,"num_object_clones":0,"num_object_copies":12972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3243,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3247,"num_write_kb":25956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1096}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'3350","reported_seq":3410,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:40.266305+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:35:40.266305+0000","last_peered":"2024-09-05T16:35:40.266305+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:35:40.266305+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3350,"log_dups_size":0,"ondisk_log_size":3350,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27399328,"num_objects":3345,"num_object_clones":0,"num_object_copies":13380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3345,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3348,"num_write_kb":26772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1106}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'3438","reported_seq":3492,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:44.232187+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:35:44.232187+0000","last_peered":"2024-09-05T16:35:44.232187+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:35:44.232187+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3438,"log_dups_size":0,"ondisk_log_size":3438,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28092736,"num_objects":3430,"num_object_clones":0,"num_object_copies":13720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3430,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3435,"num_write_kb":27457,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1169}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'3386","reported_seq":3462,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:40.184100+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:35:40.184100+0000","last_peered":"2024-09-05T16:35:40.184100+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:35:40.184100+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3386,"log_dups_size":0,"ondisk_log_size":3386,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27718816,"num_objects":3384,"num_object_clones":0,"num_object_copies":13536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3384,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3384,"num_write_kb":27070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1122}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'3499","reported_seq":3552,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:44.257621+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:35:44.257621+0000","last_peered":"2024-09-05T16:35:44.257621+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:35:44.257621+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3499,"log_dups_size":0,"ondisk_log_size":3499,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28618480,"num_objects":3494,"num_object_clones":0,"num_object_copies":13976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3494,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3496,"num_write_kb":27956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1183}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'3412","reported_seq":3446,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:44.252530+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:35:44.252530+0000","last_peered":"2024-09-05T16:35:44.252530+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:35:44.252530+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3412,"log_dups_size":0,"ondisk_log_size":3412,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27941456,"num_objects":3411,"num_object_clones":0,"num_object_copies":13644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3411,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3411,"num_write_kb":27287,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1147}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'3340","reported_seq":3413,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:40.233827+0000","last_change":"2024-09-05T16:34:56.798081+0000","last_active":"2024-09-05T16:35:40.233827+0000","last_peered":"2024-09-05T16:35:40.233827+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:35:40.233827+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3340,"log_dups_size":0,"ondisk_log_size":3340,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27351632,"num_objects":3339,"num_object_clones":0,"num_object_copies":13356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3339,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3339,"num_write_kb":26711,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1045}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'3536","reported_seq":3587,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:44.289355+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:35:44.289355+0000","last_peered":"2024-09-05T16:35:44.289355+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:35:44.289355+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3536,"log_dups_size":0,"ondisk_log_size":3536,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28931232,"num_objects":3532,"num_object_clones":0,"num_object_copies":14128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3532,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3534,"num_write_kb":28268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1175}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'3384","reported_seq":3461,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:40.110991+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:35:40.110991+0000","last_peered":"2024-09-05T16:35:40.110991+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:35:40.110991+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3384,"log_dups_size":0,"ondisk_log_size":3384,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27694240,"num_objects":3381,"num_object_clones":0,"num_object_copies":13524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3381,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3382,"num_write_kb":27053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1061}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'3313","reported_seq":3376,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:40.287461+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:35:40.287461+0000","last_peered":"2024-09-05T16:35:40.287461+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:35:40.287461+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3313,"log_dups_size":0,"ondisk_log_size":3313,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27096224,"num_objects":3308,"num_object_clones":0,"num_object_copies":13232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3308,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3311,"num_write_kb":26476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1071}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":442613952,"num_objects":54035,"num_object_clones":0,"num_object_copies":216140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35566,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54035,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54062,"num_write_kb":432378,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":657,"num_bytes_recovered":5160960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":740728832,"data_stored":740728832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54089,"ondisk_log_size":54089,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084310,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":403252,"kb_used_data":218852,"kb_used_omap":4,"kb_used_meta":184379,"kb_avail":93968588,"statfs":{"total":96636764160,"available":96223834112,"internally_reserved":0,"allocated":224104448,"data_stored":223921566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":188804819},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,3,3,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986187,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":181148,"kb_used_data":77260,"kb_used_omap":4,"kb_used_meta":103867,"kb_avail":94190692,"statfs":{"total":96636764160,"available":96451268608,"internally_reserved":0,"allocated":79114240,"data_stored":78931358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":106360530},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,0,3,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949672992,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":392576,"kb_used_data":212848,"kb_used_omap":1,"kb_used_meta":179710,"kb_avail":93979264,"statfs":{"total":96636764160,"available":96234766336,"internally_reserved":0,"allocated":217956352,"data_stored":217773470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":184023493},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,1,5,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182423,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":393244,"kb_used_data":213068,"kb_used_omap":4,"kb_used_meta":180155,"kb_avail":93978596,"statfs":{"total":96636764160,"available":96234082304,"internally_reserved":0,"allocated":218181632,"data_stored":217998750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":184479437},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,0,0,0,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":218558464,"data_stored":218558464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":79138816,"data_stored":79138816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":218447872,"data_stored":218447872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":224583680,"data_stored":224583680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:35:46.293 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-05T16:35:46.293 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'3431", 'reported_seq': 3472, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:44.176268+0000', 'last_change': '2024-09-05T16:34:56.833497+0000', 'last_active': '2024-09-05T16:35:44.176268+0000', 'last_peered': '2024-09-05T16:35:44.176268+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.828289+0000', 'last_became_peered': '2024-09-05T16:34:56.828289+0000', 'last_unstale': '2024-09-05T16:35:44.176268+0000', 'last_undegraded': '2024-09-05T16:34:56.612160+0000', 'last_fullsized': '2024-09-05T16:34:56.557632+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 3431, 'log_dups_size': 0, 'ondisk_log_size': 3431, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28106752, 'num_objects': 3431, 'num_object_clones': 0, 'num_object_copies': 13724, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2262, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3431, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3431, 'num_write_kb': 27448, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1169}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:35:46.294 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-05T16:35:46.294 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'3297", 'reported_seq': 3359, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:40.232574+0000', 'last_change': '2024-09-05T16:34:56.825031+0000', 'last_active': '2024-09-05T16:35:40.232574+0000', 'last_peered': '2024-09-05T16:35:40.232574+0000', 'last_clean': '2024-09-05T16:33:53.200762+0000', 'last_became_active': '2024-09-05T16:34:56.820292+0000', 'last_became_peered': '2024-09-05T16:34:56.820292+0000', 'last_unstale': '2024-09-05T16:35:40.232574+0000', 'last_undegraded': '2024-09-05T16:34:56.605776+0000', 'last_fullsized': '2024-09-05T16:34:56.559383+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 3297, 'log_dups_size': 0, 'ondisk_log_size': 3297, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26973344, 'num_objects': 3293, 'num_object_clones': 0, 'num_object_copies': 13172, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2269, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3293, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3295, 'num_write_kb': 26356, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 1026}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:35:46.294 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:35:46.294 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'3346", 'reported_seq': 3420, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:40.091001+0000', 'last_change': '2024-09-05T16:34:56.867900+0000', 'last_active': '2024-09-05T16:35:40.091001+0000', 'last_peered': '2024-09-05T16:35:40.091001+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:35:40.091001+0000', 'last_undegraded': '2024-09-05T16:34:56.590972+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 3346, 'log_dups_size': 0, 'ondisk_log_size': 3346, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27384400, 'num_objects': 3343, 'num_object_clones': 0, 'num_object_copies': 13372, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2251, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3343, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3345, 'num_write_kb': 26750, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1093}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:35:46.294 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-05T16:35:46.294 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'3356", 'reported_seq': 4680, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:40.244287+0000', 'last_change': '2024-09-05T16:34:56.800371+0000', 'last_active': '2024-09-05T16:35:40.244287+0000', 'last_peered': '2024-09-05T16:35:40.244287+0000', 'last_clean': '2024-09-05T16:33:53.148717+0000', 'last_became_active': '2024-09-05T16:34:56.791755+0000', 'last_became_peered': '2024-09-05T16:34:56.791755+0000', 'last_unstale': '2024-09-05T16:35:40.244287+0000', 'last_undegraded': '2024-09-05T16:34:56.597522+0000', 'last_fullsized': '2024-09-05T16:34:56.568150+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 3356, 'log_dups_size': 0, 'ondisk_log_size': 3356, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27456672, 'num_objects': 3352, 'num_object_clones': 0, 'num_object_copies': 13408, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1658, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3352, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3354, 'num_write_kb': 26821, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 632, 'num_bytes_recovered': 5160960, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1694}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:35:46.294 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:35:46.294 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'3323", 'reported_seq': 3376, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:44.148364+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:35:44.148364+0000', 'last_peered': '2024-09-05T16:35:44.148364+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:35:44.148364+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 3323, 'log_dups_size': 0, 'ondisk_log_size': 3323, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27194528, 'num_objects': 3320, 'num_object_clones': 0, 'num_object_copies': 13280, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2164, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3320, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3321, 'num_write_kb': 26565, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1158}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:35:46.294 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:35:46.294 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'3429", 'reported_seq': 3470, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:44.205571+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:35:44.205571+0000', 'last_peered': '2024-09-05T16:35:44.205571+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:35:44.205571+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 3429, 'log_dups_size': 0, 'ondisk_log_size': 3429, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28090368, 'num_objects': 3429, 'num_object_clones': 0, 'num_object_copies': 13716, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3429, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3429, 'num_write_kb': 27432, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1179}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:35:46.295 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:35:46.295 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'3249", 'reported_seq': 3312, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:40.273591+0000', 'last_change': '2024-09-05T16:34:56.903436+0000', 'last_active': '2024-09-05T16:35:40.273591+0000', 'last_peered': '2024-09-05T16:35:40.273591+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:35:40.273591+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 3249, 'log_dups_size': 0, 'ondisk_log_size': 3249, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26563744, 'num_objects': 3243, 'num_object_clones': 0, 'num_object_copies': 12972, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2149, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3243, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3247, 'num_write_kb': 25956, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1096}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:35:46.295 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-05T16:35:46.295 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'3350", 'reported_seq': 3410, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:40.266305+0000', 'last_change': '2024-09-05T16:34:56.883319+0000', 'last_active': '2024-09-05T16:35:40.266305+0000', 'last_peered': '2024-09-05T16:35:40.266305+0000', 'last_clean': '2024-09-05T16:33:53.505305+0000', 'last_became_active': '2024-09-05T16:34:56.873389+0000', 'last_became_peered': '2024-09-05T16:34:56.873389+0000', 'last_unstale': '2024-09-05T16:35:40.266305+0000', 'last_undegraded': '2024-09-05T16:34:56.602567+0000', 'last_fullsized': '2024-09-05T16:34:56.559958+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 3350, 'log_dups_size': 0, 'ondisk_log_size': 3350, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27399328, 'num_objects': 3345, 'num_object_clones': 0, 'num_object_copies': 13380, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2241, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3345, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3348, 'num_write_kb': 26772, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1106}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:35:46.295 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:35:46.295 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'3438", 'reported_seq': 3492, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:44.232187+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:35:44.232187+0000', 'last_peered': '2024-09-05T16:35:44.232187+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:35:44.232187+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 3438, 'log_dups_size': 0, 'ondisk_log_size': 3438, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28092736, 'num_objects': 3430, 'num_object_clones': 0, 'num_object_copies': 13720, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2264, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3430, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3435, 'num_write_kb': 27457, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1169}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:35:46.295 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:35:46.295 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'3386", 'reported_seq': 3462, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:40.184100+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:35:40.184100+0000', 'last_peered': '2024-09-05T16:35:40.184100+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:35:40.184100+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 3386, 'log_dups_size': 0, 'ondisk_log_size': 3386, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27718816, 'num_objects': 3384, 'num_object_clones': 0, 'num_object_copies': 13536, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2264, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3384, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3384, 'num_write_kb': 27070, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1122}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:35:46.295 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:35:46.295 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'3499", 'reported_seq': 3552, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:44.257621+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:35:44.257621+0000', 'last_peered': '2024-09-05T16:35:44.257621+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:35:44.257621+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 3499, 'log_dups_size': 0, 'ondisk_log_size': 3499, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28618480, 'num_objects': 3494, 'num_object_clones': 0, 'num_object_copies': 13976, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2314, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3494, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3496, 'num_write_kb': 27956, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1183}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:35:46.296 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:35:46.296 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'3412", 'reported_seq': 3446, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:44.252530+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:35:44.252530+0000', 'last_peered': '2024-09-05T16:35:44.252530+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:35:44.252530+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 3412, 'log_dups_size': 0, 'ondisk_log_size': 3412, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27941456, 'num_objects': 3411, 'num_object_clones': 0, 'num_object_copies': 13644, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2265, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3411, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3411, 'num_write_kb': 27287, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1147}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:35:46.296 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-05T16:35:46.296 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'3340", 'reported_seq': 3413, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:40.233827+0000', 'last_change': '2024-09-05T16:34:56.798081+0000', 'last_active': '2024-09-05T16:35:40.233827+0000', 'last_peered': '2024-09-05T16:35:40.233827+0000', 'last_clean': '2024-09-05T16:33:45.082039+0000', 'last_became_active': '2024-09-05T16:34:56.792386+0000', 'last_became_peered': '2024-09-05T16:34:56.792386+0000', 'last_unstale': '2024-09-05T16:35:40.233827+0000', 'last_undegraded': '2024-09-05T16:34:56.600595+0000', 'last_fullsized': '2024-09-05T16:34:56.558203+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 3340, 'log_dups_size': 0, 'ondisk_log_size': 3340, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27351632, 'num_objects': 3339, 'num_object_clones': 0, 'num_object_copies': 13356, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2295, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3339, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3339, 'num_write_kb': 26711, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1045}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:35:46.296 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:35:46.296 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'3536", 'reported_seq': 3587, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:44.289355+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:35:44.289355+0000', 'last_peered': '2024-09-05T16:35:44.289355+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:35:44.289355+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 3536, 'log_dups_size': 0, 'ondisk_log_size': 3536, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28931232, 'num_objects': 3532, 'num_object_clones': 0, 'num_object_copies': 14128, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2359, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3532, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3534, 'num_write_kb': 28268, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1175}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:35:46.296 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:35:46.296 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'3384", 'reported_seq': 3461, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:40.110991+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:35:40.110991+0000', 'last_peered': '2024-09-05T16:35:40.110991+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:35:40.110991+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 3384, 'log_dups_size': 0, 'ondisk_log_size': 3384, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27694240, 'num_objects': 3381, 'num_object_clones': 0, 'num_object_copies': 13524, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2322, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3381, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3382, 'num_write_kb': 27053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1061}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:35:46.296 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:35:46.296 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'3313", 'reported_seq': 3376, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:40.287461+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:35:40.287461+0000', 'last_peered': '2024-09-05T16:35:40.287461+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:35:40.287461+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 3313, 'log_dups_size': 0, 'ondisk_log_size': 3313, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27096224, 'num_objects': 3308, 'num_object_clones': 0, 'num_object_copies': 13232, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3308, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3311, 'num_write_kb': 26476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1071}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:35:46.297 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:35:46.508 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 84 20 55611 55591 5.16916 2.54688 0.0397985 0.0302097 2024-09-05T16:35:47.044 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:35:47.044 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 35495/220260 objects degraded (16.115%), 16 pgs degraded","count":16},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":150,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":55065,"data_bytes":451051712,"bytes_used":1426673664,"bytes_avail":385120382976,"bytes_total":386547056640,"degraded_objects":35495,"degraded_total":220260,"degraded_ratio":0.16115045854898752,"recovering_objects_per_sec":17,"recovering_bytes_per_sec":143308,"recovering_keys_per_sec":0,"num_objects_recovered":210,"num_bytes_recovered":1720320,"num_keys_recovered":0,"write_bytes_sec":3306338,"read_op_per_sec":0,"write_op_per_sec":403},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-05T16:35:14.802011+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-05T16:35:47.078 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:35:47.078 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:35:47.508 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 85 20 55952 55932 5.13968 2.66406 0.0406525 0.0303846 2024-09-05T16:35:47.687 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:35:47.687 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:35:47.727 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":99,"stamp":"2024-09-05T16:35:46.811962+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":451051712,"num_objects":55065,"num_object_clones":0,"num_object_copies":220260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35495,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55065,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":55092,"num_write_kb":440618,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":728,"num_bytes_recovered":5742592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55119,"ondisk_log_size":55119,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1393236,"kb_used_data":736724,"kb_used_omap":14,"kb_used_meta":656433,"kb_avail":376094124,"statfs":{"total":386547056640,"available":385120382976,"internally_reserved":0,"allocated":754405376,"data_stored":753673848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":672187959},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,3,5,9,5,14,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":39690240,"num_objects":4845,"num_object_clones":0,"num_object_copies":19380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-210,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4845,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4845,"num_write_kb":38760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":210,"num_bytes_recovered":1720320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004286"},"pg_stats":[{"pgid":"1.f","version":"33'3431","reported_seq":3472,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:44.176268+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:35:44.176268+0000","last_peered":"2024-09-05T16:35:44.176268+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:35:44.176268+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3431,"log_dups_size":0,"ondisk_log_size":3431,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28106752,"num_objects":3431,"num_object_clones":0,"num_object_copies":13724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3431,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3431,"num_write_kb":27448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1169}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'3400","reported_seq":3461,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:45.271187+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:35:45.271187+0000","last_peered":"2024-09-05T16:35:45.271187+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:35:45.271187+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3400,"log_dups_size":0,"ondisk_log_size":3400,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27817120,"num_objects":3396,"num_object_clones":0,"num_object_copies":13584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3396,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3398,"num_write_kb":27180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1129}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'3473","reported_seq":3547,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:45.231442+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:35:45.231442+0000","last_peered":"2024-09-05T16:35:45.231442+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:35:45.231442+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3473,"log_dups_size":0,"ondisk_log_size":3473,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28424784,"num_objects":3470,"num_object_clones":0,"num_object_copies":13880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3470,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3472,"num_write_kb":27766,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1220}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'3478","reported_seq":4944,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:45.215122+0000","last_change":"2024-09-05T16:34:56.800371+0000","last_active":"2024-09-05T16:35:45.215122+0000","last_peered":"2024-09-05T16:35:45.215122+0000","last_clean":"2024-09-05T16:33:53.148717+0000","last_became_active":"2024-09-05T16:34:56.791755+0000","last_became_peered":"2024-09-05T16:34:56.791755+0000","last_unstale":"2024-09-05T16:35:45.215122+0000","last_undegraded":"2024-09-05T16:34:56.597522+0000","last_fullsized":"2024-09-05T16:34:56.568150+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3478,"log_dups_size":0,"ondisk_log_size":3478,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28456096,"num_objects":3474,"num_object_clones":0,"num_object_copies":13896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1587,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3474,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3476,"num_write_kb":27797,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":703,"num_bytes_recovered":5742592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1887}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'3323","reported_seq":3376,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:44.148364+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:35:44.148364+0000","last_peered":"2024-09-05T16:35:44.148364+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:35:44.148364+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3323,"log_dups_size":0,"ondisk_log_size":3323,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27194528,"num_objects":3320,"num_object_clones":0,"num_object_copies":13280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3320,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3321,"num_write_kb":26565,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1158}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'3429","reported_seq":3470,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:44.205571+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:35:44.205571+0000","last_peered":"2024-09-05T16:35:44.205571+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:35:44.205571+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3429,"log_dups_size":0,"ondisk_log_size":3429,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28090368,"num_objects":3429,"num_object_clones":0,"num_object_copies":13716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3429,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3429,"num_write_kb":27432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1179}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'3362","reported_seq":3426,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:44.998290+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:35:44.998290+0000","last_peered":"2024-09-05T16:35:44.998290+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:35:44.998290+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3362,"log_dups_size":0,"ondisk_log_size":3362,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27489440,"num_objects":3356,"num_object_clones":0,"num_object_copies":13424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3356,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3360,"num_write_kb":26860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1209}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'3445","reported_seq":3508,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:45.217080+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:35:45.217080+0000","last_peered":"2024-09-05T16:35:45.217080+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:35:45.217080+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3445,"log_dups_size":0,"ondisk_log_size":3445,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28177568,"num_objects":3440,"num_object_clones":0,"num_object_copies":13760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3440,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3443,"num_write_kb":27532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1201}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'3438","reported_seq":3492,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:44.232187+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:35:44.232187+0000","last_peered":"2024-09-05T16:35:44.232187+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:35:44.232187+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3438,"log_dups_size":0,"ondisk_log_size":3438,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28092736,"num_objects":3430,"num_object_clones":0,"num_object_copies":13720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3430,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3435,"num_write_kb":27457,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1169}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'3508","reported_seq":3584,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:45.237716+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:35:45.237716+0000","last_peered":"2024-09-05T16:35:45.237716+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:35:45.237716+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3508,"log_dups_size":0,"ondisk_log_size":3508,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28718240,"num_objects":3506,"num_object_clones":0,"num_object_copies":14024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3506,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3506,"num_write_kb":28046,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1244}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'3499","reported_seq":3552,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:44.257621+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:35:44.257621+0000","last_peered":"2024-09-05T16:35:44.257621+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:35:44.257621+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3499,"log_dups_size":0,"ondisk_log_size":3499,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28618480,"num_objects":3494,"num_object_clones":0,"num_object_copies":13976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3494,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3496,"num_write_kb":27956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1183}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'3412","reported_seq":3446,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:44.252530+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:35:44.252530+0000","last_peered":"2024-09-05T16:35:44.252530+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:35:44.252530+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3412,"log_dups_size":0,"ondisk_log_size":3412,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27941456,"num_objects":3411,"num_object_clones":0,"num_object_copies":13644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3411,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3411,"num_write_kb":27287,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1147}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'3447","reported_seq":3518,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:45.140182+0000","last_change":"2024-09-05T16:34:56.798081+0000","last_active":"2024-09-05T16:35:45.140182+0000","last_peered":"2024-09-05T16:35:45.140182+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:35:45.140182+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3447,"log_dups_size":0,"ondisk_log_size":3447,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28228176,"num_objects":3446,"num_object_clones":0,"num_object_copies":13784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3446,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3446,"num_write_kb":27567,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1152}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'3536","reported_seq":3587,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:44.289355+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:35:44.289355+0000","last_peered":"2024-09-05T16:35:44.289355+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:35:44.289355+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3536,"log_dups_size":0,"ondisk_log_size":3536,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28931232,"num_objects":3532,"num_object_clones":0,"num_object_copies":14128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3532,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3534,"num_write_kb":28268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1175}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'3503","reported_seq":3580,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:45.270915+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:35:45.270915+0000","last_peered":"2024-09-05T16:35:45.270915+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:35:45.270915+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3503,"log_dups_size":0,"ondisk_log_size":3503,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28669088,"num_objects":3500,"num_object_clones":0,"num_object_copies":14000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3500,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3501,"num_write_kb":28005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1180}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'3435","reported_seq":3498,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:45.298916+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:35:45.298916+0000","last_peered":"2024-09-05T16:35:45.298916+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:35:45.298916+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3435,"log_dups_size":0,"ondisk_log_size":3435,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28095648,"num_objects":3430,"num_object_clones":0,"num_object_copies":13720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3430,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3433,"num_write_kb":27452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1193}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":451051712,"num_objects":55065,"num_object_clones":0,"num_object_copies":220260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35495,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55065,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":55092,"num_write_kb":440618,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":728,"num_bytes_recovered":5742592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":756002816,"data_stored":756002816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55119,"ondisk_log_size":55119,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084310,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":403252,"kb_used_data":218852,"kb_used_omap":4,"kb_used_meta":184379,"kb_avail":93968588,"statfs":{"total":96636764160,"available":96223834112,"internally_reserved":0,"allocated":224104448,"data_stored":223921566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":188804819},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,3,3,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986187,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":181148,"kb_used_data":77260,"kb_used_omap":4,"kb_used_meta":103867,"kb_avail":94190692,"statfs":{"total":96636764160,"available":96451268608,"internally_reserved":0,"allocated":79114240,"data_stored":78931358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":106360530},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,0,3,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949672993,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":404028,"kb_used_data":220140,"kb_used_omap":1,"kb_used_meta":183870,"kb_avail":93967812,"statfs":{"total":96636764160,"available":96223039488,"internally_reserved":0,"allocated":225423360,"data_stored":225240478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":188283333},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,5,1,1,7,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182424,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":404808,"kb_used_data":220472,"kb_used_omap":4,"kb_used_meta":184315,"kb_avail":93967032,"statfs":{"total":96636764160,"available":96222240768,"internally_reserved":0,"allocated":225763328,"data_stored":225580446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":188739277},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":226177024,"data_stored":226177024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":79138816,"data_stored":79138816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":226103296,"data_stored":226103296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":224583680,"data_stored":224583680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:35:48.508 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 86 20 56321 56301 5.11344 2.88281 0.0507763 0.0305423 2024-09-05T16:35:49.508 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 87 20 56703 56683 5.08897 2.98438 0.0874964 0.030688 2024-09-05T16:35:50.508 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 88 20 57080 57060 5.06461 2.94531 0.0652496 0.0308383 2024-09-05T16:35:50.729 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:35:51.336 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:35:51.336 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:35:51.363 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":101,"stamp":"2024-09-05T16:35:50.813530+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":465805504,"num_objects":56866,"num_object_clones":0,"num_object_copies":227464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35422,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56866,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":56893,"num_write_kb":455026,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":801,"num_bytes_recovered":6340608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":56920,"ondisk_log_size":56920,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1392860,"kb_used_data":765468,"kb_used_omap":14,"kb_used_meta":627313,"kb_avail":376094500,"statfs":{"total":386547056640,"available":385120768000,"internally_reserved":0,"allocated":783839232,"data_stored":783107704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":642369079},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,7,5,11,15,13,9,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":45432832,"num_objects":5546,"num_object_clones":0,"num_object_copies":22184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-209,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5546,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5546,"num_write_kb":44368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":209,"num_bytes_recovered":1712128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004335"},"pg_stats":[{"pgid":"1.f","version":"33'3529","reported_seq":3570,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:49.215913+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:35:49.215913+0000","last_peered":"2024-09-05T16:35:49.215913+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:35:49.215913+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3529,"log_dups_size":0,"ondisk_log_size":3529,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28909568,"num_objects":3529,"num_object_clones":0,"num_object_copies":14116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3529,"num_write_kb":28232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1267}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'3503","reported_seq":3566,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:50.360021+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:35:50.360021+0000","last_peered":"2024-09-05T16:35:50.360021+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:35:50.360021+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3503,"log_dups_size":0,"ondisk_log_size":3503,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28660896,"num_objects":3499,"num_object_clones":0,"num_object_copies":13996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3499,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3501,"num_write_kb":28004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1232}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'3583","reported_seq":3657,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:50.278184+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:35:50.278184+0000","last_peered":"2024-09-05T16:35:50.278184+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:35:50.278184+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3583,"log_dups_size":0,"ondisk_log_size":3583,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29325904,"num_objects":3580,"num_object_clones":0,"num_object_copies":14320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3580,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3582,"num_write_kb":28646,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1330}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'3582","reported_seq":5193,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:50.348452+0000","last_change":"2024-09-05T16:34:56.800371+0000","last_active":"2024-09-05T16:35:50.348452+0000","last_peered":"2024-09-05T16:35:50.348452+0000","last_clean":"2024-09-05T16:33:53.148717+0000","last_became_active":"2024-09-05T16:34:56.791755+0000","last_became_peered":"2024-09-05T16:34:56.791755+0000","last_unstale":"2024-09-05T16:35:50.348452+0000","last_undegraded":"2024-09-05T16:34:56.597522+0000","last_fullsized":"2024-09-05T16:34:56.568150+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3582,"log_dups_size":0,"ondisk_log_size":3582,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29308064,"num_objects":3578,"num_object_clones":0,"num_object_copies":14312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1514,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3578,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3580,"num_write_kb":28629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":776,"num_bytes_recovered":6340608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2064}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'3447","reported_seq":3500,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:49.308312+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:35:49.308312+0000","last_peered":"2024-09-05T16:35:49.308312+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:35:49.308312+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3447,"log_dups_size":0,"ondisk_log_size":3447,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28210336,"num_objects":3444,"num_object_clones":0,"num_object_copies":13776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3444,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3445,"num_write_kb":27557,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1282}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'3546","reported_seq":3585,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:49.324245+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:35:49.324245+0000","last_peered":"2024-09-05T16:35:49.324245+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:35:49.324245+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3546,"log_dups_size":0,"ondisk_log_size":3546,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29048832,"num_objects":3546,"num_object_clones":0,"num_object_copies":14184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3546,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3546,"num_write_kb":28368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1296}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'3493","reported_seq":3557,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:50.249045+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:35:50.249045+0000","last_peered":"2024-09-05T16:35:50.249045+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:35:50.249045+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3493,"log_dups_size":0,"ondisk_log_size":3493,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28562592,"num_objects":3487,"num_object_clones":0,"num_object_copies":13948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3487,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3491,"num_write_kb":27908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1340}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'3547","reported_seq":3610,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:50.341731+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:35:50.341731+0000","last_peered":"2024-09-05T16:35:50.341731+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:35:50.341731+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3547,"log_dups_size":0,"ondisk_log_size":3547,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29013152,"num_objects":3542,"num_object_clones":0,"num_object_copies":14168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3542,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3545,"num_write_kb":28348,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1303}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'3548","reported_seq":3603,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:49.092182+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:35:49.092182+0000","last_peered":"2024-09-05T16:35:49.092182+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:35:49.092182+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3548,"log_dups_size":0,"ondisk_log_size":3548,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28993856,"num_objects":3540,"num_object_clones":0,"num_object_copies":14160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3540,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3545,"num_write_kb":28337,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1279}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'3622","reported_seq":3699,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:50.277764+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:35:50.277764+0000","last_peered":"2024-09-05T16:35:50.277764+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:35:50.277764+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3622,"log_dups_size":0,"ondisk_log_size":3622,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29652128,"num_objects":3620,"num_object_clones":0,"num_object_copies":14480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3620,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3620,"num_write_kb":28958,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1358}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'3621","reported_seq":3675,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:49.276067+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:35:49.276067+0000","last_peered":"2024-09-05T16:35:49.276067+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:35:49.276067+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3621,"log_dups_size":0,"ondisk_log_size":3621,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29617904,"num_objects":3616,"num_object_clones":0,"num_object_copies":14464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3616,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3618,"num_write_kb":28932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1305}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'3501","reported_seq":3536,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:49.208122+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:35:49.208122+0000","last_peered":"2024-09-05T16:35:49.208122+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:35:49.208122+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3501,"log_dups_size":0,"ondisk_log_size":3501,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28670544,"num_objects":3500,"num_object_clones":0,"num_object_copies":14000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3500,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3500,"num_write_kb":27999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1236}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'3572","reported_seq":3645,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:50.099533+0000","last_change":"2024-09-05T16:34:56.798081+0000","last_active":"2024-09-05T16:35:50.099533+0000","last_peered":"2024-09-05T16:35:50.099533+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:35:50.099533+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3572,"log_dups_size":0,"ondisk_log_size":3572,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29252176,"num_objects":3571,"num_object_clones":0,"num_object_copies":14284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3571,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3571,"num_write_kb":28567,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1277}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'3651","reported_seq":3702,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:49.274452+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:35:49.274452+0000","last_peered":"2024-09-05T16:35:49.274452+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:35:49.274452+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3651,"log_dups_size":0,"ondisk_log_size":3651,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29873312,"num_objects":3647,"num_object_clones":0,"num_object_copies":14588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3647,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3649,"num_write_kb":29188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1290}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'3622","reported_seq":3699,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:50.182640+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:35:50.182640+0000","last_peered":"2024-09-05T16:35:50.182640+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:35:50.182640+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3622,"log_dups_size":0,"ondisk_log_size":3622,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29643936,"num_objects":3619,"num_object_clones":0,"num_object_copies":14476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3619,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3620,"num_write_kb":28957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1299}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'3553","reported_seq":3616,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:50.344092+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:35:50.344092+0000","last_peered":"2024-09-05T16:35:50.344092+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:35:50.344092+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3553,"log_dups_size":0,"ondisk_log_size":3553,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29062304,"num_objects":3548,"num_object_clones":0,"num_object_copies":14192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3551,"num_write_kb":28396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1311}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":465805504,"num_objects":56866,"num_object_clones":0,"num_object_copies":227464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35422,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56866,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":56893,"num_write_kb":455026,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":801,"num_bytes_recovered":6340608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":785854464,"data_stored":785854464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":56920,"ondisk_log_size":56920,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084311,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":397904,"kb_used_data":225984,"kb_used_omap":4,"kb_used_meta":171899,"kb_avail":93973936,"statfs":{"total":96636764160,"available":96229310464,"internally_reserved":0,"allocated":231407616,"data_stored":231224734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":176025299},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,2,2,7,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986188,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":192760,"kb_used_data":84712,"kb_used_omap":4,"kb_used_meta":108027,"kb_avail":94179080,"statfs":{"total":96636764160,"available":96439377920,"internally_reserved":0,"allocated":86745088,"data_stored":86562206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":110620370},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,2,11,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949672994,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":402940,"kb_used_data":227372,"kb_used_omap":1,"kb_used_meta":175550,"kb_avail":93968900,"statfs":{"total":96636764160,"available":96224153600,"internally_reserved":0,"allocated":232828928,"data_stored":232646046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":179763653},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,1,1,4,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182425,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":399256,"kb_used_data":227400,"kb_used_omap":4,"kb_used_meta":171835,"kb_avail":93972584,"statfs":{"total":96636764160,"available":96227926016,"internally_reserved":0,"allocated":232857600,"data_stored":232674718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":175959757},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,4,6,1,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233574400,"data_stored":233574400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86810624,"data_stored":86810624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":233476096,"data_stored":233476096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":231993344,"data_stored":231993344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:35:51.364 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:35:51.508 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 89 20 57425 57405 5.03799 2.69531 0.0798232 0.0309979 2024-09-05T16:35:51.950 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:35:51.951 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:35:51.975 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":101,"stamp":"2024-09-05T16:35:50.813530+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":465805504,"num_objects":56866,"num_object_clones":0,"num_object_copies":227464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35422,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56866,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":56893,"num_write_kb":455026,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":801,"num_bytes_recovered":6340608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":56920,"ondisk_log_size":56920,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1392860,"kb_used_data":765468,"kb_used_omap":14,"kb_used_meta":627313,"kb_avail":376094500,"statfs":{"total":386547056640,"available":385120768000,"internally_reserved":0,"allocated":783839232,"data_stored":783107704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":642369079},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,7,5,11,15,13,9,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":45432832,"num_objects":5546,"num_object_clones":0,"num_object_copies":22184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-209,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5546,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5546,"num_write_kb":44368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":209,"num_bytes_recovered":1712128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004335"},"pg_stats":[{"pgid":"1.f","version":"33'3529","reported_seq":3570,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:49.215913+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:35:49.215913+0000","last_peered":"2024-09-05T16:35:49.215913+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:35:49.215913+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3529,"log_dups_size":0,"ondisk_log_size":3529,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28909568,"num_objects":3529,"num_object_clones":0,"num_object_copies":14116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3529,"num_write_kb":28232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1267}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'3503","reported_seq":3566,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:50.360021+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:35:50.360021+0000","last_peered":"2024-09-05T16:35:50.360021+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:35:50.360021+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3503,"log_dups_size":0,"ondisk_log_size":3503,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28660896,"num_objects":3499,"num_object_clones":0,"num_object_copies":13996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3499,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3501,"num_write_kb":28004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1232}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'3583","reported_seq":3657,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:50.278184+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:35:50.278184+0000","last_peered":"2024-09-05T16:35:50.278184+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:35:50.278184+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3583,"log_dups_size":0,"ondisk_log_size":3583,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29325904,"num_objects":3580,"num_object_clones":0,"num_object_copies":14320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3580,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3582,"num_write_kb":28646,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1330}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'3582","reported_seq":5193,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:50.348452+0000","last_change":"2024-09-05T16:34:56.800371+0000","last_active":"2024-09-05T16:35:50.348452+0000","last_peered":"2024-09-05T16:35:50.348452+0000","last_clean":"2024-09-05T16:33:53.148717+0000","last_became_active":"2024-09-05T16:34:56.791755+0000","last_became_peered":"2024-09-05T16:34:56.791755+0000","last_unstale":"2024-09-05T16:35:50.348452+0000","last_undegraded":"2024-09-05T16:34:56.597522+0000","last_fullsized":"2024-09-05T16:34:56.568150+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3582,"log_dups_size":0,"ondisk_log_size":3582,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29308064,"num_objects":3578,"num_object_clones":0,"num_object_copies":14312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1514,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3578,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3580,"num_write_kb":28629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":776,"num_bytes_recovered":6340608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2064}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'3447","reported_seq":3500,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:49.308312+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:35:49.308312+0000","last_peered":"2024-09-05T16:35:49.308312+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:35:49.308312+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3447,"log_dups_size":0,"ondisk_log_size":3447,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28210336,"num_objects":3444,"num_object_clones":0,"num_object_copies":13776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3444,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3445,"num_write_kb":27557,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1282}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'3546","reported_seq":3585,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:49.324245+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:35:49.324245+0000","last_peered":"2024-09-05T16:35:49.324245+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:35:49.324245+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3546,"log_dups_size":0,"ondisk_log_size":3546,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29048832,"num_objects":3546,"num_object_clones":0,"num_object_copies":14184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3546,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3546,"num_write_kb":28368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1296}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'3493","reported_seq":3557,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:50.249045+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:35:50.249045+0000","last_peered":"2024-09-05T16:35:50.249045+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:35:50.249045+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3493,"log_dups_size":0,"ondisk_log_size":3493,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28562592,"num_objects":3487,"num_object_clones":0,"num_object_copies":13948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3487,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3491,"num_write_kb":27908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1340}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'3547","reported_seq":3610,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:50.341731+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:35:50.341731+0000","last_peered":"2024-09-05T16:35:50.341731+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:35:50.341731+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3547,"log_dups_size":0,"ondisk_log_size":3547,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29013152,"num_objects":3542,"num_object_clones":0,"num_object_copies":14168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3542,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3545,"num_write_kb":28348,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1303}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'3548","reported_seq":3603,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:49.092182+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:35:49.092182+0000","last_peered":"2024-09-05T16:35:49.092182+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:35:49.092182+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3548,"log_dups_size":0,"ondisk_log_size":3548,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28993856,"num_objects":3540,"num_object_clones":0,"num_object_copies":14160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3540,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3545,"num_write_kb":28337,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1279}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'3622","reported_seq":3699,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:50.277764+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:35:50.277764+0000","last_peered":"2024-09-05T16:35:50.277764+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:35:50.277764+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3622,"log_dups_size":0,"ondisk_log_size":3622,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29652128,"num_objects":3620,"num_object_clones":0,"num_object_copies":14480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3620,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3620,"num_write_kb":28958,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1358}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'3621","reported_seq":3675,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:49.276067+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:35:49.276067+0000","last_peered":"2024-09-05T16:35:49.276067+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:35:49.276067+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3621,"log_dups_size":0,"ondisk_log_size":3621,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29617904,"num_objects":3616,"num_object_clones":0,"num_object_copies":14464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3616,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3618,"num_write_kb":28932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1305}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'3501","reported_seq":3536,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:49.208122+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:35:49.208122+0000","last_peered":"2024-09-05T16:35:49.208122+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:35:49.208122+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3501,"log_dups_size":0,"ondisk_log_size":3501,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28670544,"num_objects":3500,"num_object_clones":0,"num_object_copies":14000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3500,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3500,"num_write_kb":27999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1236}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'3572","reported_seq":3645,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:50.099533+0000","last_change":"2024-09-05T16:34:56.798081+0000","last_active":"2024-09-05T16:35:50.099533+0000","last_peered":"2024-09-05T16:35:50.099533+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:35:50.099533+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3572,"log_dups_size":0,"ondisk_log_size":3572,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29252176,"num_objects":3571,"num_object_clones":0,"num_object_copies":14284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3571,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3571,"num_write_kb":28567,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1277}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'3651","reported_seq":3702,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:49.274452+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:35:49.274452+0000","last_peered":"2024-09-05T16:35:49.274452+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:35:49.274452+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3651,"log_dups_size":0,"ondisk_log_size":3651,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29873312,"num_objects":3647,"num_object_clones":0,"num_object_copies":14588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3647,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3649,"num_write_kb":29188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1290}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'3622","reported_seq":3699,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:50.182640+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:35:50.182640+0000","last_peered":"2024-09-05T16:35:50.182640+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:35:50.182640+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3622,"log_dups_size":0,"ondisk_log_size":3622,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29643936,"num_objects":3619,"num_object_clones":0,"num_object_copies":14476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3619,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3620,"num_write_kb":28957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1299}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'3553","reported_seq":3616,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:50.344092+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:35:50.344092+0000","last_peered":"2024-09-05T16:35:50.344092+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:35:50.344092+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3553,"log_dups_size":0,"ondisk_log_size":3553,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29062304,"num_objects":3548,"num_object_clones":0,"num_object_copies":14192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3551,"num_write_kb":28396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1311}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":465805504,"num_objects":56866,"num_object_clones":0,"num_object_copies":227464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35422,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56866,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":56893,"num_write_kb":455026,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":801,"num_bytes_recovered":6340608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":785854464,"data_stored":785854464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":56920,"ondisk_log_size":56920,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084311,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":397904,"kb_used_data":225984,"kb_used_omap":4,"kb_used_meta":171899,"kb_avail":93973936,"statfs":{"total":96636764160,"available":96229310464,"internally_reserved":0,"allocated":231407616,"data_stored":231224734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":176025299},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,2,2,7,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986188,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":192760,"kb_used_data":84712,"kb_used_omap":4,"kb_used_meta":108027,"kb_avail":94179080,"statfs":{"total":96636764160,"available":96439377920,"internally_reserved":0,"allocated":86745088,"data_stored":86562206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":110620370},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,2,11,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949672994,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":402940,"kb_used_data":227372,"kb_used_omap":1,"kb_used_meta":175550,"kb_avail":93968900,"statfs":{"total":96636764160,"available":96224153600,"internally_reserved":0,"allocated":232828928,"data_stored":232646046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":179763653},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,1,1,4,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182425,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":399256,"kb_used_data":227400,"kb_used_omap":4,"kb_used_meta":171835,"kb_avail":93972584,"statfs":{"total":96636764160,"available":96227926016,"internally_reserved":0,"allocated":232857600,"data_stored":232674718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":175959757},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,4,6,1,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233574400,"data_stored":233574400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86810624,"data_stored":86810624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":233476096,"data_stored":233476096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":231993344,"data_stored":231993344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:35:51.976 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-05T16:35:51.976 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'3529", 'reported_seq': 3570, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:49.215913+0000', 'last_change': '2024-09-05T16:34:56.833497+0000', 'last_active': '2024-09-05T16:35:49.215913+0000', 'last_peered': '2024-09-05T16:35:49.215913+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.828289+0000', 'last_became_peered': '2024-09-05T16:34:56.828289+0000', 'last_unstale': '2024-09-05T16:35:49.215913+0000', 'last_undegraded': '2024-09-05T16:34:56.612160+0000', 'last_fullsized': '2024-09-05T16:34:56.557632+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 3529, 'log_dups_size': 0, 'ondisk_log_size': 3529, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28909568, 'num_objects': 3529, 'num_object_clones': 0, 'num_object_copies': 14116, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2262, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3529, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3529, 'num_write_kb': 28232, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1267}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:35:51.977 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-05T16:35:51.977 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'3503", 'reported_seq': 3566, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:50.360021+0000', 'last_change': '2024-09-05T16:34:56.825031+0000', 'last_active': '2024-09-05T16:35:50.360021+0000', 'last_peered': '2024-09-05T16:35:50.360021+0000', 'last_clean': '2024-09-05T16:33:53.200762+0000', 'last_became_active': '2024-09-05T16:34:56.820292+0000', 'last_became_peered': '2024-09-05T16:34:56.820292+0000', 'last_unstale': '2024-09-05T16:35:50.360021+0000', 'last_undegraded': '2024-09-05T16:34:56.605776+0000', 'last_fullsized': '2024-09-05T16:34:56.559383+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 3503, 'log_dups_size': 0, 'ondisk_log_size': 3503, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28660896, 'num_objects': 3499, 'num_object_clones': 0, 'num_object_copies': 13996, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2269, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3499, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3501, 'num_write_kb': 28004, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 1232}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:35:51.977 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:35:51.977 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'3583", 'reported_seq': 3657, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:50.278184+0000', 'last_change': '2024-09-05T16:34:56.867900+0000', 'last_active': '2024-09-05T16:35:50.278184+0000', 'last_peered': '2024-09-05T16:35:50.278184+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:35:50.278184+0000', 'last_undegraded': '2024-09-05T16:34:56.590972+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 3583, 'log_dups_size': 0, 'ondisk_log_size': 3583, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29325904, 'num_objects': 3580, 'num_object_clones': 0, 'num_object_copies': 14320, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2251, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3580, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3582, 'num_write_kb': 28646, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1330}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:35:51.977 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-05T16:35:51.977 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'3582", 'reported_seq': 5193, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:50.348452+0000', 'last_change': '2024-09-05T16:34:56.800371+0000', 'last_active': '2024-09-05T16:35:50.348452+0000', 'last_peered': '2024-09-05T16:35:50.348452+0000', 'last_clean': '2024-09-05T16:33:53.148717+0000', 'last_became_active': '2024-09-05T16:34:56.791755+0000', 'last_became_peered': '2024-09-05T16:34:56.791755+0000', 'last_unstale': '2024-09-05T16:35:50.348452+0000', 'last_undegraded': '2024-09-05T16:34:56.597522+0000', 'last_fullsized': '2024-09-05T16:34:56.568150+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 3582, 'log_dups_size': 0, 'ondisk_log_size': 3582, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29308064, 'num_objects': 3578, 'num_object_clones': 0, 'num_object_copies': 14312, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1514, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3578, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3580, 'num_write_kb': 28629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 776, 'num_bytes_recovered': 6340608, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2064}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:35:51.977 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:35:51.977 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'3447", 'reported_seq': 3500, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:49.308312+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:35:49.308312+0000', 'last_peered': '2024-09-05T16:35:49.308312+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:35:49.308312+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 3447, 'log_dups_size': 0, 'ondisk_log_size': 3447, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28210336, 'num_objects': 3444, 'num_object_clones': 0, 'num_object_copies': 13776, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2164, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3444, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3445, 'num_write_kb': 27557, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1282}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:35:51.977 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:35:51.977 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'3546", 'reported_seq': 3585, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:49.324245+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:35:49.324245+0000', 'last_peered': '2024-09-05T16:35:49.324245+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:35:49.324245+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 3546, 'log_dups_size': 0, 'ondisk_log_size': 3546, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29048832, 'num_objects': 3546, 'num_object_clones': 0, 'num_object_copies': 14184, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3546, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3546, 'num_write_kb': 28368, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1296}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:35:51.978 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:35:51.978 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'3493", 'reported_seq': 3557, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:50.249045+0000', 'last_change': '2024-09-05T16:34:56.903436+0000', 'last_active': '2024-09-05T16:35:50.249045+0000', 'last_peered': '2024-09-05T16:35:50.249045+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:35:50.249045+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 3493, 'log_dups_size': 0, 'ondisk_log_size': 3493, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28562592, 'num_objects': 3487, 'num_object_clones': 0, 'num_object_copies': 13948, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2149, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3487, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3491, 'num_write_kb': 27908, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1340}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:35:51.978 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-05T16:35:51.978 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'3547", 'reported_seq': 3610, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:50.341731+0000', 'last_change': '2024-09-05T16:34:56.883319+0000', 'last_active': '2024-09-05T16:35:50.341731+0000', 'last_peered': '2024-09-05T16:35:50.341731+0000', 'last_clean': '2024-09-05T16:33:53.505305+0000', 'last_became_active': '2024-09-05T16:34:56.873389+0000', 'last_became_peered': '2024-09-05T16:34:56.873389+0000', 'last_unstale': '2024-09-05T16:35:50.341731+0000', 'last_undegraded': '2024-09-05T16:34:56.602567+0000', 'last_fullsized': '2024-09-05T16:34:56.559958+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 3547, 'log_dups_size': 0, 'ondisk_log_size': 3547, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29013152, 'num_objects': 3542, 'num_object_clones': 0, 'num_object_copies': 14168, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2241, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3542, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3545, 'num_write_kb': 28348, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1303}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:35:51.978 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:35:51.978 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'3548", 'reported_seq': 3603, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:49.092182+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:35:49.092182+0000', 'last_peered': '2024-09-05T16:35:49.092182+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:35:49.092182+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 3548, 'log_dups_size': 0, 'ondisk_log_size': 3548, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28993856, 'num_objects': 3540, 'num_object_clones': 0, 'num_object_copies': 14160, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2264, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3540, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3545, 'num_write_kb': 28337, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1279}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:35:51.978 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:35:51.978 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'3622", 'reported_seq': 3699, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:50.277764+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:35:50.277764+0000', 'last_peered': '2024-09-05T16:35:50.277764+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:35:50.277764+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 3622, 'log_dups_size': 0, 'ondisk_log_size': 3622, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29652128, 'num_objects': 3620, 'num_object_clones': 0, 'num_object_copies': 14480, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2264, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3620, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3620, 'num_write_kb': 28958, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1358}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:35:51.978 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:35:51.978 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'3621", 'reported_seq': 3675, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:49.276067+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:35:49.276067+0000', 'last_peered': '2024-09-05T16:35:49.276067+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:35:49.276067+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 3621, 'log_dups_size': 0, 'ondisk_log_size': 3621, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29617904, 'num_objects': 3616, 'num_object_clones': 0, 'num_object_copies': 14464, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2314, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3616, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3618, 'num_write_kb': 28932, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1305}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:35:51.979 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:35:51.979 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'3501", 'reported_seq': 3536, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:49.208122+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:35:49.208122+0000', 'last_peered': '2024-09-05T16:35:49.208122+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:35:49.208122+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 3501, 'log_dups_size': 0, 'ondisk_log_size': 3501, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28670544, 'num_objects': 3500, 'num_object_clones': 0, 'num_object_copies': 14000, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2265, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3500, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3500, 'num_write_kb': 27999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1236}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:35:51.979 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-05T16:35:51.979 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'3572", 'reported_seq': 3645, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:50.099533+0000', 'last_change': '2024-09-05T16:34:56.798081+0000', 'last_active': '2024-09-05T16:35:50.099533+0000', 'last_peered': '2024-09-05T16:35:50.099533+0000', 'last_clean': '2024-09-05T16:33:45.082039+0000', 'last_became_active': '2024-09-05T16:34:56.792386+0000', 'last_became_peered': '2024-09-05T16:34:56.792386+0000', 'last_unstale': '2024-09-05T16:35:50.099533+0000', 'last_undegraded': '2024-09-05T16:34:56.600595+0000', 'last_fullsized': '2024-09-05T16:34:56.558203+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 3572, 'log_dups_size': 0, 'ondisk_log_size': 3572, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29252176, 'num_objects': 3571, 'num_object_clones': 0, 'num_object_copies': 14284, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2295, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3571, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3571, 'num_write_kb': 28567, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1277}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:35:51.979 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:35:51.979 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'3651", 'reported_seq': 3702, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:49.274452+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:35:49.274452+0000', 'last_peered': '2024-09-05T16:35:49.274452+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:35:49.274452+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 3651, 'log_dups_size': 0, 'ondisk_log_size': 3651, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29873312, 'num_objects': 3647, 'num_object_clones': 0, 'num_object_copies': 14588, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2359, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3647, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3649, 'num_write_kb': 29188, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1290}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:35:51.979 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:35:51.979 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'3622", 'reported_seq': 3699, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:50.182640+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:35:50.182640+0000', 'last_peered': '2024-09-05T16:35:50.182640+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:35:50.182640+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 3622, 'log_dups_size': 0, 'ondisk_log_size': 3622, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29643936, 'num_objects': 3619, 'num_object_clones': 0, 'num_object_copies': 14476, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2322, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3619, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3620, 'num_write_kb': 28957, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1299}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:35:51.979 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:35:51.979 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'3553", 'reported_seq': 3616, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:50.344092+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:35:50.344092+0000', 'last_peered': '2024-09-05T16:35:50.344092+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:35:50.344092+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 3553, 'log_dups_size': 0, 'ondisk_log_size': 3553, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29062304, 'num_objects': 3548, 'num_object_clones': 0, 'num_object_copies': 14192, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3548, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3551, 'num_write_kb': 28396, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1311}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:35:51.980 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:35:52.508 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 90 20 57753 57733 5.01049 2.5625 0.0407037 0.0311666 2024-09-05T16:35:52.742 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:35:52.742 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 35422/227464 objects degraded (15.573%), 16 pgs degraded","count":16},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":155,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":56866,"data_bytes":465805504,"bytes_used":1426288640,"bytes_avail":385120768000,"bytes_total":386547056640,"degraded_objects":35422,"degraded_total":227464,"degraded_ratio":0.15572574121619245,"recovering_objects_per_sec":17,"recovering_bytes_per_sec":142625,"recovering_keys_per_sec":0,"num_objects_recovered":209,"num_bytes_recovered":1712128,"num_keys_recovered":0,"write_bytes_sec":3784701,"read_op_per_sec":0,"write_op_per_sec":461},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-05T16:35:14.802011+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-05T16:35:52.769 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:35:52.769 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:35:53.337 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:35:53.338 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:35:53.359 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":102,"stamp":"2024-09-05T16:35:52.814308+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":465805504,"num_objects":56866,"num_object_clones":0,"num_object_copies":227464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35422,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56866,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":56893,"num_write_kb":455026,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":801,"num_bytes_recovered":6340608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":56920,"ondisk_log_size":56920,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1392860,"kb_used_data":765468,"kb_used_omap":14,"kb_used_meta":627313,"kb_avail":376094500,"statfs":{"total":386547056640,"available":385120768000,"internally_reserved":0,"allocated":783839232,"data_stored":783107704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":642369079},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,7,5,11,15,13,9,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":29851648,"num_objects":3644,"num_object_clones":0,"num_object_copies":14576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-144,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3644,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3644,"num_write_kb":29152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":144,"num_bytes_recovered":1179648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004530"},"pg_stats":[{"pgid":"1.f","version":"33'3529","reported_seq":3570,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:49.215913+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:35:49.215913+0000","last_peered":"2024-09-05T16:35:49.215913+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:35:49.215913+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3529,"log_dups_size":0,"ondisk_log_size":3529,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28909568,"num_objects":3529,"num_object_clones":0,"num_object_copies":14116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3529,"num_write_kb":28232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1267}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'3503","reported_seq":3566,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:50.360021+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:35:50.360021+0000","last_peered":"2024-09-05T16:35:50.360021+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:35:50.360021+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3503,"log_dups_size":0,"ondisk_log_size":3503,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28660896,"num_objects":3499,"num_object_clones":0,"num_object_copies":13996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3499,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3501,"num_write_kb":28004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1232}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'3583","reported_seq":3657,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:50.278184+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:35:50.278184+0000","last_peered":"2024-09-05T16:35:50.278184+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:35:50.278184+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3583,"log_dups_size":0,"ondisk_log_size":3583,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29325904,"num_objects":3580,"num_object_clones":0,"num_object_copies":14320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3580,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3582,"num_write_kb":28646,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1330}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'3582","reported_seq":5193,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:50.348452+0000","last_change":"2024-09-05T16:34:56.800371+0000","last_active":"2024-09-05T16:35:50.348452+0000","last_peered":"2024-09-05T16:35:50.348452+0000","last_clean":"2024-09-05T16:33:53.148717+0000","last_became_active":"2024-09-05T16:34:56.791755+0000","last_became_peered":"2024-09-05T16:34:56.791755+0000","last_unstale":"2024-09-05T16:35:50.348452+0000","last_undegraded":"2024-09-05T16:34:56.597522+0000","last_fullsized":"2024-09-05T16:34:56.568150+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3582,"log_dups_size":0,"ondisk_log_size":3582,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29308064,"num_objects":3578,"num_object_clones":0,"num_object_copies":14312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1514,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3578,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3580,"num_write_kb":28629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":776,"num_bytes_recovered":6340608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2064}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'3447","reported_seq":3500,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:49.308312+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:35:49.308312+0000","last_peered":"2024-09-05T16:35:49.308312+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:35:49.308312+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3447,"log_dups_size":0,"ondisk_log_size":3447,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28210336,"num_objects":3444,"num_object_clones":0,"num_object_copies":13776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3444,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3445,"num_write_kb":27557,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1282}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'3546","reported_seq":3585,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:49.324245+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:35:49.324245+0000","last_peered":"2024-09-05T16:35:49.324245+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:35:49.324245+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3546,"log_dups_size":0,"ondisk_log_size":3546,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29048832,"num_objects":3546,"num_object_clones":0,"num_object_copies":14184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3546,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3546,"num_write_kb":28368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1296}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'3493","reported_seq":3557,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:50.249045+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:35:50.249045+0000","last_peered":"2024-09-05T16:35:50.249045+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:35:50.249045+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3493,"log_dups_size":0,"ondisk_log_size":3493,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28562592,"num_objects":3487,"num_object_clones":0,"num_object_copies":13948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3487,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3491,"num_write_kb":27908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1340}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'3547","reported_seq":3610,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:50.341731+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:35:50.341731+0000","last_peered":"2024-09-05T16:35:50.341731+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:35:50.341731+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3547,"log_dups_size":0,"ondisk_log_size":3547,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29013152,"num_objects":3542,"num_object_clones":0,"num_object_copies":14168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3542,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3545,"num_write_kb":28348,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1303}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'3548","reported_seq":3603,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:49.092182+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:35:49.092182+0000","last_peered":"2024-09-05T16:35:49.092182+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:35:49.092182+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3548,"log_dups_size":0,"ondisk_log_size":3548,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28993856,"num_objects":3540,"num_object_clones":0,"num_object_copies":14160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3540,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3545,"num_write_kb":28337,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1279}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'3622","reported_seq":3699,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:50.277764+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:35:50.277764+0000","last_peered":"2024-09-05T16:35:50.277764+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:35:50.277764+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3622,"log_dups_size":0,"ondisk_log_size":3622,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29652128,"num_objects":3620,"num_object_clones":0,"num_object_copies":14480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3620,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3620,"num_write_kb":28958,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1358}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'3621","reported_seq":3675,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:49.276067+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:35:49.276067+0000","last_peered":"2024-09-05T16:35:49.276067+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:35:49.276067+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3621,"log_dups_size":0,"ondisk_log_size":3621,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29617904,"num_objects":3616,"num_object_clones":0,"num_object_copies":14464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3616,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3618,"num_write_kb":28932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1305}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'3501","reported_seq":3536,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:49.208122+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:35:49.208122+0000","last_peered":"2024-09-05T16:35:49.208122+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:35:49.208122+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3501,"log_dups_size":0,"ondisk_log_size":3501,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28670544,"num_objects":3500,"num_object_clones":0,"num_object_copies":14000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3500,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3500,"num_write_kb":27999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1236}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'3572","reported_seq":3645,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:50.099533+0000","last_change":"2024-09-05T16:34:56.798081+0000","last_active":"2024-09-05T16:35:50.099533+0000","last_peered":"2024-09-05T16:35:50.099533+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:35:50.099533+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3572,"log_dups_size":0,"ondisk_log_size":3572,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29252176,"num_objects":3571,"num_object_clones":0,"num_object_copies":14284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3571,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3571,"num_write_kb":28567,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1277}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'3651","reported_seq":3702,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:49.274452+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:35:49.274452+0000","last_peered":"2024-09-05T16:35:49.274452+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:35:49.274452+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3651,"log_dups_size":0,"ondisk_log_size":3651,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29873312,"num_objects":3647,"num_object_clones":0,"num_object_copies":14588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3647,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3649,"num_write_kb":29188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1290}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'3622","reported_seq":3699,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:50.182640+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:35:50.182640+0000","last_peered":"2024-09-05T16:35:50.182640+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:35:50.182640+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3622,"log_dups_size":0,"ondisk_log_size":3622,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29643936,"num_objects":3619,"num_object_clones":0,"num_object_copies":14476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3619,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3620,"num_write_kb":28957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1299}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'3553","reported_seq":3616,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:50.344092+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:35:50.344092+0000","last_peered":"2024-09-05T16:35:50.344092+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:35:50.344092+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3553,"log_dups_size":0,"ondisk_log_size":3553,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29062304,"num_objects":3548,"num_object_clones":0,"num_object_copies":14192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3551,"num_write_kb":28396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1311}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":465805504,"num_objects":56866,"num_object_clones":0,"num_object_copies":227464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35422,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56866,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":56893,"num_write_kb":455026,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":801,"num_bytes_recovered":6340608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":785854464,"data_stored":785854464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":56920,"ondisk_log_size":56920,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084311,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":397904,"kb_used_data":225984,"kb_used_omap":4,"kb_used_meta":171899,"kb_avail":93973936,"statfs":{"total":96636764160,"available":96229310464,"internally_reserved":0,"allocated":231407616,"data_stored":231224734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":176025299},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,2,2,7,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986188,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":192760,"kb_used_data":84712,"kb_used_omap":4,"kb_used_meta":108027,"kb_avail":94179080,"statfs":{"total":96636764160,"available":96439377920,"internally_reserved":0,"allocated":86745088,"data_stored":86562206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":110620370},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,2,11,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949672994,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":402940,"kb_used_data":227372,"kb_used_omap":1,"kb_used_meta":175550,"kb_avail":93968900,"statfs":{"total":96636764160,"available":96224153600,"internally_reserved":0,"allocated":232828928,"data_stored":232646046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":179763653},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,1,1,4,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182425,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":399256,"kb_used_data":227400,"kb_used_omap":4,"kb_used_meta":171835,"kb_avail":93972584,"statfs":{"total":96636764160,"available":96227926016,"internally_reserved":0,"allocated":232857600,"data_stored":232674718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":175959757},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,4,6,1,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233574400,"data_stored":233574400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86810624,"data_stored":86810624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":233476096,"data_stored":233476096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":231993344,"data_stored":231993344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:35:53.508 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 91 20 58086 58066 4.98402 2.60156 0.0562787 0.0313336 2024-09-05T16:35:54.508 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 92 20 58456 58436 4.96126 2.89062 0.0460224 0.0314794 2024-09-05T16:35:55.508 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 93 20 58830 58810 4.93933 2.92188 0.0446448 0.0316147 2024-09-05T16:35:56.361 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:35:56.509 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 94 20 59199 59179 4.91746 2.88281 0.0700944 0.0317612 2024-09-05T16:35:56.974 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:35:56.975 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:35:57.000 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":104,"stamp":"2024-09-05T16:35:56.815929+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":480297152,"num_objects":58635,"num_object_clones":0,"num_object_copies":234540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35350,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58635,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":58662,"num_write_kb":469178,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":873,"num_bytes_recovered":6930432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":58689,"ondisk_log_size":58689,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1465932,"kb_used_data":794120,"kb_used_omap":14,"kb_used_meta":671729,"kb_avail":376021428,"statfs":{"total":386547056640,"available":385045942272,"internally_reserved":0,"allocated":813178880,"data_stored":812447352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":687851063},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,4,6,5,5,8,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":37683200,"num_objects":4600,"num_object_clones":0,"num_object_copies":18400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-216,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4600,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4600,"num_write_kb":36800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":216,"num_bytes_recovered":1769472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004722"},"pg_stats":[{"pgid":"1.f","version":"33'3648","reported_seq":3689,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:54.304485+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:35:54.304485+0000","last_peered":"2024-09-05T16:35:54.304485+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:35:54.304485+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3648,"log_dups_size":0,"ondisk_log_size":3648,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29884416,"num_objects":3648,"num_object_clones":0,"num_object_copies":14592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3648,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3648,"num_write_kb":29184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1386}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'3609","reported_seq":3672,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:55.375973+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:35:55.375973+0000","last_peered":"2024-09-05T16:35:55.375973+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:35:55.375973+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3609,"log_dups_size":0,"ondisk_log_size":3609,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29529248,"num_objects":3605,"num_object_clones":0,"num_object_copies":14420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3605,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3607,"num_write_kb":28852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1338}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'3682","reported_seq":3756,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:55.310356+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:35:55.310356+0000","last_peered":"2024-09-05T16:35:55.310356+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:35:55.310356+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3682,"log_dups_size":0,"ondisk_log_size":3682,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30136912,"num_objects":3679,"num_object_clones":0,"num_object_copies":14716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3679,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3681,"num_write_kb":29438,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1429}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'3702","reported_seq":5457,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:55.385312+0000","last_change":"2024-09-05T16:34:56.800371+0000","last_active":"2024-09-05T16:35:55.385312+0000","last_peered":"2024-09-05T16:35:55.385312+0000","last_clean":"2024-09-05T16:33:53.148717+0000","last_became_active":"2024-09-05T16:34:56.791755+0000","last_became_peered":"2024-09-05T16:34:56.791755+0000","last_unstale":"2024-09-05T16:35:55.385312+0000","last_undegraded":"2024-09-05T16:34:56.597522+0000","last_fullsized":"2024-09-05T16:34:56.568150+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3702,"log_dups_size":0,"ondisk_log_size":3702,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30291104,"num_objects":3698,"num_object_clones":0,"num_object_copies":14792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1442,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3698,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3700,"num_write_kb":29589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":848,"num_bytes_recovered":6930432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2256}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'3557","reported_seq":3610,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:54.381139+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:35:54.381139+0000","last_peered":"2024-09-05T16:35:54.381139+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:35:54.381139+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3557,"log_dups_size":0,"ondisk_log_size":3557,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29111456,"num_objects":3554,"num_object_clones":0,"num_object_copies":14216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3554,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3555,"num_write_kb":28437,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1392}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'3665","reported_seq":3706,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:54.241542+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:35:54.241542+0000","last_peered":"2024-09-05T16:35:54.241542+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:35:54.241542+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3665,"log_dups_size":0,"ondisk_log_size":3665,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30023680,"num_objects":3665,"num_object_clones":0,"num_object_copies":14660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3665,"num_write_kb":29320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1415}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'3598","reported_seq":3662,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:55.342353+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:35:55.342353+0000","last_peered":"2024-09-05T16:35:55.342353+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:35:55.342353+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3598,"log_dups_size":0,"ondisk_log_size":3598,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29422752,"num_objects":3592,"num_object_clones":0,"num_object_copies":14368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3592,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3596,"num_write_kb":28748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1445}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'3656","reported_seq":3719,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:55.299919+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:35:55.299919+0000","last_peered":"2024-09-05T16:35:55.299919+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:35:55.299919+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3656,"log_dups_size":0,"ondisk_log_size":3656,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29906080,"num_objects":3651,"num_object_clones":0,"num_object_copies":14604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3651,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3654,"num_write_kb":29220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1412}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'3666","reported_seq":3721,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:54.363909+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:35:54.363909+0000","last_peered":"2024-09-05T16:35:54.363909+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:35:54.363909+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3666,"log_dups_size":0,"ondisk_log_size":3666,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29960512,"num_objects":3658,"num_object_clones":0,"num_object_copies":14632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3658,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3663,"num_write_kb":29281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1397}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'3740","reported_seq":3817,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:55.264682+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:35:55.264682+0000","last_peered":"2024-09-05T16:35:55.264682+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:35:55.264682+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3740,"log_dups_size":0,"ondisk_log_size":3740,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30618784,"num_objects":3738,"num_object_clones":0,"num_object_copies":14952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3738,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3738,"num_write_kb":29902,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1476}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'3730","reported_seq":3785,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:54.249396+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:35:54.249396+0000","last_peered":"2024-09-05T16:35:54.249396+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:35:54.249396+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3730,"log_dups_size":0,"ondisk_log_size":3730,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30510832,"num_objects":3725,"num_object_clones":0,"num_object_copies":14900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3725,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3727,"num_write_kb":29804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1414}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'3598","reported_seq":3633,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:54.358646+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:35:54.358646+0000","last_peered":"2024-09-05T16:35:54.358646+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:35:54.358646+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3598,"log_dups_size":0,"ondisk_log_size":3598,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29465168,"num_objects":3597,"num_object_clones":0,"num_object_copies":14388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3597,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3597,"num_write_kb":28775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1333}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'3682","reported_seq":3755,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:55.204163+0000","last_change":"2024-09-05T16:34:56.798081+0000","last_active":"2024-09-05T16:35:55.204163+0000","last_peered":"2024-09-05T16:35:55.204163+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:35:55.204163+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3682,"log_dups_size":0,"ondisk_log_size":3682,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30153296,"num_objects":3681,"num_object_clones":0,"num_object_copies":14724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3681,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3681,"num_write_kb":29447,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1387}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'3764","reported_seq":3815,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:54.365934+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:35:54.365934+0000","last_peered":"2024-09-05T16:35:54.365934+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:35:54.365934+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3764,"log_dups_size":0,"ondisk_log_size":3764,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30799008,"num_objects":3760,"num_object_clones":0,"num_object_copies":15040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3760,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3762,"num_write_kb":30092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1403}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'3726","reported_seq":3803,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:55.290328+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:35:55.290328+0000","last_peered":"2024-09-05T16:35:55.290328+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:35:55.290328+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3726,"log_dups_size":0,"ondisk_log_size":3726,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30495904,"num_objects":3723,"num_object_clones":0,"num_object_copies":14892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3723,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3724,"num_write_kb":29789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1403}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'3666","reported_seq":3729,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:55.375680+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:35:55.375680+0000","last_peered":"2024-09-05T16:35:55.375680+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:35:55.375680+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3666,"log_dups_size":0,"ondisk_log_size":3666,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29988000,"num_objects":3661,"num_object_clones":0,"num_object_copies":14644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3661,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3664,"num_write_kb":29300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1424}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":480297152,"num_objects":58635,"num_object_clones":0,"num_object_copies":234540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35350,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58635,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":58662,"num_write_kb":469178,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":873,"num_bytes_recovered":6930432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":815132672,"data_stored":815132672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":58689,"ondisk_log_size":58689,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084312,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":409284,"kb_used_data":233204,"kb_used_omap":4,"kb_used_meta":176059,"kb_avail":93962556,"statfs":{"total":96636764160,"available":96217657344,"internally_reserved":0,"allocated":238800896,"data_stored":238618014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":180285139},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,0,1,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986189,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":209904,"kb_used_data":92124,"kb_used_omap":4,"kb_used_meta":117755,"kb_avail":94161936,"statfs":{"total":96636764160,"available":96421822464,"internally_reserved":0,"allocated":94334976,"data_stored":94152094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":120581842},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949672995,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":432200,"kb_used_data":234424,"kb_used_omap":1,"kb_used_meta":197758,"kb_avail":93939640,"statfs":{"total":96636764160,"available":96194191360,"internally_reserved":0,"allocated":240050176,"data_stored":239867294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":202504645},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182426,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":414544,"kb_used_data":234368,"kb_used_omap":4,"kb_used_meta":180155,"kb_avail":93957296,"statfs":{"total":96636764160,"available":96212271104,"internally_reserved":0,"allocated":239992832,"data_stored":239809950,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":184479437},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,2,2,3,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":240828416,"data_stored":240828416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":94326784,"data_stored":94326784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":240746496,"data_stored":240746496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":239230976,"data_stored":239230976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:35:57.002 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:35:57.500 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:35:57.501 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:35:57.508 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 95 20 59491 59471 4.88971 2.28125 0.0510547 0.0319411 2024-09-05T16:35:57.524 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":104,"stamp":"2024-09-05T16:35:56.815929+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":480297152,"num_objects":58635,"num_object_clones":0,"num_object_copies":234540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35350,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58635,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":58662,"num_write_kb":469178,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":873,"num_bytes_recovered":6930432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":58689,"ondisk_log_size":58689,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1465932,"kb_used_data":794120,"kb_used_omap":14,"kb_used_meta":671729,"kb_avail":376021428,"statfs":{"total":386547056640,"available":385045942272,"internally_reserved":0,"allocated":813178880,"data_stored":812447352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":687851063},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,4,6,5,5,8,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":37683200,"num_objects":4600,"num_object_clones":0,"num_object_copies":18400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-216,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4600,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4600,"num_write_kb":36800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":216,"num_bytes_recovered":1769472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004722"},"pg_stats":[{"pgid":"1.f","version":"33'3648","reported_seq":3689,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:54.304485+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:35:54.304485+0000","last_peered":"2024-09-05T16:35:54.304485+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:35:54.304485+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3648,"log_dups_size":0,"ondisk_log_size":3648,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29884416,"num_objects":3648,"num_object_clones":0,"num_object_copies":14592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3648,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3648,"num_write_kb":29184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1386}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'3609","reported_seq":3672,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:55.375973+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:35:55.375973+0000","last_peered":"2024-09-05T16:35:55.375973+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:35:55.375973+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3609,"log_dups_size":0,"ondisk_log_size":3609,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29529248,"num_objects":3605,"num_object_clones":0,"num_object_copies":14420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3605,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3607,"num_write_kb":28852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1338}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'3682","reported_seq":3756,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:55.310356+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:35:55.310356+0000","last_peered":"2024-09-05T16:35:55.310356+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:35:55.310356+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3682,"log_dups_size":0,"ondisk_log_size":3682,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30136912,"num_objects":3679,"num_object_clones":0,"num_object_copies":14716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3679,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3681,"num_write_kb":29438,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1429}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'3702","reported_seq":5457,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:55.385312+0000","last_change":"2024-09-05T16:34:56.800371+0000","last_active":"2024-09-05T16:35:55.385312+0000","last_peered":"2024-09-05T16:35:55.385312+0000","last_clean":"2024-09-05T16:33:53.148717+0000","last_became_active":"2024-09-05T16:34:56.791755+0000","last_became_peered":"2024-09-05T16:34:56.791755+0000","last_unstale":"2024-09-05T16:35:55.385312+0000","last_undegraded":"2024-09-05T16:34:56.597522+0000","last_fullsized":"2024-09-05T16:34:56.568150+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3702,"log_dups_size":0,"ondisk_log_size":3702,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30291104,"num_objects":3698,"num_object_clones":0,"num_object_copies":14792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1442,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3698,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3700,"num_write_kb":29589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":848,"num_bytes_recovered":6930432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2256}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'3557","reported_seq":3610,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:54.381139+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:35:54.381139+0000","last_peered":"2024-09-05T16:35:54.381139+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:35:54.381139+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3557,"log_dups_size":0,"ondisk_log_size":3557,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29111456,"num_objects":3554,"num_object_clones":0,"num_object_copies":14216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3554,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3555,"num_write_kb":28437,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1392}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'3665","reported_seq":3706,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:54.241542+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:35:54.241542+0000","last_peered":"2024-09-05T16:35:54.241542+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:35:54.241542+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3665,"log_dups_size":0,"ondisk_log_size":3665,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30023680,"num_objects":3665,"num_object_clones":0,"num_object_copies":14660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3665,"num_write_kb":29320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1415}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'3598","reported_seq":3662,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:55.342353+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:35:55.342353+0000","last_peered":"2024-09-05T16:35:55.342353+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:35:55.342353+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3598,"log_dups_size":0,"ondisk_log_size":3598,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29422752,"num_objects":3592,"num_object_clones":0,"num_object_copies":14368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3592,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3596,"num_write_kb":28748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1445}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'3656","reported_seq":3719,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:55.299919+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:35:55.299919+0000","last_peered":"2024-09-05T16:35:55.299919+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:35:55.299919+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3656,"log_dups_size":0,"ondisk_log_size":3656,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29906080,"num_objects":3651,"num_object_clones":0,"num_object_copies":14604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3651,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3654,"num_write_kb":29220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1412}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'3666","reported_seq":3721,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:54.363909+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:35:54.363909+0000","last_peered":"2024-09-05T16:35:54.363909+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:35:54.363909+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3666,"log_dups_size":0,"ondisk_log_size":3666,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29960512,"num_objects":3658,"num_object_clones":0,"num_object_copies":14632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3658,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3663,"num_write_kb":29281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1397}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'3740","reported_seq":3817,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:55.264682+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:35:55.264682+0000","last_peered":"2024-09-05T16:35:55.264682+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:35:55.264682+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3740,"log_dups_size":0,"ondisk_log_size":3740,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30618784,"num_objects":3738,"num_object_clones":0,"num_object_copies":14952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3738,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3738,"num_write_kb":29902,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1476}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'3730","reported_seq":3785,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:54.249396+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:35:54.249396+0000","last_peered":"2024-09-05T16:35:54.249396+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:35:54.249396+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3730,"log_dups_size":0,"ondisk_log_size":3730,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30510832,"num_objects":3725,"num_object_clones":0,"num_object_copies":14900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3725,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3727,"num_write_kb":29804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1414}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'3598","reported_seq":3633,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:54.358646+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:35:54.358646+0000","last_peered":"2024-09-05T16:35:54.358646+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:35:54.358646+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3598,"log_dups_size":0,"ondisk_log_size":3598,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29465168,"num_objects":3597,"num_object_clones":0,"num_object_copies":14388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3597,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3597,"num_write_kb":28775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1333}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'3682","reported_seq":3755,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:55.204163+0000","last_change":"2024-09-05T16:34:56.798081+0000","last_active":"2024-09-05T16:35:55.204163+0000","last_peered":"2024-09-05T16:35:55.204163+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:35:55.204163+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3682,"log_dups_size":0,"ondisk_log_size":3682,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30153296,"num_objects":3681,"num_object_clones":0,"num_object_copies":14724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3681,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3681,"num_write_kb":29447,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1387}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'3764","reported_seq":3815,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:54.365934+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:35:54.365934+0000","last_peered":"2024-09-05T16:35:54.365934+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:35:54.365934+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3764,"log_dups_size":0,"ondisk_log_size":3764,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30799008,"num_objects":3760,"num_object_clones":0,"num_object_copies":15040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3760,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3762,"num_write_kb":30092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1403}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'3726","reported_seq":3803,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:55.290328+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:35:55.290328+0000","last_peered":"2024-09-05T16:35:55.290328+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:35:55.290328+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3726,"log_dups_size":0,"ondisk_log_size":3726,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30495904,"num_objects":3723,"num_object_clones":0,"num_object_copies":14892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3723,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3724,"num_write_kb":29789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1403}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'3666","reported_seq":3729,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:55.375680+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:35:55.375680+0000","last_peered":"2024-09-05T16:35:55.375680+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:35:55.375680+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3666,"log_dups_size":0,"ondisk_log_size":3666,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29988000,"num_objects":3661,"num_object_clones":0,"num_object_copies":14644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3661,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3664,"num_write_kb":29300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1424}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":480297152,"num_objects":58635,"num_object_clones":0,"num_object_copies":234540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35350,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58635,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":58662,"num_write_kb":469178,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":873,"num_bytes_recovered":6930432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":815132672,"data_stored":815132672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":58689,"ondisk_log_size":58689,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084312,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":409284,"kb_used_data":233204,"kb_used_omap":4,"kb_used_meta":176059,"kb_avail":93962556,"statfs":{"total":96636764160,"available":96217657344,"internally_reserved":0,"allocated":238800896,"data_stored":238618014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":180285139},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,0,1,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986189,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":209904,"kb_used_data":92124,"kb_used_omap":4,"kb_used_meta":117755,"kb_avail":94161936,"statfs":{"total":96636764160,"available":96421822464,"internally_reserved":0,"allocated":94334976,"data_stored":94152094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":120581842},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949672995,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":432200,"kb_used_data":234424,"kb_used_omap":1,"kb_used_meta":197758,"kb_avail":93939640,"statfs":{"total":96636764160,"available":96194191360,"internally_reserved":0,"allocated":240050176,"data_stored":239867294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":202504645},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182426,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":414544,"kb_used_data":234368,"kb_used_omap":4,"kb_used_meta":180155,"kb_avail":93957296,"statfs":{"total":96636764160,"available":96212271104,"internally_reserved":0,"allocated":239992832,"data_stored":239809950,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":184479437},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,2,2,3,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":240828416,"data_stored":240828416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":94326784,"data_stored":94326784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":240746496,"data_stored":240746496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":239230976,"data_stored":239230976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:35:57.525 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-05T16:35:57.525 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'3648", 'reported_seq': 3689, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:54.304485+0000', 'last_change': '2024-09-05T16:34:56.833497+0000', 'last_active': '2024-09-05T16:35:54.304485+0000', 'last_peered': '2024-09-05T16:35:54.304485+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.828289+0000', 'last_became_peered': '2024-09-05T16:34:56.828289+0000', 'last_unstale': '2024-09-05T16:35:54.304485+0000', 'last_undegraded': '2024-09-05T16:34:56.612160+0000', 'last_fullsized': '2024-09-05T16:34:56.557632+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 3648, 'log_dups_size': 0, 'ondisk_log_size': 3648, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29884416, 'num_objects': 3648, 'num_object_clones': 0, 'num_object_copies': 14592, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2262, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3648, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3648, 'num_write_kb': 29184, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1386}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:35:57.526 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-05T16:35:57.526 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'3609", 'reported_seq': 3672, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:55.375973+0000', 'last_change': '2024-09-05T16:34:56.825031+0000', 'last_active': '2024-09-05T16:35:55.375973+0000', 'last_peered': '2024-09-05T16:35:55.375973+0000', 'last_clean': '2024-09-05T16:33:53.200762+0000', 'last_became_active': '2024-09-05T16:34:56.820292+0000', 'last_became_peered': '2024-09-05T16:34:56.820292+0000', 'last_unstale': '2024-09-05T16:35:55.375973+0000', 'last_undegraded': '2024-09-05T16:34:56.605776+0000', 'last_fullsized': '2024-09-05T16:34:56.559383+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 3609, 'log_dups_size': 0, 'ondisk_log_size': 3609, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29529248, 'num_objects': 3605, 'num_object_clones': 0, 'num_object_copies': 14420, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2269, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3605, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3607, 'num_write_kb': 28852, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 1338}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:35:57.526 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:35:57.526 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'3682", 'reported_seq': 3756, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:55.310356+0000', 'last_change': '2024-09-05T16:34:56.867900+0000', 'last_active': '2024-09-05T16:35:55.310356+0000', 'last_peered': '2024-09-05T16:35:55.310356+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:35:55.310356+0000', 'last_undegraded': '2024-09-05T16:34:56.590972+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 3682, 'log_dups_size': 0, 'ondisk_log_size': 3682, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30136912, 'num_objects': 3679, 'num_object_clones': 0, 'num_object_copies': 14716, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2251, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3679, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3681, 'num_write_kb': 29438, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1429}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:35:57.526 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-05T16:35:57.526 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'3702", 'reported_seq': 5457, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:55.385312+0000', 'last_change': '2024-09-05T16:34:56.800371+0000', 'last_active': '2024-09-05T16:35:55.385312+0000', 'last_peered': '2024-09-05T16:35:55.385312+0000', 'last_clean': '2024-09-05T16:33:53.148717+0000', 'last_became_active': '2024-09-05T16:34:56.791755+0000', 'last_became_peered': '2024-09-05T16:34:56.791755+0000', 'last_unstale': '2024-09-05T16:35:55.385312+0000', 'last_undegraded': '2024-09-05T16:34:56.597522+0000', 'last_fullsized': '2024-09-05T16:34:56.568150+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 3702, 'log_dups_size': 0, 'ondisk_log_size': 3702, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30291104, 'num_objects': 3698, 'num_object_clones': 0, 'num_object_copies': 14792, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1442, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3698, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3700, 'num_write_kb': 29589, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 848, 'num_bytes_recovered': 6930432, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2256}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:35:57.526 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:35:57.526 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'3557", 'reported_seq': 3610, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:54.381139+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:35:54.381139+0000', 'last_peered': '2024-09-05T16:35:54.381139+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:35:54.381139+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 3557, 'log_dups_size': 0, 'ondisk_log_size': 3557, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29111456, 'num_objects': 3554, 'num_object_clones': 0, 'num_object_copies': 14216, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2164, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3554, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3555, 'num_write_kb': 28437, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1392}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:35:57.526 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:35:57.526 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'3665", 'reported_seq': 3706, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:54.241542+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:35:54.241542+0000', 'last_peered': '2024-09-05T16:35:54.241542+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:35:54.241542+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 3665, 'log_dups_size': 0, 'ondisk_log_size': 3665, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30023680, 'num_objects': 3665, 'num_object_clones': 0, 'num_object_copies': 14660, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3665, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3665, 'num_write_kb': 29320, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1415}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:35:57.526 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:35:57.527 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'3598", 'reported_seq': 3662, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:55.342353+0000', 'last_change': '2024-09-05T16:34:56.903436+0000', 'last_active': '2024-09-05T16:35:55.342353+0000', 'last_peered': '2024-09-05T16:35:55.342353+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:35:55.342353+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 3598, 'log_dups_size': 0, 'ondisk_log_size': 3598, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29422752, 'num_objects': 3592, 'num_object_clones': 0, 'num_object_copies': 14368, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2149, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3592, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3596, 'num_write_kb': 28748, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1445}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:35:57.527 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-05T16:35:57.527 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'3656", 'reported_seq': 3719, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:55.299919+0000', 'last_change': '2024-09-05T16:34:56.883319+0000', 'last_active': '2024-09-05T16:35:55.299919+0000', 'last_peered': '2024-09-05T16:35:55.299919+0000', 'last_clean': '2024-09-05T16:33:53.505305+0000', 'last_became_active': '2024-09-05T16:34:56.873389+0000', 'last_became_peered': '2024-09-05T16:34:56.873389+0000', 'last_unstale': '2024-09-05T16:35:55.299919+0000', 'last_undegraded': '2024-09-05T16:34:56.602567+0000', 'last_fullsized': '2024-09-05T16:34:56.559958+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 3656, 'log_dups_size': 0, 'ondisk_log_size': 3656, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29906080, 'num_objects': 3651, 'num_object_clones': 0, 'num_object_copies': 14604, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2241, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3651, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3654, 'num_write_kb': 29220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1412}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:35:57.527 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:35:57.527 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'3666", 'reported_seq': 3721, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:54.363909+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:35:54.363909+0000', 'last_peered': '2024-09-05T16:35:54.363909+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:35:54.363909+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 3666, 'log_dups_size': 0, 'ondisk_log_size': 3666, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29960512, 'num_objects': 3658, 'num_object_clones': 0, 'num_object_copies': 14632, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2264, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3658, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3663, 'num_write_kb': 29281, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1397}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:35:57.527 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:35:57.527 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'3740", 'reported_seq': 3817, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:55.264682+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:35:55.264682+0000', 'last_peered': '2024-09-05T16:35:55.264682+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:35:55.264682+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 3740, 'log_dups_size': 0, 'ondisk_log_size': 3740, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30618784, 'num_objects': 3738, 'num_object_clones': 0, 'num_object_copies': 14952, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2264, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3738, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3738, 'num_write_kb': 29902, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1476}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:35:57.527 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:35:57.527 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'3730", 'reported_seq': 3785, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:54.249396+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:35:54.249396+0000', 'last_peered': '2024-09-05T16:35:54.249396+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:35:54.249396+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 3730, 'log_dups_size': 0, 'ondisk_log_size': 3730, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30510832, 'num_objects': 3725, 'num_object_clones': 0, 'num_object_copies': 14900, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2314, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3725, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3727, 'num_write_kb': 29804, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1414}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:35:57.527 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:35:57.527 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'3598", 'reported_seq': 3633, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:54.358646+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:35:54.358646+0000', 'last_peered': '2024-09-05T16:35:54.358646+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:35:54.358646+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 3598, 'log_dups_size': 0, 'ondisk_log_size': 3598, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29465168, 'num_objects': 3597, 'num_object_clones': 0, 'num_object_copies': 14388, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2265, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3597, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3597, 'num_write_kb': 28775, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1333}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:35:57.528 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-05T16:35:57.528 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'3682", 'reported_seq': 3755, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:55.204163+0000', 'last_change': '2024-09-05T16:34:56.798081+0000', 'last_active': '2024-09-05T16:35:55.204163+0000', 'last_peered': '2024-09-05T16:35:55.204163+0000', 'last_clean': '2024-09-05T16:33:45.082039+0000', 'last_became_active': '2024-09-05T16:34:56.792386+0000', 'last_became_peered': '2024-09-05T16:34:56.792386+0000', 'last_unstale': '2024-09-05T16:35:55.204163+0000', 'last_undegraded': '2024-09-05T16:34:56.600595+0000', 'last_fullsized': '2024-09-05T16:34:56.558203+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 3682, 'log_dups_size': 0, 'ondisk_log_size': 3682, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30153296, 'num_objects': 3681, 'num_object_clones': 0, 'num_object_copies': 14724, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2295, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3681, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3681, 'num_write_kb': 29447, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1387}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:35:57.528 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:35:57.528 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'3764", 'reported_seq': 3815, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:54.365934+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:35:54.365934+0000', 'last_peered': '2024-09-05T16:35:54.365934+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:35:54.365934+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 3764, 'log_dups_size': 0, 'ondisk_log_size': 3764, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30799008, 'num_objects': 3760, 'num_object_clones': 0, 'num_object_copies': 15040, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2359, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3760, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3762, 'num_write_kb': 30092, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1403}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:35:57.528 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:35:57.528 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'3726", 'reported_seq': 3803, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:55.290328+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:35:55.290328+0000', 'last_peered': '2024-09-05T16:35:55.290328+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:35:55.290328+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 3726, 'log_dups_size': 0, 'ondisk_log_size': 3726, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30495904, 'num_objects': 3723, 'num_object_clones': 0, 'num_object_copies': 14892, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2322, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3723, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3724, 'num_write_kb': 29789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1403}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:35:57.528 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:35:57.528 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'3666", 'reported_seq': 3729, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:55.375680+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:35:55.375680+0000', 'last_peered': '2024-09-05T16:35:55.375680+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:35:55.375680+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 3666, 'log_dups_size': 0, 'ondisk_log_size': 3666, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29988000, 'num_objects': 3661, 'num_object_clones': 0, 'num_object_copies': 14644, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3661, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3664, 'num_write_kb': 29300, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1424}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:35:57.529 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:35:58.256 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:35:58.257 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 35350/234540 objects degraded (15.072%), 16 pgs degraded, 16 pgs undersized","count":32},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":161,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":58635,"data_bytes":480297152,"bytes_used":1501114368,"bytes_avail":385045942272,"bytes_total":386547056640,"degraded_objects":35350,"degraded_total":234540,"degraded_ratio":0.15072055939285409,"recovering_objects_per_sec":17,"recovering_bytes_per_sec":147397,"recovering_keys_per_sec":0,"num_objects_recovered":216,"num_bytes_recovered":1769472,"num_keys_recovered":0,"write_bytes_sec":3139031,"read_op_per_sec":0,"write_op_per_sec":383},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:35:58.283 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:35:58.283 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:35:58.509 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 96 20 59813 59793 4.86496 2.51562 0.0277992 0.0321027 2024-09-05T16:35:58.912 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:35:58.913 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:35:58.941 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":105,"stamp":"2024-09-05T16:35:58.816614+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":480297152,"num_objects":58635,"num_object_clones":0,"num_object_copies":234540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35350,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58635,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":58662,"num_write_kb":469178,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":873,"num_bytes_recovered":6930432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":58689,"ondisk_log_size":58689,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1465932,"kb_used_data":794120,"kb_used_omap":14,"kb_used_meta":671729,"kb_avail":376021428,"statfs":{"total":386547056640,"available":385045942272,"internally_reserved":0,"allocated":813178880,"data_stored":812447352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":687851063},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,4,6,5,5,8,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":29245440,"num_objects":3570,"num_object_clones":0,"num_object_copies":14280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-145,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3570,"num_write_kb":28560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":145,"num_bytes_recovered":1187840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004651"},"pg_stats":[{"pgid":"1.f","version":"33'3648","reported_seq":3689,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:54.304485+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:35:54.304485+0000","last_peered":"2024-09-05T16:35:54.304485+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:35:54.304485+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3648,"log_dups_size":0,"ondisk_log_size":3648,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29884416,"num_objects":3648,"num_object_clones":0,"num_object_copies":14592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3648,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3648,"num_write_kb":29184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1386}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'3609","reported_seq":3672,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:55.375973+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:35:55.375973+0000","last_peered":"2024-09-05T16:35:55.375973+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:35:55.375973+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3609,"log_dups_size":0,"ondisk_log_size":3609,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29529248,"num_objects":3605,"num_object_clones":0,"num_object_copies":14420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3605,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3607,"num_write_kb":28852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1338}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'3682","reported_seq":3756,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:55.310356+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:35:55.310356+0000","last_peered":"2024-09-05T16:35:55.310356+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:35:55.310356+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3682,"log_dups_size":0,"ondisk_log_size":3682,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30136912,"num_objects":3679,"num_object_clones":0,"num_object_copies":14716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3679,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3681,"num_write_kb":29438,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1429}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'3702","reported_seq":5457,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:55.385312+0000","last_change":"2024-09-05T16:34:56.800371+0000","last_active":"2024-09-05T16:35:55.385312+0000","last_peered":"2024-09-05T16:35:55.385312+0000","last_clean":"2024-09-05T16:33:53.148717+0000","last_became_active":"2024-09-05T16:34:56.791755+0000","last_became_peered":"2024-09-05T16:34:56.791755+0000","last_unstale":"2024-09-05T16:35:55.385312+0000","last_undegraded":"2024-09-05T16:34:56.597522+0000","last_fullsized":"2024-09-05T16:34:56.568150+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3702,"log_dups_size":0,"ondisk_log_size":3702,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30291104,"num_objects":3698,"num_object_clones":0,"num_object_copies":14792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1442,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3698,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3700,"num_write_kb":29589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":848,"num_bytes_recovered":6930432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2256}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'3557","reported_seq":3610,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:54.381139+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:35:54.381139+0000","last_peered":"2024-09-05T16:35:54.381139+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:35:54.381139+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3557,"log_dups_size":0,"ondisk_log_size":3557,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29111456,"num_objects":3554,"num_object_clones":0,"num_object_copies":14216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3554,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3555,"num_write_kb":28437,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1392}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'3665","reported_seq":3706,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:54.241542+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:35:54.241542+0000","last_peered":"2024-09-05T16:35:54.241542+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:35:54.241542+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3665,"log_dups_size":0,"ondisk_log_size":3665,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30023680,"num_objects":3665,"num_object_clones":0,"num_object_copies":14660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3665,"num_write_kb":29320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1415}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'3598","reported_seq":3662,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:55.342353+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:35:55.342353+0000","last_peered":"2024-09-05T16:35:55.342353+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:35:55.342353+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3598,"log_dups_size":0,"ondisk_log_size":3598,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29422752,"num_objects":3592,"num_object_clones":0,"num_object_copies":14368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3592,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3596,"num_write_kb":28748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1445}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'3656","reported_seq":3719,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:55.299919+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:35:55.299919+0000","last_peered":"2024-09-05T16:35:55.299919+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:35:55.299919+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3656,"log_dups_size":0,"ondisk_log_size":3656,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29906080,"num_objects":3651,"num_object_clones":0,"num_object_copies":14604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3651,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3654,"num_write_kb":29220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1412}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'3666","reported_seq":3721,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:54.363909+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:35:54.363909+0000","last_peered":"2024-09-05T16:35:54.363909+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:35:54.363909+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3666,"log_dups_size":0,"ondisk_log_size":3666,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29960512,"num_objects":3658,"num_object_clones":0,"num_object_copies":14632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3658,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3663,"num_write_kb":29281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1397}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'3740","reported_seq":3817,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:55.264682+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:35:55.264682+0000","last_peered":"2024-09-05T16:35:55.264682+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:35:55.264682+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3740,"log_dups_size":0,"ondisk_log_size":3740,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30618784,"num_objects":3738,"num_object_clones":0,"num_object_copies":14952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3738,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3738,"num_write_kb":29902,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1476}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'3730","reported_seq":3785,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:54.249396+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:35:54.249396+0000","last_peered":"2024-09-05T16:35:54.249396+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:35:54.249396+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3730,"log_dups_size":0,"ondisk_log_size":3730,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30510832,"num_objects":3725,"num_object_clones":0,"num_object_copies":14900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3725,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3727,"num_write_kb":29804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1414}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'3598","reported_seq":3633,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:54.358646+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:35:54.358646+0000","last_peered":"2024-09-05T16:35:54.358646+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:35:54.358646+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3598,"log_dups_size":0,"ondisk_log_size":3598,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29465168,"num_objects":3597,"num_object_clones":0,"num_object_copies":14388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3597,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3597,"num_write_kb":28775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1333}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'3682","reported_seq":3755,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:55.204163+0000","last_change":"2024-09-05T16:34:56.798081+0000","last_active":"2024-09-05T16:35:55.204163+0000","last_peered":"2024-09-05T16:35:55.204163+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:35:55.204163+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3682,"log_dups_size":0,"ondisk_log_size":3682,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30153296,"num_objects":3681,"num_object_clones":0,"num_object_copies":14724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3681,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3681,"num_write_kb":29447,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1387}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'3764","reported_seq":3815,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:54.365934+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:35:54.365934+0000","last_peered":"2024-09-05T16:35:54.365934+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:35:54.365934+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3764,"log_dups_size":0,"ondisk_log_size":3764,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30799008,"num_objects":3760,"num_object_clones":0,"num_object_copies":15040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3760,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3762,"num_write_kb":30092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1403}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'3726","reported_seq":3803,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:55.290328+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:35:55.290328+0000","last_peered":"2024-09-05T16:35:55.290328+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:35:55.290328+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3726,"log_dups_size":0,"ondisk_log_size":3726,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30495904,"num_objects":3723,"num_object_clones":0,"num_object_copies":14892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3723,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3724,"num_write_kb":29789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1403}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'3666","reported_seq":3729,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:55.375680+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:35:55.375680+0000","last_peered":"2024-09-05T16:35:55.375680+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:35:55.375680+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3666,"log_dups_size":0,"ondisk_log_size":3666,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29988000,"num_objects":3661,"num_object_clones":0,"num_object_copies":14644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3661,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3664,"num_write_kb":29300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1424}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":480297152,"num_objects":58635,"num_object_clones":0,"num_object_copies":234540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35350,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58635,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":58662,"num_write_kb":469178,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":873,"num_bytes_recovered":6930432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":815132672,"data_stored":815132672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":58689,"ondisk_log_size":58689,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084312,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":409284,"kb_used_data":233204,"kb_used_omap":4,"kb_used_meta":176059,"kb_avail":93962556,"statfs":{"total":96636764160,"available":96217657344,"internally_reserved":0,"allocated":238800896,"data_stored":238618014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":180285139},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,0,1,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986189,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":209904,"kb_used_data":92124,"kb_used_omap":4,"kb_used_meta":117755,"kb_avail":94161936,"statfs":{"total":96636764160,"available":96421822464,"internally_reserved":0,"allocated":94334976,"data_stored":94152094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":120581842},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949672995,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":432200,"kb_used_data":234424,"kb_used_omap":1,"kb_used_meta":197758,"kb_avail":93939640,"statfs":{"total":96636764160,"available":96194191360,"internally_reserved":0,"allocated":240050176,"data_stored":239867294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":202504645},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182426,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":414544,"kb_used_data":234368,"kb_used_omap":4,"kb_used_meta":180155,"kb_avail":93957296,"statfs":{"total":96636764160,"available":96212271104,"internally_reserved":0,"allocated":239992832,"data_stored":239809950,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":184479437},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,2,2,3,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":240828416,"data_stored":240828416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":94326784,"data_stored":94326784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":240746496,"data_stored":240746496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":239230976,"data_stored":239230976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:35:59.513 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 97 20 60163 60143 4.843 2.73438 0.075284 0.0322494 2024-09-05T16:36:00.509 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 98 20 60524 60504 4.82236 2.82031 0.0569474 0.0323841 2024-09-05T16:36:01.509 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 99 20 60892 60872 4.80269 2.875 0.0751972 0.0325213 2024-09-05T16:36:01.943 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:36:02.509 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout:2024-09-05T16:36:02.517333+0000 min lat: 0.00357551 max lat: 2.29705 avg lat: 0.03266 2024-09-05T16:36:02.510 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-05T16:36:02.510 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 100 20 61241 61221 4.78193 2.72656 0.0452074 0.03266 2024-09-05T16:36:02.544 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:36:02.544 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:36:02.565 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":106,"stamp":"2024-09-05T16:36:00.817159+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":494403776,"num_objects":60357,"num_object_clones":0,"num_object_copies":241428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35274,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60357,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":60384,"num_write_kb":482954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":949,"num_bytes_recovered":7553024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":60411,"ondisk_log_size":60411,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1546492,"kb_used_data":821948,"kb_used_omap":14,"kb_used_meta":724465,"kb_avail":375940868,"statfs":{"total":386547056640,"available":384963448832,"internally_reserved":0,"allocated":841674752,"data_stored":840943224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":741852727},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[8,2,4,10,20,10,10,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":43352064,"num_objects":5292,"num_object_clones":0,"num_object_copies":21168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5292,"num_write_kb":42336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":221,"num_bytes_recovered":1810432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004451"},"pg_stats":[{"pgid":"1.f","version":"33'3776","reported_seq":3817,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:59.386183+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:35:59.386183+0000","last_peered":"2024-09-05T16:35:59.386183+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:35:59.386183+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3776,"log_dups_size":0,"ondisk_log_size":3776,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30932992,"num_objects":3776,"num_object_clones":0,"num_object_copies":15104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3776,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3776,"num_write_kb":30208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1514}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'3742","reported_seq":3805,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:00.472898+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:36:00.472898+0000","last_peered":"2024-09-05T16:36:00.472898+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:36:00.472898+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3742,"log_dups_size":0,"ondisk_log_size":3742,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30618784,"num_objects":3738,"num_object_clones":0,"num_object_copies":14952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3738,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3740,"num_write_kb":29916,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1471}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'3776","reported_seq":3851,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:00.364477+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:36:00.364477+0000","last_peered":"2024-09-05T16:36:00.364477+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:36:00.364477+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3776,"log_dups_size":0,"ondisk_log_size":3776,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30906960,"num_objects":3773,"num_object_clones":0,"num_object_copies":15092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3773,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3775,"num_write_kb":30190,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1523}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'3786","reported_seq":5694,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:00.442145+0000","last_change":"2024-09-05T16:34:56.800371+0000","last_active":"2024-09-05T16:36:00.442145+0000","last_peered":"2024-09-05T16:36:00.442145+0000","last_clean":"2024-09-05T16:33:53.148717+0000","last_became_active":"2024-09-05T16:34:56.791755+0000","last_became_peered":"2024-09-05T16:34:56.791755+0000","last_unstale":"2024-09-05T16:36:00.442145+0000","last_undegraded":"2024-09-05T16:34:56.597522+0000","last_fullsized":"2024-09-05T16:34:56.568150+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3786,"log_dups_size":0,"ondisk_log_size":3786,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30979232,"num_objects":3782,"num_object_clones":0,"num_object_copies":15128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1366,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3782,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3784,"num_write_kb":30261,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":924,"num_bytes_recovered":7553024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2416}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'3652","reported_seq":3705,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:59.232052+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:35:59.232052+0000","last_peered":"2024-09-05T16:35:59.232052+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:35:59.232052+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3652,"log_dups_size":0,"ondisk_log_size":3652,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29889696,"num_objects":3649,"num_object_clones":0,"num_object_copies":14596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3649,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3650,"num_write_kb":29197,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1487}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'3789","reported_seq":3830,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:59.355406+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:35:59.355406+0000","last_peered":"2024-09-05T16:35:59.355406+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:35:59.355406+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3789,"log_dups_size":0,"ondisk_log_size":3789,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31039488,"num_objects":3789,"num_object_clones":0,"num_object_copies":15156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3789,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3789,"num_write_kb":30312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1539}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'3704","reported_seq":3768,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:00.232101+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:36:00.232101+0000","last_peered":"2024-09-05T16:36:00.232101+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:36:00.232101+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3704,"log_dups_size":0,"ondisk_log_size":3704,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30291104,"num_objects":3698,"num_object_clones":0,"num_object_copies":14792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3698,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3702,"num_write_kb":29596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1551}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'3780","reported_seq":3843,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:00.326502+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:36:00.326502+0000","last_peered":"2024-09-05T16:36:00.326502+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:36:00.326502+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3780,"log_dups_size":0,"ondisk_log_size":3780,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30921888,"num_objects":3775,"num_object_clones":0,"num_object_copies":15100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3775,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3778,"num_write_kb":30212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1536}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'3768","reported_seq":3823,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:59.222328+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:35:59.222328+0000","last_peered":"2024-09-05T16:35:59.222328+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:35:59.222328+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3768,"log_dups_size":0,"ondisk_log_size":3768,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30796096,"num_objects":3760,"num_object_clones":0,"num_object_copies":15040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3760,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3765,"num_write_kb":30097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1499}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'3848","reported_seq":3925,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:00.335921+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:36:00.335921+0000","last_peered":"2024-09-05T16:36:00.335921+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:36:00.335921+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3848,"log_dups_size":0,"ondisk_log_size":3848,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31503520,"num_objects":3846,"num_object_clones":0,"num_object_copies":15384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3846,"num_write_kb":30766,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1584}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'3836","reported_seq":3891,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:59.420786+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:35:59.420786+0000","last_peered":"2024-09-05T16:35:59.420786+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:35:59.420786+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3836,"log_dups_size":0,"ondisk_log_size":3836,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31379184,"num_objects":3831,"num_object_clones":0,"num_object_copies":15324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3831,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3833,"num_write_kb":30652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1520}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'3690","reported_seq":3725,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:59.334871+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:35:59.334871+0000","last_peered":"2024-09-05T16:35:59.334871+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:35:59.334871+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3690,"log_dups_size":0,"ondisk_log_size":3690,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30218832,"num_objects":3689,"num_object_clones":0,"num_object_copies":14756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3689,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3689,"num_write_kb":29511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1425}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'3802","reported_seq":3875,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:00.316648+0000","last_change":"2024-09-05T16:34:56.798081+0000","last_active":"2024-09-05T16:36:00.316648+0000","last_peered":"2024-09-05T16:36:00.316648+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:36:00.316648+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3802,"log_dups_size":0,"ondisk_log_size":3802,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31136336,"num_objects":3801,"num_object_clones":0,"num_object_copies":15204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3801,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3801,"num_write_kb":30407,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1507}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'3865","reported_seq":3916,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:59.403106+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:35:59.403106+0000","last_peered":"2024-09-05T16:35:59.403106+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:35:59.403106+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3865,"log_dups_size":0,"ondisk_log_size":3865,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31626400,"num_objects":3861,"num_object_clones":0,"num_object_copies":15444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3861,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3863,"num_write_kb":30900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1504}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'3826","reported_seq":3903,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:00.250909+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:36:00.250909+0000","last_peered":"2024-09-05T16:36:00.250909+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:36:00.250909+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3826,"log_dups_size":0,"ondisk_log_size":3826,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31315104,"num_objects":3823,"num_object_clones":0,"num_object_copies":15292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3823,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3824,"num_write_kb":30589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1503}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'3771","reported_seq":3834,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:00.334477+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:36:00.334477+0000","last_peered":"2024-09-05T16:36:00.334477+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:36:00.334477+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3771,"log_dups_size":0,"ondisk_log_size":3771,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30848160,"num_objects":3766,"num_object_clones":0,"num_object_copies":15064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3766,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3769,"num_write_kb":30140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1529}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":494403776,"num_objects":60357,"num_object_clones":0,"num_object_copies":241428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35274,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60357,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":60384,"num_write_kb":482954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":949,"num_bytes_recovered":7553024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":843583488,"data_stored":843583488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":60411,"ondisk_log_size":60411,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084313,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":438336,"kb_used_data":240048,"kb_used_omap":4,"kb_used_meta":198267,"kb_avail":93933504,"statfs":{"total":96636764160,"available":96187908096,"internally_reserved":0,"allocated":245809152,"data_stored":245626270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":203026131},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,4,3,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986190,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":221252,"kb_used_data":99316,"kb_used_omap":4,"kb_used_meta":121915,"kb_avail":94150588,"statfs":{"total":96636764160,"available":96410202112,"internally_reserved":0,"allocated":101699584,"data_stored":101516702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":124841682},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,4,3,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949672996,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":443180,"kb_used_data":241244,"kb_used_omap":1,"kb_used_meta":201918,"kb_avail":93928660,"statfs":{"total":96636764160,"available":96182947840,"internally_reserved":0,"allocated":247033856,"data_stored":246850974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":206764485},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,1,7,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182427,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":443724,"kb_used_data":241340,"kb_used_omap":4,"kb_used_meta":202363,"kb_avail":93928116,"statfs":{"total":96636764160,"available":96182390784,"internally_reserved":0,"allocated":247132160,"data_stored":246949278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":207220429},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,3,5,6,2,3,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":247848960,"data_stored":247848960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":101728256,"data_stored":101728256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":247689216,"data_stored":247689216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":246317056,"data_stored":246317056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:36:02.566 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:36:03.165 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:36:03.165 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:36:03.189 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":107,"stamp":"2024-09-05T16:36:02.817797+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":494403776,"num_objects":60357,"num_object_clones":0,"num_object_copies":241428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35274,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60357,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":60384,"num_write_kb":482954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":949,"num_bytes_recovered":7553024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":60411,"ondisk_log_size":60411,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1546492,"kb_used_data":821948,"kb_used_omap":14,"kb_used_meta":724465,"kb_avail":375940868,"statfs":{"total":386547056640,"available":384963448832,"internally_reserved":0,"allocated":841674752,"data_stored":840943224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":741852727},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[8,2,4,10,20,10,10,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":28598272,"num_objects":3491,"num_object_clones":0,"num_object_copies":13964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-148,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3491,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3491,"num_write_kb":27928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":148,"num_bytes_recovered":1212416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004267"},"pg_stats":[{"pgid":"1.f","version":"33'3776","reported_seq":3817,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:59.386183+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:35:59.386183+0000","last_peered":"2024-09-05T16:35:59.386183+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:35:59.386183+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3776,"log_dups_size":0,"ondisk_log_size":3776,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30932992,"num_objects":3776,"num_object_clones":0,"num_object_copies":15104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3776,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3776,"num_write_kb":30208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1514}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'3742","reported_seq":3805,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:00.472898+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:36:00.472898+0000","last_peered":"2024-09-05T16:36:00.472898+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:36:00.472898+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3742,"log_dups_size":0,"ondisk_log_size":3742,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30618784,"num_objects":3738,"num_object_clones":0,"num_object_copies":14952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3738,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3740,"num_write_kb":29916,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1471}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'3776","reported_seq":3851,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:00.364477+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:36:00.364477+0000","last_peered":"2024-09-05T16:36:00.364477+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:36:00.364477+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3776,"log_dups_size":0,"ondisk_log_size":3776,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30906960,"num_objects":3773,"num_object_clones":0,"num_object_copies":15092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3773,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3775,"num_write_kb":30190,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1523}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'3786","reported_seq":5694,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:00.442145+0000","last_change":"2024-09-05T16:34:56.800371+0000","last_active":"2024-09-05T16:36:00.442145+0000","last_peered":"2024-09-05T16:36:00.442145+0000","last_clean":"2024-09-05T16:33:53.148717+0000","last_became_active":"2024-09-05T16:34:56.791755+0000","last_became_peered":"2024-09-05T16:34:56.791755+0000","last_unstale":"2024-09-05T16:36:00.442145+0000","last_undegraded":"2024-09-05T16:34:56.597522+0000","last_fullsized":"2024-09-05T16:34:56.568150+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3786,"log_dups_size":0,"ondisk_log_size":3786,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30979232,"num_objects":3782,"num_object_clones":0,"num_object_copies":15128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1366,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3782,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3784,"num_write_kb":30261,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":924,"num_bytes_recovered":7553024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2416}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'3652","reported_seq":3705,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:59.232052+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:35:59.232052+0000","last_peered":"2024-09-05T16:35:59.232052+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:35:59.232052+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3652,"log_dups_size":0,"ondisk_log_size":3652,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29889696,"num_objects":3649,"num_object_clones":0,"num_object_copies":14596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3649,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3650,"num_write_kb":29197,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1487}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'3789","reported_seq":3830,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:59.355406+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:35:59.355406+0000","last_peered":"2024-09-05T16:35:59.355406+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:35:59.355406+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3789,"log_dups_size":0,"ondisk_log_size":3789,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31039488,"num_objects":3789,"num_object_clones":0,"num_object_copies":15156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3789,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3789,"num_write_kb":30312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1539}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'3704","reported_seq":3768,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:00.232101+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:36:00.232101+0000","last_peered":"2024-09-05T16:36:00.232101+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:36:00.232101+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3704,"log_dups_size":0,"ondisk_log_size":3704,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30291104,"num_objects":3698,"num_object_clones":0,"num_object_copies":14792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3698,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3702,"num_write_kb":29596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1551}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'3780","reported_seq":3843,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:00.326502+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:36:00.326502+0000","last_peered":"2024-09-05T16:36:00.326502+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:36:00.326502+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3780,"log_dups_size":0,"ondisk_log_size":3780,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30921888,"num_objects":3775,"num_object_clones":0,"num_object_copies":15100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3775,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3778,"num_write_kb":30212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1536}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'3768","reported_seq":3823,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:59.222328+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:35:59.222328+0000","last_peered":"2024-09-05T16:35:59.222328+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:35:59.222328+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3768,"log_dups_size":0,"ondisk_log_size":3768,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30796096,"num_objects":3760,"num_object_clones":0,"num_object_copies":15040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3760,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3765,"num_write_kb":30097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1499}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'3848","reported_seq":3925,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:00.335921+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:36:00.335921+0000","last_peered":"2024-09-05T16:36:00.335921+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:36:00.335921+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3848,"log_dups_size":0,"ondisk_log_size":3848,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31503520,"num_objects":3846,"num_object_clones":0,"num_object_copies":15384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3846,"num_write_kb":30766,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1584}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'3836","reported_seq":3891,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:59.420786+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:35:59.420786+0000","last_peered":"2024-09-05T16:35:59.420786+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:35:59.420786+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3836,"log_dups_size":0,"ondisk_log_size":3836,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31379184,"num_objects":3831,"num_object_clones":0,"num_object_copies":15324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3831,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3833,"num_write_kb":30652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1520}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'3690","reported_seq":3725,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:59.334871+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:35:59.334871+0000","last_peered":"2024-09-05T16:35:59.334871+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:35:59.334871+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3690,"log_dups_size":0,"ondisk_log_size":3690,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30218832,"num_objects":3689,"num_object_clones":0,"num_object_copies":14756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3689,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3689,"num_write_kb":29511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1425}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'3802","reported_seq":3875,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:00.316648+0000","last_change":"2024-09-05T16:34:56.798081+0000","last_active":"2024-09-05T16:36:00.316648+0000","last_peered":"2024-09-05T16:36:00.316648+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:36:00.316648+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3802,"log_dups_size":0,"ondisk_log_size":3802,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31136336,"num_objects":3801,"num_object_clones":0,"num_object_copies":15204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3801,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3801,"num_write_kb":30407,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1507}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'3865","reported_seq":3916,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:59.403106+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:35:59.403106+0000","last_peered":"2024-09-05T16:35:59.403106+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:35:59.403106+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3865,"log_dups_size":0,"ondisk_log_size":3865,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31626400,"num_objects":3861,"num_object_clones":0,"num_object_copies":15444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3861,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3863,"num_write_kb":30900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1504}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'3826","reported_seq":3903,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:00.250909+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:36:00.250909+0000","last_peered":"2024-09-05T16:36:00.250909+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:36:00.250909+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3826,"log_dups_size":0,"ondisk_log_size":3826,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31315104,"num_objects":3823,"num_object_clones":0,"num_object_copies":15292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3823,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3824,"num_write_kb":30589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1503}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'3771","reported_seq":3834,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:00.334477+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:36:00.334477+0000","last_peered":"2024-09-05T16:36:00.334477+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:36:00.334477+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3771,"log_dups_size":0,"ondisk_log_size":3771,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30848160,"num_objects":3766,"num_object_clones":0,"num_object_copies":15064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3766,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3769,"num_write_kb":30140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1529}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":494403776,"num_objects":60357,"num_object_clones":0,"num_object_copies":241428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35274,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60357,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":60384,"num_write_kb":482954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":949,"num_bytes_recovered":7553024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":843583488,"data_stored":843583488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":60411,"ondisk_log_size":60411,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084313,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":438336,"kb_used_data":240048,"kb_used_omap":4,"kb_used_meta":198267,"kb_avail":93933504,"statfs":{"total":96636764160,"available":96187908096,"internally_reserved":0,"allocated":245809152,"data_stored":245626270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":203026131},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,4,3,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986190,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":221252,"kb_used_data":99316,"kb_used_omap":4,"kb_used_meta":121915,"kb_avail":94150588,"statfs":{"total":96636764160,"available":96410202112,"internally_reserved":0,"allocated":101699584,"data_stored":101516702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":124841682},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,4,3,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949672996,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":443180,"kb_used_data":241244,"kb_used_omap":1,"kb_used_meta":201918,"kb_avail":93928660,"statfs":{"total":96636764160,"available":96182947840,"internally_reserved":0,"allocated":247033856,"data_stored":246850974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":206764485},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,1,7,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182427,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":443724,"kb_used_data":241340,"kb_used_omap":4,"kb_used_meta":202363,"kb_avail":93928116,"statfs":{"total":96636764160,"available":96182390784,"internally_reserved":0,"allocated":247132160,"data_stored":246949278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":207220429},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,3,5,6,2,3,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":247848960,"data_stored":247848960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":101728256,"data_stored":101728256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":247689216,"data_stored":247689216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":246317056,"data_stored":246317056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:36:03.190 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-05T16:36:03.190 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'3776", 'reported_seq': 3817, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:59.386183+0000', 'last_change': '2024-09-05T16:34:56.833497+0000', 'last_active': '2024-09-05T16:35:59.386183+0000', 'last_peered': '2024-09-05T16:35:59.386183+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.828289+0000', 'last_became_peered': '2024-09-05T16:34:56.828289+0000', 'last_unstale': '2024-09-05T16:35:59.386183+0000', 'last_undegraded': '2024-09-05T16:34:56.612160+0000', 'last_fullsized': '2024-09-05T16:34:56.557632+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 3776, 'log_dups_size': 0, 'ondisk_log_size': 3776, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30932992, 'num_objects': 3776, 'num_object_clones': 0, 'num_object_copies': 15104, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2262, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3776, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3776, 'num_write_kb': 30208, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1514}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:36:03.190 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-05T16:36:03.190 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'3742", 'reported_seq': 3805, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:00.472898+0000', 'last_change': '2024-09-05T16:34:56.825031+0000', 'last_active': '2024-09-05T16:36:00.472898+0000', 'last_peered': '2024-09-05T16:36:00.472898+0000', 'last_clean': '2024-09-05T16:33:53.200762+0000', 'last_became_active': '2024-09-05T16:34:56.820292+0000', 'last_became_peered': '2024-09-05T16:34:56.820292+0000', 'last_unstale': '2024-09-05T16:36:00.472898+0000', 'last_undegraded': '2024-09-05T16:34:56.605776+0000', 'last_fullsized': '2024-09-05T16:34:56.559383+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 3742, 'log_dups_size': 0, 'ondisk_log_size': 3742, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30618784, 'num_objects': 3738, 'num_object_clones': 0, 'num_object_copies': 14952, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2269, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3738, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3740, 'num_write_kb': 29916, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 1471}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:36:03.190 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:36:03.191 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'3776", 'reported_seq': 3851, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:00.364477+0000', 'last_change': '2024-09-05T16:34:56.867900+0000', 'last_active': '2024-09-05T16:36:00.364477+0000', 'last_peered': '2024-09-05T16:36:00.364477+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:36:00.364477+0000', 'last_undegraded': '2024-09-05T16:34:56.590972+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 3776, 'log_dups_size': 0, 'ondisk_log_size': 3776, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30906960, 'num_objects': 3773, 'num_object_clones': 0, 'num_object_copies': 15092, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2251, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3773, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3775, 'num_write_kb': 30190, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1523}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:36:03.191 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-05T16:36:03.191 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'3786", 'reported_seq': 5694, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:00.442145+0000', 'last_change': '2024-09-05T16:34:56.800371+0000', 'last_active': '2024-09-05T16:36:00.442145+0000', 'last_peered': '2024-09-05T16:36:00.442145+0000', 'last_clean': '2024-09-05T16:33:53.148717+0000', 'last_became_active': '2024-09-05T16:34:56.791755+0000', 'last_became_peered': '2024-09-05T16:34:56.791755+0000', 'last_unstale': '2024-09-05T16:36:00.442145+0000', 'last_undegraded': '2024-09-05T16:34:56.597522+0000', 'last_fullsized': '2024-09-05T16:34:56.568150+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 3786, 'log_dups_size': 0, 'ondisk_log_size': 3786, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30979232, 'num_objects': 3782, 'num_object_clones': 0, 'num_object_copies': 15128, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1366, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3782, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3784, 'num_write_kb': 30261, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 924, 'num_bytes_recovered': 7553024, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2416}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:36:03.191 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:36:03.191 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'3652", 'reported_seq': 3705, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:59.232052+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:35:59.232052+0000', 'last_peered': '2024-09-05T16:35:59.232052+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:35:59.232052+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 3652, 'log_dups_size': 0, 'ondisk_log_size': 3652, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29889696, 'num_objects': 3649, 'num_object_clones': 0, 'num_object_copies': 14596, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2164, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3649, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3650, 'num_write_kb': 29197, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1487}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:36:03.191 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:36:03.191 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'3789", 'reported_seq': 3830, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:59.355406+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:35:59.355406+0000', 'last_peered': '2024-09-05T16:35:59.355406+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:35:59.355406+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 3789, 'log_dups_size': 0, 'ondisk_log_size': 3789, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31039488, 'num_objects': 3789, 'num_object_clones': 0, 'num_object_copies': 15156, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3789, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3789, 'num_write_kb': 30312, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1539}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:36:03.191 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:36:03.191 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'3704", 'reported_seq': 3768, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:00.232101+0000', 'last_change': '2024-09-05T16:34:56.903436+0000', 'last_active': '2024-09-05T16:36:00.232101+0000', 'last_peered': '2024-09-05T16:36:00.232101+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:36:00.232101+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 3704, 'log_dups_size': 0, 'ondisk_log_size': 3704, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30291104, 'num_objects': 3698, 'num_object_clones': 0, 'num_object_copies': 14792, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2149, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3698, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3702, 'num_write_kb': 29596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1551}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:36:03.191 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-05T16:36:03.191 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'3780", 'reported_seq': 3843, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:00.326502+0000', 'last_change': '2024-09-05T16:34:56.883319+0000', 'last_active': '2024-09-05T16:36:00.326502+0000', 'last_peered': '2024-09-05T16:36:00.326502+0000', 'last_clean': '2024-09-05T16:33:53.505305+0000', 'last_became_active': '2024-09-05T16:34:56.873389+0000', 'last_became_peered': '2024-09-05T16:34:56.873389+0000', 'last_unstale': '2024-09-05T16:36:00.326502+0000', 'last_undegraded': '2024-09-05T16:34:56.602567+0000', 'last_fullsized': '2024-09-05T16:34:56.559958+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 3780, 'log_dups_size': 0, 'ondisk_log_size': 3780, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30921888, 'num_objects': 3775, 'num_object_clones': 0, 'num_object_copies': 15100, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2241, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3775, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3778, 'num_write_kb': 30212, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1536}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:36:03.192 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:36:03.192 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'3768", 'reported_seq': 3823, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:59.222328+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:35:59.222328+0000', 'last_peered': '2024-09-05T16:35:59.222328+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:35:59.222328+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 3768, 'log_dups_size': 0, 'ondisk_log_size': 3768, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30796096, 'num_objects': 3760, 'num_object_clones': 0, 'num_object_copies': 15040, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2264, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3760, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3765, 'num_write_kb': 30097, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1499}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:36:03.192 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:36:03.192 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'3848", 'reported_seq': 3925, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:00.335921+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:36:00.335921+0000', 'last_peered': '2024-09-05T16:36:00.335921+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:36:00.335921+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 3848, 'log_dups_size': 0, 'ondisk_log_size': 3848, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31503520, 'num_objects': 3846, 'num_object_clones': 0, 'num_object_copies': 15384, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2264, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3846, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3846, 'num_write_kb': 30766, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1584}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:36:03.192 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:36:03.192 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'3836", 'reported_seq': 3891, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:59.420786+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:35:59.420786+0000', 'last_peered': '2024-09-05T16:35:59.420786+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:35:59.420786+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 3836, 'log_dups_size': 0, 'ondisk_log_size': 3836, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31379184, 'num_objects': 3831, 'num_object_clones': 0, 'num_object_copies': 15324, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2314, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3831, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3833, 'num_write_kb': 30652, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1520}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:36:03.192 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:36:03.192 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'3690", 'reported_seq': 3725, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:59.334871+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:35:59.334871+0000', 'last_peered': '2024-09-05T16:35:59.334871+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:35:59.334871+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 3690, 'log_dups_size': 0, 'ondisk_log_size': 3690, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30218832, 'num_objects': 3689, 'num_object_clones': 0, 'num_object_copies': 14756, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2265, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3689, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3689, 'num_write_kb': 29511, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1425}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:36:03.192 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-05T16:36:03.192 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'3802", 'reported_seq': 3875, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:00.316648+0000', 'last_change': '2024-09-05T16:34:56.798081+0000', 'last_active': '2024-09-05T16:36:00.316648+0000', 'last_peered': '2024-09-05T16:36:00.316648+0000', 'last_clean': '2024-09-05T16:33:45.082039+0000', 'last_became_active': '2024-09-05T16:34:56.792386+0000', 'last_became_peered': '2024-09-05T16:34:56.792386+0000', 'last_unstale': '2024-09-05T16:36:00.316648+0000', 'last_undegraded': '2024-09-05T16:34:56.600595+0000', 'last_fullsized': '2024-09-05T16:34:56.558203+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 3802, 'log_dups_size': 0, 'ondisk_log_size': 3802, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31136336, 'num_objects': 3801, 'num_object_clones': 0, 'num_object_copies': 15204, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2295, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3801, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3801, 'num_write_kb': 30407, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1507}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:36:03.193 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:36:03.193 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'3865", 'reported_seq': 3916, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:35:59.403106+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:35:59.403106+0000', 'last_peered': '2024-09-05T16:35:59.403106+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:35:59.403106+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 3865, 'log_dups_size': 0, 'ondisk_log_size': 3865, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31626400, 'num_objects': 3861, 'num_object_clones': 0, 'num_object_copies': 15444, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2359, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3861, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3863, 'num_write_kb': 30900, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1504}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:36:03.193 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:36:03.193 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'3826", 'reported_seq': 3903, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:00.250909+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:36:00.250909+0000', 'last_peered': '2024-09-05T16:36:00.250909+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:36:00.250909+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 3826, 'log_dups_size': 0, 'ondisk_log_size': 3826, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31315104, 'num_objects': 3823, 'num_object_clones': 0, 'num_object_copies': 15292, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2322, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3823, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3824, 'num_write_kb': 30589, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1503}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:36:03.193 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:36:03.193 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'3771", 'reported_seq': 3834, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:00.334477+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:36:00.334477+0000', 'last_peered': '2024-09-05T16:36:00.334477+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:36:00.334477+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 3771, 'log_dups_size': 0, 'ondisk_log_size': 3771, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30848160, 'num_objects': 3766, 'num_object_clones': 0, 'num_object_copies': 15064, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3766, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3769, 'num_write_kb': 30140, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1529}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:36:03.193 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:36:03.509 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 101 20 61548 61528 4.75833 2.39844 0.0541164 0.0328183 2024-09-05T16:36:03.916 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:36:03.916 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 35274/241428 objects degraded (14.611%), 16 pgs degraded, 16 pgs undersized","count":32},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":167,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":60357,"data_bytes":494403776,"bytes_used":1583607808,"bytes_avail":384963448832,"bytes_total":386547056640,"degraded_objects":35274,"degraded_total":241428,"degraded_ratio":0.14610567125602664,"recovering_objects_per_sec":12,"recovering_bytes_per_sec":100998,"recovering_keys_per_sec":0,"num_objects_recovered":148,"num_bytes_recovered":1212416,"num_keys_recovered":0,"write_bytes_sec":2382342,"read_op_per_sec":0,"write_op_per_sec":290},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:36:03.941 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:36:03.942 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:36:04.509 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 102 20 61868 61848 4.73619 2.5 0.0605035 0.0329706 2024-09-05T16:36:04.551 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:36:04.551 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:36:04.572 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":107,"stamp":"2024-09-05T16:36:02.817797+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":494403776,"num_objects":60357,"num_object_clones":0,"num_object_copies":241428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35274,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60357,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":60384,"num_write_kb":482954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":949,"num_bytes_recovered":7553024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":60411,"ondisk_log_size":60411,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1546492,"kb_used_data":821948,"kb_used_omap":14,"kb_used_meta":724465,"kb_avail":375940868,"statfs":{"total":386547056640,"available":384963448832,"internally_reserved":0,"allocated":841674752,"data_stored":840943224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":741852727},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[8,2,4,10,20,10,10,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":28598272,"num_objects":3491,"num_object_clones":0,"num_object_copies":13964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-148,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3491,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3491,"num_write_kb":27928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":148,"num_bytes_recovered":1212416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004267"},"pg_stats":[{"pgid":"1.f","version":"33'3776","reported_seq":3817,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:59.386183+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:35:59.386183+0000","last_peered":"2024-09-05T16:35:59.386183+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:35:59.386183+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3776,"log_dups_size":0,"ondisk_log_size":3776,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30932992,"num_objects":3776,"num_object_clones":0,"num_object_copies":15104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3776,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3776,"num_write_kb":30208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1514}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'3742","reported_seq":3805,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:00.472898+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:36:00.472898+0000","last_peered":"2024-09-05T16:36:00.472898+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:36:00.472898+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3742,"log_dups_size":0,"ondisk_log_size":3742,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30618784,"num_objects":3738,"num_object_clones":0,"num_object_copies":14952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3738,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3740,"num_write_kb":29916,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1471}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'3776","reported_seq":3851,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:00.364477+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:36:00.364477+0000","last_peered":"2024-09-05T16:36:00.364477+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:36:00.364477+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3776,"log_dups_size":0,"ondisk_log_size":3776,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30906960,"num_objects":3773,"num_object_clones":0,"num_object_copies":15092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3773,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3775,"num_write_kb":30190,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1523}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'3786","reported_seq":5694,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:00.442145+0000","last_change":"2024-09-05T16:34:56.800371+0000","last_active":"2024-09-05T16:36:00.442145+0000","last_peered":"2024-09-05T16:36:00.442145+0000","last_clean":"2024-09-05T16:33:53.148717+0000","last_became_active":"2024-09-05T16:34:56.791755+0000","last_became_peered":"2024-09-05T16:34:56.791755+0000","last_unstale":"2024-09-05T16:36:00.442145+0000","last_undegraded":"2024-09-05T16:34:56.597522+0000","last_fullsized":"2024-09-05T16:34:56.568150+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3786,"log_dups_size":0,"ondisk_log_size":3786,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30979232,"num_objects":3782,"num_object_clones":0,"num_object_copies":15128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1366,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3782,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3784,"num_write_kb":30261,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":924,"num_bytes_recovered":7553024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2416}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'3652","reported_seq":3705,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:59.232052+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:35:59.232052+0000","last_peered":"2024-09-05T16:35:59.232052+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:35:59.232052+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3652,"log_dups_size":0,"ondisk_log_size":3652,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29889696,"num_objects":3649,"num_object_clones":0,"num_object_copies":14596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3649,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3650,"num_write_kb":29197,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1487}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'3789","reported_seq":3830,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:59.355406+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:35:59.355406+0000","last_peered":"2024-09-05T16:35:59.355406+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:35:59.355406+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3789,"log_dups_size":0,"ondisk_log_size":3789,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31039488,"num_objects":3789,"num_object_clones":0,"num_object_copies":15156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3789,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3789,"num_write_kb":30312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1539}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'3704","reported_seq":3768,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:00.232101+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:36:00.232101+0000","last_peered":"2024-09-05T16:36:00.232101+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:36:00.232101+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3704,"log_dups_size":0,"ondisk_log_size":3704,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30291104,"num_objects":3698,"num_object_clones":0,"num_object_copies":14792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3698,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3702,"num_write_kb":29596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1551}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'3780","reported_seq":3843,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:00.326502+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:36:00.326502+0000","last_peered":"2024-09-05T16:36:00.326502+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:36:00.326502+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3780,"log_dups_size":0,"ondisk_log_size":3780,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30921888,"num_objects":3775,"num_object_clones":0,"num_object_copies":15100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3775,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3778,"num_write_kb":30212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1536}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'3768","reported_seq":3823,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:59.222328+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:35:59.222328+0000","last_peered":"2024-09-05T16:35:59.222328+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:35:59.222328+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3768,"log_dups_size":0,"ondisk_log_size":3768,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30796096,"num_objects":3760,"num_object_clones":0,"num_object_copies":15040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3760,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3765,"num_write_kb":30097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1499}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'3848","reported_seq":3925,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:00.335921+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:36:00.335921+0000","last_peered":"2024-09-05T16:36:00.335921+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:36:00.335921+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3848,"log_dups_size":0,"ondisk_log_size":3848,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31503520,"num_objects":3846,"num_object_clones":0,"num_object_copies":15384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3846,"num_write_kb":30766,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1584}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'3836","reported_seq":3891,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:59.420786+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:35:59.420786+0000","last_peered":"2024-09-05T16:35:59.420786+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:35:59.420786+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3836,"log_dups_size":0,"ondisk_log_size":3836,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31379184,"num_objects":3831,"num_object_clones":0,"num_object_copies":15324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3831,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3833,"num_write_kb":30652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1520}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'3690","reported_seq":3725,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:59.334871+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:35:59.334871+0000","last_peered":"2024-09-05T16:35:59.334871+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:35:59.334871+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3690,"log_dups_size":0,"ondisk_log_size":3690,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30218832,"num_objects":3689,"num_object_clones":0,"num_object_copies":14756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3689,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3689,"num_write_kb":29511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1425}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'3802","reported_seq":3875,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:00.316648+0000","last_change":"2024-09-05T16:34:56.798081+0000","last_active":"2024-09-05T16:36:00.316648+0000","last_peered":"2024-09-05T16:36:00.316648+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:36:00.316648+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3802,"log_dups_size":0,"ondisk_log_size":3802,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31136336,"num_objects":3801,"num_object_clones":0,"num_object_copies":15204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3801,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3801,"num_write_kb":30407,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1507}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'3865","reported_seq":3916,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:35:59.403106+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:35:59.403106+0000","last_peered":"2024-09-05T16:35:59.403106+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:35:59.403106+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3865,"log_dups_size":0,"ondisk_log_size":3865,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31626400,"num_objects":3861,"num_object_clones":0,"num_object_copies":15444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3861,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3863,"num_write_kb":30900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1504}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'3826","reported_seq":3903,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:00.250909+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:36:00.250909+0000","last_peered":"2024-09-05T16:36:00.250909+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:36:00.250909+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3826,"log_dups_size":0,"ondisk_log_size":3826,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31315104,"num_objects":3823,"num_object_clones":0,"num_object_copies":15292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3823,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3824,"num_write_kb":30589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1503}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'3771","reported_seq":3834,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:00.334477+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:36:00.334477+0000","last_peered":"2024-09-05T16:36:00.334477+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:36:00.334477+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3771,"log_dups_size":0,"ondisk_log_size":3771,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30848160,"num_objects":3766,"num_object_clones":0,"num_object_copies":15064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3766,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3769,"num_write_kb":30140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1529}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":494403776,"num_objects":60357,"num_object_clones":0,"num_object_copies":241428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35274,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60357,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":60384,"num_write_kb":482954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":949,"num_bytes_recovered":7553024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":843583488,"data_stored":843583488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":60411,"ondisk_log_size":60411,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084313,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":438336,"kb_used_data":240048,"kb_used_omap":4,"kb_used_meta":198267,"kb_avail":93933504,"statfs":{"total":96636764160,"available":96187908096,"internally_reserved":0,"allocated":245809152,"data_stored":245626270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":203026131},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,4,3,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986190,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":221252,"kb_used_data":99316,"kb_used_omap":4,"kb_used_meta":121915,"kb_avail":94150588,"statfs":{"total":96636764160,"available":96410202112,"internally_reserved":0,"allocated":101699584,"data_stored":101516702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":124841682},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,4,3,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949672996,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":443180,"kb_used_data":241244,"kb_used_omap":1,"kb_used_meta":201918,"kb_avail":93928660,"statfs":{"total":96636764160,"available":96182947840,"internally_reserved":0,"allocated":247033856,"data_stored":246850974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":206764485},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,1,7,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182427,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":443724,"kb_used_data":241340,"kb_used_omap":4,"kb_used_meta":202363,"kb_avail":93928116,"statfs":{"total":96636764160,"available":96182390784,"internally_reserved":0,"allocated":247132160,"data_stored":246949278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":207220429},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,3,5,6,2,3,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":247848960,"data_stored":247848960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":101728256,"data_stored":101728256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":247689216,"data_stored":247689216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":246317056,"data_stored":246317056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:36:05.510 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 103 20 62233 62213 4.71789 2.85156 0.0488015 0.0331075 2024-09-05T16:36:06.510 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 104 20 62598 62578 4.69994 2.85156 0.0880745 0.0332316 2024-09-05T16:36:07.510 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 105 20 62957 62937 4.68189 2.80469 0.0874654 0.0333584 2024-09-05T16:36:07.573 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:36:08.188 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:36:08.189 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:36:08.219 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":109,"stamp":"2024-09-05T16:36:06.819305+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":508436672,"num_objects":62070,"num_object_clones":0,"num_object_copies":248280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35207,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62070,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62097,"num_write_kb":496658,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1016,"num_bytes_recovered":8101888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":62124,"ondisk_log_size":62124,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1616112,"kb_used_data":849904,"kb_used_omap":14,"kb_used_meta":766129,"kb_avail":375871248,"statfs":{"total":386547056640,"available":384892157952,"internally_reserved":0,"allocated":870301696,"data_stored":869570168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":784516663},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,7,10,13,10,9],"upper_bound":128},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":36200448,"num_objects":4419,"num_object_clones":0,"num_object_copies":17676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4419,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4419,"num_write_kb":35352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":215,"num_bytes_recovered":1761280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004211"},"pg_stats":[{"pgid":"1.f","version":"33'3883","reported_seq":3924,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:04.386184+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:36:04.386184+0000","last_peered":"2024-09-05T16:36:04.386184+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:36:04.386184+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3883,"log_dups_size":0,"ondisk_log_size":3883,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31809536,"num_objects":3883,"num_object_clones":0,"num_object_copies":15532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3883,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3883,"num_write_kb":31064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1621}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'3845","reported_seq":3908,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:05.433300+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:36:05.433300+0000","last_peered":"2024-09-05T16:36:05.433300+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:36:05.433300+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3845,"log_dups_size":0,"ondisk_log_size":3845,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31462560,"num_objects":3841,"num_object_clones":0,"num_object_copies":15364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3841,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3843,"num_write_kb":30740,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1574}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'3886","reported_seq":3961,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:05.363702+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:36:05.363702+0000","last_peered":"2024-09-05T16:36:05.363702+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:36:05.363702+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3886,"log_dups_size":0,"ondisk_log_size":3886,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31808080,"num_objects":3883,"num_object_clones":0,"num_object_copies":15532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3883,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3885,"num_write_kb":31070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1633}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'3896","reported_seq":5938,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:05.484730+0000","last_change":"2024-09-05T16:34:56.800371+0000","last_active":"2024-09-05T16:36:05.484730+0000","last_peered":"2024-09-05T16:36:05.484730+0000","last_clean":"2024-09-05T16:33:53.148717+0000","last_became_active":"2024-09-05T16:34:56.791755+0000","last_became_peered":"2024-09-05T16:34:56.791755+0000","last_unstale":"2024-09-05T16:36:05.484730+0000","last_undegraded":"2024-09-05T16:34:56.597522+0000","last_fullsized":"2024-09-05T16:34:56.568150+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3896,"log_dups_size":0,"ondisk_log_size":3896,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31880352,"num_objects":3892,"num_object_clones":0,"num_object_copies":15568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1299,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3892,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3894,"num_write_kb":31141,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":991,"num_bytes_recovered":8101888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2593}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'3754","reported_seq":3807,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:04.338695+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:36:04.338695+0000","last_peered":"2024-09-05T16:36:04.338695+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:36:04.338695+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3754,"log_dups_size":0,"ondisk_log_size":3754,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30725280,"num_objects":3751,"num_object_clones":0,"num_object_copies":15004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3751,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3752,"num_write_kb":30013,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1589}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'3902","reported_seq":3943,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:04.405715+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:36:04.405715+0000","last_peered":"2024-09-05T16:36:04.405715+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:36:04.405715+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3902,"log_dups_size":0,"ondisk_log_size":3902,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31965184,"num_objects":3902,"num_object_clones":0,"num_object_copies":15608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3902,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3902,"num_write_kb":31216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1652}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'3805","reported_seq":3869,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:05.469424+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:36:05.469424+0000","last_peered":"2024-09-05T16:36:05.469424+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:36:05.469424+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3805,"log_dups_size":0,"ondisk_log_size":3805,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31118496,"num_objects":3799,"num_object_clones":0,"num_object_copies":15196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3799,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3803,"num_write_kb":30404,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1652}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'3896","reported_seq":3959,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:05.419806+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:36:05.419806+0000","last_peered":"2024-09-05T16:36:05.419806+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:36:05.419806+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3896,"log_dups_size":0,"ondisk_log_size":3896,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31872160,"num_objects":3891,"num_object_clones":0,"num_object_copies":15564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3891,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3894,"num_write_kb":31140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1652}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'3875","reported_seq":3930,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:04.432012+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:36:04.432012+0000","last_peered":"2024-09-05T16:36:04.432012+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:36:04.432012+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3875,"log_dups_size":0,"ondisk_log_size":3875,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31672640,"num_objects":3867,"num_object_clones":0,"num_object_copies":15468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3867,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3872,"num_write_kb":30953,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1606}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'3943","reported_seq":4020,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:05.383469+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:36:05.383469+0000","last_peered":"2024-09-05T16:36:05.383469+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:36:05.383469+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3943,"log_dups_size":0,"ondisk_log_size":3943,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32281760,"num_objects":3941,"num_object_clones":0,"num_object_copies":15764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3941,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3941,"num_write_kb":31526,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1679}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'3955","reported_seq":4010,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:04.416930+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:36:04.416930+0000","last_peered":"2024-09-05T16:36:04.416930+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:36:04.416930+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3955,"log_dups_size":0,"ondisk_log_size":3955,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32354032,"num_objects":3950,"num_object_clones":0,"num_object_copies":15800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3950,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3952,"num_write_kb":31604,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1639}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'3792","reported_seq":3824,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:04.448301+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:36:04.448301+0000","last_peered":"2024-09-05T16:36:04.448301+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:36:04.448301+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3792,"log_dups_size":0,"ondisk_log_size":3792,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31054416,"num_objects":3791,"num_object_clones":0,"num_object_copies":15164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3791,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3791,"num_write_kb":30327,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1527}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'3912","reported_seq":3985,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:05.262099+0000","last_change":"2024-09-05T16:34:56.798081+0000","last_active":"2024-09-05T16:36:05.262099+0000","last_peered":"2024-09-05T16:36:05.262099+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:36:05.262099+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3912,"log_dups_size":0,"ondisk_log_size":3912,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32037456,"num_objects":3911,"num_object_clones":0,"num_object_copies":15644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3911,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3911,"num_write_kb":31287,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1617}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'3970","reported_seq":4021,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:04.418944+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:36:04.418944+0000","last_peered":"2024-09-05T16:36:04.418944+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:36:04.418944+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3970,"log_dups_size":0,"ondisk_log_size":3970,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32486560,"num_objects":3966,"num_object_clones":0,"num_object_copies":15864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3966,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3968,"num_write_kb":31740,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1609}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'3932","reported_seq":4008,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:05.338649+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:36:05.338649+0000","last_peered":"2024-09-05T16:36:05.338649+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:36:05.338649+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3932,"log_dups_size":0,"ondisk_log_size":3932,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32183456,"num_objects":3929,"num_object_clones":0,"num_object_copies":15716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3929,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3930,"num_write_kb":31437,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1609}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'3878","reported_seq":3941,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:05.368520+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:36:05.368520+0000","last_peered":"2024-09-05T16:36:05.368520+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:36:05.368520+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3878,"log_dups_size":0,"ondisk_log_size":3878,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31724704,"num_objects":3873,"num_object_clones":0,"num_object_copies":15492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3873,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3876,"num_write_kb":30996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1636}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":508436672,"num_objects":62070,"num_object_clones":0,"num_object_copies":248280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35207,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62070,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62097,"num_write_kb":496658,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1016,"num_bytes_recovered":8101888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":872079360,"data_stored":872079360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":62124,"ondisk_log_size":62124,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084314,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":467520,"kb_used_data":247024,"kb_used_omap":4,"kb_used_meta":220475,"kb_avail":93904320,"statfs":{"total":96636764160,"available":96158023680,"internally_reserved":0,"allocated":252952576,"data_stored":252769694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":225767123},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,7,3,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986191,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":236880,"kb_used_data":106624,"kb_used_omap":4,"kb_used_meta":130235,"kb_avail":94134960,"statfs":{"total":96636764160,"available":96394199040,"internally_reserved":0,"allocated":109182976,"data_stored":109000094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":133361362},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,0,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949672997,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":455748,"kb_used_data":248244,"kb_used_omap":1,"kb_used_meta":207486,"kb_avail":93916092,"statfs":{"total":96636764160,"available":96170078208,"internally_reserved":0,"allocated":254201856,"data_stored":254018974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":212466117},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,1,3,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182428,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":455964,"kb_used_data":248012,"kb_used_omap":4,"kb_used_meta":207931,"kb_avail":93915876,"statfs":{"total":96636764160,"available":96169857024,"internally_reserved":0,"allocated":253964288,"data_stored":253781406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":212922061},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,3,4,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":254836736,"data_stored":254836736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":109117440,"data_stored":109117440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":254767104,"data_stored":254767104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":253358080,"data_stored":253358080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:36:08.220 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:36:08.511 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 106 20 63275 63255 4.66116 2.48438 0.0968686 0.0335069 2024-09-05T16:36:08.803 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:36:08.804 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:36:08.825 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":109,"stamp":"2024-09-05T16:36:06.819305+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":508436672,"num_objects":62070,"num_object_clones":0,"num_object_copies":248280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35207,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62070,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62097,"num_write_kb":496658,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1016,"num_bytes_recovered":8101888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":62124,"ondisk_log_size":62124,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1616112,"kb_used_data":849904,"kb_used_omap":14,"kb_used_meta":766129,"kb_avail":375871248,"statfs":{"total":386547056640,"available":384892157952,"internally_reserved":0,"allocated":870301696,"data_stored":869570168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":784516663},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,7,10,13,10,9],"upper_bound":128},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":36200448,"num_objects":4419,"num_object_clones":0,"num_object_copies":17676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4419,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4419,"num_write_kb":35352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":215,"num_bytes_recovered":1761280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004211"},"pg_stats":[{"pgid":"1.f","version":"33'3883","reported_seq":3924,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:04.386184+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:36:04.386184+0000","last_peered":"2024-09-05T16:36:04.386184+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:36:04.386184+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3883,"log_dups_size":0,"ondisk_log_size":3883,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31809536,"num_objects":3883,"num_object_clones":0,"num_object_copies":15532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3883,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3883,"num_write_kb":31064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1621}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'3845","reported_seq":3908,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:05.433300+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:36:05.433300+0000","last_peered":"2024-09-05T16:36:05.433300+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:36:05.433300+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3845,"log_dups_size":0,"ondisk_log_size":3845,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31462560,"num_objects":3841,"num_object_clones":0,"num_object_copies":15364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3841,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3843,"num_write_kb":30740,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1574}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'3886","reported_seq":3961,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:05.363702+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:36:05.363702+0000","last_peered":"2024-09-05T16:36:05.363702+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:36:05.363702+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3886,"log_dups_size":0,"ondisk_log_size":3886,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31808080,"num_objects":3883,"num_object_clones":0,"num_object_copies":15532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3883,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3885,"num_write_kb":31070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1633}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'3896","reported_seq":5938,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:05.484730+0000","last_change":"2024-09-05T16:34:56.800371+0000","last_active":"2024-09-05T16:36:05.484730+0000","last_peered":"2024-09-05T16:36:05.484730+0000","last_clean":"2024-09-05T16:33:53.148717+0000","last_became_active":"2024-09-05T16:34:56.791755+0000","last_became_peered":"2024-09-05T16:34:56.791755+0000","last_unstale":"2024-09-05T16:36:05.484730+0000","last_undegraded":"2024-09-05T16:34:56.597522+0000","last_fullsized":"2024-09-05T16:34:56.568150+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3896,"log_dups_size":0,"ondisk_log_size":3896,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31880352,"num_objects":3892,"num_object_clones":0,"num_object_copies":15568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1299,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3892,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3894,"num_write_kb":31141,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":991,"num_bytes_recovered":8101888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2593}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'3754","reported_seq":3807,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:04.338695+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:36:04.338695+0000","last_peered":"2024-09-05T16:36:04.338695+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:36:04.338695+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3754,"log_dups_size":0,"ondisk_log_size":3754,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30725280,"num_objects":3751,"num_object_clones":0,"num_object_copies":15004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3751,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3752,"num_write_kb":30013,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1589}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'3902","reported_seq":3943,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:04.405715+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:36:04.405715+0000","last_peered":"2024-09-05T16:36:04.405715+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:36:04.405715+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3902,"log_dups_size":0,"ondisk_log_size":3902,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31965184,"num_objects":3902,"num_object_clones":0,"num_object_copies":15608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3902,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3902,"num_write_kb":31216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1652}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'3805","reported_seq":3869,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:05.469424+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:36:05.469424+0000","last_peered":"2024-09-05T16:36:05.469424+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:36:05.469424+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3805,"log_dups_size":0,"ondisk_log_size":3805,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31118496,"num_objects":3799,"num_object_clones":0,"num_object_copies":15196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3799,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3803,"num_write_kb":30404,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1652}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'3896","reported_seq":3959,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:05.419806+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:36:05.419806+0000","last_peered":"2024-09-05T16:36:05.419806+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:36:05.419806+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3896,"log_dups_size":0,"ondisk_log_size":3896,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31872160,"num_objects":3891,"num_object_clones":0,"num_object_copies":15564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3891,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3894,"num_write_kb":31140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1652}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'3875","reported_seq":3930,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:04.432012+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:36:04.432012+0000","last_peered":"2024-09-05T16:36:04.432012+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:36:04.432012+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3875,"log_dups_size":0,"ondisk_log_size":3875,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31672640,"num_objects":3867,"num_object_clones":0,"num_object_copies":15468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3867,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3872,"num_write_kb":30953,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1606}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'3943","reported_seq":4020,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:05.383469+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:36:05.383469+0000","last_peered":"2024-09-05T16:36:05.383469+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:36:05.383469+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3943,"log_dups_size":0,"ondisk_log_size":3943,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32281760,"num_objects":3941,"num_object_clones":0,"num_object_copies":15764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3941,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3941,"num_write_kb":31526,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1679}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'3955","reported_seq":4010,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:04.416930+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:36:04.416930+0000","last_peered":"2024-09-05T16:36:04.416930+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:36:04.416930+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3955,"log_dups_size":0,"ondisk_log_size":3955,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32354032,"num_objects":3950,"num_object_clones":0,"num_object_copies":15800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3950,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3952,"num_write_kb":31604,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1639}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'3792","reported_seq":3824,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:04.448301+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:36:04.448301+0000","last_peered":"2024-09-05T16:36:04.448301+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:36:04.448301+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3792,"log_dups_size":0,"ondisk_log_size":3792,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31054416,"num_objects":3791,"num_object_clones":0,"num_object_copies":15164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3791,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3791,"num_write_kb":30327,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1527}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'3912","reported_seq":3985,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:05.262099+0000","last_change":"2024-09-05T16:34:56.798081+0000","last_active":"2024-09-05T16:36:05.262099+0000","last_peered":"2024-09-05T16:36:05.262099+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:36:05.262099+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3912,"log_dups_size":0,"ondisk_log_size":3912,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32037456,"num_objects":3911,"num_object_clones":0,"num_object_copies":15644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3911,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3911,"num_write_kb":31287,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1617}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'3970","reported_seq":4021,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:04.418944+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:36:04.418944+0000","last_peered":"2024-09-05T16:36:04.418944+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:36:04.418944+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3970,"log_dups_size":0,"ondisk_log_size":3970,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32486560,"num_objects":3966,"num_object_clones":0,"num_object_copies":15864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3966,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3968,"num_write_kb":31740,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1609}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'3932","reported_seq":4008,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:05.338649+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:36:05.338649+0000","last_peered":"2024-09-05T16:36:05.338649+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:36:05.338649+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3932,"log_dups_size":0,"ondisk_log_size":3932,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32183456,"num_objects":3929,"num_object_clones":0,"num_object_copies":15716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3929,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3930,"num_write_kb":31437,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1609}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'3878","reported_seq":3941,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:05.368520+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:36:05.368520+0000","last_peered":"2024-09-05T16:36:05.368520+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:36:05.368520+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3878,"log_dups_size":0,"ondisk_log_size":3878,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31724704,"num_objects":3873,"num_object_clones":0,"num_object_copies":15492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3873,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3876,"num_write_kb":30996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1636}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":508436672,"num_objects":62070,"num_object_clones":0,"num_object_copies":248280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35207,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62070,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62097,"num_write_kb":496658,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1016,"num_bytes_recovered":8101888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":872079360,"data_stored":872079360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":62124,"ondisk_log_size":62124,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084314,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":467520,"kb_used_data":247024,"kb_used_omap":4,"kb_used_meta":220475,"kb_avail":93904320,"statfs":{"total":96636764160,"available":96158023680,"internally_reserved":0,"allocated":252952576,"data_stored":252769694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":225767123},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,7,3,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986191,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":236880,"kb_used_data":106624,"kb_used_omap":4,"kb_used_meta":130235,"kb_avail":94134960,"statfs":{"total":96636764160,"available":96394199040,"internally_reserved":0,"allocated":109182976,"data_stored":109000094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":133361362},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,0,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949672997,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":455748,"kb_used_data":248244,"kb_used_omap":1,"kb_used_meta":207486,"kb_avail":93916092,"statfs":{"total":96636764160,"available":96170078208,"internally_reserved":0,"allocated":254201856,"data_stored":254018974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":212466117},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,1,3,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182428,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":455964,"kb_used_data":248012,"kb_used_omap":4,"kb_used_meta":207931,"kb_avail":93915876,"statfs":{"total":96636764160,"available":96169857024,"internally_reserved":0,"allocated":253964288,"data_stored":253781406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":212922061},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,3,4,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":254836736,"data_stored":254836736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":109117440,"data_stored":109117440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":254767104,"data_stored":254767104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":253358080,"data_stored":253358080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:36:08.826 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-05T16:36:08.826 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'3883", 'reported_seq': 3924, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:04.386184+0000', 'last_change': '2024-09-05T16:34:56.833497+0000', 'last_active': '2024-09-05T16:36:04.386184+0000', 'last_peered': '2024-09-05T16:36:04.386184+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.828289+0000', 'last_became_peered': '2024-09-05T16:34:56.828289+0000', 'last_unstale': '2024-09-05T16:36:04.386184+0000', 'last_undegraded': '2024-09-05T16:34:56.612160+0000', 'last_fullsized': '2024-09-05T16:34:56.557632+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 3883, 'log_dups_size': 0, 'ondisk_log_size': 3883, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31809536, 'num_objects': 3883, 'num_object_clones': 0, 'num_object_copies': 15532, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2262, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3883, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3883, 'num_write_kb': 31064, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1621}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:36:08.826 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-05T16:36:08.827 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'3845", 'reported_seq': 3908, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:05.433300+0000', 'last_change': '2024-09-05T16:34:56.825031+0000', 'last_active': '2024-09-05T16:36:05.433300+0000', 'last_peered': '2024-09-05T16:36:05.433300+0000', 'last_clean': '2024-09-05T16:33:53.200762+0000', 'last_became_active': '2024-09-05T16:34:56.820292+0000', 'last_became_peered': '2024-09-05T16:34:56.820292+0000', 'last_unstale': '2024-09-05T16:36:05.433300+0000', 'last_undegraded': '2024-09-05T16:34:56.605776+0000', 'last_fullsized': '2024-09-05T16:34:56.559383+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 3845, 'log_dups_size': 0, 'ondisk_log_size': 3845, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31462560, 'num_objects': 3841, 'num_object_clones': 0, 'num_object_copies': 15364, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2269, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3841, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3843, 'num_write_kb': 30740, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 1574}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:36:08.827 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:36:08.827 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'3886", 'reported_seq': 3961, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:05.363702+0000', 'last_change': '2024-09-05T16:34:56.867900+0000', 'last_active': '2024-09-05T16:36:05.363702+0000', 'last_peered': '2024-09-05T16:36:05.363702+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:36:05.363702+0000', 'last_undegraded': '2024-09-05T16:34:56.590972+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 3886, 'log_dups_size': 0, 'ondisk_log_size': 3886, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31808080, 'num_objects': 3883, 'num_object_clones': 0, 'num_object_copies': 15532, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2251, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3883, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3885, 'num_write_kb': 31070, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1633}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:36:08.827 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-05T16:36:08.827 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'3896", 'reported_seq': 5938, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:05.484730+0000', 'last_change': '2024-09-05T16:34:56.800371+0000', 'last_active': '2024-09-05T16:36:05.484730+0000', 'last_peered': '2024-09-05T16:36:05.484730+0000', 'last_clean': '2024-09-05T16:33:53.148717+0000', 'last_became_active': '2024-09-05T16:34:56.791755+0000', 'last_became_peered': '2024-09-05T16:34:56.791755+0000', 'last_unstale': '2024-09-05T16:36:05.484730+0000', 'last_undegraded': '2024-09-05T16:34:56.597522+0000', 'last_fullsized': '2024-09-05T16:34:56.568150+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 3896, 'log_dups_size': 0, 'ondisk_log_size': 3896, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31880352, 'num_objects': 3892, 'num_object_clones': 0, 'num_object_copies': 15568, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1299, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3892, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3894, 'num_write_kb': 31141, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 991, 'num_bytes_recovered': 8101888, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:36:08.828 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:36:08.828 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'3754", 'reported_seq': 3807, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:04.338695+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:36:04.338695+0000', 'last_peered': '2024-09-05T16:36:04.338695+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:36:04.338695+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 3754, 'log_dups_size': 0, 'ondisk_log_size': 3754, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30725280, 'num_objects': 3751, 'num_object_clones': 0, 'num_object_copies': 15004, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2164, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3751, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3752, 'num_write_kb': 30013, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1589}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:36:08.828 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:36:08.828 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'3902", 'reported_seq': 3943, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:04.405715+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:36:04.405715+0000', 'last_peered': '2024-09-05T16:36:04.405715+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:36:04.405715+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 3902, 'log_dups_size': 0, 'ondisk_log_size': 3902, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31965184, 'num_objects': 3902, 'num_object_clones': 0, 'num_object_copies': 15608, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3902, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3902, 'num_write_kb': 31216, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1652}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:36:08.828 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:36:08.828 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'3805", 'reported_seq': 3869, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:05.469424+0000', 'last_change': '2024-09-05T16:34:56.903436+0000', 'last_active': '2024-09-05T16:36:05.469424+0000', 'last_peered': '2024-09-05T16:36:05.469424+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:36:05.469424+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 3805, 'log_dups_size': 0, 'ondisk_log_size': 3805, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31118496, 'num_objects': 3799, 'num_object_clones': 0, 'num_object_copies': 15196, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2149, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3799, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3803, 'num_write_kb': 30404, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1652}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:36:08.828 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-05T16:36:08.828 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'3896", 'reported_seq': 3959, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:05.419806+0000', 'last_change': '2024-09-05T16:34:56.883319+0000', 'last_active': '2024-09-05T16:36:05.419806+0000', 'last_peered': '2024-09-05T16:36:05.419806+0000', 'last_clean': '2024-09-05T16:33:53.505305+0000', 'last_became_active': '2024-09-05T16:34:56.873389+0000', 'last_became_peered': '2024-09-05T16:34:56.873389+0000', 'last_unstale': '2024-09-05T16:36:05.419806+0000', 'last_undegraded': '2024-09-05T16:34:56.602567+0000', 'last_fullsized': '2024-09-05T16:34:56.559958+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 3896, 'log_dups_size': 0, 'ondisk_log_size': 3896, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31872160, 'num_objects': 3891, 'num_object_clones': 0, 'num_object_copies': 15564, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2241, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3891, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3894, 'num_write_kb': 31140, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1652}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:36:08.828 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:36:08.828 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'3875", 'reported_seq': 3930, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:04.432012+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:36:04.432012+0000', 'last_peered': '2024-09-05T16:36:04.432012+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:36:04.432012+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 3875, 'log_dups_size': 0, 'ondisk_log_size': 3875, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31672640, 'num_objects': 3867, 'num_object_clones': 0, 'num_object_copies': 15468, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2264, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3867, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3872, 'num_write_kb': 30953, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1606}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:36:08.829 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:36:08.829 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'3943", 'reported_seq': 4020, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:05.383469+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:36:05.383469+0000', 'last_peered': '2024-09-05T16:36:05.383469+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:36:05.383469+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 3943, 'log_dups_size': 0, 'ondisk_log_size': 3943, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32281760, 'num_objects': 3941, 'num_object_clones': 0, 'num_object_copies': 15764, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2264, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3941, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3941, 'num_write_kb': 31526, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1679}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:36:08.829 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:36:08.829 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'3955", 'reported_seq': 4010, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:04.416930+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:36:04.416930+0000', 'last_peered': '2024-09-05T16:36:04.416930+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:36:04.416930+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 3955, 'log_dups_size': 0, 'ondisk_log_size': 3955, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32354032, 'num_objects': 3950, 'num_object_clones': 0, 'num_object_copies': 15800, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2314, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3950, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3952, 'num_write_kb': 31604, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1639}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:36:08.829 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:36:08.829 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'3792", 'reported_seq': 3824, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:04.448301+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:36:04.448301+0000', 'last_peered': '2024-09-05T16:36:04.448301+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:36:04.448301+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 3792, 'log_dups_size': 0, 'ondisk_log_size': 3792, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31054416, 'num_objects': 3791, 'num_object_clones': 0, 'num_object_copies': 15164, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2265, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3791, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3791, 'num_write_kb': 30327, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1527}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:36:08.829 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-05T16:36:08.829 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'3912", 'reported_seq': 3985, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:05.262099+0000', 'last_change': '2024-09-05T16:34:56.798081+0000', 'last_active': '2024-09-05T16:36:05.262099+0000', 'last_peered': '2024-09-05T16:36:05.262099+0000', 'last_clean': '2024-09-05T16:33:45.082039+0000', 'last_became_active': '2024-09-05T16:34:56.792386+0000', 'last_became_peered': '2024-09-05T16:34:56.792386+0000', 'last_unstale': '2024-09-05T16:36:05.262099+0000', 'last_undegraded': '2024-09-05T16:34:56.600595+0000', 'last_fullsized': '2024-09-05T16:34:56.558203+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 3912, 'log_dups_size': 0, 'ondisk_log_size': 3912, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32037456, 'num_objects': 3911, 'num_object_clones': 0, 'num_object_copies': 15644, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2295, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3911, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3911, 'num_write_kb': 31287, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1617}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:36:08.829 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:36:08.829 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'3970", 'reported_seq': 4021, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:04.418944+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:36:04.418944+0000', 'last_peered': '2024-09-05T16:36:04.418944+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:36:04.418944+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 3970, 'log_dups_size': 0, 'ondisk_log_size': 3970, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32486560, 'num_objects': 3966, 'num_object_clones': 0, 'num_object_copies': 15864, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2359, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3966, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3968, 'num_write_kb': 31740, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1609}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:36:08.830 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:36:08.830 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'3932", 'reported_seq': 4008, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:05.338649+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:36:05.338649+0000', 'last_peered': '2024-09-05T16:36:05.338649+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:36:05.338649+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 3932, 'log_dups_size': 0, 'ondisk_log_size': 3932, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32183456, 'num_objects': 3929, 'num_object_clones': 0, 'num_object_copies': 15716, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2322, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3929, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3930, 'num_write_kb': 31437, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1609}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:36:08.830 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:36:08.830 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'3878", 'reported_seq': 3941, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:05.368520+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:36:05.368520+0000', 'last_peered': '2024-09-05T16:36:05.368520+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:36:05.368520+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 3878, 'log_dups_size': 0, 'ondisk_log_size': 3878, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31724704, 'num_objects': 3873, 'num_object_clones': 0, 'num_object_copies': 15492, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3873, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3876, 'num_write_kb': 30996, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1636}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:36:08.830 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:36:09.510 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 107 20 63600 63580 4.64133 2.53906 0.0432677 0.0336522 2024-09-05T16:36:09.595 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:36:09.596 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 35207/248280 objects degraded (14.180%), 16 pgs degraded, 16 pgs undersized","count":32},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":172,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":62070,"data_bytes":508436672,"bytes_used":1654898688,"bytes_avail":384892157952,"bytes_total":386547056640,"degraded_objects":35207,"degraded_total":248280,"degraded_ratio":0.14180360882874174,"recovering_objects_per_sec":11,"recovering_bytes_per_sec":97586,"recovering_keys_per_sec":0,"num_objects_recovered":143,"num_bytes_recovered":1171456,"num_keys_recovered":0,"write_bytes_sec":2344114,"read_op_per_sec":0,"write_op_per_sec":286},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:36:09.636 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:36:09.636 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:36:10.220 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:36:10.220 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:36:10.251 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":110,"stamp":"2024-09-05T16:36:08.820258+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":508436672,"num_objects":62070,"num_object_clones":0,"num_object_copies":248280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35207,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62070,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62097,"num_write_kb":496658,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1016,"num_bytes_recovered":8101888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":62124,"ondisk_log_size":62124,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1616112,"kb_used_data":849904,"kb_used_omap":14,"kb_used_meta":766129,"kb_avail":375871248,"statfs":{"total":386547056640,"available":384892157952,"internally_reserved":0,"allocated":870301696,"data_stored":869570168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":784516663},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,7,10,13,10,9],"upper_bound":128},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":28139520,"num_objects":3435,"num_object_clones":0,"num_object_copies":13740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3435,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3435,"num_write_kb":27480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":143,"num_bytes_recovered":1171456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004329"},"pg_stats":[{"pgid":"1.f","version":"33'3883","reported_seq":3924,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:04.386184+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:36:04.386184+0000","last_peered":"2024-09-05T16:36:04.386184+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:36:04.386184+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3883,"log_dups_size":0,"ondisk_log_size":3883,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31809536,"num_objects":3883,"num_object_clones":0,"num_object_copies":15532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3883,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3883,"num_write_kb":31064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1621}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'3845","reported_seq":3908,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:05.433300+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:36:05.433300+0000","last_peered":"2024-09-05T16:36:05.433300+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:36:05.433300+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3845,"log_dups_size":0,"ondisk_log_size":3845,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31462560,"num_objects":3841,"num_object_clones":0,"num_object_copies":15364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3841,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3843,"num_write_kb":30740,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1574}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'3886","reported_seq":3961,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:05.363702+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:36:05.363702+0000","last_peered":"2024-09-05T16:36:05.363702+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:36:05.363702+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3886,"log_dups_size":0,"ondisk_log_size":3886,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31808080,"num_objects":3883,"num_object_clones":0,"num_object_copies":15532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3883,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3885,"num_write_kb":31070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1633}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'3896","reported_seq":5938,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:05.484730+0000","last_change":"2024-09-05T16:34:56.800371+0000","last_active":"2024-09-05T16:36:05.484730+0000","last_peered":"2024-09-05T16:36:05.484730+0000","last_clean":"2024-09-05T16:33:53.148717+0000","last_became_active":"2024-09-05T16:34:56.791755+0000","last_became_peered":"2024-09-05T16:34:56.791755+0000","last_unstale":"2024-09-05T16:36:05.484730+0000","last_undegraded":"2024-09-05T16:34:56.597522+0000","last_fullsized":"2024-09-05T16:34:56.568150+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3896,"log_dups_size":0,"ondisk_log_size":3896,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31880352,"num_objects":3892,"num_object_clones":0,"num_object_copies":15568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1299,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3892,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3894,"num_write_kb":31141,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":991,"num_bytes_recovered":8101888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2593}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'3754","reported_seq":3807,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:04.338695+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:36:04.338695+0000","last_peered":"2024-09-05T16:36:04.338695+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:36:04.338695+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3754,"log_dups_size":0,"ondisk_log_size":3754,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30725280,"num_objects":3751,"num_object_clones":0,"num_object_copies":15004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3751,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3752,"num_write_kb":30013,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1589}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'3902","reported_seq":3943,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:04.405715+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:36:04.405715+0000","last_peered":"2024-09-05T16:36:04.405715+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:36:04.405715+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3902,"log_dups_size":0,"ondisk_log_size":3902,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31965184,"num_objects":3902,"num_object_clones":0,"num_object_copies":15608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3902,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3902,"num_write_kb":31216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1652}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'3805","reported_seq":3869,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:05.469424+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:36:05.469424+0000","last_peered":"2024-09-05T16:36:05.469424+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:36:05.469424+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3805,"log_dups_size":0,"ondisk_log_size":3805,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31118496,"num_objects":3799,"num_object_clones":0,"num_object_copies":15196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3799,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3803,"num_write_kb":30404,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1652}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'3896","reported_seq":3959,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:05.419806+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:36:05.419806+0000","last_peered":"2024-09-05T16:36:05.419806+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:36:05.419806+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3896,"log_dups_size":0,"ondisk_log_size":3896,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31872160,"num_objects":3891,"num_object_clones":0,"num_object_copies":15564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3891,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3894,"num_write_kb":31140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1652}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'3875","reported_seq":3930,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:04.432012+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:36:04.432012+0000","last_peered":"2024-09-05T16:36:04.432012+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:36:04.432012+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3875,"log_dups_size":0,"ondisk_log_size":3875,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31672640,"num_objects":3867,"num_object_clones":0,"num_object_copies":15468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3867,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3872,"num_write_kb":30953,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1606}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'3943","reported_seq":4020,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:05.383469+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:36:05.383469+0000","last_peered":"2024-09-05T16:36:05.383469+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:36:05.383469+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3943,"log_dups_size":0,"ondisk_log_size":3943,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32281760,"num_objects":3941,"num_object_clones":0,"num_object_copies":15764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3941,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3941,"num_write_kb":31526,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1679}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'3955","reported_seq":4010,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:04.416930+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:36:04.416930+0000","last_peered":"2024-09-05T16:36:04.416930+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:36:04.416930+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3955,"log_dups_size":0,"ondisk_log_size":3955,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32354032,"num_objects":3950,"num_object_clones":0,"num_object_copies":15800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3950,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3952,"num_write_kb":31604,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1639}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'3792","reported_seq":3824,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:04.448301+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:36:04.448301+0000","last_peered":"2024-09-05T16:36:04.448301+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:36:04.448301+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3792,"log_dups_size":0,"ondisk_log_size":3792,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31054416,"num_objects":3791,"num_object_clones":0,"num_object_copies":15164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3791,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3791,"num_write_kb":30327,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1527}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'3912","reported_seq":3985,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:05.262099+0000","last_change":"2024-09-05T16:34:56.798081+0000","last_active":"2024-09-05T16:36:05.262099+0000","last_peered":"2024-09-05T16:36:05.262099+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:36:05.262099+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3912,"log_dups_size":0,"ondisk_log_size":3912,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32037456,"num_objects":3911,"num_object_clones":0,"num_object_copies":15644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3911,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3911,"num_write_kb":31287,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1617}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'3970","reported_seq":4021,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:04.418944+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:36:04.418944+0000","last_peered":"2024-09-05T16:36:04.418944+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:36:04.418944+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3970,"log_dups_size":0,"ondisk_log_size":3970,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32486560,"num_objects":3966,"num_object_clones":0,"num_object_copies":15864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3966,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3968,"num_write_kb":31740,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1609}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'3932","reported_seq":4008,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:05.338649+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:36:05.338649+0000","last_peered":"2024-09-05T16:36:05.338649+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:36:05.338649+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3932,"log_dups_size":0,"ondisk_log_size":3932,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32183456,"num_objects":3929,"num_object_clones":0,"num_object_copies":15716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3929,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3930,"num_write_kb":31437,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1609}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'3878","reported_seq":3941,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:05.368520+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:36:05.368520+0000","last_peered":"2024-09-05T16:36:05.368520+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:36:05.368520+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3878,"log_dups_size":0,"ondisk_log_size":3878,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31724704,"num_objects":3873,"num_object_clones":0,"num_object_copies":15492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3873,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3876,"num_write_kb":30996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1636}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":508436672,"num_objects":62070,"num_object_clones":0,"num_object_copies":248280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35207,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62070,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62097,"num_write_kb":496658,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1016,"num_bytes_recovered":8101888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":872079360,"data_stored":872079360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":62124,"ondisk_log_size":62124,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084314,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":467520,"kb_used_data":247024,"kb_used_omap":4,"kb_used_meta":220475,"kb_avail":93904320,"statfs":{"total":96636764160,"available":96158023680,"internally_reserved":0,"allocated":252952576,"data_stored":252769694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":225767123},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,7,3,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986191,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":236880,"kb_used_data":106624,"kb_used_omap":4,"kb_used_meta":130235,"kb_avail":94134960,"statfs":{"total":96636764160,"available":96394199040,"internally_reserved":0,"allocated":109182976,"data_stored":109000094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":133361362},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,0,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949672997,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":455748,"kb_used_data":248244,"kb_used_omap":1,"kb_used_meta":207486,"kb_avail":93916092,"statfs":{"total":96636764160,"available":96170078208,"internally_reserved":0,"allocated":254201856,"data_stored":254018974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":212466117},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,1,3,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182428,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":455964,"kb_used_data":248012,"kb_used_omap":4,"kb_used_meta":207931,"kb_avail":93915876,"statfs":{"total":96636764160,"available":96169857024,"internally_reserved":0,"allocated":253964288,"data_stored":253781406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":212922061},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,3,4,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":254836736,"data_stored":254836736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":109117440,"data_stored":109117440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":254767104,"data_stored":254767104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":253358080,"data_stored":253358080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:36:10.510 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 108 20 63931 63911 4.6223 2.58594 0.0383644 0.0337885 2024-09-05T16:36:11.510 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 109 20 64290 64270 4.60562 2.80469 0.0573979 0.0339132 2024-09-05T16:36:12.510 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 110 20 64648 64628 4.58918 2.79688 0.0976648 0.034034 2024-09-05T16:36:13.253 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:36:13.512 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 111 20 64994 64974 4.57219 2.70312 0.075001 0.034155 2024-09-05T16:36:13.898 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:36:13.899 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:36:13.930 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":112,"stamp":"2024-09-05T16:36:12.822030+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":522600640,"num_objects":63799,"num_object_clones":0,"num_object_copies":255196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63799,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":63826,"num_write_kb":510490,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1087,"num_bytes_recovered":8683520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":63853,"ondisk_log_size":63853,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1635888,"kb_used_data":878000,"kb_used_omap":14,"kb_used_meta":757809,"kb_avail":375851472,"statfs":{"total":386547056640,"available":384871907328,"internally_reserved":0,"allocated":899072000,"data_stored":898340472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":775996983},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,3,5,16,10,6,9,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":28196864,"num_objects":3442,"num_object_clones":0,"num_object_copies":13768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3442,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3442,"num_write_kb":27536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":138,"num_bytes_recovered":1130496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004870"},"pg_stats":[{"pgid":"1.f","version":"33'3993","reported_seq":4034,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:09.394198+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:36:09.394198+0000","last_peered":"2024-09-05T16:36:09.394198+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:36:09.394198+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3993,"log_dups_size":0,"ondisk_log_size":3993,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32710656,"num_objects":3993,"num_object_clones":0,"num_object_copies":15972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3993,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3993,"num_write_kb":31944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1731}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'3946","reported_seq":4009,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:10.420005+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:36:10.420005+0000","last_peered":"2024-09-05T16:36:10.420005+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:36:10.420005+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3946,"log_dups_size":0,"ondisk_log_size":3946,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32289952,"num_objects":3942,"num_object_clones":0,"num_object_copies":15768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3942,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3944,"num_write_kb":31548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1675}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'3989","reported_seq":4064,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:10.431943+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:36:10.431943+0000","last_peered":"2024-09-05T16:36:10.431943+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:36:10.431943+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3989,"log_dups_size":0,"ondisk_log_size":3989,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32651856,"num_objects":3986,"num_object_clones":0,"num_object_copies":15944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":31894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1736}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'4015","reported_seq":6198,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:10.503771+0000","last_change":"2024-09-05T16:34:56.800371+0000","last_active":"2024-09-05T16:36:10.503771+0000","last_peered":"2024-09-05T16:36:10.503771+0000","last_clean":"2024-09-05T16:33:53.148717+0000","last_became_active":"2024-09-05T16:34:56.791755+0000","last_became_peered":"2024-09-05T16:34:56.791755+0000","last_unstale":"2024-09-05T16:36:10.503771+0000","last_undegraded":"2024-09-05T16:34:56.597522+0000","last_fullsized":"2024-09-05T16:34:56.568150+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4015,"log_dups_size":0,"ondisk_log_size":4015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32855200,"num_objects":4011,"num_object_clones":0,"num_object_copies":16044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4011,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4013,"num_write_kb":32093,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1062,"num_bytes_recovered":8683520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2783}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'3861","reported_seq":3914,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:09.370695+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:36:09.370695+0000","last_peered":"2024-09-05T16:36:09.370695+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:36:09.370695+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3861,"log_dups_size":0,"ondisk_log_size":3861,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31601824,"num_objects":3858,"num_object_clones":0,"num_object_copies":15432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3858,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":30869,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1696}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'3997","reported_seq":4038,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:09.382508+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:36:09.382508+0000","last_peered":"2024-09-05T16:36:09.382508+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:36:09.382508+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3997,"log_dups_size":0,"ondisk_log_size":3997,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32743424,"num_objects":3997,"num_object_clones":0,"num_object_copies":15988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3997,"num_write_kb":31976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1747}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'3915","reported_seq":3979,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:10.545177+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:36:10.545177+0000","last_peered":"2024-09-05T16:36:10.545177+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:36:10.545177+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3915,"log_dups_size":0,"ondisk_log_size":3915,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32019616,"num_objects":3909,"num_object_clones":0,"num_object_copies":15636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3909,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3913,"num_write_kb":31284,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1762}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'3996","reported_seq":4059,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:10.292158+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:36:10.292158+0000","last_peered":"2024-09-05T16:36:10.292158+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:36:10.292158+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3996,"log_dups_size":0,"ondisk_log_size":3996,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32691360,"num_objects":3991,"num_object_clones":0,"num_object_copies":15964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3991,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3994,"num_write_kb":31940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1752}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'3997","reported_seq":4052,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:09.403813+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:36:09.403813+0000","last_peered":"2024-09-05T16:36:09.403813+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:36:09.403813+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3997,"log_dups_size":0,"ondisk_log_size":3997,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32672064,"num_objects":3989,"num_object_clones":0,"num_object_copies":15956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3989,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3994,"num_write_kb":31929,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1728}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'4039","reported_seq":4116,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:10.354655+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:36:10.354655+0000","last_peered":"2024-09-05T16:36:10.354655+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:36:10.354655+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4039,"log_dups_size":0,"ondisk_log_size":4039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33068192,"num_objects":4037,"num_object_clones":0,"num_object_copies":16148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":32294,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1775}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'4082","reported_seq":4137,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:09.334987+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:36:09.334987+0000","last_peered":"2024-09-05T16:36:09.334987+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:36:09.334987+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4082,"log_dups_size":0,"ondisk_log_size":4082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33394416,"num_objects":4077,"num_object_clones":0,"num_object_copies":16308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4077,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4079,"num_write_kb":32620,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1766}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'3894","reported_seq":3929,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:09.484033+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:36:09.484033+0000","last_peered":"2024-09-05T16:36:09.484033+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:36:09.484033+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3894,"log_dups_size":0,"ondisk_log_size":3894,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31890000,"num_objects":3893,"num_object_clones":0,"num_object_copies":15572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3893,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3893,"num_write_kb":31143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1629}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'4016","reported_seq":4090,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:10.429538+0000","last_change":"2024-09-05T16:34:56.798081+0000","last_active":"2024-09-05T16:36:10.429538+0000","last_peered":"2024-09-05T16:36:10.429538+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:36:10.429538+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4016,"log_dups_size":0,"ondisk_log_size":4016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32889424,"num_objects":4015,"num_object_clones":0,"num_object_copies":16060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4015,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4015,"num_write_kb":32119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1721}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'4083","reported_seq":4134,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:09.385435+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:36:09.385435+0000","last_peered":"2024-09-05T16:36:09.385435+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:36:09.385435+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4083,"log_dups_size":0,"ondisk_log_size":4083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33412256,"num_objects":4079,"num_object_clones":0,"num_object_copies":16316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4079,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4081,"num_write_kb":32644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1722}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'4041","reported_seq":4116,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:10.451296+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:36:10.451296+0000","last_peered":"2024-09-05T16:36:10.451296+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:36:10.451296+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4041,"log_dups_size":0,"ondisk_log_size":4041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33076384,"num_objects":4038,"num_object_clones":0,"num_object_copies":16152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4038,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4039,"num_write_kb":32309,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1718}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'3989","reported_seq":4052,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:10.489978+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:36:10.489978+0000","last_peered":"2024-09-05T16:36:10.489978+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:36:10.489978+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3989,"log_dups_size":0,"ondisk_log_size":3989,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32634016,"num_objects":3984,"num_object_clones":0,"num_object_copies":15936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3984,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3987,"num_write_kb":31884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1747}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":522600640,"num_objects":63799,"num_object_clones":0,"num_object_copies":255196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63799,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":63826,"num_write_kb":510490,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1087,"num_bytes_recovered":8683520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":900694016,"data_stored":900694016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":63853,"ondisk_log_size":63853,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084315,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":466232,"kb_used_data":254056,"kb_used_omap":4,"kb_used_meta":212155,"kb_avail":93905608,"statfs":{"total":96636764160,"available":96159342592,"internally_reserved":0,"allocated":260153344,"data_stored":259970462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":217247443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,6,3,4,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986192,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":235732,"kb_used_data":113796,"kb_used_omap":4,"kb_used_meta":121915,"kb_avail":94136108,"statfs":{"total":96636764160,"available":96395374592,"internally_reserved":0,"allocated":116527104,"data_stored":116344222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":124841682},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,2,2,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949672998,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":466864,"kb_used_data":255200,"kb_used_omap":1,"kb_used_meta":211646,"kb_avail":93904976,"statfs":{"total":96636764160,"available":96158695424,"internally_reserved":0,"allocated":261324800,"data_stored":261141918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":216725957},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,5,4,1,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182429,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":467060,"kb_used_data":254948,"kb_used_omap":4,"kb_used_meta":212091,"kb_avail":93904780,"statfs":{"total":96636764160,"available":96158494720,"internally_reserved":0,"allocated":261066752,"data_stored":260883870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":217181901},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,0,3,0,0,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":261890048,"data_stored":261890048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":116514816,"data_stored":116514816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":261820416,"data_stored":261820416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":260468736,"data_stored":260468736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:36:13.931 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:36:14.510 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 112 20 65307 65287 4.5532 2.44531 0.0686917 0.034302 2024-09-05T16:36:14.551 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:36:14.551 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:36:14.580 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":112,"stamp":"2024-09-05T16:36:12.822030+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":522600640,"num_objects":63799,"num_object_clones":0,"num_object_copies":255196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63799,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":63826,"num_write_kb":510490,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1087,"num_bytes_recovered":8683520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":63853,"ondisk_log_size":63853,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1635888,"kb_used_data":878000,"kb_used_omap":14,"kb_used_meta":757809,"kb_avail":375851472,"statfs":{"total":386547056640,"available":384871907328,"internally_reserved":0,"allocated":899072000,"data_stored":898340472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":775996983},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,3,5,16,10,6,9,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":28196864,"num_objects":3442,"num_object_clones":0,"num_object_copies":13768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3442,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3442,"num_write_kb":27536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":138,"num_bytes_recovered":1130496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004870"},"pg_stats":[{"pgid":"1.f","version":"33'3993","reported_seq":4034,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:09.394198+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:36:09.394198+0000","last_peered":"2024-09-05T16:36:09.394198+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:36:09.394198+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3993,"log_dups_size":0,"ondisk_log_size":3993,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32710656,"num_objects":3993,"num_object_clones":0,"num_object_copies":15972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3993,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3993,"num_write_kb":31944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1731}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'3946","reported_seq":4009,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:10.420005+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:36:10.420005+0000","last_peered":"2024-09-05T16:36:10.420005+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:36:10.420005+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3946,"log_dups_size":0,"ondisk_log_size":3946,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32289952,"num_objects":3942,"num_object_clones":0,"num_object_copies":15768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3942,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3944,"num_write_kb":31548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1675}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'3989","reported_seq":4064,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:10.431943+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:36:10.431943+0000","last_peered":"2024-09-05T16:36:10.431943+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:36:10.431943+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3989,"log_dups_size":0,"ondisk_log_size":3989,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32651856,"num_objects":3986,"num_object_clones":0,"num_object_copies":15944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":31894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1736}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'4015","reported_seq":6198,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:10.503771+0000","last_change":"2024-09-05T16:34:56.800371+0000","last_active":"2024-09-05T16:36:10.503771+0000","last_peered":"2024-09-05T16:36:10.503771+0000","last_clean":"2024-09-05T16:33:53.148717+0000","last_became_active":"2024-09-05T16:34:56.791755+0000","last_became_peered":"2024-09-05T16:34:56.791755+0000","last_unstale":"2024-09-05T16:36:10.503771+0000","last_undegraded":"2024-09-05T16:34:56.597522+0000","last_fullsized":"2024-09-05T16:34:56.568150+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4015,"log_dups_size":0,"ondisk_log_size":4015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32855200,"num_objects":4011,"num_object_clones":0,"num_object_copies":16044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4011,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4013,"num_write_kb":32093,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1062,"num_bytes_recovered":8683520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2783}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'3861","reported_seq":3914,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:09.370695+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:36:09.370695+0000","last_peered":"2024-09-05T16:36:09.370695+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:36:09.370695+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3861,"log_dups_size":0,"ondisk_log_size":3861,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31601824,"num_objects":3858,"num_object_clones":0,"num_object_copies":15432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3858,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":30869,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1696}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'3997","reported_seq":4038,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:09.382508+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:36:09.382508+0000","last_peered":"2024-09-05T16:36:09.382508+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:36:09.382508+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3997,"log_dups_size":0,"ondisk_log_size":3997,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32743424,"num_objects":3997,"num_object_clones":0,"num_object_copies":15988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3997,"num_write_kb":31976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1747}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'3915","reported_seq":3979,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:10.545177+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:36:10.545177+0000","last_peered":"2024-09-05T16:36:10.545177+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:36:10.545177+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3915,"log_dups_size":0,"ondisk_log_size":3915,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32019616,"num_objects":3909,"num_object_clones":0,"num_object_copies":15636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3909,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3913,"num_write_kb":31284,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1762}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'3996","reported_seq":4059,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:10.292158+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:36:10.292158+0000","last_peered":"2024-09-05T16:36:10.292158+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:36:10.292158+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3996,"log_dups_size":0,"ondisk_log_size":3996,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32691360,"num_objects":3991,"num_object_clones":0,"num_object_copies":15964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3991,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3994,"num_write_kb":31940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1752}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'3997","reported_seq":4052,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:09.403813+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:36:09.403813+0000","last_peered":"2024-09-05T16:36:09.403813+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:36:09.403813+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3997,"log_dups_size":0,"ondisk_log_size":3997,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32672064,"num_objects":3989,"num_object_clones":0,"num_object_copies":15956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3989,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3994,"num_write_kb":31929,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1728}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'4039","reported_seq":4116,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:10.354655+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:36:10.354655+0000","last_peered":"2024-09-05T16:36:10.354655+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:36:10.354655+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4039,"log_dups_size":0,"ondisk_log_size":4039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33068192,"num_objects":4037,"num_object_clones":0,"num_object_copies":16148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":32294,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1775}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'4082","reported_seq":4137,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:09.334987+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:36:09.334987+0000","last_peered":"2024-09-05T16:36:09.334987+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:36:09.334987+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4082,"log_dups_size":0,"ondisk_log_size":4082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33394416,"num_objects":4077,"num_object_clones":0,"num_object_copies":16308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4077,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4079,"num_write_kb":32620,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1766}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'3894","reported_seq":3929,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:09.484033+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:36:09.484033+0000","last_peered":"2024-09-05T16:36:09.484033+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:36:09.484033+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3894,"log_dups_size":0,"ondisk_log_size":3894,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31890000,"num_objects":3893,"num_object_clones":0,"num_object_copies":15572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3893,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3893,"num_write_kb":31143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1629}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'4016","reported_seq":4090,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:10.429538+0000","last_change":"2024-09-05T16:34:56.798081+0000","last_active":"2024-09-05T16:36:10.429538+0000","last_peered":"2024-09-05T16:36:10.429538+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:36:10.429538+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4016,"log_dups_size":0,"ondisk_log_size":4016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32889424,"num_objects":4015,"num_object_clones":0,"num_object_copies":16060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4015,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4015,"num_write_kb":32119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1721}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'4083","reported_seq":4134,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:09.385435+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:36:09.385435+0000","last_peered":"2024-09-05T16:36:09.385435+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:36:09.385435+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4083,"log_dups_size":0,"ondisk_log_size":4083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33412256,"num_objects":4079,"num_object_clones":0,"num_object_copies":16316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4079,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4081,"num_write_kb":32644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1722}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'4041","reported_seq":4116,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:10.451296+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:36:10.451296+0000","last_peered":"2024-09-05T16:36:10.451296+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:36:10.451296+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4041,"log_dups_size":0,"ondisk_log_size":4041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33076384,"num_objects":4038,"num_object_clones":0,"num_object_copies":16152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4038,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4039,"num_write_kb":32309,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1718}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'3989","reported_seq":4052,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:10.489978+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:36:10.489978+0000","last_peered":"2024-09-05T16:36:10.489978+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:36:10.489978+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3989,"log_dups_size":0,"ondisk_log_size":3989,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32634016,"num_objects":3984,"num_object_clones":0,"num_object_copies":15936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3984,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3987,"num_write_kb":31884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1747}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":522600640,"num_objects":63799,"num_object_clones":0,"num_object_copies":255196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63799,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":63826,"num_write_kb":510490,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1087,"num_bytes_recovered":8683520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":900694016,"data_stored":900694016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":63853,"ondisk_log_size":63853,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084315,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":466232,"kb_used_data":254056,"kb_used_omap":4,"kb_used_meta":212155,"kb_avail":93905608,"statfs":{"total":96636764160,"available":96159342592,"internally_reserved":0,"allocated":260153344,"data_stored":259970462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":217247443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,6,3,4,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986192,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":235732,"kb_used_data":113796,"kb_used_omap":4,"kb_used_meta":121915,"kb_avail":94136108,"statfs":{"total":96636764160,"available":96395374592,"internally_reserved":0,"allocated":116527104,"data_stored":116344222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":124841682},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,2,2,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949672998,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":466864,"kb_used_data":255200,"kb_used_omap":1,"kb_used_meta":211646,"kb_avail":93904976,"statfs":{"total":96636764160,"available":96158695424,"internally_reserved":0,"allocated":261324800,"data_stored":261141918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":216725957},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,5,4,1,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182429,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":467060,"kb_used_data":254948,"kb_used_omap":4,"kb_used_meta":212091,"kb_avail":93904780,"statfs":{"total":96636764160,"available":96158494720,"internally_reserved":0,"allocated":261066752,"data_stored":260883870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":217181901},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,0,3,0,0,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":261890048,"data_stored":261890048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":116514816,"data_stored":116514816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":261820416,"data_stored":261820416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":260468736,"data_stored":260468736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:36:14.583 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-05T16:36:14.583 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'3993", 'reported_seq': 4034, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:09.394198+0000', 'last_change': '2024-09-05T16:34:56.833497+0000', 'last_active': '2024-09-05T16:36:09.394198+0000', 'last_peered': '2024-09-05T16:36:09.394198+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.828289+0000', 'last_became_peered': '2024-09-05T16:34:56.828289+0000', 'last_unstale': '2024-09-05T16:36:09.394198+0000', 'last_undegraded': '2024-09-05T16:34:56.612160+0000', 'last_fullsized': '2024-09-05T16:34:56.557632+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 3993, 'log_dups_size': 0, 'ondisk_log_size': 3993, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32710656, 'num_objects': 3993, 'num_object_clones': 0, 'num_object_copies': 15972, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2262, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3993, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3993, 'num_write_kb': 31944, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1731}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:36:14.583 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-05T16:36:14.583 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'3946", 'reported_seq': 4009, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:10.420005+0000', 'last_change': '2024-09-05T16:34:56.825031+0000', 'last_active': '2024-09-05T16:36:10.420005+0000', 'last_peered': '2024-09-05T16:36:10.420005+0000', 'last_clean': '2024-09-05T16:33:53.200762+0000', 'last_became_active': '2024-09-05T16:34:56.820292+0000', 'last_became_peered': '2024-09-05T16:34:56.820292+0000', 'last_unstale': '2024-09-05T16:36:10.420005+0000', 'last_undegraded': '2024-09-05T16:34:56.605776+0000', 'last_fullsized': '2024-09-05T16:34:56.559383+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 3946, 'log_dups_size': 0, 'ondisk_log_size': 3946, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32289952, 'num_objects': 3942, 'num_object_clones': 0, 'num_object_copies': 15768, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2269, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3942, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3944, 'num_write_kb': 31548, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 1675}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:36:14.583 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:36:14.583 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'3989", 'reported_seq': 4064, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:10.431943+0000', 'last_change': '2024-09-05T16:34:56.867900+0000', 'last_active': '2024-09-05T16:36:10.431943+0000', 'last_peered': '2024-09-05T16:36:10.431943+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:36:10.431943+0000', 'last_undegraded': '2024-09-05T16:34:56.590972+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 3989, 'log_dups_size': 0, 'ondisk_log_size': 3989, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32651856, 'num_objects': 3986, 'num_object_clones': 0, 'num_object_copies': 15944, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2251, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3986, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3988, 'num_write_kb': 31894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1736}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:36:14.583 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-05T16:36:14.584 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'4015", 'reported_seq': 6198, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:10.503771+0000', 'last_change': '2024-09-05T16:34:56.800371+0000', 'last_active': '2024-09-05T16:36:10.503771+0000', 'last_peered': '2024-09-05T16:36:10.503771+0000', 'last_clean': '2024-09-05T16:33:53.148717+0000', 'last_became_active': '2024-09-05T16:34:56.791755+0000', 'last_became_peered': '2024-09-05T16:34:56.791755+0000', 'last_unstale': '2024-09-05T16:36:10.503771+0000', 'last_undegraded': '2024-09-05T16:34:56.597522+0000', 'last_fullsized': '2024-09-05T16:34:56.568150+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4015, 'log_dups_size': 0, 'ondisk_log_size': 4015, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32855200, 'num_objects': 4011, 'num_object_clones': 0, 'num_object_copies': 16044, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1228, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4011, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4013, 'num_write_kb': 32093, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1062, 'num_bytes_recovered': 8683520, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2783}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:36:14.584 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:36:14.584 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'3861", 'reported_seq': 3914, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:09.370695+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:36:09.370695+0000', 'last_peered': '2024-09-05T16:36:09.370695+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:36:09.370695+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 3861, 'log_dups_size': 0, 'ondisk_log_size': 3861, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31601824, 'num_objects': 3858, 'num_object_clones': 0, 'num_object_copies': 15432, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2164, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3858, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3859, 'num_write_kb': 30869, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1696}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:36:14.584 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:36:14.584 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'3997", 'reported_seq': 4038, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:09.382508+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:36:09.382508+0000', 'last_peered': '2024-09-05T16:36:09.382508+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:36:09.382508+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 3997, 'log_dups_size': 0, 'ondisk_log_size': 3997, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32743424, 'num_objects': 3997, 'num_object_clones': 0, 'num_object_copies': 15988, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3997, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3997, 'num_write_kb': 31976, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1747}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:36:14.584 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:36:14.584 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'3915", 'reported_seq': 3979, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:10.545177+0000', 'last_change': '2024-09-05T16:34:56.903436+0000', 'last_active': '2024-09-05T16:36:10.545177+0000', 'last_peered': '2024-09-05T16:36:10.545177+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:36:10.545177+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 3915, 'log_dups_size': 0, 'ondisk_log_size': 3915, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32019616, 'num_objects': 3909, 'num_object_clones': 0, 'num_object_copies': 15636, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2149, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3909, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3913, 'num_write_kb': 31284, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1762}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:36:14.584 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-05T16:36:14.584 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'3996", 'reported_seq': 4059, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:10.292158+0000', 'last_change': '2024-09-05T16:34:56.883319+0000', 'last_active': '2024-09-05T16:36:10.292158+0000', 'last_peered': '2024-09-05T16:36:10.292158+0000', 'last_clean': '2024-09-05T16:33:53.505305+0000', 'last_became_active': '2024-09-05T16:34:56.873389+0000', 'last_became_peered': '2024-09-05T16:34:56.873389+0000', 'last_unstale': '2024-09-05T16:36:10.292158+0000', 'last_undegraded': '2024-09-05T16:34:56.602567+0000', 'last_fullsized': '2024-09-05T16:34:56.559958+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 3996, 'log_dups_size': 0, 'ondisk_log_size': 3996, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32691360, 'num_objects': 3991, 'num_object_clones': 0, 'num_object_copies': 15964, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2241, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3991, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3994, 'num_write_kb': 31940, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1752}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:36:14.584 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:36:14.585 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'3997", 'reported_seq': 4052, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:09.403813+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:36:09.403813+0000', 'last_peered': '2024-09-05T16:36:09.403813+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:36:09.403813+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 3997, 'log_dups_size': 0, 'ondisk_log_size': 3997, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32672064, 'num_objects': 3989, 'num_object_clones': 0, 'num_object_copies': 15956, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2264, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3989, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3994, 'num_write_kb': 31929, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1728}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:36:14.585 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:36:14.585 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'4039", 'reported_seq': 4116, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:10.354655+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:36:10.354655+0000', 'last_peered': '2024-09-05T16:36:10.354655+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:36:10.354655+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4039, 'log_dups_size': 0, 'ondisk_log_size': 4039, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33068192, 'num_objects': 4037, 'num_object_clones': 0, 'num_object_copies': 16148, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2264, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4037, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4037, 'num_write_kb': 32294, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1775}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:36:14.585 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:36:14.585 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'4082", 'reported_seq': 4137, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:09.334987+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:36:09.334987+0000', 'last_peered': '2024-09-05T16:36:09.334987+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:36:09.334987+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4082, 'log_dups_size': 0, 'ondisk_log_size': 4082, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33394416, 'num_objects': 4077, 'num_object_clones': 0, 'num_object_copies': 16308, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2314, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4077, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4079, 'num_write_kb': 32620, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1766}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:36:14.585 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:36:14.585 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'3894", 'reported_seq': 3929, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:09.484033+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:36:09.484033+0000', 'last_peered': '2024-09-05T16:36:09.484033+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:36:09.484033+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 3894, 'log_dups_size': 0, 'ondisk_log_size': 3894, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31890000, 'num_objects': 3893, 'num_object_clones': 0, 'num_object_copies': 15572, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2265, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3893, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3893, 'num_write_kb': 31143, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1629}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:36:14.585 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-05T16:36:14.585 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'4016", 'reported_seq': 4090, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:10.429538+0000', 'last_change': '2024-09-05T16:34:56.798081+0000', 'last_active': '2024-09-05T16:36:10.429538+0000', 'last_peered': '2024-09-05T16:36:10.429538+0000', 'last_clean': '2024-09-05T16:33:45.082039+0000', 'last_became_active': '2024-09-05T16:34:56.792386+0000', 'last_became_peered': '2024-09-05T16:34:56.792386+0000', 'last_unstale': '2024-09-05T16:36:10.429538+0000', 'last_undegraded': '2024-09-05T16:34:56.600595+0000', 'last_fullsized': '2024-09-05T16:34:56.558203+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4016, 'log_dups_size': 0, 'ondisk_log_size': 4016, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32889424, 'num_objects': 4015, 'num_object_clones': 0, 'num_object_copies': 16060, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2295, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4015, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4015, 'num_write_kb': 32119, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1721}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:36:14.585 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:36:14.585 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'4083", 'reported_seq': 4134, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:09.385435+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:36:09.385435+0000', 'last_peered': '2024-09-05T16:36:09.385435+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:36:09.385435+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4083, 'log_dups_size': 0, 'ondisk_log_size': 4083, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33412256, 'num_objects': 4079, 'num_object_clones': 0, 'num_object_copies': 16316, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2359, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4079, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4081, 'num_write_kb': 32644, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1722}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:36:14.586 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:36:14.586 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'4041", 'reported_seq': 4116, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:10.451296+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:36:10.451296+0000', 'last_peered': '2024-09-05T16:36:10.451296+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:36:10.451296+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4041, 'log_dups_size': 0, 'ondisk_log_size': 4041, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33076384, 'num_objects': 4038, 'num_object_clones': 0, 'num_object_copies': 16152, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2322, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4038, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4039, 'num_write_kb': 32309, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1718}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:36:14.586 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:36:14.586 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'3989", 'reported_seq': 4052, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:10.489978+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:36:10.489978+0000', 'last_peered': '2024-09-05T16:36:10.489978+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:36:10.489978+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 3989, 'log_dups_size': 0, 'ondisk_log_size': 3989, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32634016, 'num_objects': 3984, 'num_object_clones': 0, 'num_object_copies': 15936, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3984, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3987, 'num_write_kb': 31884, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1747}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:36:14.586 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:36:15.296 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:36:15.297 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 35136/258168 objects degraded (13.610%), 16 pgs degraded, 16 pgs undersized","count":32},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":178,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":64542,"data_bytes":528687296,"bytes_used":1697947648,"bytes_avail":384849108992,"bytes_total":386547056640,"degraded_objects":35136,"degraded_total":258168,"degraded_ratio":0.13609742493260202,"recovering_objects_per_sec":11,"recovering_bytes_per_sec":94153,"recovering_keys_per_sec":0,"num_objects_recovered":138,"num_bytes_recovered":1130496,"num_keys_recovered":0,"write_bytes_sec":2855315,"read_op_per_sec":0,"write_op_per_sec":348},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:36:15.322 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:36:15.323 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:36:15.512 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 113 20 65617 65597 4.53434 2.42188 0.0310008 0.0344441 2024-09-05T16:36:15.947 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:36:15.947 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:36:15.969 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":113,"stamp":"2024-09-05T16:36:14.824708+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":528687296,"num_objects":64542,"num_object_clones":0,"num_object_copies":258168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64542,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":64569,"num_write_kb":516434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1087,"num_bytes_recovered":8683520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64596,"ondisk_log_size":64596,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1658152,"kb_used_data":891944,"kb_used_omap":14,"kb_used_meta":766129,"kb_avail":375829208,"statfs":{"total":386547056640,"available":384849108992,"internally_reserved":0,"allocated":913350656,"data_stored":912619128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":784516663},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,3,15,18,11,13,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":34283520,"num_objects":4185,"num_object_clones":0,"num_object_copies":16740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4185,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4185,"num_write_kb":33480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":138,"num_bytes_recovered":1130496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006911"},"pg_stats":[{"pgid":"1.f","version":"33'4090","reported_seq":4131,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:14.470661+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:36:14.470661+0000","last_peered":"2024-09-05T16:36:14.470661+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:36:14.470661+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4090,"log_dups_size":0,"ondisk_log_size":4090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33505280,"num_objects":4090,"num_object_clones":0,"num_object_copies":16360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4090,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4090,"num_write_kb":32720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1828}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'3946","reported_seq":4009,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:10.420005+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:36:10.420005+0000","last_peered":"2024-09-05T16:36:10.420005+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:36:10.420005+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3946,"log_dups_size":0,"ondisk_log_size":3946,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32289952,"num_objects":3942,"num_object_clones":0,"num_object_copies":15768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3942,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3944,"num_write_kb":31548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1675}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'3989","reported_seq":4064,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:10.431943+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:36:10.431943+0000","last_peered":"2024-09-05T16:36:10.431943+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:36:10.431943+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3989,"log_dups_size":0,"ondisk_log_size":3989,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32651856,"num_objects":3986,"num_object_clones":0,"num_object_copies":15944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":31894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1736}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'4015","reported_seq":6198,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:10.503771+0000","last_change":"2024-09-05T16:34:56.800371+0000","last_active":"2024-09-05T16:36:10.503771+0000","last_peered":"2024-09-05T16:36:10.503771+0000","last_clean":"2024-09-05T16:33:53.148717+0000","last_became_active":"2024-09-05T16:34:56.791755+0000","last_became_peered":"2024-09-05T16:34:56.791755+0000","last_unstale":"2024-09-05T16:36:10.503771+0000","last_undegraded":"2024-09-05T16:34:56.597522+0000","last_fullsized":"2024-09-05T16:34:56.568150+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4015,"log_dups_size":0,"ondisk_log_size":4015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32855200,"num_objects":4011,"num_object_clones":0,"num_object_copies":16044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4011,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4013,"num_write_kb":32093,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1062,"num_bytes_recovered":8683520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2783}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'3971","reported_seq":4024,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:14.484321+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:36:14.484321+0000","last_peered":"2024-09-05T16:36:14.484321+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:36:14.484321+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3971,"log_dups_size":0,"ondisk_log_size":3971,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32502944,"num_objects":3968,"num_object_clones":0,"num_object_copies":15872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3968,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3969,"num_write_kb":31749,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1806}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'4112","reported_seq":4153,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:14.446783+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:36:14.446783+0000","last_peered":"2024-09-05T16:36:14.446783+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:36:14.446783+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4112,"log_dups_size":0,"ondisk_log_size":4112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33685504,"num_objects":4112,"num_object_clones":0,"num_object_copies":16448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4112,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4112,"num_write_kb":32896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1862}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'3915","reported_seq":3979,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:10.545177+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:36:10.545177+0000","last_peered":"2024-09-05T16:36:10.545177+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:36:10.545177+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3915,"log_dups_size":0,"ondisk_log_size":3915,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32019616,"num_objects":3909,"num_object_clones":0,"num_object_copies":15636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3909,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3913,"num_write_kb":31284,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1762}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'3996","reported_seq":4059,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:10.292158+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:36:10.292158+0000","last_peered":"2024-09-05T16:36:10.292158+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:36:10.292158+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3996,"log_dups_size":0,"ondisk_log_size":3996,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32691360,"num_objects":3991,"num_object_clones":0,"num_object_copies":15964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3991,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3994,"num_write_kb":31940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1752}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'4094","reported_seq":4148,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:14.383061+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:36:14.383061+0000","last_peered":"2024-09-05T16:36:14.383061+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:36:14.383061+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4094,"log_dups_size":0,"ondisk_log_size":4094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33466688,"num_objects":4086,"num_object_clones":0,"num_object_copies":16344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4086,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4091,"num_write_kb":32705,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1825}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'4039","reported_seq":4116,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:10.354655+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:36:10.354655+0000","last_peered":"2024-09-05T16:36:10.354655+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:36:10.354655+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4039,"log_dups_size":0,"ondisk_log_size":4039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33068192,"num_objects":4037,"num_object_clones":0,"num_object_copies":16148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":32294,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1775}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'4185","reported_seq":4240,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:14.447242+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:36:14.447242+0000","last_peered":"2024-09-05T16:36:14.447242+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:36:14.447242+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4185,"log_dups_size":0,"ondisk_log_size":4185,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34238192,"num_objects":4180,"num_object_clones":0,"num_object_copies":16720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4180,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4182,"num_write_kb":33444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1869}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'4003","reported_seq":4038,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:14.524942+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:36:14.524942+0000","last_peered":"2024-09-05T16:36:14.524942+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:36:14.524942+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4003,"log_dups_size":0,"ondisk_log_size":4003,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32782928,"num_objects":4002,"num_object_clones":0,"num_object_copies":16008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4002,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4002,"num_write_kb":32015,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1738}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'4016","reported_seq":4090,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:10.429538+0000","last_change":"2024-09-05T16:34:56.798081+0000","last_active":"2024-09-05T16:36:10.429538+0000","last_peered":"2024-09-05T16:36:10.429538+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:36:10.429538+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4016,"log_dups_size":0,"ondisk_log_size":4016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32889424,"num_objects":4015,"num_object_clones":0,"num_object_copies":16060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4015,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4015,"num_write_kb":32119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1721}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'4195","reported_seq":4246,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:14.453299+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:36:14.453299+0000","last_peered":"2024-09-05T16:36:14.453299+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:36:14.453299+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4195,"log_dups_size":0,"ondisk_log_size":4195,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34329760,"num_objects":4191,"num_object_clones":0,"num_object_copies":16764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4191,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4193,"num_write_kb":33540,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1834}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'4041","reported_seq":4116,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:10.451296+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:36:10.451296+0000","last_peered":"2024-09-05T16:36:10.451296+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:36:10.451296+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4041,"log_dups_size":0,"ondisk_log_size":4041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33076384,"num_objects":4038,"num_object_clones":0,"num_object_copies":16152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4038,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4039,"num_write_kb":32309,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1718}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'3989","reported_seq":4052,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:10.489978+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:36:10.489978+0000","last_peered":"2024-09-05T16:36:10.489978+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:36:10.489978+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3989,"log_dups_size":0,"ondisk_log_size":3989,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32634016,"num_objects":3984,"num_object_clones":0,"num_object_copies":15936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3984,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3987,"num_write_kb":31884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1747}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":528687296,"num_objects":64542,"num_object_clones":0,"num_object_copies":258168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64542,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":64569,"num_write_kb":516434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1087,"num_bytes_recovered":8683520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":915066880,"data_stored":915066880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64596,"ondisk_log_size":64596,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084316,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":477276,"kb_used_data":260940,"kb_used_omap":4,"kb_used_meta":216315,"kb_avail":93894564,"statfs":{"total":96636764160,"available":96148033536,"internally_reserved":0,"allocated":267202560,"data_stored":267019678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":221507283},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,10,8,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986193,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":246952,"kb_used_data":120856,"kb_used_omap":4,"kb_used_meta":126075,"kb_avail":94124888,"statfs":{"total":96636764160,"available":96383885312,"internally_reserved":0,"allocated":123756544,"data_stored":123573662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":129101522},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,4,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949672998,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":466864,"kb_used_data":255200,"kb_used_omap":1,"kb_used_meta":211646,"kb_avail":93904976,"statfs":{"total":96636764160,"available":96158695424,"internally_reserved":0,"allocated":261324800,"data_stored":261141918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":216725957},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,5,4,1,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182429,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":467060,"kb_used_data":254948,"kb_used_omap":4,"kb_used_meta":212091,"kb_avail":93904780,"statfs":{"total":96636764160,"available":96158494720,"internally_reserved":0,"allocated":261066752,"data_stored":260883870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":217181901},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,0,3,0,0,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":261890048,"data_stored":261890048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":123789312,"data_stored":123789312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":261820416,"data_stored":261820416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":267567104,"data_stored":267567104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:36:16.510 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 114 20 65948 65928 4.51725 2.58594 0.0427799 0.034577 2024-09-05T16:36:17.510 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 115 20 66292 66272 4.50133 2.6875 0.0689785 0.0346981 2024-09-05T16:36:18.511 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 116 20 66651 66631 4.48671 2.80469 0.0258475 0.034812 2024-09-05T16:36:18.971 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:36:19.512 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 117 20 66990 66970 4.471 2.64844 0.0805341 0.0349342 2024-09-05T16:36:19.577 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:36:19.577 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:36:19.598 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":115,"stamp":"2024-09-05T16:36:18.825950+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":536543424,"num_objects":65501,"num_object_clones":0,"num_object_copies":262004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35068,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65501,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":65528,"num_write_kb":524106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1157,"num_bytes_recovered":9240576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65555,"ondisk_log_size":65555,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1680208,"kb_used_data":905680,"kb_used_omap":14,"kb_used_meta":774449,"kb_avail":375807152,"statfs":{"total":386547056640,"available":384826523648,"internally_reserved":0,"allocated":927416320,"data_stored":926684792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":793036343},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,0,12,16,13,14,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":28106752,"num_objects":3431,"num_object_clones":0,"num_object_copies":13724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-139,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3431,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3431,"num_write_kb":27448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":141,"num_bytes_recovered":1138688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006645"},"pg_stats":[{"pgid":"1.f","version":"33'4090","reported_seq":4131,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:14.470661+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:36:14.470661+0000","last_peered":"2024-09-05T16:36:14.470661+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:36:14.470661+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4090,"log_dups_size":0,"ondisk_log_size":4090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33505280,"num_objects":4090,"num_object_clones":0,"num_object_copies":16360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4090,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4090,"num_write_kb":32720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1828}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'4055","reported_seq":4118,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:15.513485+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:36:15.513485+0000","last_peered":"2024-09-05T16:36:15.513485+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:36:15.513485+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4055,"log_dups_size":0,"ondisk_log_size":4055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33182880,"num_objects":4051,"num_object_clones":0,"num_object_copies":16204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4051,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4053,"num_write_kb":32420,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1784}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'4099","reported_seq":4174,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:15.519560+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:36:15.519560+0000","last_peered":"2024-09-05T16:36:15.519560+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:36:15.519560+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4099,"log_dups_size":0,"ondisk_log_size":4099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33552976,"num_objects":4096,"num_object_clones":0,"num_object_copies":16384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4096,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4098,"num_write_kb":32774,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1846}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'4114","reported_seq":6438,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:15.560106+0000","last_change":"2024-09-05T16:34:56.800371+0000","last_active":"2024-09-05T16:36:15.560106+0000","last_peered":"2024-09-05T16:36:15.560106+0000","last_clean":"2024-09-05T16:33:53.148717+0000","last_became_active":"2024-09-05T16:34:56.791755+0000","last_became_peered":"2024-09-05T16:34:56.791755+0000","last_unstale":"2024-09-05T16:36:15.560106+0000","last_undegraded":"2024-09-05T16:34:56.597522+0000","last_fullsized":"2024-09-05T16:34:56.568150+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4114,"log_dups_size":0,"ondisk_log_size":4114,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33666208,"num_objects":4110,"num_object_clones":0,"num_object_copies":16440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4110,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4112,"num_write_kb":32885,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1132,"num_bytes_recovered":9240576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2950}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'3971","reported_seq":4024,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:14.484321+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:36:14.484321+0000","last_peered":"2024-09-05T16:36:14.484321+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:36:14.484321+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3971,"log_dups_size":0,"ondisk_log_size":3971,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32502944,"num_objects":3968,"num_object_clones":0,"num_object_copies":15872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3968,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3969,"num_write_kb":31749,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1806}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'4112","reported_seq":4153,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:14.446783+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:36:14.446783+0000","last_peered":"2024-09-05T16:36:14.446783+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:36:14.446783+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4112,"log_dups_size":0,"ondisk_log_size":4112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33685504,"num_objects":4112,"num_object_clones":0,"num_object_copies":16448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4112,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4112,"num_write_kb":32896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1862}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'4024","reported_seq":4089,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:15.549719+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:36:15.549719+0000","last_peered":"2024-09-05T16:36:15.549719+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:36:15.549719+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32912544,"num_objects":4018,"num_object_clones":0,"num_object_copies":16072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4018,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4022,"num_write_kb":32156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1871}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'4081","reported_seq":4145,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:15.558554+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:36:15.558554+0000","last_peered":"2024-09-05T16:36:15.558554+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:36:15.558554+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4081,"log_dups_size":0,"ondisk_log_size":4081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33387680,"num_objects":4076,"num_object_clones":0,"num_object_copies":16304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4076,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4079,"num_write_kb":32620,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1837}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'4094","reported_seq":4148,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:14.383061+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:36:14.383061+0000","last_peered":"2024-09-05T16:36:14.383061+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:36:14.383061+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4094,"log_dups_size":0,"ondisk_log_size":4094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33466688,"num_objects":4086,"num_object_clones":0,"num_object_copies":16344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4086,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4091,"num_write_kb":32705,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1825}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'4145","reported_seq":4222,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:15.453663+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:36:15.453663+0000","last_peered":"2024-09-05T16:36:15.453663+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:36:15.453663+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4145,"log_dups_size":0,"ondisk_log_size":4145,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33936544,"num_objects":4143,"num_object_clones":0,"num_object_copies":16572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4143,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4143,"num_write_kb":33142,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1881}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'4185","reported_seq":4240,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:14.447242+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:36:14.447242+0000","last_peered":"2024-09-05T16:36:14.447242+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:36:14.447242+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4185,"log_dups_size":0,"ondisk_log_size":4185,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34238192,"num_objects":4180,"num_object_clones":0,"num_object_copies":16720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4180,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4182,"num_write_kb":33444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1869}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'4003","reported_seq":4038,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:14.524942+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:36:14.524942+0000","last_peered":"2024-09-05T16:36:14.524942+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:36:14.524942+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4003,"log_dups_size":0,"ondisk_log_size":4003,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32782928,"num_objects":4002,"num_object_clones":0,"num_object_copies":16008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4002,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4002,"num_write_kb":32015,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1738}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'4128","reported_seq":4202,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:15.444571+0000","last_change":"2024-09-05T16:34:56.798081+0000","last_active":"2024-09-05T16:36:15.444571+0000","last_peered":"2024-09-05T16:36:15.444571+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:36:15.444571+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4128,"log_dups_size":0,"ondisk_log_size":4128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33806928,"num_objects":4127,"num_object_clones":0,"num_object_copies":16508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4127,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4127,"num_write_kb":33015,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1833}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'4195","reported_seq":4246,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:14.453299+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:36:14.453299+0000","last_peered":"2024-09-05T16:36:14.453299+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:36:14.453299+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4195,"log_dups_size":0,"ondisk_log_size":4195,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34329760,"num_objects":4191,"num_object_clones":0,"num_object_copies":16764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4191,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4193,"num_write_kb":33540,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1834}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'4158","reported_seq":4236,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:15.511925+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:36:15.511925+0000","last_peered":"2024-09-05T16:36:15.511925+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:36:15.511925+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4158,"log_dups_size":0,"ondisk_log_size":4158,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34034848,"num_objects":4155,"num_object_clones":0,"num_object_copies":16620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4155,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4156,"num_write_kb":33245,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1835}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'4101","reported_seq":4164,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:15.433881+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:36:15.433881+0000","last_peered":"2024-09-05T16:36:15.433881+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:36:15.433881+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4101,"log_dups_size":0,"ondisk_log_size":4101,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33551520,"num_objects":4096,"num_object_clones":0,"num_object_copies":16384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4096,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4099,"num_write_kb":32780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1859}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":536543424,"num_objects":65501,"num_object_clones":0,"num_object_copies":262004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35068,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65501,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":65528,"num_write_kb":524106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1157,"num_bytes_recovered":9240576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":928952320,"data_stored":928952320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65555,"ondisk_log_size":65555,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084316,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":477276,"kb_used_data":260940,"kb_used_omap":4,"kb_used_meta":216315,"kb_avail":93894564,"statfs":{"total":96636764160,"available":96148033536,"internally_reserved":0,"allocated":267202560,"data_stored":267019678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":221507283},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,10,8,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986193,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":246952,"kb_used_data":120856,"kb_used_omap":4,"kb_used_meta":126075,"kb_avail":94124888,"statfs":{"total":96636764160,"available":96383885312,"internally_reserved":0,"allocated":123756544,"data_stored":123573662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":129101522},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,4,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949672999,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":478000,"kb_used_data":262176,"kb_used_omap":1,"kb_used_meta":215806,"kb_avail":93893840,"statfs":{"total":96636764160,"available":96147292160,"internally_reserved":0,"allocated":268468224,"data_stored":268285342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":220985797},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,3,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182430,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":477980,"kb_used_data":261708,"kb_used_omap":4,"kb_used_meta":216251,"kb_avail":93893860,"statfs":{"total":96636764160,"available":96147312640,"internally_reserved":0,"allocated":267988992,"data_stored":267806110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":221441741},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,0,3,0,0,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":268816384,"data_stored":268816384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":123789312,"data_stored":123789312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":268779520,"data_stored":268779520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":267567104,"data_stored":267567104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:36:19.600 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:36:20.233 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:36:20.234 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:36:20.258 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":115,"stamp":"2024-09-05T16:36:18.825950+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":536543424,"num_objects":65501,"num_object_clones":0,"num_object_copies":262004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35068,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65501,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":65528,"num_write_kb":524106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1157,"num_bytes_recovered":9240576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65555,"ondisk_log_size":65555,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1680208,"kb_used_data":905680,"kb_used_omap":14,"kb_used_meta":774449,"kb_avail":375807152,"statfs":{"total":386547056640,"available":384826523648,"internally_reserved":0,"allocated":927416320,"data_stored":926684792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":793036343},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,0,12,16,13,14,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":28106752,"num_objects":3431,"num_object_clones":0,"num_object_copies":13724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-139,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3431,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3431,"num_write_kb":27448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":141,"num_bytes_recovered":1138688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006645"},"pg_stats":[{"pgid":"1.f","version":"33'4090","reported_seq":4131,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:14.470661+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:36:14.470661+0000","last_peered":"2024-09-05T16:36:14.470661+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:36:14.470661+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4090,"log_dups_size":0,"ondisk_log_size":4090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33505280,"num_objects":4090,"num_object_clones":0,"num_object_copies":16360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4090,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4090,"num_write_kb":32720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1828}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'4055","reported_seq":4118,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:15.513485+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:36:15.513485+0000","last_peered":"2024-09-05T16:36:15.513485+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:36:15.513485+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4055,"log_dups_size":0,"ondisk_log_size":4055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33182880,"num_objects":4051,"num_object_clones":0,"num_object_copies":16204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4051,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4053,"num_write_kb":32420,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1784}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'4099","reported_seq":4174,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:15.519560+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:36:15.519560+0000","last_peered":"2024-09-05T16:36:15.519560+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:36:15.519560+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4099,"log_dups_size":0,"ondisk_log_size":4099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33552976,"num_objects":4096,"num_object_clones":0,"num_object_copies":16384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4096,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4098,"num_write_kb":32774,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1846}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'4114","reported_seq":6438,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:15.560106+0000","last_change":"2024-09-05T16:34:56.800371+0000","last_active":"2024-09-05T16:36:15.560106+0000","last_peered":"2024-09-05T16:36:15.560106+0000","last_clean":"2024-09-05T16:33:53.148717+0000","last_became_active":"2024-09-05T16:34:56.791755+0000","last_became_peered":"2024-09-05T16:34:56.791755+0000","last_unstale":"2024-09-05T16:36:15.560106+0000","last_undegraded":"2024-09-05T16:34:56.597522+0000","last_fullsized":"2024-09-05T16:34:56.568150+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4114,"log_dups_size":0,"ondisk_log_size":4114,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33666208,"num_objects":4110,"num_object_clones":0,"num_object_copies":16440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4110,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4112,"num_write_kb":32885,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1132,"num_bytes_recovered":9240576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2950}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'3971","reported_seq":4024,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:14.484321+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:36:14.484321+0000","last_peered":"2024-09-05T16:36:14.484321+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:36:14.484321+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":3971,"log_dups_size":0,"ondisk_log_size":3971,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32502944,"num_objects":3968,"num_object_clones":0,"num_object_copies":15872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3968,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3969,"num_write_kb":31749,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1806}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'4112","reported_seq":4153,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:14.446783+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:36:14.446783+0000","last_peered":"2024-09-05T16:36:14.446783+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:36:14.446783+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4112,"log_dups_size":0,"ondisk_log_size":4112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33685504,"num_objects":4112,"num_object_clones":0,"num_object_copies":16448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4112,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4112,"num_write_kb":32896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1862}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'4024","reported_seq":4089,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:15.549719+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:36:15.549719+0000","last_peered":"2024-09-05T16:36:15.549719+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:36:15.549719+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32912544,"num_objects":4018,"num_object_clones":0,"num_object_copies":16072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4018,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4022,"num_write_kb":32156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1871}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'4081","reported_seq":4145,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:15.558554+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:36:15.558554+0000","last_peered":"2024-09-05T16:36:15.558554+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:36:15.558554+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4081,"log_dups_size":0,"ondisk_log_size":4081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33387680,"num_objects":4076,"num_object_clones":0,"num_object_copies":16304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4076,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4079,"num_write_kb":32620,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1837}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'4094","reported_seq":4148,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:14.383061+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:36:14.383061+0000","last_peered":"2024-09-05T16:36:14.383061+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:36:14.383061+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4094,"log_dups_size":0,"ondisk_log_size":4094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33466688,"num_objects":4086,"num_object_clones":0,"num_object_copies":16344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4086,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4091,"num_write_kb":32705,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1825}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'4145","reported_seq":4222,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:15.453663+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:36:15.453663+0000","last_peered":"2024-09-05T16:36:15.453663+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:36:15.453663+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4145,"log_dups_size":0,"ondisk_log_size":4145,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33936544,"num_objects":4143,"num_object_clones":0,"num_object_copies":16572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4143,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4143,"num_write_kb":33142,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1881}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'4185","reported_seq":4240,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:14.447242+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:36:14.447242+0000","last_peered":"2024-09-05T16:36:14.447242+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:36:14.447242+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4185,"log_dups_size":0,"ondisk_log_size":4185,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34238192,"num_objects":4180,"num_object_clones":0,"num_object_copies":16720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4180,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4182,"num_write_kb":33444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1869}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'4003","reported_seq":4038,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:14.524942+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:36:14.524942+0000","last_peered":"2024-09-05T16:36:14.524942+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:36:14.524942+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4003,"log_dups_size":0,"ondisk_log_size":4003,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32782928,"num_objects":4002,"num_object_clones":0,"num_object_copies":16008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4002,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4002,"num_write_kb":32015,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1738}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'4128","reported_seq":4202,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:15.444571+0000","last_change":"2024-09-05T16:34:56.798081+0000","last_active":"2024-09-05T16:36:15.444571+0000","last_peered":"2024-09-05T16:36:15.444571+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:36:15.444571+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4128,"log_dups_size":0,"ondisk_log_size":4128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33806928,"num_objects":4127,"num_object_clones":0,"num_object_copies":16508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4127,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4127,"num_write_kb":33015,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1833}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'4195","reported_seq":4246,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:14.453299+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:36:14.453299+0000","last_peered":"2024-09-05T16:36:14.453299+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:36:14.453299+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4195,"log_dups_size":0,"ondisk_log_size":4195,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34329760,"num_objects":4191,"num_object_clones":0,"num_object_copies":16764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4191,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4193,"num_write_kb":33540,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1834}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'4158","reported_seq":4236,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:15.511925+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:36:15.511925+0000","last_peered":"2024-09-05T16:36:15.511925+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:36:15.511925+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4158,"log_dups_size":0,"ondisk_log_size":4158,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34034848,"num_objects":4155,"num_object_clones":0,"num_object_copies":16620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4155,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4156,"num_write_kb":33245,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1835}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'4101","reported_seq":4164,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:15.433881+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:36:15.433881+0000","last_peered":"2024-09-05T16:36:15.433881+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:36:15.433881+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4101,"log_dups_size":0,"ondisk_log_size":4101,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33551520,"num_objects":4096,"num_object_clones":0,"num_object_copies":16384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4096,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4099,"num_write_kb":32780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1859}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":536543424,"num_objects":65501,"num_object_clones":0,"num_object_copies":262004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35068,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65501,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":65528,"num_write_kb":524106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1157,"num_bytes_recovered":9240576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":928952320,"data_stored":928952320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65555,"ondisk_log_size":65555,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084316,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":477276,"kb_used_data":260940,"kb_used_omap":4,"kb_used_meta":216315,"kb_avail":93894564,"statfs":{"total":96636764160,"available":96148033536,"internally_reserved":0,"allocated":267202560,"data_stored":267019678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":221507283},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,10,8,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986193,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":246952,"kb_used_data":120856,"kb_used_omap":4,"kb_used_meta":126075,"kb_avail":94124888,"statfs":{"total":96636764160,"available":96383885312,"internally_reserved":0,"allocated":123756544,"data_stored":123573662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":129101522},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,4,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949672999,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":478000,"kb_used_data":262176,"kb_used_omap":1,"kb_used_meta":215806,"kb_avail":93893840,"statfs":{"total":96636764160,"available":96147292160,"internally_reserved":0,"allocated":268468224,"data_stored":268285342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":220985797},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,3,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182430,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":477980,"kb_used_data":261708,"kb_used_omap":4,"kb_used_meta":216251,"kb_avail":93893860,"statfs":{"total":96636764160,"available":96147312640,"internally_reserved":0,"allocated":267988992,"data_stored":267806110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":221441741},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,0,3,0,0,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":268816384,"data_stored":268816384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":123789312,"data_stored":123789312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":268779520,"data_stored":268779520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":267567104,"data_stored":267567104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:36:20.259 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-05T16:36:20.259 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'4090", 'reported_seq': 4131, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:14.470661+0000', 'last_change': '2024-09-05T16:34:56.833497+0000', 'last_active': '2024-09-05T16:36:14.470661+0000', 'last_peered': '2024-09-05T16:36:14.470661+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.828289+0000', 'last_became_peered': '2024-09-05T16:34:56.828289+0000', 'last_unstale': '2024-09-05T16:36:14.470661+0000', 'last_undegraded': '2024-09-05T16:34:56.612160+0000', 'last_fullsized': '2024-09-05T16:34:56.557632+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4090, 'log_dups_size': 0, 'ondisk_log_size': 4090, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33505280, 'num_objects': 4090, 'num_object_clones': 0, 'num_object_copies': 16360, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2262, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4090, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4090, 'num_write_kb': 32720, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1828}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:36:20.259 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-05T16:36:20.259 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'4055", 'reported_seq': 4118, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:15.513485+0000', 'last_change': '2024-09-05T16:34:56.825031+0000', 'last_active': '2024-09-05T16:36:15.513485+0000', 'last_peered': '2024-09-05T16:36:15.513485+0000', 'last_clean': '2024-09-05T16:33:53.200762+0000', 'last_became_active': '2024-09-05T16:34:56.820292+0000', 'last_became_peered': '2024-09-05T16:34:56.820292+0000', 'last_unstale': '2024-09-05T16:36:15.513485+0000', 'last_undegraded': '2024-09-05T16:34:56.605776+0000', 'last_fullsized': '2024-09-05T16:34:56.559383+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4055, 'log_dups_size': 0, 'ondisk_log_size': 4055, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33182880, 'num_objects': 4051, 'num_object_clones': 0, 'num_object_copies': 16204, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2269, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4051, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4053, 'num_write_kb': 32420, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 1784}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:36:20.260 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:36:20.260 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'4099", 'reported_seq': 4174, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:15.519560+0000', 'last_change': '2024-09-05T16:34:56.867900+0000', 'last_active': '2024-09-05T16:36:15.519560+0000', 'last_peered': '2024-09-05T16:36:15.519560+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:36:15.519560+0000', 'last_undegraded': '2024-09-05T16:34:56.590972+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4099, 'log_dups_size': 0, 'ondisk_log_size': 4099, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33552976, 'num_objects': 4096, 'num_object_clones': 0, 'num_object_copies': 16384, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2251, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4096, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4098, 'num_write_kb': 32774, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1846}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:36:20.260 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-05T16:36:20.260 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'4114", 'reported_seq': 6438, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:15.560106+0000', 'last_change': '2024-09-05T16:34:56.800371+0000', 'last_active': '2024-09-05T16:36:15.560106+0000', 'last_peered': '2024-09-05T16:36:15.560106+0000', 'last_clean': '2024-09-05T16:33:53.148717+0000', 'last_became_active': '2024-09-05T16:34:56.791755+0000', 'last_became_peered': '2024-09-05T16:34:56.791755+0000', 'last_unstale': '2024-09-05T16:36:15.560106+0000', 'last_undegraded': '2024-09-05T16:34:56.597522+0000', 'last_fullsized': '2024-09-05T16:34:56.568150+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4114, 'log_dups_size': 0, 'ondisk_log_size': 4114, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33666208, 'num_objects': 4110, 'num_object_clones': 0, 'num_object_copies': 16440, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1160, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4110, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4112, 'num_write_kb': 32885, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1132, 'num_bytes_recovered': 9240576, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2950}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:36:20.260 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:36:20.260 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'3971", 'reported_seq': 4024, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:14.484321+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:36:14.484321+0000', 'last_peered': '2024-09-05T16:36:14.484321+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:36:14.484321+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 3971, 'log_dups_size': 0, 'ondisk_log_size': 3971, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32502944, 'num_objects': 3968, 'num_object_clones': 0, 'num_object_copies': 15872, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2164, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3968, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3969, 'num_write_kb': 31749, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1806}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:36:20.260 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:36:20.260 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'4112", 'reported_seq': 4153, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:14.446783+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:36:14.446783+0000', 'last_peered': '2024-09-05T16:36:14.446783+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:36:14.446783+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4112, 'log_dups_size': 0, 'ondisk_log_size': 4112, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33685504, 'num_objects': 4112, 'num_object_clones': 0, 'num_object_copies': 16448, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4112, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4112, 'num_write_kb': 32896, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1862}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:36:20.260 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:36:20.260 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'4024", 'reported_seq': 4089, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:15.549719+0000', 'last_change': '2024-09-05T16:34:56.903436+0000', 'last_active': '2024-09-05T16:36:15.549719+0000', 'last_peered': '2024-09-05T16:36:15.549719+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:36:15.549719+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4024, 'log_dups_size': 0, 'ondisk_log_size': 4024, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32912544, 'num_objects': 4018, 'num_object_clones': 0, 'num_object_copies': 16072, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2149, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4018, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4022, 'num_write_kb': 32156, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1871}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:36:20.261 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-05T16:36:20.261 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'4081", 'reported_seq': 4145, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:15.558554+0000', 'last_change': '2024-09-05T16:34:56.883319+0000', 'last_active': '2024-09-05T16:36:15.558554+0000', 'last_peered': '2024-09-05T16:36:15.558554+0000', 'last_clean': '2024-09-05T16:33:53.505305+0000', 'last_became_active': '2024-09-05T16:34:56.873389+0000', 'last_became_peered': '2024-09-05T16:34:56.873389+0000', 'last_unstale': '2024-09-05T16:36:15.558554+0000', 'last_undegraded': '2024-09-05T16:34:56.602567+0000', 'last_fullsized': '2024-09-05T16:34:56.559958+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4081, 'log_dups_size': 0, 'ondisk_log_size': 4081, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33387680, 'num_objects': 4076, 'num_object_clones': 0, 'num_object_copies': 16304, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2241, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4076, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4079, 'num_write_kb': 32620, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1837}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:36:20.261 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:36:20.261 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'4094", 'reported_seq': 4148, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:14.383061+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:36:14.383061+0000', 'last_peered': '2024-09-05T16:36:14.383061+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:36:14.383061+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4094, 'log_dups_size': 0, 'ondisk_log_size': 4094, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33466688, 'num_objects': 4086, 'num_object_clones': 0, 'num_object_copies': 16344, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2264, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4086, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4091, 'num_write_kb': 32705, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1825}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:36:20.261 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:36:20.261 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'4145", 'reported_seq': 4222, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:15.453663+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:36:15.453663+0000', 'last_peered': '2024-09-05T16:36:15.453663+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:36:15.453663+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4145, 'log_dups_size': 0, 'ondisk_log_size': 4145, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33936544, 'num_objects': 4143, 'num_object_clones': 0, 'num_object_copies': 16572, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2264, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4143, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4143, 'num_write_kb': 33142, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1881}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:36:20.261 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:36:20.261 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'4185", 'reported_seq': 4240, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:14.447242+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:36:14.447242+0000', 'last_peered': '2024-09-05T16:36:14.447242+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:36:14.447242+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4185, 'log_dups_size': 0, 'ondisk_log_size': 4185, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34238192, 'num_objects': 4180, 'num_object_clones': 0, 'num_object_copies': 16720, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2314, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4180, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4182, 'num_write_kb': 33444, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1869}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:36:20.261 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:36:20.261 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'4003", 'reported_seq': 4038, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:14.524942+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:36:14.524942+0000', 'last_peered': '2024-09-05T16:36:14.524942+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:36:14.524942+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4003, 'log_dups_size': 0, 'ondisk_log_size': 4003, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32782928, 'num_objects': 4002, 'num_object_clones': 0, 'num_object_copies': 16008, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2265, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4002, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4002, 'num_write_kb': 32015, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1738}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:36:20.262 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-05T16:36:20.262 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'4128", 'reported_seq': 4202, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:15.444571+0000', 'last_change': '2024-09-05T16:34:56.798081+0000', 'last_active': '2024-09-05T16:36:15.444571+0000', 'last_peered': '2024-09-05T16:36:15.444571+0000', 'last_clean': '2024-09-05T16:33:45.082039+0000', 'last_became_active': '2024-09-05T16:34:56.792386+0000', 'last_became_peered': '2024-09-05T16:34:56.792386+0000', 'last_unstale': '2024-09-05T16:36:15.444571+0000', 'last_undegraded': '2024-09-05T16:34:56.600595+0000', 'last_fullsized': '2024-09-05T16:34:56.558203+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4128, 'log_dups_size': 0, 'ondisk_log_size': 4128, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33806928, 'num_objects': 4127, 'num_object_clones': 0, 'num_object_copies': 16508, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2295, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4127, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4127, 'num_write_kb': 33015, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1833}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:36:20.262 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:36:20.262 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'4195", 'reported_seq': 4246, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:14.453299+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:36:14.453299+0000', 'last_peered': '2024-09-05T16:36:14.453299+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:36:14.453299+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4195, 'log_dups_size': 0, 'ondisk_log_size': 4195, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34329760, 'num_objects': 4191, 'num_object_clones': 0, 'num_object_copies': 16764, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2359, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4191, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4193, 'num_write_kb': 33540, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1834}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:36:20.262 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:36:20.262 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'4158", 'reported_seq': 4236, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:15.511925+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:36:15.511925+0000', 'last_peered': '2024-09-05T16:36:15.511925+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:36:15.511925+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4158, 'log_dups_size': 0, 'ondisk_log_size': 4158, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34034848, 'num_objects': 4155, 'num_object_clones': 0, 'num_object_copies': 16620, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2322, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4155, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4156, 'num_write_kb': 33245, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1835}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:36:20.262 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:36:20.262 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'4101", 'reported_seq': 4164, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:15.433881+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:36:15.433881+0000', 'last_peered': '2024-09-05T16:36:15.433881+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:36:15.433881+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4101, 'log_dups_size': 0, 'ondisk_log_size': 4101, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33551520, 'num_objects': 4096, 'num_object_clones': 0, 'num_object_copies': 16384, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4096, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4099, 'num_write_kb': 32780, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1859}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:36:20.262 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:36:20.515 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 118 20 67297 67277 4.45343 2.39844 0.0344035 0.0350728 2024-09-05T16:36:20.996 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:36:20.997 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 34994/268776 objects degraded (13.020%), 16 pgs degraded, 16 pgs undersized","count":32},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":184,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":67194,"data_bytes":550412480,"bytes_used":1775476736,"bytes_avail":384771579904,"bytes_total":386547056640,"degraded_objects":34994,"degraded_total":268776,"degraded_ratio":0.13019763669375242,"recovering_objects_per_sec":17,"recovering_bytes_per_sec":145330,"recovering_keys_per_sec":0,"num_objects_recovered":215,"num_bytes_recovered":1744896,"num_keys_recovered":0,"write_bytes_sec":3496122,"read_op_per_sec":0,"write_op_per_sec":426},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:36:21.036 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:36:21.036 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:36:21.511 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 119 20 67602 67582 4.43603 2.38281 0.0733021 0.0352107 2024-09-05T16:36:21.648 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:36:21.648 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:36:21.679 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":116,"stamp":"2024-09-05T16:36:20.826649+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":550412480,"num_objects":67194,"num_object_clones":0,"num_object_copies":268776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34994,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67194,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":67221,"num_write_kb":537650,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1231,"num_bytes_recovered":9846784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":67248,"ondisk_log_size":67248,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1733864,"kb_used_data":932968,"kb_used_omap":14,"kb_used_meta":800817,"kb_avail":375753496,"statfs":{"total":386547056640,"available":384771579904,"internally_reserved":0,"allocated":955359232,"data_stored":954627704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":820037175},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,10,3,4,8,12,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":41975808,"num_objects":5124,"num_object_clones":0,"num_object_copies":20496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5124,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5124,"num_write_kb":40992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":215,"num_bytes_recovered":1744896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006390"},"pg_stats":[{"pgid":"1.f","version":"33'4201","reported_seq":4242,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:19.428029+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:36:19.428029+0000","last_peered":"2024-09-05T16:36:19.428029+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:36:19.428029+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4201,"log_dups_size":0,"ondisk_log_size":4201,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34414592,"num_objects":4201,"num_object_clones":0,"num_object_copies":16804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4201,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4201,"num_write_kb":33608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1939}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'4159","reported_seq":4222,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:20.481523+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:36:20.481523+0000","last_peered":"2024-09-05T16:36:20.481523+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:36:20.481523+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4159,"log_dups_size":0,"ondisk_log_size":4159,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34034848,"num_objects":4155,"num_object_clones":0,"num_object_copies":16620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4155,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4157,"num_write_kb":33252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1888}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'4208","reported_seq":4282,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:20.376263+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:36:20.376263+0000","last_peered":"2024-09-05T16:36:20.376263+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:36:20.376263+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4208,"log_dups_size":0,"ondisk_log_size":4208,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34445904,"num_objects":4205,"num_object_clones":0,"num_object_copies":16820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4205,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4207,"num_write_kb":33646,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1955}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'4218","reported_seq":6691,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:20.564001+0000","last_change":"2024-09-05T16:34:56.800371+0000","last_active":"2024-09-05T16:36:20.564001+0000","last_peered":"2024-09-05T16:36:20.564001+0000","last_clean":"2024-09-05T16:33:53.148717+0000","last_became_active":"2024-09-05T16:34:56.791755+0000","last_became_peered":"2024-09-05T16:34:56.791755+0000","last_unstale":"2024-09-05T16:36:20.564001+0000","last_undegraded":"2024-09-05T16:34:56.597522+0000","last_fullsized":"2024-09-05T16:34:56.568150+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4218,"log_dups_size":0,"ondisk_log_size":4218,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34518176,"num_objects":4214,"num_object_clones":0,"num_object_copies":16856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4214,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4216,"num_write_kb":33717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1206,"num_bytes_recovered":9846784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3128}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'4075","reported_seq":4127,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:19.523229+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:36:19.523229+0000","last_peered":"2024-09-05T16:36:19.523229+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:36:19.523229+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4075,"log_dups_size":0,"ondisk_log_size":4075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33354912,"num_objects":4072,"num_object_clones":0,"num_object_copies":16288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4072,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4073,"num_write_kb":32581,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1910}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'4212","reported_seq":4253,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:19.457961+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:36:19.457961+0000","last_peered":"2024-09-05T16:36:19.457961+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:36:19.457961+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4212,"log_dups_size":0,"ondisk_log_size":4212,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34504704,"num_objects":4212,"num_object_clones":0,"num_object_copies":16848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4212,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4212,"num_write_kb":33696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1962}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'4109","reported_seq":4174,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:20.491618+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:36:20.491618+0000","last_peered":"2024-09-05T16:36:20.491618+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:36:20.491618+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4109,"log_dups_size":0,"ondisk_log_size":4109,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33608864,"num_objects":4103,"num_object_clones":0,"num_object_copies":16412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4103,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4107,"num_write_kb":32836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1956}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'4187","reported_seq":4251,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:20.466281+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:36:20.466281+0000","last_peered":"2024-09-05T16:36:20.466281+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:36:20.466281+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4187,"log_dups_size":0,"ondisk_log_size":4187,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34256032,"num_objects":4182,"num_object_clones":0,"num_object_copies":16728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4182,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4185,"num_write_kb":33468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1943}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'4209","reported_seq":4264,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:19.506351+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:36:19.506351+0000","last_peered":"2024-09-05T16:36:19.506351+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:36:19.506351+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4209,"log_dups_size":0,"ondisk_log_size":4209,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34408768,"num_objects":4201,"num_object_clones":0,"num_object_copies":16804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4201,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4206,"num_write_kb":33625,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1940}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'4241","reported_seq":4318,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:20.472330+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:36:20.472330+0000","last_peered":"2024-09-05T16:36:20.472330+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:36:20.472330+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4241,"log_dups_size":0,"ondisk_log_size":4241,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34722976,"num_objects":4239,"num_object_clones":0,"num_object_copies":16956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4239,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4239,"num_write_kb":33910,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1977}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'4298","reported_seq":4353,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:19.557460+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:36:19.557460+0000","last_peered":"2024-09-05T16:36:19.557460+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:36:19.557460+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4298,"log_dups_size":0,"ondisk_log_size":4298,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35163888,"num_objects":4293,"num_object_clones":0,"num_object_copies":17172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4293,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4295,"num_write_kb":34348,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1982}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'4111","reported_seq":4146,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:19.425340+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:36:19.425340+0000","last_peered":"2024-09-05T16:36:19.425340+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:36:19.425340+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4111,"log_dups_size":0,"ondisk_log_size":4111,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33667664,"num_objects":4110,"num_object_clones":0,"num_object_copies":16440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4110,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4110,"num_write_kb":32879,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1846}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'4248","reported_seq":4322,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:20.531176+0000","last_change":"2024-09-05T16:34:56.798081+0000","last_active":"2024-09-05T16:36:20.531176+0000","last_peered":"2024-09-05T16:36:20.531176+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:36:20.531176+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4248,"log_dups_size":0,"ondisk_log_size":4248,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34789968,"num_objects":4247,"num_object_clones":0,"num_object_copies":16988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4247,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4247,"num_write_kb":33975,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1953}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'4301","reported_seq":4352,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:19.511940+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:36:19.511940+0000","last_peered":"2024-09-05T16:36:19.511940+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:36:19.511940+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4301,"log_dups_size":0,"ondisk_log_size":4301,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35198112,"num_objects":4297,"num_object_clones":0,"num_object_copies":17188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4297,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4299,"num_write_kb":34388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1940}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'4265","reported_seq":4341,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:20.540371+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:36:20.540371+0000","last_peered":"2024-09-05T16:36:20.540371+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:36:20.540371+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4265,"log_dups_size":0,"ondisk_log_size":4265,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34911392,"num_objects":4262,"num_object_clones":0,"num_object_copies":17048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4262,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4263,"num_write_kb":34101,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1942}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'4206","reported_seq":4269,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:20.414212+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:36:20.414212+0000","last_peered":"2024-09-05T16:36:20.414212+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:36:20.414212+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4206,"log_dups_size":0,"ondisk_log_size":4206,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34411680,"num_objects":4201,"num_object_clones":0,"num_object_copies":16804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4201,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4204,"num_write_kb":33620,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1964}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":550412480,"num_objects":67194,"num_object_clones":0,"num_object_copies":268776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34994,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67194,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":67221,"num_write_kb":537650,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1231,"num_bytes_recovered":9846784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":956964864,"data_stored":956964864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":67248,"ondisk_log_size":67248,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084317,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":488160,"kb_used_data":267664,"kb_used_omap":4,"kb_used_meta":220475,"kb_avail":93883680,"statfs":{"total":96636764160,"available":96136888320,"internally_reserved":0,"allocated":274087936,"data_stored":273905054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":225767123},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,2,1,1,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986194,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":263664,"kb_used_data":127840,"kb_used_omap":4,"kb_used_meta":135803,"kb_avail":94108176,"statfs":{"total":96636764160,"available":96366772224,"internally_reserved":0,"allocated":130908160,"data_stored":130725278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":139062994},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673000,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":493128,"kb_used_data":268984,"kb_used_omap":1,"kb_used_meta":224126,"kb_avail":93878712,"statfs":{"total":96636764160,"available":96131801088,"internally_reserved":0,"allocated":275439616,"data_stored":275256734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":229505477},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,6,0,1,5,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182431,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":488912,"kb_used_data":268480,"kb_used_omap":4,"kb_used_meta":220411,"kb_avail":93882928,"statfs":{"total":96636764160,"available":96136118272,"internally_reserved":0,"allocated":274923520,"data_stored":274740638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":225701581},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":275738624,"data_stored":275738624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":131043328,"data_stored":131043328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":275693568,"data_stored":275693568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":274489344,"data_stored":274489344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:36:22.511 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout:2024-09-05T16:36:22.519003+0000 min lat: 0.00357551 max lat: 2.29705 avg lat: 0.0353287 2024-09-05T16:36:22.511 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-05T16:36:22.511 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 120 20 67944 67924 4.42133 2.67188 0.0229384 0.0353287 2024-09-05T16:36:23.516 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 121 20 68290 68270 4.40705 2.70312 0.0585429 0.0354421 2024-09-05T16:36:24.513 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 122 20 68638 68618 4.39321 2.71875 0.0773389 0.0355519 2024-09-05T16:36:24.681 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:36:25.290 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:36:25.291 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:36:25.318 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":118,"stamp":"2024-09-05T16:36:24.828133+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":556277952,"num_objects":67910,"num_object_clones":0,"num_object_copies":271640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34994,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":67937,"num_write_kb":543378,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1231,"num_bytes_recovered":9846784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":67964,"ondisk_log_size":67964,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1757248,"kb_used_data":946624,"kb_used_omap":14,"kb_used_meta":810545,"kb_avail":375730112,"statfs":{"total":386547056640,"available":384747634688,"internally_reserved":0,"allocated":969342976,"data_stored":968611448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":829998647},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,7,7,20,12,11,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":33677312,"num_objects":4111,"num_object_clones":0,"num_object_copies":16444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-142,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4111,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4111,"num_write_kb":32888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":144,"num_bytes_recovered":1163264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006103"},"pg_stats":[{"pgid":"1.f","version":"33'4309","reported_seq":4350,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:24.521283+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:36:24.521283+0000","last_peered":"2024-09-05T16:36:24.521283+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:36:24.521283+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4309,"log_dups_size":0,"ondisk_log_size":4309,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35299328,"num_objects":4309,"num_object_clones":0,"num_object_copies":17236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4309,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4309,"num_write_kb":34472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2047}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'4159","reported_seq":4222,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:20.481523+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:36:20.481523+0000","last_peered":"2024-09-05T16:36:20.481523+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:36:20.481523+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4159,"log_dups_size":0,"ondisk_log_size":4159,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34034848,"num_objects":4155,"num_object_clones":0,"num_object_copies":16620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4155,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4157,"num_write_kb":33252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1888}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'4208","reported_seq":4282,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:20.376263+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:36:20.376263+0000","last_peered":"2024-09-05T16:36:20.376263+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:36:20.376263+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4208,"log_dups_size":0,"ondisk_log_size":4208,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34445904,"num_objects":4205,"num_object_clones":0,"num_object_copies":16820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4205,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4207,"num_write_kb":33646,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1955}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'4218","reported_seq":6691,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:20.564001+0000","last_change":"2024-09-05T16:34:56.800371+0000","last_active":"2024-09-05T16:36:20.564001+0000","last_peered":"2024-09-05T16:36:20.564001+0000","last_clean":"2024-09-05T16:33:53.148717+0000","last_became_active":"2024-09-05T16:34:56.791755+0000","last_became_peered":"2024-09-05T16:34:56.791755+0000","last_unstale":"2024-09-05T16:36:20.564001+0000","last_undegraded":"2024-09-05T16:34:56.597522+0000","last_fullsized":"2024-09-05T16:34:56.568150+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4218,"log_dups_size":0,"ondisk_log_size":4218,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34518176,"num_objects":4214,"num_object_clones":0,"num_object_copies":16856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4214,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4216,"num_write_kb":33717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1206,"num_bytes_recovered":9846784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3128}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'4175","reported_seq":4229,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:24.496971+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:36:24.496971+0000","last_peered":"2024-09-05T16:36:24.496971+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:36:24.496971+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4175,"log_dups_size":0,"ondisk_log_size":4175,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34174112,"num_objects":4172,"num_object_clones":0,"num_object_copies":16688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4172,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4173,"num_write_kb":33381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2010}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'4326","reported_seq":4366,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:24.630345+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:36:24.630345+0000","last_peered":"2024-09-05T16:36:24.630345+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:36:24.630345+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4326,"log_dups_size":0,"ondisk_log_size":4326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35438592,"num_objects":4326,"num_object_clones":0,"num_object_copies":17304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4326,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4326,"num_write_kb":34608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2076}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'4109","reported_seq":4174,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:20.491618+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:36:20.491618+0000","last_peered":"2024-09-05T16:36:20.491618+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:36:20.491618+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4109,"log_dups_size":0,"ondisk_log_size":4109,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33608864,"num_objects":4103,"num_object_clones":0,"num_object_copies":16412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4103,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4107,"num_write_kb":32836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1956}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'4187","reported_seq":4251,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:20.466281+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:36:20.466281+0000","last_peered":"2024-09-05T16:36:20.466281+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:36:20.466281+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4187,"log_dups_size":0,"ondisk_log_size":4187,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34256032,"num_objects":4182,"num_object_clones":0,"num_object_copies":16728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4182,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4185,"num_write_kb":33468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1943}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'4298","reported_seq":4354,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:24.547084+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:36:24.547084+0000","last_peered":"2024-09-05T16:36:24.547084+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:36:24.547084+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4298,"log_dups_size":0,"ondisk_log_size":4298,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35137856,"num_objects":4290,"num_object_clones":0,"num_object_copies":17160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4290,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4295,"num_write_kb":34337,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2029}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'4241","reported_seq":4318,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:20.472330+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:36:20.472330+0000","last_peered":"2024-09-05T16:36:20.472330+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:36:20.472330+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4241,"log_dups_size":0,"ondisk_log_size":4241,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34722976,"num_objects":4239,"num_object_clones":0,"num_object_copies":16956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4239,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4239,"num_write_kb":33910,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1977}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'4404","reported_seq":4457,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:24.612101+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:36:24.612101+0000","last_peered":"2024-09-05T16:36:24.612101+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:36:24.612101+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4404,"log_dups_size":0,"ondisk_log_size":4404,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36032240,"num_objects":4399,"num_object_clones":0,"num_object_copies":17596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4399,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4401,"num_write_kb":35196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2088}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'4224","reported_seq":4259,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:24.596868+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:36:24.596868+0000","last_peered":"2024-09-05T16:36:24.596868+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:36:24.596868+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4224,"log_dups_size":0,"ondisk_log_size":4224,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34593360,"num_objects":4223,"num_object_clones":0,"num_object_copies":16892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4223,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4223,"num_write_kb":33783,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1959}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'4248","reported_seq":4322,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:20.531176+0000","last_change":"2024-09-05T16:34:56.798081+0000","last_active":"2024-09-05T16:36:20.531176+0000","last_peered":"2024-09-05T16:36:20.531176+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:36:20.531176+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4248,"log_dups_size":0,"ondisk_log_size":4248,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34789968,"num_objects":4247,"num_object_clones":0,"num_object_copies":16988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4247,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4247,"num_write_kb":33975,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1953}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'4387","reported_seq":4438,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:24.527123+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:36:24.527123+0000","last_peered":"2024-09-05T16:36:24.527123+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:36:24.527123+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4387,"log_dups_size":0,"ondisk_log_size":4387,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35902624,"num_objects":4383,"num_object_clones":0,"num_object_copies":17532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4383,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4385,"num_write_kb":35076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2026}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'4265","reported_seq":4341,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:20.540371+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:36:20.540371+0000","last_peered":"2024-09-05T16:36:20.540371+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:36:20.540371+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4265,"log_dups_size":0,"ondisk_log_size":4265,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34911392,"num_objects":4262,"num_object_clones":0,"num_object_copies":17048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4262,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4263,"num_write_kb":34101,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1942}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'4206","reported_seq":4269,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:20.414212+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:36:20.414212+0000","last_peered":"2024-09-05T16:36:20.414212+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:36:20.414212+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4206,"log_dups_size":0,"ondisk_log_size":4206,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34411680,"num_objects":4201,"num_object_clones":0,"num_object_copies":16804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4201,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4204,"num_write_kb":33620,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1964}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":556277952,"num_objects":67910,"num_object_clones":0,"num_object_copies":271640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34994,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":67937,"num_write_kb":543378,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1231,"num_bytes_recovered":9846784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":970854400,"data_stored":970854400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":67964,"ondisk_log_size":67964,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084318,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":482416,"kb_used_data":274400,"kb_used_omap":4,"kb_used_meta":207995,"kb_avail":93889424,"statfs":{"total":96636764160,"available":96142770176,"internally_reserved":0,"allocated":280985600,"data_stored":280802718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":212987603},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,7,17,0,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986195,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":292792,"kb_used_data":134760,"kb_used_omap":4,"kb_used_meta":158011,"kb_avail":94079048,"statfs":{"total":96636764160,"available":96336945152,"internally_reserved":0,"allocated":137994240,"data_stored":137811358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":161803986},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673000,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":493128,"kb_used_data":268984,"kb_used_omap":1,"kb_used_meta":224126,"kb_avail":93878712,"statfs":{"total":96636764160,"available":96131801088,"internally_reserved":0,"allocated":275439616,"data_stored":275256734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":229505477},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,6,0,1,5,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182431,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":488912,"kb_used_data":268480,"kb_used_omap":4,"kb_used_meta":220411,"kb_avail":93882928,"statfs":{"total":96636764160,"available":96136118272,"internally_reserved":0,"allocated":274923520,"data_stored":274740638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":225701581},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":275738624,"data_stored":275738624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":138100736,"data_stored":138100736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":275693568,"data_stored":275693568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":281321472,"data_stored":281321472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:36:25.319 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:36:25.513 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 123 20 68943 68923 4.37686 2.38281 0.0686668 0.0356809 2024-09-05T16:36:25.952 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:36:25.953 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:36:25.975 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":118,"stamp":"2024-09-05T16:36:24.828133+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":556277952,"num_objects":67910,"num_object_clones":0,"num_object_copies":271640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34994,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":67937,"num_write_kb":543378,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1231,"num_bytes_recovered":9846784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":67964,"ondisk_log_size":67964,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1757248,"kb_used_data":946624,"kb_used_omap":14,"kb_used_meta":810545,"kb_avail":375730112,"statfs":{"total":386547056640,"available":384747634688,"internally_reserved":0,"allocated":969342976,"data_stored":968611448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":829998647},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,7,7,20,12,11,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":33677312,"num_objects":4111,"num_object_clones":0,"num_object_copies":16444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-142,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4111,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4111,"num_write_kb":32888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":144,"num_bytes_recovered":1163264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006103"},"pg_stats":[{"pgid":"1.f","version":"33'4309","reported_seq":4350,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:24.521283+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:36:24.521283+0000","last_peered":"2024-09-05T16:36:24.521283+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:36:24.521283+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4309,"log_dups_size":0,"ondisk_log_size":4309,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35299328,"num_objects":4309,"num_object_clones":0,"num_object_copies":17236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4309,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4309,"num_write_kb":34472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2047}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'4159","reported_seq":4222,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:20.481523+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:36:20.481523+0000","last_peered":"2024-09-05T16:36:20.481523+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:36:20.481523+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4159,"log_dups_size":0,"ondisk_log_size":4159,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34034848,"num_objects":4155,"num_object_clones":0,"num_object_copies":16620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4155,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4157,"num_write_kb":33252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1888}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'4208","reported_seq":4282,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:20.376263+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:36:20.376263+0000","last_peered":"2024-09-05T16:36:20.376263+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:36:20.376263+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4208,"log_dups_size":0,"ondisk_log_size":4208,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34445904,"num_objects":4205,"num_object_clones":0,"num_object_copies":16820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4205,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4207,"num_write_kb":33646,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1955}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'4218","reported_seq":6691,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:20.564001+0000","last_change":"2024-09-05T16:34:56.800371+0000","last_active":"2024-09-05T16:36:20.564001+0000","last_peered":"2024-09-05T16:36:20.564001+0000","last_clean":"2024-09-05T16:33:53.148717+0000","last_became_active":"2024-09-05T16:34:56.791755+0000","last_became_peered":"2024-09-05T16:34:56.791755+0000","last_unstale":"2024-09-05T16:36:20.564001+0000","last_undegraded":"2024-09-05T16:34:56.597522+0000","last_fullsized":"2024-09-05T16:34:56.568150+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4218,"log_dups_size":0,"ondisk_log_size":4218,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34518176,"num_objects":4214,"num_object_clones":0,"num_object_copies":16856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4214,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4216,"num_write_kb":33717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1206,"num_bytes_recovered":9846784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3128}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'4175","reported_seq":4229,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:24.496971+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:36:24.496971+0000","last_peered":"2024-09-05T16:36:24.496971+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:36:24.496971+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4175,"log_dups_size":0,"ondisk_log_size":4175,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34174112,"num_objects":4172,"num_object_clones":0,"num_object_copies":16688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4172,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4173,"num_write_kb":33381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2010}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'4326","reported_seq":4366,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:24.630345+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:36:24.630345+0000","last_peered":"2024-09-05T16:36:24.630345+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:36:24.630345+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4326,"log_dups_size":0,"ondisk_log_size":4326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35438592,"num_objects":4326,"num_object_clones":0,"num_object_copies":17304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4326,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4326,"num_write_kb":34608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2076}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'4109","reported_seq":4174,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:20.491618+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:36:20.491618+0000","last_peered":"2024-09-05T16:36:20.491618+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:36:20.491618+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4109,"log_dups_size":0,"ondisk_log_size":4109,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33608864,"num_objects":4103,"num_object_clones":0,"num_object_copies":16412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4103,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4107,"num_write_kb":32836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1956}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'4187","reported_seq":4251,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:20.466281+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:36:20.466281+0000","last_peered":"2024-09-05T16:36:20.466281+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:36:20.466281+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4187,"log_dups_size":0,"ondisk_log_size":4187,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34256032,"num_objects":4182,"num_object_clones":0,"num_object_copies":16728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4182,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4185,"num_write_kb":33468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1943}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'4298","reported_seq":4354,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:24.547084+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:36:24.547084+0000","last_peered":"2024-09-05T16:36:24.547084+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:36:24.547084+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4298,"log_dups_size":0,"ondisk_log_size":4298,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35137856,"num_objects":4290,"num_object_clones":0,"num_object_copies":17160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4290,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4295,"num_write_kb":34337,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2029}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'4241","reported_seq":4318,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:20.472330+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:36:20.472330+0000","last_peered":"2024-09-05T16:36:20.472330+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:36:20.472330+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4241,"log_dups_size":0,"ondisk_log_size":4241,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34722976,"num_objects":4239,"num_object_clones":0,"num_object_copies":16956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4239,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4239,"num_write_kb":33910,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1977}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'4404","reported_seq":4457,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:24.612101+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:36:24.612101+0000","last_peered":"2024-09-05T16:36:24.612101+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:36:24.612101+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4404,"log_dups_size":0,"ondisk_log_size":4404,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36032240,"num_objects":4399,"num_object_clones":0,"num_object_copies":17596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4399,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4401,"num_write_kb":35196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2088}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'4224","reported_seq":4259,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:24.596868+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:36:24.596868+0000","last_peered":"2024-09-05T16:36:24.596868+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:36:24.596868+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4224,"log_dups_size":0,"ondisk_log_size":4224,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34593360,"num_objects":4223,"num_object_clones":0,"num_object_copies":16892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4223,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4223,"num_write_kb":33783,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1959}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'4248","reported_seq":4322,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:20.531176+0000","last_change":"2024-09-05T16:34:56.798081+0000","last_active":"2024-09-05T16:36:20.531176+0000","last_peered":"2024-09-05T16:36:20.531176+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:36:20.531176+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4248,"log_dups_size":0,"ondisk_log_size":4248,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34789968,"num_objects":4247,"num_object_clones":0,"num_object_copies":16988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4247,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4247,"num_write_kb":33975,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1953}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'4387","reported_seq":4438,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:24.527123+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:36:24.527123+0000","last_peered":"2024-09-05T16:36:24.527123+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:36:24.527123+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4387,"log_dups_size":0,"ondisk_log_size":4387,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35902624,"num_objects":4383,"num_object_clones":0,"num_object_copies":17532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4383,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4385,"num_write_kb":35076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2026}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'4265","reported_seq":4341,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:20.540371+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:36:20.540371+0000","last_peered":"2024-09-05T16:36:20.540371+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:36:20.540371+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4265,"log_dups_size":0,"ondisk_log_size":4265,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34911392,"num_objects":4262,"num_object_clones":0,"num_object_copies":17048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4262,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4263,"num_write_kb":34101,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1942}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'4206","reported_seq":4269,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:20.414212+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:36:20.414212+0000","last_peered":"2024-09-05T16:36:20.414212+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:36:20.414212+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4206,"log_dups_size":0,"ondisk_log_size":4206,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34411680,"num_objects":4201,"num_object_clones":0,"num_object_copies":16804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4201,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4204,"num_write_kb":33620,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1964}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":556277952,"num_objects":67910,"num_object_clones":0,"num_object_copies":271640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34994,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":67937,"num_write_kb":543378,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1231,"num_bytes_recovered":9846784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":970854400,"data_stored":970854400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":67964,"ondisk_log_size":67964,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084318,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":482416,"kb_used_data":274400,"kb_used_omap":4,"kb_used_meta":207995,"kb_avail":93889424,"statfs":{"total":96636764160,"available":96142770176,"internally_reserved":0,"allocated":280985600,"data_stored":280802718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":212987603},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,7,17,0,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986195,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":292792,"kb_used_data":134760,"kb_used_omap":4,"kb_used_meta":158011,"kb_avail":94079048,"statfs":{"total":96636764160,"available":96336945152,"internally_reserved":0,"allocated":137994240,"data_stored":137811358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":161803986},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673000,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":493128,"kb_used_data":268984,"kb_used_omap":1,"kb_used_meta":224126,"kb_avail":93878712,"statfs":{"total":96636764160,"available":96131801088,"internally_reserved":0,"allocated":275439616,"data_stored":275256734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":229505477},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,6,0,1,5,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182431,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":488912,"kb_used_data":268480,"kb_used_omap":4,"kb_used_meta":220411,"kb_avail":93882928,"statfs":{"total":96636764160,"available":96136118272,"internally_reserved":0,"allocated":274923520,"data_stored":274740638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":225701581},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":275738624,"data_stored":275738624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":138100736,"data_stored":138100736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":275693568,"data_stored":275693568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":281321472,"data_stored":281321472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:36:25.976 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-05T16:36:25.977 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'4309", 'reported_seq': 4350, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:24.521283+0000', 'last_change': '2024-09-05T16:34:56.833497+0000', 'last_active': '2024-09-05T16:36:24.521283+0000', 'last_peered': '2024-09-05T16:36:24.521283+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.828289+0000', 'last_became_peered': '2024-09-05T16:34:56.828289+0000', 'last_unstale': '2024-09-05T16:36:24.521283+0000', 'last_undegraded': '2024-09-05T16:34:56.612160+0000', 'last_fullsized': '2024-09-05T16:34:56.557632+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4309, 'log_dups_size': 0, 'ondisk_log_size': 4309, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35299328, 'num_objects': 4309, 'num_object_clones': 0, 'num_object_copies': 17236, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2262, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4309, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4309, 'num_write_kb': 34472, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2047}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:36:25.977 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-05T16:36:25.977 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'4159", 'reported_seq': 4222, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:20.481523+0000', 'last_change': '2024-09-05T16:34:56.825031+0000', 'last_active': '2024-09-05T16:36:20.481523+0000', 'last_peered': '2024-09-05T16:36:20.481523+0000', 'last_clean': '2024-09-05T16:33:53.200762+0000', 'last_became_active': '2024-09-05T16:34:56.820292+0000', 'last_became_peered': '2024-09-05T16:34:56.820292+0000', 'last_unstale': '2024-09-05T16:36:20.481523+0000', 'last_undegraded': '2024-09-05T16:34:56.605776+0000', 'last_fullsized': '2024-09-05T16:34:56.559383+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4159, 'log_dups_size': 0, 'ondisk_log_size': 4159, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34034848, 'num_objects': 4155, 'num_object_clones': 0, 'num_object_copies': 16620, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2269, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4155, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4157, 'num_write_kb': 33252, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 1888}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:36:25.977 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:36:25.977 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'4208", 'reported_seq': 4282, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:20.376263+0000', 'last_change': '2024-09-05T16:34:56.867900+0000', 'last_active': '2024-09-05T16:36:20.376263+0000', 'last_peered': '2024-09-05T16:36:20.376263+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:36:20.376263+0000', 'last_undegraded': '2024-09-05T16:34:56.590972+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4208, 'log_dups_size': 0, 'ondisk_log_size': 4208, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34445904, 'num_objects': 4205, 'num_object_clones': 0, 'num_object_copies': 16820, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2251, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4205, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4207, 'num_write_kb': 33646, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1955}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:36:25.977 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-05T16:36:25.977 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'4218", 'reported_seq': 6691, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:20.564001+0000', 'last_change': '2024-09-05T16:34:56.800371+0000', 'last_active': '2024-09-05T16:36:20.564001+0000', 'last_peered': '2024-09-05T16:36:20.564001+0000', 'last_clean': '2024-09-05T16:33:53.148717+0000', 'last_became_active': '2024-09-05T16:34:56.791755+0000', 'last_became_peered': '2024-09-05T16:34:56.791755+0000', 'last_unstale': '2024-09-05T16:36:20.564001+0000', 'last_undegraded': '2024-09-05T16:34:56.597522+0000', 'last_fullsized': '2024-09-05T16:34:56.568150+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4218, 'log_dups_size': 0, 'ondisk_log_size': 4218, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34518176, 'num_objects': 4214, 'num_object_clones': 0, 'num_object_copies': 16856, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1086, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4214, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4216, 'num_write_kb': 33717, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1206, 'num_bytes_recovered': 9846784, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3128}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:36:25.977 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:36:25.977 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'4175", 'reported_seq': 4229, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:24.496971+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:36:24.496971+0000', 'last_peered': '2024-09-05T16:36:24.496971+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:36:24.496971+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4175, 'log_dups_size': 0, 'ondisk_log_size': 4175, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34174112, 'num_objects': 4172, 'num_object_clones': 0, 'num_object_copies': 16688, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2164, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4172, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4173, 'num_write_kb': 33381, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2010}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:36:25.978 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:36:25.978 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'4326", 'reported_seq': 4366, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:24.630345+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:36:24.630345+0000', 'last_peered': '2024-09-05T16:36:24.630345+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:36:24.630345+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4326, 'log_dups_size': 0, 'ondisk_log_size': 4326, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35438592, 'num_objects': 4326, 'num_object_clones': 0, 'num_object_copies': 17304, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4326, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4326, 'num_write_kb': 34608, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2076}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:36:25.978 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:36:25.978 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'4109", 'reported_seq': 4174, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:20.491618+0000', 'last_change': '2024-09-05T16:34:56.903436+0000', 'last_active': '2024-09-05T16:36:20.491618+0000', 'last_peered': '2024-09-05T16:36:20.491618+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:36:20.491618+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4109, 'log_dups_size': 0, 'ondisk_log_size': 4109, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33608864, 'num_objects': 4103, 'num_object_clones': 0, 'num_object_copies': 16412, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2149, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4103, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4107, 'num_write_kb': 32836, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1956}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:36:25.978 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-05T16:36:25.978 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'4187", 'reported_seq': 4251, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:20.466281+0000', 'last_change': '2024-09-05T16:34:56.883319+0000', 'last_active': '2024-09-05T16:36:20.466281+0000', 'last_peered': '2024-09-05T16:36:20.466281+0000', 'last_clean': '2024-09-05T16:33:53.505305+0000', 'last_became_active': '2024-09-05T16:34:56.873389+0000', 'last_became_peered': '2024-09-05T16:34:56.873389+0000', 'last_unstale': '2024-09-05T16:36:20.466281+0000', 'last_undegraded': '2024-09-05T16:34:56.602567+0000', 'last_fullsized': '2024-09-05T16:34:56.559958+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4187, 'log_dups_size': 0, 'ondisk_log_size': 4187, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34256032, 'num_objects': 4182, 'num_object_clones': 0, 'num_object_copies': 16728, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2241, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4182, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4185, 'num_write_kb': 33468, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1943}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:36:25.978 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:36:25.978 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'4298", 'reported_seq': 4354, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:24.547084+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:36:24.547084+0000', 'last_peered': '2024-09-05T16:36:24.547084+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:36:24.547084+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4298, 'log_dups_size': 0, 'ondisk_log_size': 4298, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35137856, 'num_objects': 4290, 'num_object_clones': 0, 'num_object_copies': 17160, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2264, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4290, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4295, 'num_write_kb': 34337, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2029}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:36:25.978 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:36:25.978 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'4241", 'reported_seq': 4318, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:20.472330+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:36:20.472330+0000', 'last_peered': '2024-09-05T16:36:20.472330+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:36:20.472330+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4241, 'log_dups_size': 0, 'ondisk_log_size': 4241, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34722976, 'num_objects': 4239, 'num_object_clones': 0, 'num_object_copies': 16956, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2264, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4239, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4239, 'num_write_kb': 33910, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1977}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:36:25.978 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:36:25.979 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'4404", 'reported_seq': 4457, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:24.612101+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:36:24.612101+0000', 'last_peered': '2024-09-05T16:36:24.612101+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:36:24.612101+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4404, 'log_dups_size': 0, 'ondisk_log_size': 4404, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36032240, 'num_objects': 4399, 'num_object_clones': 0, 'num_object_copies': 17596, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2314, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4399, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4401, 'num_write_kb': 35196, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2088}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:36:25.979 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:36:25.979 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'4224", 'reported_seq': 4259, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:24.596868+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:36:24.596868+0000', 'last_peered': '2024-09-05T16:36:24.596868+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:36:24.596868+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4224, 'log_dups_size': 0, 'ondisk_log_size': 4224, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34593360, 'num_objects': 4223, 'num_object_clones': 0, 'num_object_copies': 16892, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2265, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4223, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4223, 'num_write_kb': 33783, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1959}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:36:25.979 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-05T16:36:25.979 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'4248", 'reported_seq': 4322, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:20.531176+0000', 'last_change': '2024-09-05T16:34:56.798081+0000', 'last_active': '2024-09-05T16:36:20.531176+0000', 'last_peered': '2024-09-05T16:36:20.531176+0000', 'last_clean': '2024-09-05T16:33:45.082039+0000', 'last_became_active': '2024-09-05T16:34:56.792386+0000', 'last_became_peered': '2024-09-05T16:34:56.792386+0000', 'last_unstale': '2024-09-05T16:36:20.531176+0000', 'last_undegraded': '2024-09-05T16:34:56.600595+0000', 'last_fullsized': '2024-09-05T16:34:56.558203+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4248, 'log_dups_size': 0, 'ondisk_log_size': 4248, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34789968, 'num_objects': 4247, 'num_object_clones': 0, 'num_object_copies': 16988, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2295, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4247, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4247, 'num_write_kb': 33975, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1953}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:36:25.979 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:36:25.979 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'4387", 'reported_seq': 4438, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:24.527123+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:36:24.527123+0000', 'last_peered': '2024-09-05T16:36:24.527123+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:36:24.527123+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4387, 'log_dups_size': 0, 'ondisk_log_size': 4387, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35902624, 'num_objects': 4383, 'num_object_clones': 0, 'num_object_copies': 17532, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2359, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4383, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4385, 'num_write_kb': 35076, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2026}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:36:25.979 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:36:25.979 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'4265", 'reported_seq': 4341, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:20.540371+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:36:20.540371+0000', 'last_peered': '2024-09-05T16:36:20.540371+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:36:20.540371+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4265, 'log_dups_size': 0, 'ondisk_log_size': 4265, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34911392, 'num_objects': 4262, 'num_object_clones': 0, 'num_object_copies': 17048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2322, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4262, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4263, 'num_write_kb': 34101, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1942}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:36:25.979 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:36:25.979 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'4206", 'reported_seq': 4269, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:20.414212+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:36:20.414212+0000', 'last_peered': '2024-09-05T16:36:20.414212+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:36:20.414212+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4206, 'log_dups_size': 0, 'ondisk_log_size': 4206, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34411680, 'num_objects': 4201, 'num_object_clones': 0, 'num_object_copies': 16804, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4201, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4204, 'num_write_kb': 33620, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1964}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:36:25.980 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:36:26.513 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 124 20 69249 69229 4.36085 2.39062 0.0409343 0.0358187 2024-09-05T16:36:26.698 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:36:26.698 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 34994/271640 objects degraded (12.882%), 16 pgs degraded, 16 pgs undersized","count":32},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":189,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":67910,"data_bytes":556277952,"bytes_used":1799421952,"bytes_avail":384747634688,"bytes_total":386547056640,"degraded_objects":34994,"degraded_total":271640,"degraded_ratio":0.12882491532911206,"recovering_objects_per_sec":11,"recovering_bytes_per_sec":96889,"recovering_keys_per_sec":0,"num_objects_recovered":144,"num_bytes_recovered":1163264,"num_keys_recovered":0,"write_bytes_sec":2805016,"read_op_per_sec":0,"write_op_per_sec":342},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:36:26.729 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:36:26.730 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:36:27.393 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:36:27.393 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:36:27.431 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":119,"stamp":"2024-09-05T16:36:26.828948+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":564134080,"num_objects":68869,"num_object_clones":0,"num_object_copies":275476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34917,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68869,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68896,"num_write_kb":551050,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1308,"num_bytes_recovered":10477568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":68923,"ondisk_log_size":68923,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1745424,"kb_used_data":959760,"kb_used_omap":14,"kb_used_meta":785585,"kb_avail":375741936,"statfs":{"total":386547056640,"available":384759742464,"internally_reserved":0,"allocated":982794240,"data_stored":982062712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":804439607},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,15,23,26,10,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":35446784,"num_objects":4327,"num_object_clones":0,"num_object_copies":17308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4327,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4327,"num_write_kb":34616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":221,"num_bytes_recovered":1794048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004239"},"pg_stats":[{"pgid":"1.f","version":"33'4309","reported_seq":4350,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:24.521283+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:36:24.521283+0000","last_peered":"2024-09-05T16:36:24.521283+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:36:24.521283+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4309,"log_dups_size":0,"ondisk_log_size":4309,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35299328,"num_objects":4309,"num_object_clones":0,"num_object_copies":17236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4309,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4309,"num_write_kb":34472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2047}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'4268","reported_seq":4329,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:25.602915+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:36:25.602915+0000","last_peered":"2024-09-05T16:36:25.602915+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:36:25.602915+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4268,"log_dups_size":0,"ondisk_log_size":4268,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34927776,"num_objects":4264,"num_object_clones":0,"num_object_copies":17056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4266,"num_write_kb":34124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1997}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'4304","reported_seq":4378,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:25.573714+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:36:25.573714+0000","last_peered":"2024-09-05T16:36:25.573714+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:36:25.573714+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4304,"log_dups_size":0,"ondisk_log_size":4304,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35232336,"num_objects":4301,"num_object_clones":0,"num_object_copies":17204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4301,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4303,"num_write_kb":34414,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2051}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'4325","reported_seq":6952,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:25.630651+0000","last_change":"2024-09-05T16:34:56.800371+0000","last_active":"2024-09-05T16:36:25.630651+0000","last_peered":"2024-09-05T16:36:25.630651+0000","last_clean":"2024-09-05T16:33:53.148717+0000","last_became_active":"2024-09-05T16:34:56.791755+0000","last_became_peered":"2024-09-05T16:34:56.791755+0000","last_unstale":"2024-09-05T16:36:25.630651+0000","last_undegraded":"2024-09-05T16:34:56.597522+0000","last_fullsized":"2024-09-05T16:34:56.568150+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4325,"log_dups_size":0,"ondisk_log_size":4325,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35394720,"num_objects":4321,"num_object_clones":0,"num_object_copies":17284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1009,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4321,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4323,"num_write_kb":34573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1283,"num_bytes_recovered":10477568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3312}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'4175","reported_seq":4229,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:24.496971+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:36:24.496971+0000","last_peered":"2024-09-05T16:36:24.496971+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:36:24.496971+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4175,"log_dups_size":0,"ondisk_log_size":4175,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34174112,"num_objects":4172,"num_object_clones":0,"num_object_copies":16688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4172,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4173,"num_write_kb":33381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2010}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'4326","reported_seq":4366,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:24.630345+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:36:24.630345+0000","last_peered":"2024-09-05T16:36:24.630345+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:36:24.630345+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4326,"log_dups_size":0,"ondisk_log_size":4326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35438592,"num_objects":4326,"num_object_clones":0,"num_object_copies":17304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4326,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4326,"num_write_kb":34608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2076}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'4211","reported_seq":4276,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:25.593271+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:36:25.593271+0000","last_peered":"2024-09-05T16:36:25.593271+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:36:25.593271+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4211,"log_dups_size":0,"ondisk_log_size":4211,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34444448,"num_objects":4205,"num_object_clones":0,"num_object_copies":16820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4205,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4209,"num_write_kb":33652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2058}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'4290","reported_seq":4354,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:25.576054+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:36:25.576054+0000","last_peered":"2024-09-05T16:36:25.576054+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:36:25.576054+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4290,"log_dups_size":0,"ondisk_log_size":4290,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35099808,"num_objects":4285,"num_object_clones":0,"num_object_copies":17140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4285,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4288,"num_write_kb":34292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2046}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'4298","reported_seq":4354,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:24.547084+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:36:24.547084+0000","last_peered":"2024-09-05T16:36:24.547084+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:36:24.547084+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4298,"log_dups_size":0,"ondisk_log_size":4298,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35137856,"num_objects":4290,"num_object_clones":0,"num_object_copies":17160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4290,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4295,"num_write_kb":34337,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2029}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'4340","reported_seq":4417,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:25.436663+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:36:25.436663+0000","last_peered":"2024-09-05T16:36:25.436663+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:36:25.436663+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4340,"log_dups_size":0,"ondisk_log_size":4340,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35533984,"num_objects":4338,"num_object_clones":0,"num_object_copies":17352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4338,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4338,"num_write_kb":34702,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2076}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'4404","reported_seq":4457,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:24.612101+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:36:24.612101+0000","last_peered":"2024-09-05T16:36:24.612101+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:36:24.612101+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4404,"log_dups_size":0,"ondisk_log_size":4404,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36032240,"num_objects":4399,"num_object_clones":0,"num_object_copies":17596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4399,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4401,"num_write_kb":35196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2088}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'4224","reported_seq":4259,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:24.596868+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:36:24.596868+0000","last_peered":"2024-09-05T16:36:24.596868+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:36:24.596868+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4224,"log_dups_size":0,"ondisk_log_size":4224,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34593360,"num_objects":4223,"num_object_clones":0,"num_object_copies":16892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4223,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4223,"num_write_kb":33783,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1959}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'4365","reported_seq":4439,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:25.454213+0000","last_change":"2024-09-05T16:34:56.798081+0000","last_active":"2024-09-05T16:36:25.454213+0000","last_peered":"2024-09-05T16:36:25.454213+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:36:25.454213+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4365,"log_dups_size":0,"ondisk_log_size":4365,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35748432,"num_objects":4364,"num_object_clones":0,"num_object_copies":17456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4364,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4364,"num_write_kb":34911,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2070}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'4387","reported_seq":4438,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:24.527123+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:36:24.527123+0000","last_peered":"2024-09-05T16:36:24.527123+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:36:24.527123+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4387,"log_dups_size":0,"ondisk_log_size":4387,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35902624,"num_objects":4383,"num_object_clones":0,"num_object_copies":17532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4383,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4385,"num_write_kb":35076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2026}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'4359","reported_seq":4437,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:25.433917+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:36:25.433917+0000","last_peered":"2024-09-05T16:36:25.433917+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:36:25.433917+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4359,"log_dups_size":0,"ondisk_log_size":4359,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35681440,"num_objects":4356,"num_object_clones":0,"num_object_copies":17424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4356,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4357,"num_write_kb":34853,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2036}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'4338","reported_seq":4401,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:25.639181+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:36:25.639181+0000","last_peered":"2024-09-05T16:36:25.639181+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:36:25.639181+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4338,"log_dups_size":0,"ondisk_log_size":4338,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35493024,"num_objects":4333,"num_object_clones":0,"num_object_copies":17332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4333,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4336,"num_write_kb":34676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2096}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":564134080,"num_objects":68869,"num_object_clones":0,"num_object_copies":275476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34917,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68869,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68896,"num_write_kb":551050,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1308,"num_bytes_recovered":10477568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":984444928,"data_stored":984444928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":68923,"ondisk_log_size":68923,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084318,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":482416,"kb_used_data":274400,"kb_used_omap":4,"kb_used_meta":207995,"kb_avail":93889424,"statfs":{"total":96636764160,"available":96142770176,"internally_reserved":0,"allocated":280985600,"data_stored":280802718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":212987603},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,7,17,0,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986195,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":292792,"kb_used_data":134760,"kb_used_omap":4,"kb_used_meta":158011,"kb_avail":94079048,"statfs":{"total":96636764160,"available":96336945152,"internally_reserved":0,"allocated":137994240,"data_stored":137811358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":161803986},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673001,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":487136,"kb_used_data":275472,"kb_used_omap":1,"kb_used_meta":211646,"kb_avail":93884704,"statfs":{"total":96636764160,"available":96137936896,"internally_reserved":0,"allocated":282083328,"data_stored":281900446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":216725957},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,0,8,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182432,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":483080,"kb_used_data":275128,"kb_used_omap":4,"kb_used_meta":207931,"kb_avail":93888760,"statfs":{"total":96636764160,"available":96142090240,"internally_reserved":0,"allocated":281731072,"data_stored":281548190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":212922061},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,4,4,13,4,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":282558464,"data_stored":282558464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":138100736,"data_stored":138100736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":282464256,"data_stored":282464256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":281321472,"data_stored":281321472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:36:27.514 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 125 20 69567 69547 4.34583 2.48438 0.0436215 0.0359447 2024-09-05T16:36:28.514 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 126 20 69917 69897 4.33305 2.73438 0.0548955 0.036044 2024-09-05T16:36:29.514 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 127 20 70261 70241 4.32009 2.6875 0.0401336 0.0361546 2024-09-05T16:36:30.433 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:36:30.514 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 128 20 70608 70588 4.30752 2.71094 0.0399844 0.0362613 2024-09-05T16:36:31.037 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:36:31.038 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:36:31.056 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":121,"stamp":"2024-09-05T16:36:30.831132+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":577470656,"num_objects":70497,"num_object_clones":0,"num_object_copies":281988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34836,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70497,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":70524,"num_write_kb":564074,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1389,"num_bytes_recovered":11141120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70551,"ondisk_log_size":70551,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1792872,"kb_used_data":986408,"kb_used_omap":14,"kb_used_meta":806385,"kb_avail":375694488,"statfs":{"total":386547056640,"available":384711155712,"internally_reserved":0,"allocated":1010081792,"data_stored":1009350264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":825738807},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,5,9,11,28,8,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":40927232,"num_objects":4996,"num_object_clones":0,"num_object_copies":19984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4996,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4996,"num_write_kb":39968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":232,"num_bytes_recovered":1900544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005181"},"pg_stats":[{"pgid":"1.f","version":"33'4418","reported_seq":4459,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:29.693545+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:36:29.693545+0000","last_peered":"2024-09-05T16:36:29.693545+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:36:29.693545+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4418,"log_dups_size":0,"ondisk_log_size":4418,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36192256,"num_objects":4418,"num_object_clones":0,"num_object_copies":17672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4418,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4418,"num_write_kb":35344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2156}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'4350","reported_seq":4414,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:30.425269+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:36:30.425269+0000","last_peered":"2024-09-05T16:36:30.425269+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:36:30.425269+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4350,"log_dups_size":0,"ondisk_log_size":4350,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35599520,"num_objects":4346,"num_object_clones":0,"num_object_copies":17384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4346,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4348,"num_write_kb":34780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2079}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'4414","reported_seq":4488,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:30.512415+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:36:30.512415+0000","last_peered":"2024-09-05T16:36:30.512415+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:36:30.512415+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4414,"log_dups_size":0,"ondisk_log_size":4414,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36133456,"num_objects":4411,"num_object_clones":0,"num_object_copies":17644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4411,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4413,"num_write_kb":35294,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2161}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'4445","reported_seq":7233,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:30.654994+0000","last_change":"2024-09-05T16:34:56.800371+0000","last_active":"2024-09-05T16:36:30.654994+0000","last_peered":"2024-09-05T16:36:30.654994+0000","last_clean":"2024-09-05T16:33:53.148717+0000","last_became_active":"2024-09-05T16:34:56.791755+0000","last_became_peered":"2024-09-05T16:34:56.791755+0000","last_unstale":"2024-09-05T16:36:30.654994+0000","last_undegraded":"2024-09-05T16:34:56.597522+0000","last_fullsized":"2024-09-05T16:34:56.568150+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4445,"log_dups_size":0,"ondisk_log_size":4445,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36377760,"num_objects":4441,"num_object_clones":0,"num_object_copies":17764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":928,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4441,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4443,"num_write_kb":35533,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1364,"num_bytes_recovered":11141120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3513}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'4282","reported_seq":4336,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:29.615534+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:36:29.615534+0000","last_peered":"2024-09-05T16:36:29.615534+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:36:29.615534+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4282,"log_dups_size":0,"ondisk_log_size":4282,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35050656,"num_objects":4279,"num_object_clones":0,"num_object_copies":17116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4279,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4280,"num_write_kb":34237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2117}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'4429","reported_seq":4471,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:29.669260+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:36:29.669260+0000","last_peered":"2024-09-05T16:36:29.669260+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:36:29.669260+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4429,"log_dups_size":0,"ondisk_log_size":4429,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36282368,"num_objects":4429,"num_object_clones":0,"num_object_copies":17716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4429,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4429,"num_write_kb":35432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2179}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'4313","reported_seq":4378,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:30.536638+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:36:30.536638+0000","last_peered":"2024-09-05T16:36:30.536638+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:36:30.536638+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4313,"log_dups_size":0,"ondisk_log_size":4313,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35280032,"num_objects":4307,"num_object_clones":0,"num_object_copies":17228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4307,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4311,"num_write_kb":34468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2160}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'4398","reported_seq":4462,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:30.550986+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:36:30.550986+0000","last_peered":"2024-09-05T16:36:30.550986+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:36:30.550986+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4398,"log_dups_size":0,"ondisk_log_size":4398,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35984544,"num_objects":4393,"num_object_clones":0,"num_object_copies":17572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4393,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4396,"num_write_kb":35156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2154}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'4408","reported_seq":4464,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:29.650525+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:36:29.650525+0000","last_peered":"2024-09-05T16:36:29.650525+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:36:29.650525+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4408,"log_dups_size":0,"ondisk_log_size":4408,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36038976,"num_objects":4400,"num_object_clones":0,"num_object_copies":17600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4400,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4405,"num_write_kb":35217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2139}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'4441","reported_seq":4518,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:30.300290+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:36:30.300290+0000","last_peered":"2024-09-05T16:36:30.300290+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:36:30.300290+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4441,"log_dups_size":0,"ondisk_log_size":4441,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36361376,"num_objects":4439,"num_object_clones":0,"num_object_copies":17756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4439,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4439,"num_write_kb":35510,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2177}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'4499","reported_seq":4555,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:29.582457+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:36:29.582457+0000","last_peered":"2024-09-05T16:36:29.582457+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:36:29.582457+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4499,"log_dups_size":0,"ondisk_log_size":4499,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36810480,"num_objects":4494,"num_object_clones":0,"num_object_copies":17976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4494,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4496,"num_write_kb":35956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2183}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'4336","reported_seq":4370,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:29.692707+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:36:29.692707+0000","last_peered":"2024-09-05T16:36:29.692707+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:36:29.692707+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4336,"log_dups_size":0,"ondisk_log_size":4336,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35510864,"num_objects":4335,"num_object_clones":0,"num_object_copies":17340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4335,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4335,"num_write_kb":34679,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2071}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'4456","reported_seq":4530,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:30.495248+0000","last_change":"2024-09-05T16:34:56.798081+0000","last_active":"2024-09-05T16:36:30.495248+0000","last_peered":"2024-09-05T16:36:30.495248+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:36:30.495248+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4456,"log_dups_size":0,"ondisk_log_size":4456,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36493904,"num_objects":4455,"num_object_clones":0,"num_object_copies":17820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4455,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4455,"num_write_kb":35639,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2161}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'4497","reported_seq":4547,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:29.650576+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:36:29.650576+0000","last_peered":"2024-09-05T16:36:29.650576+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:36:29.650576+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4497,"log_dups_size":0,"ondisk_log_size":4497,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36803744,"num_objects":4493,"num_object_clones":0,"num_object_copies":17972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4493,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4495,"num_write_kb":35956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2136}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'4437","reported_seq":4515,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:30.221511+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:36:30.221511+0000","last_peered":"2024-09-05T16:36:30.221511+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:36:30.221511+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4437,"log_dups_size":0,"ondisk_log_size":4437,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36320416,"num_objects":4434,"num_object_clones":0,"num_object_copies":17736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4434,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4435,"num_write_kb":35477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2114}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'4428","reported_seq":4492,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:30.593263+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:36:30.593263+0000","last_peered":"2024-09-05T16:36:30.593263+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:36:30.593263+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4428,"log_dups_size":0,"ondisk_log_size":4428,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36230304,"num_objects":4423,"num_object_clones":0,"num_object_copies":17692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4423,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4426,"num_write_kb":35396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2186}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":577470656,"num_objects":70497,"num_object_clones":0,"num_object_copies":281988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34836,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70497,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":70524,"num_write_kb":564074,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1389,"num_bytes_recovered":11141120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1011802112,"data_stored":1011802112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70551,"ondisk_log_size":70551,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084319,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":493240,"kb_used_data":281064,"kb_used_omap":4,"kb_used_meta":212155,"kb_avail":93878600,"statfs":{"total":96636764160,"available":96131686400,"internally_reserved":0,"allocated":287809536,"data_stored":287626654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":217247443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,0,8,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986196,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307888,"kb_used_data":141536,"kb_used_omap":4,"kb_used_meta":166331,"kb_avail":94063952,"statfs":{"total":96636764160,"available":96321486848,"internally_reserved":0,"allocated":144932864,"data_stored":144749982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":170323666},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,6,6],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673002,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":497864,"kb_used_data":282040,"kb_used_omap":1,"kb_used_meta":215806,"kb_avail":93873976,"statfs":{"total":96636764160,"available":96126951424,"internally_reserved":0,"allocated":288808960,"data_stored":288626078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":220985797},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182433,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":493880,"kb_used_data":281768,"kb_used_omap":4,"kb_used_meta":212091,"kb_avail":93877960,"statfs":{"total":96636764160,"available":96131031040,"internally_reserved":0,"allocated":288530432,"data_stored":288347550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":217181901},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,4,4,13,4,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":289378304,"data_stored":289378304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":145133568,"data_stored":145133568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":289255424,"data_stored":289255424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":288034816,"data_stored":288034816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:36:31.057 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:36:31.514 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 129 20 70906 70886 4.29217 2.32812 0.0581449 0.0363901 2024-09-05T16:36:31.616 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:36:31.617 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:36:31.643 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":121,"stamp":"2024-09-05T16:36:30.831132+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":577470656,"num_objects":70497,"num_object_clones":0,"num_object_copies":281988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34836,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70497,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":70524,"num_write_kb":564074,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1389,"num_bytes_recovered":11141120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70551,"ondisk_log_size":70551,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1792872,"kb_used_data":986408,"kb_used_omap":14,"kb_used_meta":806385,"kb_avail":375694488,"statfs":{"total":386547056640,"available":384711155712,"internally_reserved":0,"allocated":1010081792,"data_stored":1009350264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":825738807},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,5,9,11,28,8,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":40927232,"num_objects":4996,"num_object_clones":0,"num_object_copies":19984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4996,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4996,"num_write_kb":39968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":232,"num_bytes_recovered":1900544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005181"},"pg_stats":[{"pgid":"1.f","version":"33'4418","reported_seq":4459,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:29.693545+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:36:29.693545+0000","last_peered":"2024-09-05T16:36:29.693545+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:36:29.693545+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4418,"log_dups_size":0,"ondisk_log_size":4418,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36192256,"num_objects":4418,"num_object_clones":0,"num_object_copies":17672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4418,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4418,"num_write_kb":35344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2156}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'4350","reported_seq":4414,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:30.425269+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:36:30.425269+0000","last_peered":"2024-09-05T16:36:30.425269+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:36:30.425269+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4350,"log_dups_size":0,"ondisk_log_size":4350,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35599520,"num_objects":4346,"num_object_clones":0,"num_object_copies":17384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4346,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4348,"num_write_kb":34780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2079}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'4414","reported_seq":4488,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:30.512415+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:36:30.512415+0000","last_peered":"2024-09-05T16:36:30.512415+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:36:30.512415+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4414,"log_dups_size":0,"ondisk_log_size":4414,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36133456,"num_objects":4411,"num_object_clones":0,"num_object_copies":17644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4411,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4413,"num_write_kb":35294,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2161}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'4445","reported_seq":7233,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:30.654994+0000","last_change":"2024-09-05T16:34:56.800371+0000","last_active":"2024-09-05T16:36:30.654994+0000","last_peered":"2024-09-05T16:36:30.654994+0000","last_clean":"2024-09-05T16:33:53.148717+0000","last_became_active":"2024-09-05T16:34:56.791755+0000","last_became_peered":"2024-09-05T16:34:56.791755+0000","last_unstale":"2024-09-05T16:36:30.654994+0000","last_undegraded":"2024-09-05T16:34:56.597522+0000","last_fullsized":"2024-09-05T16:34:56.568150+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4445,"log_dups_size":0,"ondisk_log_size":4445,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36377760,"num_objects":4441,"num_object_clones":0,"num_object_copies":17764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":928,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4441,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4443,"num_write_kb":35533,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1364,"num_bytes_recovered":11141120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3513}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'4282","reported_seq":4336,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:29.615534+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:36:29.615534+0000","last_peered":"2024-09-05T16:36:29.615534+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:36:29.615534+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4282,"log_dups_size":0,"ondisk_log_size":4282,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35050656,"num_objects":4279,"num_object_clones":0,"num_object_copies":17116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4279,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4280,"num_write_kb":34237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2117}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'4429","reported_seq":4471,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:29.669260+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:36:29.669260+0000","last_peered":"2024-09-05T16:36:29.669260+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:36:29.669260+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4429,"log_dups_size":0,"ondisk_log_size":4429,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36282368,"num_objects":4429,"num_object_clones":0,"num_object_copies":17716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4429,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4429,"num_write_kb":35432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2179}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'4313","reported_seq":4378,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:30.536638+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:36:30.536638+0000","last_peered":"2024-09-05T16:36:30.536638+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:36:30.536638+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4313,"log_dups_size":0,"ondisk_log_size":4313,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35280032,"num_objects":4307,"num_object_clones":0,"num_object_copies":17228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4307,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4311,"num_write_kb":34468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2160}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'4398","reported_seq":4462,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:30.550986+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:36:30.550986+0000","last_peered":"2024-09-05T16:36:30.550986+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:36:30.550986+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4398,"log_dups_size":0,"ondisk_log_size":4398,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35984544,"num_objects":4393,"num_object_clones":0,"num_object_copies":17572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4393,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4396,"num_write_kb":35156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2154}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'4408","reported_seq":4464,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:29.650525+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:36:29.650525+0000","last_peered":"2024-09-05T16:36:29.650525+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:36:29.650525+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4408,"log_dups_size":0,"ondisk_log_size":4408,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36038976,"num_objects":4400,"num_object_clones":0,"num_object_copies":17600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4400,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4405,"num_write_kb":35217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2139}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'4441","reported_seq":4518,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:30.300290+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:36:30.300290+0000","last_peered":"2024-09-05T16:36:30.300290+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:36:30.300290+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4441,"log_dups_size":0,"ondisk_log_size":4441,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36361376,"num_objects":4439,"num_object_clones":0,"num_object_copies":17756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4439,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4439,"num_write_kb":35510,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2177}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'4499","reported_seq":4555,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:29.582457+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:36:29.582457+0000","last_peered":"2024-09-05T16:36:29.582457+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:36:29.582457+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4499,"log_dups_size":0,"ondisk_log_size":4499,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36810480,"num_objects":4494,"num_object_clones":0,"num_object_copies":17976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4494,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4496,"num_write_kb":35956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2183}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'4336","reported_seq":4370,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:29.692707+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:36:29.692707+0000","last_peered":"2024-09-05T16:36:29.692707+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:36:29.692707+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4336,"log_dups_size":0,"ondisk_log_size":4336,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35510864,"num_objects":4335,"num_object_clones":0,"num_object_copies":17340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4335,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4335,"num_write_kb":34679,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2071}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'4456","reported_seq":4530,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:30.495248+0000","last_change":"2024-09-05T16:34:56.798081+0000","last_active":"2024-09-05T16:36:30.495248+0000","last_peered":"2024-09-05T16:36:30.495248+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:36:30.495248+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4456,"log_dups_size":0,"ondisk_log_size":4456,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36493904,"num_objects":4455,"num_object_clones":0,"num_object_copies":17820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4455,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4455,"num_write_kb":35639,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2161}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'4497","reported_seq":4547,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:29.650576+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:36:29.650576+0000","last_peered":"2024-09-05T16:36:29.650576+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:36:29.650576+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4497,"log_dups_size":0,"ondisk_log_size":4497,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36803744,"num_objects":4493,"num_object_clones":0,"num_object_copies":17972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4493,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4495,"num_write_kb":35956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2136}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'4437","reported_seq":4515,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:30.221511+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:36:30.221511+0000","last_peered":"2024-09-05T16:36:30.221511+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:36:30.221511+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4437,"log_dups_size":0,"ondisk_log_size":4437,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36320416,"num_objects":4434,"num_object_clones":0,"num_object_copies":17736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4434,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4435,"num_write_kb":35477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2114}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'4428","reported_seq":4492,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:30.593263+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:36:30.593263+0000","last_peered":"2024-09-05T16:36:30.593263+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:36:30.593263+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4428,"log_dups_size":0,"ondisk_log_size":4428,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36230304,"num_objects":4423,"num_object_clones":0,"num_object_copies":17692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4423,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4426,"num_write_kb":35396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2186}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":577470656,"num_objects":70497,"num_object_clones":0,"num_object_copies":281988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34836,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70497,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":70524,"num_write_kb":564074,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1389,"num_bytes_recovered":11141120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1011802112,"data_stored":1011802112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70551,"ondisk_log_size":70551,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084319,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":493240,"kb_used_data":281064,"kb_used_omap":4,"kb_used_meta":212155,"kb_avail":93878600,"statfs":{"total":96636764160,"available":96131686400,"internally_reserved":0,"allocated":287809536,"data_stored":287626654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":217247443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,0,8,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986196,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307888,"kb_used_data":141536,"kb_used_omap":4,"kb_used_meta":166331,"kb_avail":94063952,"statfs":{"total":96636764160,"available":96321486848,"internally_reserved":0,"allocated":144932864,"data_stored":144749982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":170323666},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,6,6],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673002,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":497864,"kb_used_data":282040,"kb_used_omap":1,"kb_used_meta":215806,"kb_avail":93873976,"statfs":{"total":96636764160,"available":96126951424,"internally_reserved":0,"allocated":288808960,"data_stored":288626078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":220985797},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182433,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":493880,"kb_used_data":281768,"kb_used_omap":4,"kb_used_meta":212091,"kb_avail":93877960,"statfs":{"total":96636764160,"available":96131031040,"internally_reserved":0,"allocated":288530432,"data_stored":288347550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":217181901},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,4,4,13,4,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":289378304,"data_stored":289378304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":145133568,"data_stored":145133568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":289255424,"data_stored":289255424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":288034816,"data_stored":288034816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:36:31.644 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-05T16:36:31.645 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'4418", 'reported_seq': 4459, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:29.693545+0000', 'last_change': '2024-09-05T16:34:56.833497+0000', 'last_active': '2024-09-05T16:36:29.693545+0000', 'last_peered': '2024-09-05T16:36:29.693545+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.828289+0000', 'last_became_peered': '2024-09-05T16:34:56.828289+0000', 'last_unstale': '2024-09-05T16:36:29.693545+0000', 'last_undegraded': '2024-09-05T16:34:56.612160+0000', 'last_fullsized': '2024-09-05T16:34:56.557632+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4418, 'log_dups_size': 0, 'ondisk_log_size': 4418, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36192256, 'num_objects': 4418, 'num_object_clones': 0, 'num_object_copies': 17672, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2262, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4418, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4418, 'num_write_kb': 35344, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2156}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:36:31.645 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-05T16:36:31.645 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'4350", 'reported_seq': 4414, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:30.425269+0000', 'last_change': '2024-09-05T16:34:56.825031+0000', 'last_active': '2024-09-05T16:36:30.425269+0000', 'last_peered': '2024-09-05T16:36:30.425269+0000', 'last_clean': '2024-09-05T16:33:53.200762+0000', 'last_became_active': '2024-09-05T16:34:56.820292+0000', 'last_became_peered': '2024-09-05T16:34:56.820292+0000', 'last_unstale': '2024-09-05T16:36:30.425269+0000', 'last_undegraded': '2024-09-05T16:34:56.605776+0000', 'last_fullsized': '2024-09-05T16:34:56.559383+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4350, 'log_dups_size': 0, 'ondisk_log_size': 4350, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35599520, 'num_objects': 4346, 'num_object_clones': 0, 'num_object_copies': 17384, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2269, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4346, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4348, 'num_write_kb': 34780, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2079}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:36:31.645 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:36:31.645 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'4414", 'reported_seq': 4488, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:30.512415+0000', 'last_change': '2024-09-05T16:34:56.867900+0000', 'last_active': '2024-09-05T16:36:30.512415+0000', 'last_peered': '2024-09-05T16:36:30.512415+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:36:30.512415+0000', 'last_undegraded': '2024-09-05T16:34:56.590972+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4414, 'log_dups_size': 0, 'ondisk_log_size': 4414, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36133456, 'num_objects': 4411, 'num_object_clones': 0, 'num_object_copies': 17644, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2251, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4411, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4413, 'num_write_kb': 35294, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2161}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:36:31.645 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-05T16:36:31.645 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'4445", 'reported_seq': 7233, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:30.654994+0000', 'last_change': '2024-09-05T16:34:56.800371+0000', 'last_active': '2024-09-05T16:36:30.654994+0000', 'last_peered': '2024-09-05T16:36:30.654994+0000', 'last_clean': '2024-09-05T16:33:53.148717+0000', 'last_became_active': '2024-09-05T16:34:56.791755+0000', 'last_became_peered': '2024-09-05T16:34:56.791755+0000', 'last_unstale': '2024-09-05T16:36:30.654994+0000', 'last_undegraded': '2024-09-05T16:34:56.597522+0000', 'last_fullsized': '2024-09-05T16:34:56.568150+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4445, 'log_dups_size': 0, 'ondisk_log_size': 4445, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36377760, 'num_objects': 4441, 'num_object_clones': 0, 'num_object_copies': 17764, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 928, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4441, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4443, 'num_write_kb': 35533, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1364, 'num_bytes_recovered': 11141120, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3513}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:36:31.645 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:36:31.645 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'4282", 'reported_seq': 4336, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:29.615534+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:36:29.615534+0000', 'last_peered': '2024-09-05T16:36:29.615534+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:36:29.615534+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4282, 'log_dups_size': 0, 'ondisk_log_size': 4282, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35050656, 'num_objects': 4279, 'num_object_clones': 0, 'num_object_copies': 17116, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2164, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4279, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4280, 'num_write_kb': 34237, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2117}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:36:31.646 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:36:31.646 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'4429", 'reported_seq': 4471, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:29.669260+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:36:29.669260+0000', 'last_peered': '2024-09-05T16:36:29.669260+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:36:29.669260+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4429, 'log_dups_size': 0, 'ondisk_log_size': 4429, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36282368, 'num_objects': 4429, 'num_object_clones': 0, 'num_object_copies': 17716, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4429, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4429, 'num_write_kb': 35432, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2179}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:36:31.646 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:36:31.646 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'4313", 'reported_seq': 4378, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:30.536638+0000', 'last_change': '2024-09-05T16:34:56.903436+0000', 'last_active': '2024-09-05T16:36:30.536638+0000', 'last_peered': '2024-09-05T16:36:30.536638+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:36:30.536638+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4313, 'log_dups_size': 0, 'ondisk_log_size': 4313, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35280032, 'num_objects': 4307, 'num_object_clones': 0, 'num_object_copies': 17228, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2149, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4307, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4311, 'num_write_kb': 34468, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2160}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:36:31.646 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-05T16:36:31.646 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'4398", 'reported_seq': 4462, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:30.550986+0000', 'last_change': '2024-09-05T16:34:56.883319+0000', 'last_active': '2024-09-05T16:36:30.550986+0000', 'last_peered': '2024-09-05T16:36:30.550986+0000', 'last_clean': '2024-09-05T16:33:53.505305+0000', 'last_became_active': '2024-09-05T16:34:56.873389+0000', 'last_became_peered': '2024-09-05T16:34:56.873389+0000', 'last_unstale': '2024-09-05T16:36:30.550986+0000', 'last_undegraded': '2024-09-05T16:34:56.602567+0000', 'last_fullsized': '2024-09-05T16:34:56.559958+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4398, 'log_dups_size': 0, 'ondisk_log_size': 4398, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35984544, 'num_objects': 4393, 'num_object_clones': 0, 'num_object_copies': 17572, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2241, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4393, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4396, 'num_write_kb': 35156, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2154}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:36:31.646 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:36:31.646 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'4408", 'reported_seq': 4464, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:29.650525+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:36:29.650525+0000', 'last_peered': '2024-09-05T16:36:29.650525+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:36:29.650525+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4408, 'log_dups_size': 0, 'ondisk_log_size': 4408, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36038976, 'num_objects': 4400, 'num_object_clones': 0, 'num_object_copies': 17600, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2264, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4400, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4405, 'num_write_kb': 35217, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2139}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:36:31.646 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:36:31.646 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'4441", 'reported_seq': 4518, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:30.300290+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:36:30.300290+0000', 'last_peered': '2024-09-05T16:36:30.300290+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:36:30.300290+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4441, 'log_dups_size': 0, 'ondisk_log_size': 4441, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36361376, 'num_objects': 4439, 'num_object_clones': 0, 'num_object_copies': 17756, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2264, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4439, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4439, 'num_write_kb': 35510, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2177}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:36:31.646 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:36:31.647 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'4499", 'reported_seq': 4555, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:29.582457+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:36:29.582457+0000', 'last_peered': '2024-09-05T16:36:29.582457+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:36:29.582457+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4499, 'log_dups_size': 0, 'ondisk_log_size': 4499, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36810480, 'num_objects': 4494, 'num_object_clones': 0, 'num_object_copies': 17976, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2314, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4494, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4496, 'num_write_kb': 35956, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2183}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:36:31.647 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:36:31.647 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'4336", 'reported_seq': 4370, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:29.692707+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:36:29.692707+0000', 'last_peered': '2024-09-05T16:36:29.692707+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:36:29.692707+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4336, 'log_dups_size': 0, 'ondisk_log_size': 4336, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35510864, 'num_objects': 4335, 'num_object_clones': 0, 'num_object_copies': 17340, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2265, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4335, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4335, 'num_write_kb': 34679, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2071}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:36:31.647 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-05T16:36:31.647 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'4456", 'reported_seq': 4530, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:30.495248+0000', 'last_change': '2024-09-05T16:34:56.798081+0000', 'last_active': '2024-09-05T16:36:30.495248+0000', 'last_peered': '2024-09-05T16:36:30.495248+0000', 'last_clean': '2024-09-05T16:33:45.082039+0000', 'last_became_active': '2024-09-05T16:34:56.792386+0000', 'last_became_peered': '2024-09-05T16:34:56.792386+0000', 'last_unstale': '2024-09-05T16:36:30.495248+0000', 'last_undegraded': '2024-09-05T16:34:56.600595+0000', 'last_fullsized': '2024-09-05T16:34:56.558203+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4456, 'log_dups_size': 0, 'ondisk_log_size': 4456, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36493904, 'num_objects': 4455, 'num_object_clones': 0, 'num_object_copies': 17820, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2295, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4455, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4455, 'num_write_kb': 35639, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2161}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:36:31.647 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:36:31.647 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'4497", 'reported_seq': 4547, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:29.650576+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:36:29.650576+0000', 'last_peered': '2024-09-05T16:36:29.650576+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:36:29.650576+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4497, 'log_dups_size': 0, 'ondisk_log_size': 4497, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36803744, 'num_objects': 4493, 'num_object_clones': 0, 'num_object_copies': 17972, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2359, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4493, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4495, 'num_write_kb': 35956, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2136}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:36:31.647 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:36:31.647 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'4437", 'reported_seq': 4515, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:30.221511+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:36:30.221511+0000', 'last_peered': '2024-09-05T16:36:30.221511+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:36:30.221511+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4437, 'log_dups_size': 0, 'ondisk_log_size': 4437, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36320416, 'num_objects': 4434, 'num_object_clones': 0, 'num_object_copies': 17736, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2322, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4434, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4435, 'num_write_kb': 35477, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2114}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:36:31.647 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:36:31.647 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'4428", 'reported_seq': 4492, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:30.593263+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:36:30.593263+0000', 'last_peered': '2024-09-05T16:36:30.593263+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:36:30.593263+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4428, 'log_dups_size': 0, 'ondisk_log_size': 4428, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36230304, 'num_objects': 4423, 'num_object_clones': 0, 'num_object_copies': 17692, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4423, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4426, 'num_write_kb': 35396, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2186}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:36:31.648 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:36:32.295 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:36:32.295 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 34836/281988 objects degraded (12.354%), 16 pgs degraded, 16 pgs undersized","count":32},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":195,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":70497,"data_bytes":577470656,"bytes_used":1835900928,"bytes_avail":384711155712,"bytes_total":386547056640,"degraded_objects":34836,"degraded_total":281988,"degraded_ratio":0.12353717179454443,"recovering_objects_per_sec":19,"recovering_bytes_per_sec":158310,"recovering_keys_per_sec":0,"num_objects_recovered":232,"num_bytes_recovered":1900544,"num_keys_recovered":0,"write_bytes_sec":3409130,"read_op_per_sec":0,"write_op_per_sec":416},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:36:32.324 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:36:32.324 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:36:32.514 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 130 20 71171 71151 4.27508 2.07031 0.059971 0.0365337 2024-09-05T16:36:32.959 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:36:32.959 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:36:32.984 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":122,"stamp":"2024-09-05T16:36:32.832082+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":577470656,"num_objects":70497,"num_object_clones":0,"num_object_copies":281988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34836,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70497,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":70524,"num_write_kb":564074,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1389,"num_bytes_recovered":11141120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70551,"ondisk_log_size":70551,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1792872,"kb_used_data":986408,"kb_used_omap":14,"kb_used_meta":806385,"kb_avail":375694488,"statfs":{"total":386547056640,"available":384711155712,"internally_reserved":0,"allocated":1010081792,"data_stored":1009350264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":825738807},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,5,9,11,28,8,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":27058176,"num_objects":3303,"num_object_clones":0,"num_object_copies":13212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3303,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3303,"num_write_kb":26424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":158,"num_bytes_recovered":1294336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005432"},"pg_stats":[{"pgid":"1.f","version":"33'4418","reported_seq":4459,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:29.693545+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:36:29.693545+0000","last_peered":"2024-09-05T16:36:29.693545+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:36:29.693545+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4418,"log_dups_size":0,"ondisk_log_size":4418,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36192256,"num_objects":4418,"num_object_clones":0,"num_object_copies":17672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4418,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4418,"num_write_kb":35344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2156}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'4350","reported_seq":4414,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:30.425269+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:36:30.425269+0000","last_peered":"2024-09-05T16:36:30.425269+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:36:30.425269+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4350,"log_dups_size":0,"ondisk_log_size":4350,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35599520,"num_objects":4346,"num_object_clones":0,"num_object_copies":17384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4346,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4348,"num_write_kb":34780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2079}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'4414","reported_seq":4488,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:30.512415+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:36:30.512415+0000","last_peered":"2024-09-05T16:36:30.512415+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:36:30.512415+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4414,"log_dups_size":0,"ondisk_log_size":4414,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36133456,"num_objects":4411,"num_object_clones":0,"num_object_copies":17644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4411,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4413,"num_write_kb":35294,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2161}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'4445","reported_seq":7233,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:30.654994+0000","last_change":"2024-09-05T16:34:56.800371+0000","last_active":"2024-09-05T16:36:30.654994+0000","last_peered":"2024-09-05T16:36:30.654994+0000","last_clean":"2024-09-05T16:33:53.148717+0000","last_became_active":"2024-09-05T16:34:56.791755+0000","last_became_peered":"2024-09-05T16:34:56.791755+0000","last_unstale":"2024-09-05T16:36:30.654994+0000","last_undegraded":"2024-09-05T16:34:56.597522+0000","last_fullsized":"2024-09-05T16:34:56.568150+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4445,"log_dups_size":0,"ondisk_log_size":4445,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36377760,"num_objects":4441,"num_object_clones":0,"num_object_copies":17764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":928,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4441,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4443,"num_write_kb":35533,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1364,"num_bytes_recovered":11141120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3513}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'4282","reported_seq":4336,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:29.615534+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:36:29.615534+0000","last_peered":"2024-09-05T16:36:29.615534+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:36:29.615534+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4282,"log_dups_size":0,"ondisk_log_size":4282,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35050656,"num_objects":4279,"num_object_clones":0,"num_object_copies":17116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4279,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4280,"num_write_kb":34237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2117}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'4429","reported_seq":4471,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:29.669260+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:36:29.669260+0000","last_peered":"2024-09-05T16:36:29.669260+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:36:29.669260+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4429,"log_dups_size":0,"ondisk_log_size":4429,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36282368,"num_objects":4429,"num_object_clones":0,"num_object_copies":17716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4429,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4429,"num_write_kb":35432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2179}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'4313","reported_seq":4378,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:30.536638+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:36:30.536638+0000","last_peered":"2024-09-05T16:36:30.536638+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:36:30.536638+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4313,"log_dups_size":0,"ondisk_log_size":4313,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35280032,"num_objects":4307,"num_object_clones":0,"num_object_copies":17228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4307,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4311,"num_write_kb":34468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2160}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'4398","reported_seq":4462,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:30.550986+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:36:30.550986+0000","last_peered":"2024-09-05T16:36:30.550986+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:36:30.550986+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4398,"log_dups_size":0,"ondisk_log_size":4398,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35984544,"num_objects":4393,"num_object_clones":0,"num_object_copies":17572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4393,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4396,"num_write_kb":35156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2154}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'4408","reported_seq":4464,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:29.650525+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:36:29.650525+0000","last_peered":"2024-09-05T16:36:29.650525+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:36:29.650525+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4408,"log_dups_size":0,"ondisk_log_size":4408,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36038976,"num_objects":4400,"num_object_clones":0,"num_object_copies":17600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4400,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4405,"num_write_kb":35217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2139}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'4441","reported_seq":4518,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:30.300290+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:36:30.300290+0000","last_peered":"2024-09-05T16:36:30.300290+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:36:30.300290+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4441,"log_dups_size":0,"ondisk_log_size":4441,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36361376,"num_objects":4439,"num_object_clones":0,"num_object_copies":17756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4439,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4439,"num_write_kb":35510,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2177}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'4499","reported_seq":4555,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:29.582457+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:36:29.582457+0000","last_peered":"2024-09-05T16:36:29.582457+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:36:29.582457+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4499,"log_dups_size":0,"ondisk_log_size":4499,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36810480,"num_objects":4494,"num_object_clones":0,"num_object_copies":17976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4494,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4496,"num_write_kb":35956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2183}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'4336","reported_seq":4370,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:29.692707+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:36:29.692707+0000","last_peered":"2024-09-05T16:36:29.692707+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:36:29.692707+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4336,"log_dups_size":0,"ondisk_log_size":4336,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35510864,"num_objects":4335,"num_object_clones":0,"num_object_copies":17340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4335,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4335,"num_write_kb":34679,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2071}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'4456","reported_seq":4530,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:30.495248+0000","last_change":"2024-09-05T16:34:56.798081+0000","last_active":"2024-09-05T16:36:30.495248+0000","last_peered":"2024-09-05T16:36:30.495248+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:36:30.495248+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4456,"log_dups_size":0,"ondisk_log_size":4456,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36493904,"num_objects":4455,"num_object_clones":0,"num_object_copies":17820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4455,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4455,"num_write_kb":35639,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2161}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'4497","reported_seq":4547,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:29.650576+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:36:29.650576+0000","last_peered":"2024-09-05T16:36:29.650576+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:36:29.650576+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4497,"log_dups_size":0,"ondisk_log_size":4497,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36803744,"num_objects":4493,"num_object_clones":0,"num_object_copies":17972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4493,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4495,"num_write_kb":35956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2136}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'4437","reported_seq":4515,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:30.221511+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:36:30.221511+0000","last_peered":"2024-09-05T16:36:30.221511+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:36:30.221511+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4437,"log_dups_size":0,"ondisk_log_size":4437,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36320416,"num_objects":4434,"num_object_clones":0,"num_object_copies":17736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4434,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4435,"num_write_kb":35477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2114}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'4428","reported_seq":4492,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:30.593263+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:36:30.593263+0000","last_peered":"2024-09-05T16:36:30.593263+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:36:30.593263+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4428,"log_dups_size":0,"ondisk_log_size":4428,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36230304,"num_objects":4423,"num_object_clones":0,"num_object_copies":17692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4423,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4426,"num_write_kb":35396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2186}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":577470656,"num_objects":70497,"num_object_clones":0,"num_object_copies":281988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34836,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70497,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":70524,"num_write_kb":564074,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1389,"num_bytes_recovered":11141120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1011802112,"data_stored":1011802112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70551,"ondisk_log_size":70551,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084319,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":493240,"kb_used_data":281064,"kb_used_omap":4,"kb_used_meta":212155,"kb_avail":93878600,"statfs":{"total":96636764160,"available":96131686400,"internally_reserved":0,"allocated":287809536,"data_stored":287626654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":217247443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,0,8,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986196,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307888,"kb_used_data":141536,"kb_used_omap":4,"kb_used_meta":166331,"kb_avail":94063952,"statfs":{"total":96636764160,"available":96321486848,"internally_reserved":0,"allocated":144932864,"data_stored":144749982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":170323666},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,6,6],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673002,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":497864,"kb_used_data":282040,"kb_used_omap":1,"kb_used_meta":215806,"kb_avail":93873976,"statfs":{"total":96636764160,"available":96126951424,"internally_reserved":0,"allocated":288808960,"data_stored":288626078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":220985797},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182433,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":493880,"kb_used_data":281768,"kb_used_omap":4,"kb_used_meta":212091,"kb_avail":93877960,"statfs":{"total":96636764160,"available":96131031040,"internally_reserved":0,"allocated":288530432,"data_stored":288347550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":217181901},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,4,4,13,4,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":289378304,"data_stored":289378304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":145133568,"data_stored":145133568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":289255424,"data_stored":289255424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":288034816,"data_stored":288034816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:36:33.514 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 131 20 71495 71475 4.26177 2.53125 0.0423012 0.0366502 2024-09-05T16:36:34.514 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 132 20 71835 71815 4.24961 2.65625 0.0389274 0.0367516 2024-09-05T16:36:35.515 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 133 20 72180 72160 4.2379 2.69531 0.0649149 0.0368554 2024-09-05T16:36:35.986 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:36:36.515 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 134 20 72498 72478 4.22482 2.48438 0.0774092 0.0369666 2024-09-05T16:36:36.608 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:36:36.608 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:36:36.641 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":123,"stamp":"2024-09-05T16:36:34.832703+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":583131328,"num_objects":71188,"num_object_clones":0,"num_object_copies":284752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34836,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71188,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71215,"num_write_kb":569602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1389,"num_bytes_recovered":11141120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71242,"ondisk_log_size":71242,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1832276,"kb_used_data":999444,"kb_used_omap":14,"kb_used_meta":832753,"kb_avail":375655084,"statfs":{"total":386547056640,"available":384670806016,"internally_reserved":0,"allocated":1023430656,"data_stored":1022699128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":852739639},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,2,8,13,24,13,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":32718848,"num_objects":3994,"num_object_clones":0,"num_object_copies":15976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3994,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3994,"num_write_kb":31952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":158,"num_bytes_recovered":1294336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005127"},"pg_stats":[{"pgid":"1.f","version":"33'4522","reported_seq":4564,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:34.458700+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:36:34.458700+0000","last_peered":"2024-09-05T16:36:34.458700+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:36:34.458700+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4522,"log_dups_size":0,"ondisk_log_size":4522,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37044224,"num_objects":4522,"num_object_clones":0,"num_object_copies":18088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4522,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4522,"num_write_kb":36176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2260}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'4350","reported_seq":4414,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:30.425269+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:36:30.425269+0000","last_peered":"2024-09-05T16:36:30.425269+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:36:30.425269+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4350,"log_dups_size":0,"ondisk_log_size":4350,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35599520,"num_objects":4346,"num_object_clones":0,"num_object_copies":17384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4346,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4348,"num_write_kb":34780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2079}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'4414","reported_seq":4488,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:30.512415+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:36:30.512415+0000","last_peered":"2024-09-05T16:36:30.512415+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:36:30.512415+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4414,"log_dups_size":0,"ondisk_log_size":4414,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36133456,"num_objects":4411,"num_object_clones":0,"num_object_copies":17644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4411,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4413,"num_write_kb":35294,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2161}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'4445","reported_seq":7233,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:30.654994+0000","last_change":"2024-09-05T16:34:56.800371+0000","last_active":"2024-09-05T16:36:30.654994+0000","last_peered":"2024-09-05T16:36:30.654994+0000","last_clean":"2024-09-05T16:33:53.148717+0000","last_became_active":"2024-09-05T16:34:56.791755+0000","last_became_peered":"2024-09-05T16:34:56.791755+0000","last_unstale":"2024-09-05T16:36:30.654994+0000","last_undegraded":"2024-09-05T16:34:56.597522+0000","last_fullsized":"2024-09-05T16:34:56.568150+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4445,"log_dups_size":0,"ondisk_log_size":4445,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36377760,"num_objects":4441,"num_object_clones":0,"num_object_copies":17764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":928,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4441,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4443,"num_write_kb":35533,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1364,"num_bytes_recovered":11141120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3513}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'4381","reported_seq":4435,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:34.465595+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:36:34.465595+0000","last_peered":"2024-09-05T16:36:34.465595+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:36:34.465595+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4381,"log_dups_size":0,"ondisk_log_size":4381,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35861664,"num_objects":4378,"num_object_clones":0,"num_object_copies":17512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4378,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4379,"num_write_kb":35029,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2216}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'4511","reported_seq":4553,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:34.669787+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:36:34.669787+0000","last_peered":"2024-09-05T16:36:34.669787+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:36:34.669787+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4511,"log_dups_size":0,"ondisk_log_size":4511,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36954112,"num_objects":4511,"num_object_clones":0,"num_object_copies":18044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4511,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4511,"num_write_kb":36088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2261}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'4313","reported_seq":4378,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:30.536638+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:36:30.536638+0000","last_peered":"2024-09-05T16:36:30.536638+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:36:30.536638+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4313,"log_dups_size":0,"ondisk_log_size":4313,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35280032,"num_objects":4307,"num_object_clones":0,"num_object_copies":17228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4307,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4311,"num_write_kb":34468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2160}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'4398","reported_seq":4462,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:30.550986+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:36:30.550986+0000","last_peered":"2024-09-05T16:36:30.550986+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:36:30.550986+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4398,"log_dups_size":0,"ondisk_log_size":4398,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35984544,"num_objects":4393,"num_object_clones":0,"num_object_copies":17572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4393,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4396,"num_write_kb":35156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2154}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'4502","reported_seq":4558,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:34.597909+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:36:34.597909+0000","last_peered":"2024-09-05T16:36:34.597909+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:36:34.597909+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4502,"log_dups_size":0,"ondisk_log_size":4502,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36809024,"num_objects":4494,"num_object_clones":0,"num_object_copies":17976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4494,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4499,"num_write_kb":35969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2233}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'4441","reported_seq":4518,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:30.300290+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:36:30.300290+0000","last_peered":"2024-09-05T16:36:30.300290+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:36:30.300290+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4441,"log_dups_size":0,"ondisk_log_size":4441,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36361376,"num_objects":4439,"num_object_clones":0,"num_object_copies":17756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4439,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4439,"num_write_kb":35510,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2177}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'4597","reported_seq":4653,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:34.659491+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:36:34.659491+0000","last_peered":"2024-09-05T16:36:34.659491+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:36:34.659491+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4597,"log_dups_size":0,"ondisk_log_size":4597,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37613296,"num_objects":4592,"num_object_clones":0,"num_object_copies":18368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4592,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4594,"num_write_kb":36740,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2281}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'4439","reported_seq":4475,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:34.691271+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:36:34.691271+0000","last_peered":"2024-09-05T16:36:34.691271+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:36:34.691271+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4439,"log_dups_size":0,"ondisk_log_size":4439,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36354640,"num_objects":4438,"num_object_clones":0,"num_object_copies":17752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4438,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4438,"num_write_kb":35503,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2174}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'4456","reported_seq":4530,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:30.495248+0000","last_change":"2024-09-05T16:34:56.798081+0000","last_active":"2024-09-05T16:36:30.495248+0000","last_peered":"2024-09-05T16:36:30.495248+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:36:30.495248+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4456,"log_dups_size":0,"ondisk_log_size":4456,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36493904,"num_objects":4455,"num_object_clones":0,"num_object_copies":17820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4455,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4455,"num_write_kb":35639,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2161}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'4608","reported_seq":4660,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:34.664754+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:36:34.664754+0000","last_peered":"2024-09-05T16:36:34.664754+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:36:34.664754+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4608,"log_dups_size":0,"ondisk_log_size":4608,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37713056,"num_objects":4604,"num_object_clones":0,"num_object_copies":18416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4604,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4606,"num_write_kb":36844,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2247}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'4437","reported_seq":4515,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:30.221511+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:36:30.221511+0000","last_peered":"2024-09-05T16:36:30.221511+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:36:30.221511+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4437,"log_dups_size":0,"ondisk_log_size":4437,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36320416,"num_objects":4434,"num_object_clones":0,"num_object_copies":17736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4434,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4435,"num_write_kb":35477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2114}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'4428","reported_seq":4492,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:30.593263+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:36:30.593263+0000","last_peered":"2024-09-05T16:36:30.593263+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:36:30.593263+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4428,"log_dups_size":0,"ondisk_log_size":4428,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36230304,"num_objects":4423,"num_object_clones":0,"num_object_copies":17692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4423,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4426,"num_write_kb":35396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2186}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":583131328,"num_objects":71188,"num_object_clones":0,"num_object_copies":284752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34836,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71188,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71215,"num_write_kb":569602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1389,"num_bytes_recovered":11141120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1025159168,"data_stored":1025159168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71242,"ondisk_log_size":71242,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084320,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":521808,"kb_used_data":287424,"kb_used_omap":4,"kb_used_meta":234363,"kb_avail":93850032,"statfs":{"total":96636764160,"available":96102432768,"internally_reserved":0,"allocated":294322176,"data_stored":294139294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":239988435},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,3,7,4,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986197,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":318724,"kb_used_data":148212,"kb_used_omap":4,"kb_used_meta":170491,"kb_avail":94053116,"statfs":{"total":96636764160,"available":96310390784,"internally_reserved":0,"allocated":151769088,"data_stored":151586206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":174583506},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,5,3,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673002,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":497864,"kb_used_data":282040,"kb_used_omap":1,"kb_used_meta":215806,"kb_avail":93873976,"statfs":{"total":96636764160,"available":96126951424,"internally_reserved":0,"allocated":288808960,"data_stored":288626078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":220985797},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182433,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":493880,"kb_used_data":281768,"kb_used_omap":4,"kb_used_meta":212091,"kb_avail":93877960,"statfs":{"total":96636764160,"available":96131031040,"internally_reserved":0,"allocated":288530432,"data_stored":288347550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":217181901},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,4,4,13,4,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":289378304,"data_stored":289378304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":152014848,"data_stored":152014848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":289255424,"data_stored":289255424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":294510592,"data_stored":294510592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:36:36.642 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:36:37.254 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:36:37.256 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:36:37.281 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":124,"stamp":"2024-09-05T16:36:36.833374+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590618816,"num_objects":72102,"num_object_clones":0,"num_object_copies":288408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34746,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72102,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":72129,"num_write_kb":576914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1479,"num_bytes_recovered":11878400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72156,"ondisk_log_size":72156,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1889376,"kb_used_data":1012128,"kb_used_omap":14,"kb_used_meta":877169,"kb_avail":375597984,"statfs":{"total":386547056640,"available":384612335616,"internally_reserved":0,"allocated":1036419072,"data_stored":1035687544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":898221623},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,1,14,13,15,14,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":34340864,"num_objects":4192,"num_object_clones":0,"num_object_copies":16768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-248,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4192,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4192,"num_write_kb":33536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":248,"num_bytes_recovered":2031616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005241"},"pg_stats":[{"pgid":"1.f","version":"33'4522","reported_seq":4564,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:34.458700+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:36:34.458700+0000","last_peered":"2024-09-05T16:36:34.458700+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:36:34.458700+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4522,"log_dups_size":0,"ondisk_log_size":4522,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37044224,"num_objects":4522,"num_object_clones":0,"num_object_copies":18088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4522,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4522,"num_write_kb":36176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2260}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'4450","reported_seq":4513,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:35.665970+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:36:35.665970+0000","last_peered":"2024-09-05T16:36:35.665970+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:36:35.665970+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4450,"log_dups_size":0,"ondisk_log_size":4450,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36418720,"num_objects":4446,"num_object_clones":0,"num_object_copies":17784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4446,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4448,"num_write_kb":35580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2179}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'4521","reported_seq":4596,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:35.428528+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:36:35.428528+0000","last_peered":"2024-09-05T16:36:35.428528+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:36:35.428528+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4521,"log_dups_size":0,"ondisk_log_size":4521,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37010000,"num_objects":4518,"num_object_clones":0,"num_object_copies":18072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4518,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4520,"num_write_kb":36150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2268}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'4550","reported_seq":7519,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:35.651961+0000","last_change":"2024-09-05T16:34:56.800371+0000","last_active":"2024-09-05T16:36:35.651961+0000","last_peered":"2024-09-05T16:36:35.651961+0000","last_clean":"2024-09-05T16:33:53.148717+0000","last_became_active":"2024-09-05T16:34:56.791755+0000","last_became_peered":"2024-09-05T16:34:56.791755+0000","last_unstale":"2024-09-05T16:36:35.651961+0000","last_undegraded":"2024-09-05T16:34:56.597522+0000","last_fullsized":"2024-09-05T16:34:56.568150+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4550,"log_dups_size":0,"ondisk_log_size":4550,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37237920,"num_objects":4546,"num_object_clones":0,"num_object_copies":18184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":838,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4546,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":36373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1454,"num_bytes_recovered":11878400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3708}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'4381","reported_seq":4435,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:34.465595+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:36:34.465595+0000","last_peered":"2024-09-05T16:36:34.465595+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:36:34.465595+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4381,"log_dups_size":0,"ondisk_log_size":4381,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35861664,"num_objects":4378,"num_object_clones":0,"num_object_copies":17512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4378,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4379,"num_write_kb":35029,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2216}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'4511","reported_seq":4553,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:34.669787+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:36:34.669787+0000","last_peered":"2024-09-05T16:36:34.669787+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:36:34.669787+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4511,"log_dups_size":0,"ondisk_log_size":4511,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36954112,"num_objects":4511,"num_object_clones":0,"num_object_copies":18044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4511,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4511,"num_write_kb":36088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2261}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'4420","reported_seq":4485,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:35.594236+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:36:35.594236+0000","last_peered":"2024-09-05T16:36:35.594236+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:36:35.594236+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4420,"log_dups_size":0,"ondisk_log_size":4420,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36156576,"num_objects":4414,"num_object_clones":0,"num_object_copies":17656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4414,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4418,"num_write_kb":35324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2267}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'4488","reported_seq":4552,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:35.641076+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:36:35.641076+0000","last_peered":"2024-09-05T16:36:35.641076+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:36:35.641076+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4488,"log_dups_size":0,"ondisk_log_size":4488,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36721824,"num_objects":4483,"num_object_clones":0,"num_object_copies":17932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4483,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4486,"num_write_kb":35876,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2244}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'4502","reported_seq":4558,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:34.597909+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:36:34.597909+0000","last_peered":"2024-09-05T16:36:34.597909+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:36:34.597909+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4502,"log_dups_size":0,"ondisk_log_size":4502,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36809024,"num_objects":4494,"num_object_clones":0,"num_object_copies":17976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4494,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4499,"num_write_kb":35969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2233}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'4535","reported_seq":4612,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:35.357375+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:36:35.357375+0000","last_peered":"2024-09-05T16:36:35.357375+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:36:35.357375+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4535,"log_dups_size":0,"ondisk_log_size":4535,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37131424,"num_objects":4533,"num_object_clones":0,"num_object_copies":18132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4533,"num_write_kb":36262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2271}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'4597","reported_seq":4653,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:34.659491+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:36:34.659491+0000","last_peered":"2024-09-05T16:36:34.659491+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:36:34.659491+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4597,"log_dups_size":0,"ondisk_log_size":4597,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37613296,"num_objects":4592,"num_object_clones":0,"num_object_copies":18368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4592,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4594,"num_write_kb":36740,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2281}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'4439","reported_seq":4475,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:34.691271+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:36:34.691271+0000","last_peered":"2024-09-05T16:36:34.691271+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:36:34.691271+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4439,"log_dups_size":0,"ondisk_log_size":4439,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36354640,"num_objects":4438,"num_object_clones":0,"num_object_copies":17752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4438,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4438,"num_write_kb":35503,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2174}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'4560","reported_seq":4634,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:35.351128+0000","last_change":"2024-09-05T16:34:56.798081+0000","last_active":"2024-09-05T16:36:35.351128+0000","last_peered":"2024-09-05T16:36:35.351128+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:36:35.351128+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4560,"log_dups_size":0,"ondisk_log_size":4560,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37345872,"num_objects":4559,"num_object_clones":0,"num_object_copies":18236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4559,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4559,"num_write_kb":36471,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2265}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'4608","reported_seq":4660,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:34.664754+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:36:34.664754+0000","last_peered":"2024-09-05T16:36:34.664754+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:36:34.664754+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4608,"log_dups_size":0,"ondisk_log_size":4608,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37713056,"num_objects":4604,"num_object_clones":0,"num_object_copies":18416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4604,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4606,"num_write_kb":36844,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2247}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'4544","reported_seq":4622,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:35.562280+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:36:35.562280+0000","last_peered":"2024-09-05T16:36:35.562280+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:36:35.562280+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4544,"log_dups_size":0,"ondisk_log_size":4544,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37196960,"num_objects":4541,"num_object_clones":0,"num_object_copies":18164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4541,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4542,"num_write_kb":36333,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2221}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'4528","reported_seq":4592,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:35.643532+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:36:35.643532+0000","last_peered":"2024-09-05T16:36:35.643532+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:36:35.643532+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4528,"log_dups_size":0,"ondisk_log_size":4528,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37049504,"num_objects":4523,"num_object_clones":0,"num_object_copies":18092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4523,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4526,"num_write_kb":36196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2286}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":590618816,"num_objects":72102,"num_object_clones":0,"num_object_copies":288408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34746,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72102,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":72129,"num_write_kb":576914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1479,"num_bytes_recovered":11878400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1038159872,"data_stored":1038159872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72156,"ondisk_log_size":72156,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084320,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":521808,"kb_used_data":287424,"kb_used_omap":4,"kb_used_meta":234363,"kb_avail":93850032,"statfs":{"total":96636764160,"available":96102432768,"internally_reserved":0,"allocated":294322176,"data_stored":294139294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":239988435},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,3,7,4,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986197,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":318724,"kb_used_data":148212,"kb_used_omap":4,"kb_used_meta":170491,"kb_avail":94053116,"statfs":{"total":96636764160,"available":96310390784,"internally_reserved":0,"allocated":151769088,"data_stored":151586206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":174583506},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,5,3,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673003,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":526484,"kb_used_data":288452,"kb_used_omap":1,"kb_used_meta":238014,"kb_avail":93845356,"statfs":{"total":96636764160,"available":96097644544,"internally_reserved":0,"allocated":295374848,"data_stored":295191966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":243726789},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,8,0,0,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182434,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":522360,"kb_used_data":288040,"kb_used_omap":4,"kb_used_meta":234299,"kb_avail":93849480,"statfs":{"total":96636764160,"available":96101867520,"internally_reserved":0,"allocated":294952960,"data_stored":294770078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":239922893},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,2,5,5,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":295890944,"data_stored":295890944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":152014848,"data_stored":152014848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":295743488,"data_stored":295743488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":294510592,"data_stored":294510592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:36:37.283 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-05T16:36:37.283 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'4522", 'reported_seq': 4564, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:34.458700+0000', 'last_change': '2024-09-05T16:34:56.833497+0000', 'last_active': '2024-09-05T16:36:34.458700+0000', 'last_peered': '2024-09-05T16:36:34.458700+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.828289+0000', 'last_became_peered': '2024-09-05T16:34:56.828289+0000', 'last_unstale': '2024-09-05T16:36:34.458700+0000', 'last_undegraded': '2024-09-05T16:34:56.612160+0000', 'last_fullsized': '2024-09-05T16:34:56.557632+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4522, 'log_dups_size': 0, 'ondisk_log_size': 4522, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37044224, 'num_objects': 4522, 'num_object_clones': 0, 'num_object_copies': 18088, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2262, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4522, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4522, 'num_write_kb': 36176, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2260}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:36:37.283 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-05T16:36:37.283 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'4450", 'reported_seq': 4513, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:35.665970+0000', 'last_change': '2024-09-05T16:34:56.825031+0000', 'last_active': '2024-09-05T16:36:35.665970+0000', 'last_peered': '2024-09-05T16:36:35.665970+0000', 'last_clean': '2024-09-05T16:33:53.200762+0000', 'last_became_active': '2024-09-05T16:34:56.820292+0000', 'last_became_peered': '2024-09-05T16:34:56.820292+0000', 'last_unstale': '2024-09-05T16:36:35.665970+0000', 'last_undegraded': '2024-09-05T16:34:56.605776+0000', 'last_fullsized': '2024-09-05T16:34:56.559383+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4450, 'log_dups_size': 0, 'ondisk_log_size': 4450, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36418720, 'num_objects': 4446, 'num_object_clones': 0, 'num_object_copies': 17784, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2269, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4446, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4448, 'num_write_kb': 35580, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2179}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:36:37.283 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:36:37.284 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'4521", 'reported_seq': 4596, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:35.428528+0000', 'last_change': '2024-09-05T16:34:56.867900+0000', 'last_active': '2024-09-05T16:36:35.428528+0000', 'last_peered': '2024-09-05T16:36:35.428528+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:36:35.428528+0000', 'last_undegraded': '2024-09-05T16:34:56.590972+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4521, 'log_dups_size': 0, 'ondisk_log_size': 4521, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37010000, 'num_objects': 4518, 'num_object_clones': 0, 'num_object_copies': 18072, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2251, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4518, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4520, 'num_write_kb': 36150, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2268}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:36:37.284 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-05T16:36:37.284 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'4550", 'reported_seq': 7519, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:35.651961+0000', 'last_change': '2024-09-05T16:34:56.800371+0000', 'last_active': '2024-09-05T16:36:35.651961+0000', 'last_peered': '2024-09-05T16:36:35.651961+0000', 'last_clean': '2024-09-05T16:33:53.148717+0000', 'last_became_active': '2024-09-05T16:34:56.791755+0000', 'last_became_peered': '2024-09-05T16:34:56.791755+0000', 'last_unstale': '2024-09-05T16:36:35.651961+0000', 'last_undegraded': '2024-09-05T16:34:56.597522+0000', 'last_fullsized': '2024-09-05T16:34:56.568150+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4550, 'log_dups_size': 0, 'ondisk_log_size': 4550, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37237920, 'num_objects': 4546, 'num_object_clones': 0, 'num_object_copies': 18184, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 838, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4546, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4548, 'num_write_kb': 36373, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1454, 'num_bytes_recovered': 11878400, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3708}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:36:37.284 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:36:37.284 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'4381", 'reported_seq': 4435, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:34.465595+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:36:34.465595+0000', 'last_peered': '2024-09-05T16:36:34.465595+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:36:34.465595+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4381, 'log_dups_size': 0, 'ondisk_log_size': 4381, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35861664, 'num_objects': 4378, 'num_object_clones': 0, 'num_object_copies': 17512, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2164, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4378, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4379, 'num_write_kb': 35029, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2216}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:36:37.284 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:36:37.284 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'4511", 'reported_seq': 4553, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:34.669787+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:36:34.669787+0000', 'last_peered': '2024-09-05T16:36:34.669787+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:36:34.669787+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4511, 'log_dups_size': 0, 'ondisk_log_size': 4511, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36954112, 'num_objects': 4511, 'num_object_clones': 0, 'num_object_copies': 18044, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4511, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4511, 'num_write_kb': 36088, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2261}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:36:37.284 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:36:37.284 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'4420", 'reported_seq': 4485, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:35.594236+0000', 'last_change': '2024-09-05T16:34:56.903436+0000', 'last_active': '2024-09-05T16:36:35.594236+0000', 'last_peered': '2024-09-05T16:36:35.594236+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:36:35.594236+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4420, 'log_dups_size': 0, 'ondisk_log_size': 4420, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36156576, 'num_objects': 4414, 'num_object_clones': 0, 'num_object_copies': 17656, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2149, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4414, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4418, 'num_write_kb': 35324, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2267}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:36:37.284 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-05T16:36:37.285 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'4488", 'reported_seq': 4552, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:35.641076+0000', 'last_change': '2024-09-05T16:34:56.883319+0000', 'last_active': '2024-09-05T16:36:35.641076+0000', 'last_peered': '2024-09-05T16:36:35.641076+0000', 'last_clean': '2024-09-05T16:33:53.505305+0000', 'last_became_active': '2024-09-05T16:34:56.873389+0000', 'last_became_peered': '2024-09-05T16:34:56.873389+0000', 'last_unstale': '2024-09-05T16:36:35.641076+0000', 'last_undegraded': '2024-09-05T16:34:56.602567+0000', 'last_fullsized': '2024-09-05T16:34:56.559958+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4488, 'log_dups_size': 0, 'ondisk_log_size': 4488, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36721824, 'num_objects': 4483, 'num_object_clones': 0, 'num_object_copies': 17932, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2241, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4483, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4486, 'num_write_kb': 35876, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2244}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:36:37.286 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:36:37.286 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'4502", 'reported_seq': 4558, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:34.597909+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:36:34.597909+0000', 'last_peered': '2024-09-05T16:36:34.597909+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:36:34.597909+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4502, 'log_dups_size': 0, 'ondisk_log_size': 4502, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36809024, 'num_objects': 4494, 'num_object_clones': 0, 'num_object_copies': 17976, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2264, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4494, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4499, 'num_write_kb': 35969, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2233}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:36:37.286 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:36:37.286 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'4535", 'reported_seq': 4612, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:35.357375+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:36:35.357375+0000', 'last_peered': '2024-09-05T16:36:35.357375+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:36:35.357375+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4535, 'log_dups_size': 0, 'ondisk_log_size': 4535, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37131424, 'num_objects': 4533, 'num_object_clones': 0, 'num_object_copies': 18132, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2264, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4533, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4533, 'num_write_kb': 36262, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2271}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:36:37.286 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:36:37.286 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'4597", 'reported_seq': 4653, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:34.659491+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:36:34.659491+0000', 'last_peered': '2024-09-05T16:36:34.659491+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:36:34.659491+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4597, 'log_dups_size': 0, 'ondisk_log_size': 4597, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37613296, 'num_objects': 4592, 'num_object_clones': 0, 'num_object_copies': 18368, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2314, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4592, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4594, 'num_write_kb': 36740, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2281}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:36:37.286 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:36:37.286 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'4439", 'reported_seq': 4475, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:34.691271+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:36:34.691271+0000', 'last_peered': '2024-09-05T16:36:34.691271+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:36:34.691271+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4439, 'log_dups_size': 0, 'ondisk_log_size': 4439, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36354640, 'num_objects': 4438, 'num_object_clones': 0, 'num_object_copies': 17752, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2265, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4438, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4438, 'num_write_kb': 35503, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2174}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:36:37.287 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-05T16:36:37.287 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'4560", 'reported_seq': 4634, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:35.351128+0000', 'last_change': '2024-09-05T16:34:56.798081+0000', 'last_active': '2024-09-05T16:36:35.351128+0000', 'last_peered': '2024-09-05T16:36:35.351128+0000', 'last_clean': '2024-09-05T16:33:45.082039+0000', 'last_became_active': '2024-09-05T16:34:56.792386+0000', 'last_became_peered': '2024-09-05T16:34:56.792386+0000', 'last_unstale': '2024-09-05T16:36:35.351128+0000', 'last_undegraded': '2024-09-05T16:34:56.600595+0000', 'last_fullsized': '2024-09-05T16:34:56.558203+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4560, 'log_dups_size': 0, 'ondisk_log_size': 4560, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37345872, 'num_objects': 4559, 'num_object_clones': 0, 'num_object_copies': 18236, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2295, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4559, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4559, 'num_write_kb': 36471, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2265}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:36:37.287 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:36:37.287 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'4608", 'reported_seq': 4660, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:34.664754+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:36:34.664754+0000', 'last_peered': '2024-09-05T16:36:34.664754+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:36:34.664754+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4608, 'log_dups_size': 0, 'ondisk_log_size': 4608, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37713056, 'num_objects': 4604, 'num_object_clones': 0, 'num_object_copies': 18416, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2359, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4604, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4606, 'num_write_kb': 36844, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2247}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:36:37.287 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:36:37.288 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'4544", 'reported_seq': 4622, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:35.562280+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:36:35.562280+0000', 'last_peered': '2024-09-05T16:36:35.562280+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:36:35.562280+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4544, 'log_dups_size': 0, 'ondisk_log_size': 4544, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37196960, 'num_objects': 4541, 'num_object_clones': 0, 'num_object_copies': 18164, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2322, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4541, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4542, 'num_write_kb': 36333, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2221}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:36:37.288 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:36:37.288 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'4528", 'reported_seq': 4592, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:35.643532+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:36:35.643532+0000', 'last_peered': '2024-09-05T16:36:35.643532+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:36:35.643532+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4528, 'log_dups_size': 0, 'ondisk_log_size': 4528, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37049504, 'num_objects': 4523, 'num_object_clones': 0, 'num_object_copies': 18092, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4523, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4526, 'num_write_kb': 36196, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2286}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:36:37.288 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:36:37.515 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 135 20 72797 72777 4.21082 2.33594 0.0457376 0.037094 2024-09-05T16:36:38.018 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:36:38.018 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 34746/288408 objects degraded (12.048%), 16 pgs degraded, 16 pgs undersized","count":32},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":201,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":72102,"data_bytes":590618816,"bytes_used":1934721024,"bytes_avail":384612335616,"bytes_total":386547056640,"degraded_objects":34746,"degraded_total":288408,"degraded_ratio":0.12047516018973121,"recovering_objects_per_sec":20,"recovering_bytes_per_sec":169227,"recovering_keys_per_sec":0,"num_objects_recovered":248,"num_bytes_recovered":2031616,"num_keys_recovered":0,"write_bytes_sec":2860489,"read_op_per_sec":0,"write_op_per_sec":349},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:36:38.047 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:36:38.047 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:36:38.515 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 136 20 73088 73068 4.19658 2.27344 0.0805868 0.0372207 2024-09-05T16:36:38.637 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:36:38.637 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:36:38.668 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":124,"stamp":"2024-09-05T16:36:36.833374+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590618816,"num_objects":72102,"num_object_clones":0,"num_object_copies":288408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34746,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72102,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":72129,"num_write_kb":576914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1479,"num_bytes_recovered":11878400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72156,"ondisk_log_size":72156,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1889376,"kb_used_data":1012128,"kb_used_omap":14,"kb_used_meta":877169,"kb_avail":375597984,"statfs":{"total":386547056640,"available":384612335616,"internally_reserved":0,"allocated":1036419072,"data_stored":1035687544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":898221623},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,1,14,13,15,14,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":34340864,"num_objects":4192,"num_object_clones":0,"num_object_copies":16768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-248,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4192,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4192,"num_write_kb":33536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":248,"num_bytes_recovered":2031616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005241"},"pg_stats":[{"pgid":"1.f","version":"33'4522","reported_seq":4564,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:34.458700+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:36:34.458700+0000","last_peered":"2024-09-05T16:36:34.458700+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:36:34.458700+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4522,"log_dups_size":0,"ondisk_log_size":4522,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37044224,"num_objects":4522,"num_object_clones":0,"num_object_copies":18088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4522,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4522,"num_write_kb":36176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2260}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'4450","reported_seq":4513,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:35.665970+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:36:35.665970+0000","last_peered":"2024-09-05T16:36:35.665970+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:36:35.665970+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4450,"log_dups_size":0,"ondisk_log_size":4450,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36418720,"num_objects":4446,"num_object_clones":0,"num_object_copies":17784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4446,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4448,"num_write_kb":35580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2179}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'4521","reported_seq":4596,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:35.428528+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:36:35.428528+0000","last_peered":"2024-09-05T16:36:35.428528+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:36:35.428528+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4521,"log_dups_size":0,"ondisk_log_size":4521,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37010000,"num_objects":4518,"num_object_clones":0,"num_object_copies":18072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4518,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4520,"num_write_kb":36150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2268}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'4550","reported_seq":7519,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:35.651961+0000","last_change":"2024-09-05T16:34:56.800371+0000","last_active":"2024-09-05T16:36:35.651961+0000","last_peered":"2024-09-05T16:36:35.651961+0000","last_clean":"2024-09-05T16:33:53.148717+0000","last_became_active":"2024-09-05T16:34:56.791755+0000","last_became_peered":"2024-09-05T16:34:56.791755+0000","last_unstale":"2024-09-05T16:36:35.651961+0000","last_undegraded":"2024-09-05T16:34:56.597522+0000","last_fullsized":"2024-09-05T16:34:56.568150+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4550,"log_dups_size":0,"ondisk_log_size":4550,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37237920,"num_objects":4546,"num_object_clones":0,"num_object_copies":18184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":838,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4546,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":36373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1454,"num_bytes_recovered":11878400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3708}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'4381","reported_seq":4435,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:34.465595+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:36:34.465595+0000","last_peered":"2024-09-05T16:36:34.465595+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:36:34.465595+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4381,"log_dups_size":0,"ondisk_log_size":4381,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35861664,"num_objects":4378,"num_object_clones":0,"num_object_copies":17512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4378,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4379,"num_write_kb":35029,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2216}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'4511","reported_seq":4553,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:34.669787+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:36:34.669787+0000","last_peered":"2024-09-05T16:36:34.669787+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:36:34.669787+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4511,"log_dups_size":0,"ondisk_log_size":4511,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36954112,"num_objects":4511,"num_object_clones":0,"num_object_copies":18044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4511,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4511,"num_write_kb":36088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2261}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'4420","reported_seq":4485,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:35.594236+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:36:35.594236+0000","last_peered":"2024-09-05T16:36:35.594236+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:36:35.594236+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4420,"log_dups_size":0,"ondisk_log_size":4420,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36156576,"num_objects":4414,"num_object_clones":0,"num_object_copies":17656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4414,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4418,"num_write_kb":35324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2267}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'4488","reported_seq":4552,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:35.641076+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:36:35.641076+0000","last_peered":"2024-09-05T16:36:35.641076+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:36:35.641076+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4488,"log_dups_size":0,"ondisk_log_size":4488,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36721824,"num_objects":4483,"num_object_clones":0,"num_object_copies":17932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4483,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4486,"num_write_kb":35876,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2244}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'4502","reported_seq":4558,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:34.597909+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:36:34.597909+0000","last_peered":"2024-09-05T16:36:34.597909+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:36:34.597909+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4502,"log_dups_size":0,"ondisk_log_size":4502,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36809024,"num_objects":4494,"num_object_clones":0,"num_object_copies":17976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4494,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4499,"num_write_kb":35969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2233}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'4535","reported_seq":4612,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:35.357375+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:36:35.357375+0000","last_peered":"2024-09-05T16:36:35.357375+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:36:35.357375+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4535,"log_dups_size":0,"ondisk_log_size":4535,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37131424,"num_objects":4533,"num_object_clones":0,"num_object_copies":18132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4533,"num_write_kb":36262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2271}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'4597","reported_seq":4653,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:34.659491+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:36:34.659491+0000","last_peered":"2024-09-05T16:36:34.659491+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:36:34.659491+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4597,"log_dups_size":0,"ondisk_log_size":4597,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37613296,"num_objects":4592,"num_object_clones":0,"num_object_copies":18368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4592,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4594,"num_write_kb":36740,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2281}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'4439","reported_seq":4475,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:34.691271+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:36:34.691271+0000","last_peered":"2024-09-05T16:36:34.691271+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:36:34.691271+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4439,"log_dups_size":0,"ondisk_log_size":4439,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36354640,"num_objects":4438,"num_object_clones":0,"num_object_copies":17752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4438,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4438,"num_write_kb":35503,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2174}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'4560","reported_seq":4634,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:35.351128+0000","last_change":"2024-09-05T16:34:56.798081+0000","last_active":"2024-09-05T16:36:35.351128+0000","last_peered":"2024-09-05T16:36:35.351128+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:36:35.351128+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4560,"log_dups_size":0,"ondisk_log_size":4560,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37345872,"num_objects":4559,"num_object_clones":0,"num_object_copies":18236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4559,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4559,"num_write_kb":36471,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2265}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'4608","reported_seq":4660,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:34.664754+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:36:34.664754+0000","last_peered":"2024-09-05T16:36:34.664754+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:36:34.664754+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4608,"log_dups_size":0,"ondisk_log_size":4608,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37713056,"num_objects":4604,"num_object_clones":0,"num_object_copies":18416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4604,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4606,"num_write_kb":36844,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2247}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'4544","reported_seq":4622,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:35.562280+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:36:35.562280+0000","last_peered":"2024-09-05T16:36:35.562280+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:36:35.562280+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4544,"log_dups_size":0,"ondisk_log_size":4544,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37196960,"num_objects":4541,"num_object_clones":0,"num_object_copies":18164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4541,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4542,"num_write_kb":36333,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2221}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'4528","reported_seq":4592,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:35.643532+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:36:35.643532+0000","last_peered":"2024-09-05T16:36:35.643532+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:36:35.643532+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4528,"log_dups_size":0,"ondisk_log_size":4528,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37049504,"num_objects":4523,"num_object_clones":0,"num_object_copies":18092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4523,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4526,"num_write_kb":36196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2286}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":590618816,"num_objects":72102,"num_object_clones":0,"num_object_copies":288408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34746,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72102,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":72129,"num_write_kb":576914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1479,"num_bytes_recovered":11878400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1038159872,"data_stored":1038159872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72156,"ondisk_log_size":72156,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084320,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":521808,"kb_used_data":287424,"kb_used_omap":4,"kb_used_meta":234363,"kb_avail":93850032,"statfs":{"total":96636764160,"available":96102432768,"internally_reserved":0,"allocated":294322176,"data_stored":294139294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":239988435},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,3,7,4,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986197,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":318724,"kb_used_data":148212,"kb_used_omap":4,"kb_used_meta":170491,"kb_avail":94053116,"statfs":{"total":96636764160,"available":96310390784,"internally_reserved":0,"allocated":151769088,"data_stored":151586206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":174583506},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,5,3,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673003,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":526484,"kb_used_data":288452,"kb_used_omap":1,"kb_used_meta":238014,"kb_avail":93845356,"statfs":{"total":96636764160,"available":96097644544,"internally_reserved":0,"allocated":295374848,"data_stored":295191966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":243726789},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,8,0,0,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182434,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":522360,"kb_used_data":288040,"kb_used_omap":4,"kb_used_meta":234299,"kb_avail":93849480,"statfs":{"total":96636764160,"available":96101867520,"internally_reserved":0,"allocated":294952960,"data_stored":294770078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":239922893},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,2,5,5,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":295890944,"data_stored":295890944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":152014848,"data_stored":152014848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":295743488,"data_stored":295743488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":294510592,"data_stored":294510592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:36:39.517 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 137 20 73409 73389 4.18425 2.50781 0.0637948 0.0373292 2024-09-05T16:36:40.515 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 138 20 73742 73722 4.17278 2.60156 0.0291703 0.0374323 2024-09-05T16:36:41.515 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 139 20 74074 74054 4.16142 2.59375 0.0396145 0.0375347 2024-09-05T16:36:41.670 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:36:42.260 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:36:42.263 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:36:42.289 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":126,"stamp":"2024-09-05T16:36:40.835296+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":603562176,"num_objects":73682,"num_object_clones":0,"num_object_copies":294728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34675,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73682,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":73709,"num_write_kb":589554,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1550,"num_bytes_recovered":12460032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":73736,"ondisk_log_size":73736,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1940756,"kb_used_data":1038480,"kb_used_omap":14,"kb_used_meta":902193,"kb_avail":375546604,"statfs":{"total":386547056640,"available":384559722496,"internally_reserved":0,"allocated":1063403520,"data_stored":1062671992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":923846199},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,10,0,5,15,11,10,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":39428096,"num_objects":4813,"num_object_clones":0,"num_object_copies":19252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-242,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4813,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4813,"num_write_kb":38504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":242,"num_bytes_recovered":1982464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004814"},"pg_stats":[{"pgid":"1.f","version":"33'4618","reported_seq":4660,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:39.677975+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:36:39.677975+0000","last_peered":"2024-09-05T16:36:39.677975+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:36:39.677975+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4618,"log_dups_size":0,"ondisk_log_size":4618,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37830656,"num_objects":4618,"num_object_clones":0,"num_object_copies":18472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4618,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4618,"num_write_kb":36944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2356}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'4556","reported_seq":4620,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:40.734354+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:36:40.734354+0000","last_peered":"2024-09-05T16:36:40.734354+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:36:40.734354+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4556,"log_dups_size":0,"ondisk_log_size":4556,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37287072,"num_objects":4552,"num_object_clones":0,"num_object_copies":18208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4552,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4554,"num_write_kb":36428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2285}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'4625","reported_seq":4700,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:40.516960+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:36:40.516960+0000","last_peered":"2024-09-05T16:36:40.516960+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:36:40.516960+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4625,"log_dups_size":0,"ondisk_log_size":4625,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37861968,"num_objects":4622,"num_object_clones":0,"num_object_copies":18488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4622,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4624,"num_write_kb":36982,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2372}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'4656","reported_seq":7767,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:40.706260+0000","last_change":"2024-09-05T16:34:56.800371+0000","last_active":"2024-09-05T16:36:40.706260+0000","last_peered":"2024-09-05T16:36:40.706260+0000","last_clean":"2024-09-05T16:33:53.148717+0000","last_became_active":"2024-09-05T16:34:56.791755+0000","last_became_peered":"2024-09-05T16:34:56.791755+0000","last_unstale":"2024-09-05T16:36:40.706260+0000","last_undegraded":"2024-09-05T16:34:56.597522+0000","last_fullsized":"2024-09-05T16:34:56.568150+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4656,"log_dups_size":0,"ondisk_log_size":4656,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38106272,"num_objects":4652,"num_object_clones":0,"num_object_copies":18608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":767,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4652,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4654,"num_write_kb":37221,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1525,"num_bytes_recovered":12460032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3885}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'4488","reported_seq":4542,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:39.673923+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:36:39.673923+0000","last_peered":"2024-09-05T16:36:39.673923+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:36:39.673923+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4488,"log_dups_size":0,"ondisk_log_size":4488,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36738208,"num_objects":4485,"num_object_clones":0,"num_object_copies":17940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4485,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4486,"num_write_kb":35885,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2323}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'4613","reported_seq":4651,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:39.711820+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:36:39.711820+0000","last_peered":"2024-09-05T16:36:39.711820+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:36:39.711820+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4613,"log_dups_size":0,"ondisk_log_size":4613,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37789696,"num_objects":4613,"num_object_clones":0,"num_object_copies":18452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4613,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4613,"num_write_kb":36904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2363}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'4498","reported_seq":4563,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:40.638481+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:36:40.638481+0000","last_peered":"2024-09-05T16:36:40.638481+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:36:40.638481+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4498,"log_dups_size":0,"ondisk_log_size":4498,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36795552,"num_objects":4492,"num_object_clones":0,"num_object_copies":17968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4492,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4496,"num_write_kb":35948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2345}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'4594","reported_seq":4658,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:40.707884+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:36:40.707884+0000","last_peered":"2024-09-05T16:36:40.707884+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:36:40.707884+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4594,"log_dups_size":0,"ondisk_log_size":4594,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37590176,"num_objects":4589,"num_object_clones":0,"num_object_copies":18356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4589,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4592,"num_write_kb":36724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2350}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'4599","reported_seq":4655,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:39.691580+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:36:39.691580+0000","last_peered":"2024-09-05T16:36:39.691580+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:36:39.691580+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4599,"log_dups_size":0,"ondisk_log_size":4599,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37603648,"num_objects":4591,"num_object_clones":0,"num_object_copies":18364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4591,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4596,"num_write_kb":36745,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2330}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'4623","reported_seq":4700,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:40.401405+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:36:40.401405+0000","last_peered":"2024-09-05T16:36:40.401405+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:36:40.401405+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4623,"log_dups_size":0,"ondisk_log_size":4623,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37852320,"num_objects":4621,"num_object_clones":0,"num_object_copies":18484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4621,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4621,"num_write_kb":36966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2359}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'4693","reported_seq":4749,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:39.478933+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:36:39.478933+0000","last_peered":"2024-09-05T16:36:39.478933+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:36:39.478933+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4693,"log_dups_size":0,"ondisk_log_size":4693,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38399728,"num_objects":4688,"num_object_clones":0,"num_object_copies":18752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4690,"num_write_kb":37508,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2377}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'4534","reported_seq":4569,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:39.758638+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:36:39.758638+0000","last_peered":"2024-09-05T16:36:39.758638+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:36:39.758638+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4534,"log_dups_size":0,"ondisk_log_size":4534,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37132880,"num_objects":4533,"num_object_clones":0,"num_object_copies":18132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4533,"num_write_kb":36263,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2269}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'4676","reported_seq":4749,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:40.602077+0000","last_change":"2024-09-05T16:34:56.798081+0000","last_active":"2024-09-05T16:36:40.602077+0000","last_peered":"2024-09-05T16:36:40.602077+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:36:40.602077+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4676,"log_dups_size":0,"ondisk_log_size":4676,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38296144,"num_objects":4675,"num_object_clones":0,"num_object_copies":18700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4675,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4675,"num_write_kb":37399,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2381}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'4709","reported_seq":4761,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:39.661595+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:36:39.661595+0000","last_peered":"2024-09-05T16:36:39.661595+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:36:39.661595+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4709,"log_dups_size":0,"ondisk_log_size":4709,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38540448,"num_objects":4705,"num_object_clones":0,"num_object_copies":18820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4705,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4707,"num_write_kb":37652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2348}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'4623","reported_seq":4701,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:40.526143+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:36:40.526143+0000","last_peered":"2024-09-05T16:36:40.526143+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:36:40.526143+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4623,"log_dups_size":0,"ondisk_log_size":4623,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37844128,"num_objects":4620,"num_object_clones":0,"num_object_copies":18480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4620,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4621,"num_write_kb":36965,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2300}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'4631","reported_seq":4695,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:40.722070+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:36:40.722070+0000","last_peered":"2024-09-05T16:36:40.722070+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:36:40.722070+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4631,"log_dups_size":0,"ondisk_log_size":4631,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37893280,"num_objects":4626,"num_object_clones":0,"num_object_copies":18504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4626,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4629,"num_write_kb":37020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2389}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":603562176,"num_objects":73682,"num_object_clones":0,"num_object_copies":294728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34675,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73682,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":73709,"num_write_kb":589554,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1550,"num_bytes_recovered":12460032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064321024,"data_stored":1064321024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":73736,"ondisk_log_size":73736,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084321,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":550228,"kb_used_data":293636,"kb_used_omap":4,"kb_used_meta":256571,"kb_avail":93821612,"statfs":{"total":96636764160,"available":96073330688,"internally_reserved":0,"allocated":300683264,"data_stored":300500382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":262729427},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,7,0,2,1,6,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986198,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312724,"kb_used_data":154692,"kb_used_omap":4,"kb_used_meta":158011,"kb_avail":94059116,"statfs":{"total":96636764160,"available":96316534784,"internally_reserved":0,"allocated":158404608,"data_stored":158221726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":161803986},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,2,0,5,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673004,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":538256,"kb_used_data":294656,"kb_used_omap":1,"kb_used_meta":243582,"kb_avail":93833584,"statfs":{"total":96636764160,"available":96085590016,"internally_reserved":0,"allocated":301727744,"data_stored":301544862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":249428421},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,2,1,2,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182435,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":539548,"kb_used_data":295496,"kb_used_omap":4,"kb_used_meta":244027,"kb_avail":93832292,"statfs":{"total":96636764160,"available":96084267008,"internally_reserved":0,"allocated":302587904,"data_stored":302405022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":249884365},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,11,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":302346240,"data_stored":302346240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":158777344,"data_stored":158777344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":302153728,"data_stored":302153728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":301043712,"data_stored":301043712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:36:42.291 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:36:42.515 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout:2024-09-05T16:36:42.523504+0000 min lat: 0.00357551 max lat: 2.29705 avg lat: 0.0376527 2024-09-05T16:36:42.515 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-05T16:36:42.515 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 140 20 74367 74347 4.14805 2.28906 0.0710931 0.0376527 2024-09-05T16:36:42.866 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:36:42.867 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:36:42.902 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":127,"stamp":"2024-09-05T16:36:42.836292+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":603562176,"num_objects":73682,"num_object_clones":0,"num_object_copies":294728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34675,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73682,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":73709,"num_write_kb":589554,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1550,"num_bytes_recovered":12460032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":73736,"ondisk_log_size":73736,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1940756,"kb_used_data":1038480,"kb_used_omap":14,"kb_used_meta":902193,"kb_avail":375546604,"statfs":{"total":386547056640,"available":384559722496,"internally_reserved":0,"allocated":1063403520,"data_stored":1062671992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":923846199},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,10,0,5,15,11,10,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":26091520,"num_objects":3185,"num_object_clones":0,"num_object_copies":12740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-161,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3185,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3185,"num_write_kb":25480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":161,"num_bytes_recovered":1318912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005160"},"pg_stats":[{"pgid":"1.f","version":"33'4618","reported_seq":4660,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:39.677975+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:36:39.677975+0000","last_peered":"2024-09-05T16:36:39.677975+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:36:39.677975+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4618,"log_dups_size":0,"ondisk_log_size":4618,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37830656,"num_objects":4618,"num_object_clones":0,"num_object_copies":18472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4618,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4618,"num_write_kb":36944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2356}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'4556","reported_seq":4620,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:40.734354+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:36:40.734354+0000","last_peered":"2024-09-05T16:36:40.734354+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:36:40.734354+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4556,"log_dups_size":0,"ondisk_log_size":4556,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37287072,"num_objects":4552,"num_object_clones":0,"num_object_copies":18208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4552,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4554,"num_write_kb":36428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2285}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'4625","reported_seq":4700,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:40.516960+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:36:40.516960+0000","last_peered":"2024-09-05T16:36:40.516960+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:36:40.516960+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4625,"log_dups_size":0,"ondisk_log_size":4625,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37861968,"num_objects":4622,"num_object_clones":0,"num_object_copies":18488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4622,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4624,"num_write_kb":36982,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2372}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'4656","reported_seq":7767,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:40.706260+0000","last_change":"2024-09-05T16:34:56.800371+0000","last_active":"2024-09-05T16:36:40.706260+0000","last_peered":"2024-09-05T16:36:40.706260+0000","last_clean":"2024-09-05T16:33:53.148717+0000","last_became_active":"2024-09-05T16:34:56.791755+0000","last_became_peered":"2024-09-05T16:34:56.791755+0000","last_unstale":"2024-09-05T16:36:40.706260+0000","last_undegraded":"2024-09-05T16:34:56.597522+0000","last_fullsized":"2024-09-05T16:34:56.568150+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4656,"log_dups_size":0,"ondisk_log_size":4656,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38106272,"num_objects":4652,"num_object_clones":0,"num_object_copies":18608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":767,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4652,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4654,"num_write_kb":37221,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1525,"num_bytes_recovered":12460032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3885}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'4488","reported_seq":4542,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:39.673923+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:36:39.673923+0000","last_peered":"2024-09-05T16:36:39.673923+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:36:39.673923+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4488,"log_dups_size":0,"ondisk_log_size":4488,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36738208,"num_objects":4485,"num_object_clones":0,"num_object_copies":17940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4485,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4486,"num_write_kb":35885,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2323}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'4613","reported_seq":4651,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:39.711820+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:36:39.711820+0000","last_peered":"2024-09-05T16:36:39.711820+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:36:39.711820+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4613,"log_dups_size":0,"ondisk_log_size":4613,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37789696,"num_objects":4613,"num_object_clones":0,"num_object_copies":18452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4613,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4613,"num_write_kb":36904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2363}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'4498","reported_seq":4563,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:40.638481+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:36:40.638481+0000","last_peered":"2024-09-05T16:36:40.638481+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:36:40.638481+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4498,"log_dups_size":0,"ondisk_log_size":4498,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36795552,"num_objects":4492,"num_object_clones":0,"num_object_copies":17968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4492,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4496,"num_write_kb":35948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2345}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'4594","reported_seq":4658,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:40.707884+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:36:40.707884+0000","last_peered":"2024-09-05T16:36:40.707884+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:36:40.707884+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4594,"log_dups_size":0,"ondisk_log_size":4594,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37590176,"num_objects":4589,"num_object_clones":0,"num_object_copies":18356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4589,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4592,"num_write_kb":36724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2350}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'4599","reported_seq":4655,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:39.691580+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:36:39.691580+0000","last_peered":"2024-09-05T16:36:39.691580+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:36:39.691580+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4599,"log_dups_size":0,"ondisk_log_size":4599,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37603648,"num_objects":4591,"num_object_clones":0,"num_object_copies":18364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4591,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4596,"num_write_kb":36745,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2330}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'4623","reported_seq":4700,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:40.401405+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:36:40.401405+0000","last_peered":"2024-09-05T16:36:40.401405+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:36:40.401405+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4623,"log_dups_size":0,"ondisk_log_size":4623,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37852320,"num_objects":4621,"num_object_clones":0,"num_object_copies":18484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4621,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4621,"num_write_kb":36966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2359}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'4693","reported_seq":4749,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:39.478933+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:36:39.478933+0000","last_peered":"2024-09-05T16:36:39.478933+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:36:39.478933+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4693,"log_dups_size":0,"ondisk_log_size":4693,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38399728,"num_objects":4688,"num_object_clones":0,"num_object_copies":18752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4690,"num_write_kb":37508,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2377}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'4534","reported_seq":4569,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:39.758638+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:36:39.758638+0000","last_peered":"2024-09-05T16:36:39.758638+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:36:39.758638+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4534,"log_dups_size":0,"ondisk_log_size":4534,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37132880,"num_objects":4533,"num_object_clones":0,"num_object_copies":18132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4533,"num_write_kb":36263,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2269}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'4676","reported_seq":4749,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:40.602077+0000","last_change":"2024-09-05T16:34:56.798081+0000","last_active":"2024-09-05T16:36:40.602077+0000","last_peered":"2024-09-05T16:36:40.602077+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:36:40.602077+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4676,"log_dups_size":0,"ondisk_log_size":4676,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38296144,"num_objects":4675,"num_object_clones":0,"num_object_copies":18700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4675,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4675,"num_write_kb":37399,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2381}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'4709","reported_seq":4761,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:39.661595+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:36:39.661595+0000","last_peered":"2024-09-05T16:36:39.661595+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:36:39.661595+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4709,"log_dups_size":0,"ondisk_log_size":4709,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38540448,"num_objects":4705,"num_object_clones":0,"num_object_copies":18820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4705,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4707,"num_write_kb":37652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2348}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'4623","reported_seq":4701,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:40.526143+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:36:40.526143+0000","last_peered":"2024-09-05T16:36:40.526143+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:36:40.526143+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4623,"log_dups_size":0,"ondisk_log_size":4623,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37844128,"num_objects":4620,"num_object_clones":0,"num_object_copies":18480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4620,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4621,"num_write_kb":36965,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2300}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'4631","reported_seq":4695,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:40.722070+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:36:40.722070+0000","last_peered":"2024-09-05T16:36:40.722070+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:36:40.722070+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4631,"log_dups_size":0,"ondisk_log_size":4631,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37893280,"num_objects":4626,"num_object_clones":0,"num_object_copies":18504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4626,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4629,"num_write_kb":37020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2389}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":603562176,"num_objects":73682,"num_object_clones":0,"num_object_copies":294728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34675,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73682,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":73709,"num_write_kb":589554,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1550,"num_bytes_recovered":12460032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064321024,"data_stored":1064321024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":73736,"ondisk_log_size":73736,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084321,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":550228,"kb_used_data":293636,"kb_used_omap":4,"kb_used_meta":256571,"kb_avail":93821612,"statfs":{"total":96636764160,"available":96073330688,"internally_reserved":0,"allocated":300683264,"data_stored":300500382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":262729427},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,7,0,2,1,6,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986198,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312724,"kb_used_data":154692,"kb_used_omap":4,"kb_used_meta":158011,"kb_avail":94059116,"statfs":{"total":96636764160,"available":96316534784,"internally_reserved":0,"allocated":158404608,"data_stored":158221726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":161803986},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,2,0,5,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673004,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":538256,"kb_used_data":294656,"kb_used_omap":1,"kb_used_meta":243582,"kb_avail":93833584,"statfs":{"total":96636764160,"available":96085590016,"internally_reserved":0,"allocated":301727744,"data_stored":301544862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":249428421},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,2,1,2,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182435,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":539548,"kb_used_data":295496,"kb_used_omap":4,"kb_used_meta":244027,"kb_avail":93832292,"statfs":{"total":96636764160,"available":96084267008,"internally_reserved":0,"allocated":302587904,"data_stored":302405022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":249884365},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,11,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":302346240,"data_stored":302346240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":158777344,"data_stored":158777344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":302153728,"data_stored":302153728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":301043712,"data_stored":301043712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:36:42.903 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-05T16:36:42.904 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'4618", 'reported_seq': 4660, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:39.677975+0000', 'last_change': '2024-09-05T16:34:56.833497+0000', 'last_active': '2024-09-05T16:36:39.677975+0000', 'last_peered': '2024-09-05T16:36:39.677975+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.828289+0000', 'last_became_peered': '2024-09-05T16:34:56.828289+0000', 'last_unstale': '2024-09-05T16:36:39.677975+0000', 'last_undegraded': '2024-09-05T16:34:56.612160+0000', 'last_fullsized': '2024-09-05T16:34:56.557632+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4618, 'log_dups_size': 0, 'ondisk_log_size': 4618, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37830656, 'num_objects': 4618, 'num_object_clones': 0, 'num_object_copies': 18472, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2262, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4618, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4618, 'num_write_kb': 36944, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2356}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:36:42.904 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-05T16:36:42.904 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'4556", 'reported_seq': 4620, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:40.734354+0000', 'last_change': '2024-09-05T16:34:56.825031+0000', 'last_active': '2024-09-05T16:36:40.734354+0000', 'last_peered': '2024-09-05T16:36:40.734354+0000', 'last_clean': '2024-09-05T16:33:53.200762+0000', 'last_became_active': '2024-09-05T16:34:56.820292+0000', 'last_became_peered': '2024-09-05T16:34:56.820292+0000', 'last_unstale': '2024-09-05T16:36:40.734354+0000', 'last_undegraded': '2024-09-05T16:34:56.605776+0000', 'last_fullsized': '2024-09-05T16:34:56.559383+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4556, 'log_dups_size': 0, 'ondisk_log_size': 4556, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37287072, 'num_objects': 4552, 'num_object_clones': 0, 'num_object_copies': 18208, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2269, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4552, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4554, 'num_write_kb': 36428, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2285}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:36:42.904 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:36:42.904 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'4625", 'reported_seq': 4700, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:40.516960+0000', 'last_change': '2024-09-05T16:34:56.867900+0000', 'last_active': '2024-09-05T16:36:40.516960+0000', 'last_peered': '2024-09-05T16:36:40.516960+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:36:40.516960+0000', 'last_undegraded': '2024-09-05T16:34:56.590972+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4625, 'log_dups_size': 0, 'ondisk_log_size': 4625, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37861968, 'num_objects': 4622, 'num_object_clones': 0, 'num_object_copies': 18488, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2251, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4622, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4624, 'num_write_kb': 36982, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2372}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:36:42.904 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-05T16:36:42.904 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'4656", 'reported_seq': 7767, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:40.706260+0000', 'last_change': '2024-09-05T16:34:56.800371+0000', 'last_active': '2024-09-05T16:36:40.706260+0000', 'last_peered': '2024-09-05T16:36:40.706260+0000', 'last_clean': '2024-09-05T16:33:53.148717+0000', 'last_became_active': '2024-09-05T16:34:56.791755+0000', 'last_became_peered': '2024-09-05T16:34:56.791755+0000', 'last_unstale': '2024-09-05T16:36:40.706260+0000', 'last_undegraded': '2024-09-05T16:34:56.597522+0000', 'last_fullsized': '2024-09-05T16:34:56.568150+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4656, 'log_dups_size': 0, 'ondisk_log_size': 4656, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38106272, 'num_objects': 4652, 'num_object_clones': 0, 'num_object_copies': 18608, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 767, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4652, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4654, 'num_write_kb': 37221, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1525, 'num_bytes_recovered': 12460032, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3885}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:36:42.904 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:36:42.904 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'4488", 'reported_seq': 4542, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:39.673923+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:36:39.673923+0000', 'last_peered': '2024-09-05T16:36:39.673923+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:36:39.673923+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4488, 'log_dups_size': 0, 'ondisk_log_size': 4488, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36738208, 'num_objects': 4485, 'num_object_clones': 0, 'num_object_copies': 17940, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2164, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4485, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4486, 'num_write_kb': 35885, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2323}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:36:42.905 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:36:42.905 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'4613", 'reported_seq': 4651, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:39.711820+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:36:39.711820+0000', 'last_peered': '2024-09-05T16:36:39.711820+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:36:39.711820+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4613, 'log_dups_size': 0, 'ondisk_log_size': 4613, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37789696, 'num_objects': 4613, 'num_object_clones': 0, 'num_object_copies': 18452, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4613, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4613, 'num_write_kb': 36904, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2363}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:36:42.905 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:36:42.905 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'4498", 'reported_seq': 4563, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:40.638481+0000', 'last_change': '2024-09-05T16:34:56.903436+0000', 'last_active': '2024-09-05T16:36:40.638481+0000', 'last_peered': '2024-09-05T16:36:40.638481+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:36:40.638481+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4498, 'log_dups_size': 0, 'ondisk_log_size': 4498, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36795552, 'num_objects': 4492, 'num_object_clones': 0, 'num_object_copies': 17968, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2149, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4492, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4496, 'num_write_kb': 35948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2345}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:36:42.905 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-05T16:36:42.905 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'4594", 'reported_seq': 4658, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:40.707884+0000', 'last_change': '2024-09-05T16:34:56.883319+0000', 'last_active': '2024-09-05T16:36:40.707884+0000', 'last_peered': '2024-09-05T16:36:40.707884+0000', 'last_clean': '2024-09-05T16:33:53.505305+0000', 'last_became_active': '2024-09-05T16:34:56.873389+0000', 'last_became_peered': '2024-09-05T16:34:56.873389+0000', 'last_unstale': '2024-09-05T16:36:40.707884+0000', 'last_undegraded': '2024-09-05T16:34:56.602567+0000', 'last_fullsized': '2024-09-05T16:34:56.559958+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4594, 'log_dups_size': 0, 'ondisk_log_size': 4594, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37590176, 'num_objects': 4589, 'num_object_clones': 0, 'num_object_copies': 18356, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2241, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4589, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4592, 'num_write_kb': 36724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2350}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:36:42.905 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:36:42.905 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'4599", 'reported_seq': 4655, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:39.691580+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:36:39.691580+0000', 'last_peered': '2024-09-05T16:36:39.691580+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:36:39.691580+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4599, 'log_dups_size': 0, 'ondisk_log_size': 4599, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37603648, 'num_objects': 4591, 'num_object_clones': 0, 'num_object_copies': 18364, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2264, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4591, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4596, 'num_write_kb': 36745, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2330}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:36:42.905 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:36:42.905 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'4623", 'reported_seq': 4700, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:40.401405+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:36:40.401405+0000', 'last_peered': '2024-09-05T16:36:40.401405+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:36:40.401405+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4623, 'log_dups_size': 0, 'ondisk_log_size': 4623, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37852320, 'num_objects': 4621, 'num_object_clones': 0, 'num_object_copies': 18484, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2264, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4621, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4621, 'num_write_kb': 36966, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2359}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:36:42.906 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:36:42.906 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'4693", 'reported_seq': 4749, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:39.478933+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:36:39.478933+0000', 'last_peered': '2024-09-05T16:36:39.478933+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:36:39.478933+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4693, 'log_dups_size': 0, 'ondisk_log_size': 4693, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38399728, 'num_objects': 4688, 'num_object_clones': 0, 'num_object_copies': 18752, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2314, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4688, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4690, 'num_write_kb': 37508, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2377}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:36:42.906 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:36:42.906 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'4534", 'reported_seq': 4569, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:39.758638+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:36:39.758638+0000', 'last_peered': '2024-09-05T16:36:39.758638+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:36:39.758638+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4534, 'log_dups_size': 0, 'ondisk_log_size': 4534, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37132880, 'num_objects': 4533, 'num_object_clones': 0, 'num_object_copies': 18132, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2265, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4533, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4533, 'num_write_kb': 36263, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2269}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:36:42.906 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-05T16:36:42.906 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'4676", 'reported_seq': 4749, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:40.602077+0000', 'last_change': '2024-09-05T16:34:56.798081+0000', 'last_active': '2024-09-05T16:36:40.602077+0000', 'last_peered': '2024-09-05T16:36:40.602077+0000', 'last_clean': '2024-09-05T16:33:45.082039+0000', 'last_became_active': '2024-09-05T16:34:56.792386+0000', 'last_became_peered': '2024-09-05T16:34:56.792386+0000', 'last_unstale': '2024-09-05T16:36:40.602077+0000', 'last_undegraded': '2024-09-05T16:34:56.600595+0000', 'last_fullsized': '2024-09-05T16:34:56.558203+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4676, 'log_dups_size': 0, 'ondisk_log_size': 4676, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38296144, 'num_objects': 4675, 'num_object_clones': 0, 'num_object_copies': 18700, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2295, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4675, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4675, 'num_write_kb': 37399, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2381}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:36:42.906 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:36:42.906 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'4709", 'reported_seq': 4761, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:39.661595+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:36:39.661595+0000', 'last_peered': '2024-09-05T16:36:39.661595+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:36:39.661595+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4709, 'log_dups_size': 0, 'ondisk_log_size': 4709, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38540448, 'num_objects': 4705, 'num_object_clones': 0, 'num_object_copies': 18820, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2359, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4705, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4707, 'num_write_kb': 37652, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2348}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:36:42.906 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:36:42.906 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'4623", 'reported_seq': 4701, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:40.526143+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:36:40.526143+0000', 'last_peered': '2024-09-05T16:36:40.526143+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:36:40.526143+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4623, 'log_dups_size': 0, 'ondisk_log_size': 4623, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37844128, 'num_objects': 4620, 'num_object_clones': 0, 'num_object_copies': 18480, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2322, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4620, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4621, 'num_write_kb': 36965, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2300}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:36:42.907 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:36:42.907 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'4631", 'reported_seq': 4695, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:40.722070+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:36:40.722070+0000', 'last_peered': '2024-09-05T16:36:40.722070+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:36:40.722070+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4631, 'log_dups_size': 0, 'ondisk_log_size': 4631, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37893280, 'num_objects': 4626, 'num_object_clones': 0, 'num_object_copies': 18504, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4626, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4629, 'num_write_kb': 37020, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2389}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:36:42.907 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:36:43.515 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 141 20 74654 74634 4.13453 2.24219 0.122811 0.0377756 2024-09-05T16:36:43.656 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:36:43.657 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 34675/294728 objects degraded (11.765%), 16 pgs degraded, 16 pgs undersized","count":32},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":206,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":73682,"data_bytes":603562176,"bytes_used":1987334144,"bytes_avail":384559722496,"bytes_total":386547056640,"degraded_objects":34675,"degraded_total":294728,"degraded_ratio":0.11765085095410005,"recovering_objects_per_sec":13,"recovering_bytes_per_sec":109862,"recovering_keys_per_sec":0,"num_objects_recovered":161,"num_bytes_recovered":1318912,"num_keys_recovered":0,"write_bytes_sec":2173358,"read_op_per_sec":0,"write_op_per_sec":265},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:36:43.688 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:36:43.688 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:36:44.309 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:36:44.309 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:36:44.327 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":127,"stamp":"2024-09-05T16:36:42.836292+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":603562176,"num_objects":73682,"num_object_clones":0,"num_object_copies":294728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34675,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73682,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":73709,"num_write_kb":589554,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1550,"num_bytes_recovered":12460032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":73736,"ondisk_log_size":73736,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1940756,"kb_used_data":1038480,"kb_used_omap":14,"kb_used_meta":902193,"kb_avail":375546604,"statfs":{"total":386547056640,"available":384559722496,"internally_reserved":0,"allocated":1063403520,"data_stored":1062671992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":923846199},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,10,0,5,15,11,10,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":26091520,"num_objects":3185,"num_object_clones":0,"num_object_copies":12740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-161,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3185,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3185,"num_write_kb":25480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":161,"num_bytes_recovered":1318912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005160"},"pg_stats":[{"pgid":"1.f","version":"33'4618","reported_seq":4660,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:39.677975+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:36:39.677975+0000","last_peered":"2024-09-05T16:36:39.677975+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:36:39.677975+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4618,"log_dups_size":0,"ondisk_log_size":4618,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37830656,"num_objects":4618,"num_object_clones":0,"num_object_copies":18472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4618,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4618,"num_write_kb":36944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2356}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'4556","reported_seq":4620,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:40.734354+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:36:40.734354+0000","last_peered":"2024-09-05T16:36:40.734354+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:36:40.734354+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4556,"log_dups_size":0,"ondisk_log_size":4556,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37287072,"num_objects":4552,"num_object_clones":0,"num_object_copies":18208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4552,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4554,"num_write_kb":36428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2285}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'4625","reported_seq":4700,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:40.516960+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:36:40.516960+0000","last_peered":"2024-09-05T16:36:40.516960+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:36:40.516960+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4625,"log_dups_size":0,"ondisk_log_size":4625,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37861968,"num_objects":4622,"num_object_clones":0,"num_object_copies":18488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4622,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4624,"num_write_kb":36982,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2372}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'4656","reported_seq":7767,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:40.706260+0000","last_change":"2024-09-05T16:34:56.800371+0000","last_active":"2024-09-05T16:36:40.706260+0000","last_peered":"2024-09-05T16:36:40.706260+0000","last_clean":"2024-09-05T16:33:53.148717+0000","last_became_active":"2024-09-05T16:34:56.791755+0000","last_became_peered":"2024-09-05T16:34:56.791755+0000","last_unstale":"2024-09-05T16:36:40.706260+0000","last_undegraded":"2024-09-05T16:34:56.597522+0000","last_fullsized":"2024-09-05T16:34:56.568150+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4656,"log_dups_size":0,"ondisk_log_size":4656,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38106272,"num_objects":4652,"num_object_clones":0,"num_object_copies":18608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":767,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4652,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4654,"num_write_kb":37221,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1525,"num_bytes_recovered":12460032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3885}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'4488","reported_seq":4542,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:39.673923+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:36:39.673923+0000","last_peered":"2024-09-05T16:36:39.673923+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:36:39.673923+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4488,"log_dups_size":0,"ondisk_log_size":4488,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36738208,"num_objects":4485,"num_object_clones":0,"num_object_copies":17940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4485,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4486,"num_write_kb":35885,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2323}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'4613","reported_seq":4651,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:39.711820+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:36:39.711820+0000","last_peered":"2024-09-05T16:36:39.711820+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:36:39.711820+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4613,"log_dups_size":0,"ondisk_log_size":4613,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37789696,"num_objects":4613,"num_object_clones":0,"num_object_copies":18452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4613,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4613,"num_write_kb":36904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2363}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'4498","reported_seq":4563,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:40.638481+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:36:40.638481+0000","last_peered":"2024-09-05T16:36:40.638481+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:36:40.638481+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4498,"log_dups_size":0,"ondisk_log_size":4498,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36795552,"num_objects":4492,"num_object_clones":0,"num_object_copies":17968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4492,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4496,"num_write_kb":35948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2345}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'4594","reported_seq":4658,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:40.707884+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:36:40.707884+0000","last_peered":"2024-09-05T16:36:40.707884+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:36:40.707884+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4594,"log_dups_size":0,"ondisk_log_size":4594,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37590176,"num_objects":4589,"num_object_clones":0,"num_object_copies":18356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4589,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4592,"num_write_kb":36724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2350}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'4599","reported_seq":4655,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:39.691580+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:36:39.691580+0000","last_peered":"2024-09-05T16:36:39.691580+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:36:39.691580+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4599,"log_dups_size":0,"ondisk_log_size":4599,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37603648,"num_objects":4591,"num_object_clones":0,"num_object_copies":18364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4591,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4596,"num_write_kb":36745,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2330}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'4623","reported_seq":4700,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:40.401405+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:36:40.401405+0000","last_peered":"2024-09-05T16:36:40.401405+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:36:40.401405+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4623,"log_dups_size":0,"ondisk_log_size":4623,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37852320,"num_objects":4621,"num_object_clones":0,"num_object_copies":18484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4621,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4621,"num_write_kb":36966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2359}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'4693","reported_seq":4749,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:39.478933+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:36:39.478933+0000","last_peered":"2024-09-05T16:36:39.478933+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:36:39.478933+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4693,"log_dups_size":0,"ondisk_log_size":4693,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38399728,"num_objects":4688,"num_object_clones":0,"num_object_copies":18752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4690,"num_write_kb":37508,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2377}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'4534","reported_seq":4569,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:39.758638+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:36:39.758638+0000","last_peered":"2024-09-05T16:36:39.758638+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:36:39.758638+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4534,"log_dups_size":0,"ondisk_log_size":4534,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37132880,"num_objects":4533,"num_object_clones":0,"num_object_copies":18132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4533,"num_write_kb":36263,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2269}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'4676","reported_seq":4749,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:40.602077+0000","last_change":"2024-09-05T16:34:56.798081+0000","last_active":"2024-09-05T16:36:40.602077+0000","last_peered":"2024-09-05T16:36:40.602077+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:36:40.602077+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4676,"log_dups_size":0,"ondisk_log_size":4676,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38296144,"num_objects":4675,"num_object_clones":0,"num_object_copies":18700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4675,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4675,"num_write_kb":37399,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2381}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'4709","reported_seq":4761,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:39.661595+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:36:39.661595+0000","last_peered":"2024-09-05T16:36:39.661595+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:36:39.661595+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4709,"log_dups_size":0,"ondisk_log_size":4709,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38540448,"num_objects":4705,"num_object_clones":0,"num_object_copies":18820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4705,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4707,"num_write_kb":37652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2348}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'4623","reported_seq":4701,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:40.526143+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:36:40.526143+0000","last_peered":"2024-09-05T16:36:40.526143+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:36:40.526143+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4623,"log_dups_size":0,"ondisk_log_size":4623,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37844128,"num_objects":4620,"num_object_clones":0,"num_object_copies":18480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4620,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4621,"num_write_kb":36965,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2300}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'4631","reported_seq":4695,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:40.722070+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:36:40.722070+0000","last_peered":"2024-09-05T16:36:40.722070+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:36:40.722070+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4631,"log_dups_size":0,"ondisk_log_size":4631,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37893280,"num_objects":4626,"num_object_clones":0,"num_object_copies":18504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4626,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4629,"num_write_kb":37020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2389}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":603562176,"num_objects":73682,"num_object_clones":0,"num_object_copies":294728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34675,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73682,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":73709,"num_write_kb":589554,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1550,"num_bytes_recovered":12460032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064321024,"data_stored":1064321024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":73736,"ondisk_log_size":73736,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084321,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":550228,"kb_used_data":293636,"kb_used_omap":4,"kb_used_meta":256571,"kb_avail":93821612,"statfs":{"total":96636764160,"available":96073330688,"internally_reserved":0,"allocated":300683264,"data_stored":300500382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":262729427},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,7,0,2,1,6,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986198,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312724,"kb_used_data":154692,"kb_used_omap":4,"kb_used_meta":158011,"kb_avail":94059116,"statfs":{"total":96636764160,"available":96316534784,"internally_reserved":0,"allocated":158404608,"data_stored":158221726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":161803986},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,2,0,5,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673004,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":538256,"kb_used_data":294656,"kb_used_omap":1,"kb_used_meta":243582,"kb_avail":93833584,"statfs":{"total":96636764160,"available":96085590016,"internally_reserved":0,"allocated":301727744,"data_stored":301544862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":249428421},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,2,1,2,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182435,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":539548,"kb_used_data":295496,"kb_used_omap":4,"kb_used_meta":244027,"kb_avail":93832292,"statfs":{"total":96636764160,"available":96084267008,"internally_reserved":0,"allocated":302587904,"data_stored":302405022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":249884365},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,11,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":302346240,"data_stored":302346240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":158777344,"data_stored":158777344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":302153728,"data_stored":302153728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":301043712,"data_stored":301043712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:36:44.515 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 142 20 74954 74934 4.12192 2.34375 0.0871738 0.0378969 2024-09-05T16:36:45.519 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 143 20 75280 75260 4.11091 2.54688 0.0442198 0.0379988 2024-09-05T16:36:46.516 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 144 20 75606 75586 4.10004 2.54688 0.0790633 0.0380983 2024-09-05T16:36:47.330 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:36:47.516 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 145 20 75927 75907 4.08906 2.50781 0.0821354 0.0381986 2024-09-05T16:36:47.948 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:36:47.948 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:36:47.982 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":129,"stamp":"2024-09-05T16:36:46.837628+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":616210624,"num_objects":75226,"num_object_clones":0,"num_object_copies":300904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34600,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":75226,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":75253,"num_write_kb":601906,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1625,"num_bytes_recovered":13074432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75280,"ondisk_log_size":75280,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1964576,"kb_used_data":1062300,"kb_used_omap":14,"kb_used_meta":902193,"kb_avail":375522784,"statfs":{"total":386547056640,"available":384535330816,"internally_reserved":0,"allocated":1087795200,"data_stored":1087063672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":923846199},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,7,3,6,19,17,12,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":33079296,"num_objects":4038,"num_object_clones":0,"num_object_copies":16152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4038,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4038,"num_write_kb":32304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":236,"num_bytes_recovered":1933312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004925"},"pg_stats":[{"pgid":"1.f","version":"33'4724","reported_seq":4766,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:44.733631+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:36:44.733631+0000","last_peered":"2024-09-05T16:36:44.733631+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:36:44.733631+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4724,"log_dups_size":0,"ondisk_log_size":4724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38699008,"num_objects":4724,"num_object_clones":0,"num_object_copies":18896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4724,"num_write_kb":37792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2462}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'4639","reported_seq":4703,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:45.722157+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:36:45.722157+0000","last_peered":"2024-09-05T16:36:45.722157+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:36:45.722157+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4639,"log_dups_size":0,"ondisk_log_size":4639,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37967008,"num_objects":4635,"num_object_clones":0,"num_object_copies":18540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4635,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":37092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2368}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'4703","reported_seq":4778,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:45.403521+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:36:45.403521+0000","last_peered":"2024-09-05T16:36:45.403521+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:36:45.403521+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4703,"log_dups_size":0,"ondisk_log_size":4703,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38500944,"num_objects":4700,"num_object_clones":0,"num_object_copies":18800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4700,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4702,"num_write_kb":37606,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2450}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'4759","reported_seq":8020,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:45.734961+0000","last_change":"2024-09-05T16:34:56.800371+0000","last_active":"2024-09-05T16:36:45.734961+0000","last_peered":"2024-09-05T16:36:45.734961+0000","last_clean":"2024-09-05T16:33:53.148717+0000","last_became_active":"2024-09-05T16:34:56.791755+0000","last_became_peered":"2024-09-05T16:34:56.791755+0000","last_unstale":"2024-09-05T16:36:45.734961+0000","last_undegraded":"2024-09-05T16:34:56.597522+0000","last_fullsized":"2024-09-05T16:34:56.568150+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4759,"log_dups_size":0,"ondisk_log_size":4759,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38950048,"num_objects":4755,"num_object_clones":0,"num_object_copies":19020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":692,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4755,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4757,"num_write_kb":38045,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1600,"num_bytes_recovered":13074432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4063}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'4590","reported_seq":4644,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:44.704437+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:36:44.704437+0000","last_peered":"2024-09-05T16:36:44.704437+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:36:44.704437+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4590,"log_dups_size":0,"ondisk_log_size":4590,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37573792,"num_objects":4587,"num_object_clones":0,"num_object_copies":18348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4587,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4588,"num_write_kb":36701,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2425}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'4719","reported_seq":4761,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:44.793023+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:36:44.793023+0000","last_peered":"2024-09-05T16:36:44.793023+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:36:44.793023+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4719,"log_dups_size":0,"ondisk_log_size":4719,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38658048,"num_objects":4719,"num_object_clones":0,"num_object_copies":18876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4719,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4719,"num_write_kb":37752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2469}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'4580","reported_seq":4645,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:45.759980+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:36:45.759980+0000","last_peered":"2024-09-05T16:36:45.759980+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:36:45.759980+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4580,"log_dups_size":0,"ondisk_log_size":4580,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37467296,"num_objects":4574,"num_object_clones":0,"num_object_copies":18296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4574,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4578,"num_write_kb":36604,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2427}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'4696","reported_seq":4760,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:45.760353+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:36:45.760353+0000","last_peered":"2024-09-05T16:36:45.760353+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:36:45.760353+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4696,"log_dups_size":0,"ondisk_log_size":4696,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38425760,"num_objects":4691,"num_object_clones":0,"num_object_copies":18764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4694,"num_write_kb":37540,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2452}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'4711","reported_seq":4767,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:44.802371+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:36:44.802371+0000","last_peered":"2024-09-05T16:36:44.802371+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:36:44.802371+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4711,"log_dups_size":0,"ondisk_log_size":4711,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38521152,"num_objects":4703,"num_object_clones":0,"num_object_copies":18812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4703,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4708,"num_write_kb":37641,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2442}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'4722","reported_seq":4798,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:45.574536+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:36:45.574536+0000","last_peered":"2024-09-05T16:36:45.574536+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:36:45.574536+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4722,"log_dups_size":0,"ondisk_log_size":4722,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38663328,"num_objects":4720,"num_object_clones":0,"num_object_copies":18880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4720,"num_write_kb":37758,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2458}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'4783","reported_seq":4839,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:44.729999+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:36:44.729999+0000","last_peered":"2024-09-05T16:36:44.729999+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:36:44.729999+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4783,"log_dups_size":0,"ondisk_log_size":4783,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39137008,"num_objects":4778,"num_object_clones":0,"num_object_copies":19112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4778,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4780,"num_write_kb":38228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2467}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'4638","reported_seq":4674,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:44.816857+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:36:44.816857+0000","last_peered":"2024-09-05T16:36:44.816857+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:36:44.816857+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4638,"log_dups_size":0,"ondisk_log_size":4638,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37984848,"num_objects":4637,"num_object_clones":0,"num_object_copies":18548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4637,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":37095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2373}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'4756","reported_seq":4830,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:45.507315+0000","last_change":"2024-09-05T16:34:56.798081+0000","last_active":"2024-09-05T16:36:45.507315+0000","last_peered":"2024-09-05T16:36:45.507315+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:36:45.507315+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4756,"log_dups_size":0,"ondisk_log_size":4756,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38951504,"num_objects":4755,"num_object_clones":0,"num_object_copies":19020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4755,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4755,"num_write_kb":38039,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2461}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'4806","reported_seq":4858,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:44.825020+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:36:44.825020+0000","last_peered":"2024-09-05T16:36:44.825020+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:36:44.825020+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4806,"log_dups_size":0,"ondisk_log_size":4806,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39335072,"num_objects":4802,"num_object_clones":0,"num_object_copies":19208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4802,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4804,"num_write_kb":38428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2445}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'4718","reported_seq":4796,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:45.504119+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:36:45.504119+0000","last_peered":"2024-09-05T16:36:45.504119+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:36:45.504119+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4718,"log_dups_size":0,"ondisk_log_size":4718,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38622368,"num_objects":4715,"num_object_clones":0,"num_object_copies":18860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4715,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4716,"num_write_kb":37725,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2395}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'4736","reported_seq":4800,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:45.516768+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:36:45.516768+0000","last_peered":"2024-09-05T16:36:45.516768+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:36:45.516768+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4736,"log_dups_size":0,"ondisk_log_size":4736,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38753440,"num_objects":4731,"num_object_clones":0,"num_object_copies":18924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4731,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4734,"num_write_kb":37860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2494}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":616210624,"num_objects":75226,"num_object_clones":0,"num_object_copies":300904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34600,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":75226,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":75253,"num_write_kb":601906,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1625,"num_bytes_recovered":13074432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1090097152,"data_stored":1090097152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75280,"ondisk_log_size":75280,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084322,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":543940,"kb_used_data":299828,"kb_used_omap":4,"kb_used_meta":244091,"kb_avail":93827900,"statfs":{"total":96636764160,"available":96079769600,"internally_reserved":0,"allocated":307023872,"data_stored":306840990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":249949907},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,0,11,6,4,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986199,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":323292,"kb_used_data":161100,"kb_used_omap":4,"kb_used_meta":162171,"kb_avail":94048548,"statfs":{"total":96636764160,"available":96305713152,"internally_reserved":0,"allocated":164966400,"data_stored":164783518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":166063826},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,1,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673005,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":548740,"kb_used_data":300980,"kb_used_omap":1,"kb_used_meta":247742,"kb_avail":93823100,"statfs":{"total":96636764160,"available":96074854400,"internally_reserved":0,"allocated":308203520,"data_stored":308020638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":253688261},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,5,0,0,1,7,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182436,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":548604,"kb_used_data":300392,"kb_used_omap":4,"kb_used_meta":248187,"kb_avail":93823236,"statfs":{"total":96636764160,"available":96074993664,"internally_reserved":0,"allocated":307601408,"data_stored":307418526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":254144205},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,6,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":308674560,"data_stored":308674560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":165482496,"data_stored":165482496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":308482048,"data_stored":308482048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":307458048,"data_stored":307458048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:36:47.984 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:36:48.516 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 146 20 76217 76197 4.07657 2.26562 0.0532067 0.0383153 2024-09-05T16:36:48.607 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:36:48.608 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:36:48.634 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":129,"stamp":"2024-09-05T16:36:46.837628+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":616210624,"num_objects":75226,"num_object_clones":0,"num_object_copies":300904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34600,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":75226,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":75253,"num_write_kb":601906,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1625,"num_bytes_recovered":13074432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75280,"ondisk_log_size":75280,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1964576,"kb_used_data":1062300,"kb_used_omap":14,"kb_used_meta":902193,"kb_avail":375522784,"statfs":{"total":386547056640,"available":384535330816,"internally_reserved":0,"allocated":1087795200,"data_stored":1087063672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":923846199},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,7,3,6,19,17,12,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":33079296,"num_objects":4038,"num_object_clones":0,"num_object_copies":16152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4038,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4038,"num_write_kb":32304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":236,"num_bytes_recovered":1933312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004925"},"pg_stats":[{"pgid":"1.f","version":"33'4724","reported_seq":4766,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:44.733631+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:36:44.733631+0000","last_peered":"2024-09-05T16:36:44.733631+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:36:44.733631+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4724,"log_dups_size":0,"ondisk_log_size":4724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38699008,"num_objects":4724,"num_object_clones":0,"num_object_copies":18896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4724,"num_write_kb":37792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2462}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'4639","reported_seq":4703,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:45.722157+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:36:45.722157+0000","last_peered":"2024-09-05T16:36:45.722157+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:36:45.722157+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4639,"log_dups_size":0,"ondisk_log_size":4639,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37967008,"num_objects":4635,"num_object_clones":0,"num_object_copies":18540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4635,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":37092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2368}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'4703","reported_seq":4778,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:45.403521+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:36:45.403521+0000","last_peered":"2024-09-05T16:36:45.403521+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:36:45.403521+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4703,"log_dups_size":0,"ondisk_log_size":4703,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38500944,"num_objects":4700,"num_object_clones":0,"num_object_copies":18800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4700,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4702,"num_write_kb":37606,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2450}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'4759","reported_seq":8020,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:45.734961+0000","last_change":"2024-09-05T16:34:56.800371+0000","last_active":"2024-09-05T16:36:45.734961+0000","last_peered":"2024-09-05T16:36:45.734961+0000","last_clean":"2024-09-05T16:33:53.148717+0000","last_became_active":"2024-09-05T16:34:56.791755+0000","last_became_peered":"2024-09-05T16:34:56.791755+0000","last_unstale":"2024-09-05T16:36:45.734961+0000","last_undegraded":"2024-09-05T16:34:56.597522+0000","last_fullsized":"2024-09-05T16:34:56.568150+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4759,"log_dups_size":0,"ondisk_log_size":4759,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38950048,"num_objects":4755,"num_object_clones":0,"num_object_copies":19020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":692,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4755,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4757,"num_write_kb":38045,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1600,"num_bytes_recovered":13074432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4063}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'4590","reported_seq":4644,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:44.704437+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:36:44.704437+0000","last_peered":"2024-09-05T16:36:44.704437+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:36:44.704437+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4590,"log_dups_size":0,"ondisk_log_size":4590,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37573792,"num_objects":4587,"num_object_clones":0,"num_object_copies":18348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4587,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4588,"num_write_kb":36701,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2425}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'4719","reported_seq":4761,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:44.793023+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:36:44.793023+0000","last_peered":"2024-09-05T16:36:44.793023+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:36:44.793023+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4719,"log_dups_size":0,"ondisk_log_size":4719,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38658048,"num_objects":4719,"num_object_clones":0,"num_object_copies":18876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4719,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4719,"num_write_kb":37752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2469}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'4580","reported_seq":4645,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:45.759980+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:36:45.759980+0000","last_peered":"2024-09-05T16:36:45.759980+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:36:45.759980+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4580,"log_dups_size":0,"ondisk_log_size":4580,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37467296,"num_objects":4574,"num_object_clones":0,"num_object_copies":18296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4574,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4578,"num_write_kb":36604,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2427}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'4696","reported_seq":4760,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:45.760353+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:36:45.760353+0000","last_peered":"2024-09-05T16:36:45.760353+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:36:45.760353+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4696,"log_dups_size":0,"ondisk_log_size":4696,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38425760,"num_objects":4691,"num_object_clones":0,"num_object_copies":18764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4694,"num_write_kb":37540,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2452}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'4711","reported_seq":4767,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:44.802371+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:36:44.802371+0000","last_peered":"2024-09-05T16:36:44.802371+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:36:44.802371+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4711,"log_dups_size":0,"ondisk_log_size":4711,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38521152,"num_objects":4703,"num_object_clones":0,"num_object_copies":18812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4703,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4708,"num_write_kb":37641,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2442}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'4722","reported_seq":4798,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:45.574536+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:36:45.574536+0000","last_peered":"2024-09-05T16:36:45.574536+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:36:45.574536+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4722,"log_dups_size":0,"ondisk_log_size":4722,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38663328,"num_objects":4720,"num_object_clones":0,"num_object_copies":18880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4720,"num_write_kb":37758,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2458}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'4783","reported_seq":4839,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:44.729999+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:36:44.729999+0000","last_peered":"2024-09-05T16:36:44.729999+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:36:44.729999+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4783,"log_dups_size":0,"ondisk_log_size":4783,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39137008,"num_objects":4778,"num_object_clones":0,"num_object_copies":19112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4778,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4780,"num_write_kb":38228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2467}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'4638","reported_seq":4674,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:44.816857+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:36:44.816857+0000","last_peered":"2024-09-05T16:36:44.816857+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:36:44.816857+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4638,"log_dups_size":0,"ondisk_log_size":4638,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37984848,"num_objects":4637,"num_object_clones":0,"num_object_copies":18548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4637,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":37095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2373}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'4756","reported_seq":4830,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:45.507315+0000","last_change":"2024-09-05T16:34:56.798081+0000","last_active":"2024-09-05T16:36:45.507315+0000","last_peered":"2024-09-05T16:36:45.507315+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:36:45.507315+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4756,"log_dups_size":0,"ondisk_log_size":4756,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38951504,"num_objects":4755,"num_object_clones":0,"num_object_copies":19020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4755,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4755,"num_write_kb":38039,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2461}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'4806","reported_seq":4858,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:44.825020+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:36:44.825020+0000","last_peered":"2024-09-05T16:36:44.825020+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:36:44.825020+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4806,"log_dups_size":0,"ondisk_log_size":4806,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39335072,"num_objects":4802,"num_object_clones":0,"num_object_copies":19208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4802,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4804,"num_write_kb":38428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2445}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'4718","reported_seq":4796,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:45.504119+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:36:45.504119+0000","last_peered":"2024-09-05T16:36:45.504119+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:36:45.504119+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4718,"log_dups_size":0,"ondisk_log_size":4718,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38622368,"num_objects":4715,"num_object_clones":0,"num_object_copies":18860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4715,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4716,"num_write_kb":37725,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2395}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'4736","reported_seq":4800,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:45.516768+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:36:45.516768+0000","last_peered":"2024-09-05T16:36:45.516768+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:36:45.516768+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4736,"log_dups_size":0,"ondisk_log_size":4736,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38753440,"num_objects":4731,"num_object_clones":0,"num_object_copies":18924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4731,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4734,"num_write_kb":37860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2494}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":616210624,"num_objects":75226,"num_object_clones":0,"num_object_copies":300904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34600,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":75226,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":75253,"num_write_kb":601906,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1625,"num_bytes_recovered":13074432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1090097152,"data_stored":1090097152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75280,"ondisk_log_size":75280,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084322,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":543940,"kb_used_data":299828,"kb_used_omap":4,"kb_used_meta":244091,"kb_avail":93827900,"statfs":{"total":96636764160,"available":96079769600,"internally_reserved":0,"allocated":307023872,"data_stored":306840990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":249949907},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,0,11,6,4,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986199,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":323292,"kb_used_data":161100,"kb_used_omap":4,"kb_used_meta":162171,"kb_avail":94048548,"statfs":{"total":96636764160,"available":96305713152,"internally_reserved":0,"allocated":164966400,"data_stored":164783518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":166063826},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,1,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673005,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":548740,"kb_used_data":300980,"kb_used_omap":1,"kb_used_meta":247742,"kb_avail":93823100,"statfs":{"total":96636764160,"available":96074854400,"internally_reserved":0,"allocated":308203520,"data_stored":308020638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":253688261},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,5,0,0,1,7,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182436,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":548604,"kb_used_data":300392,"kb_used_omap":4,"kb_used_meta":248187,"kb_avail":93823236,"statfs":{"total":96636764160,"available":96074993664,"internally_reserved":0,"allocated":307601408,"data_stored":307418526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":254144205},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,6,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":308674560,"data_stored":308674560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":165482496,"data_stored":165482496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":308482048,"data_stored":308482048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":307458048,"data_stored":307458048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:36:48.635 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-05T16:36:48.635 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'4724", 'reported_seq': 4766, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:44.733631+0000', 'last_change': '2024-09-05T16:34:56.833497+0000', 'last_active': '2024-09-05T16:36:44.733631+0000', 'last_peered': '2024-09-05T16:36:44.733631+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.828289+0000', 'last_became_peered': '2024-09-05T16:34:56.828289+0000', 'last_unstale': '2024-09-05T16:36:44.733631+0000', 'last_undegraded': '2024-09-05T16:34:56.612160+0000', 'last_fullsized': '2024-09-05T16:34:56.557632+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4724, 'log_dups_size': 0, 'ondisk_log_size': 4724, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38699008, 'num_objects': 4724, 'num_object_clones': 0, 'num_object_copies': 18896, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2262, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4724, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4724, 'num_write_kb': 37792, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2462}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:36:48.635 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-05T16:36:48.635 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'4639", 'reported_seq': 4703, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:45.722157+0000', 'last_change': '2024-09-05T16:34:56.825031+0000', 'last_active': '2024-09-05T16:36:45.722157+0000', 'last_peered': '2024-09-05T16:36:45.722157+0000', 'last_clean': '2024-09-05T16:33:53.200762+0000', 'last_became_active': '2024-09-05T16:34:56.820292+0000', 'last_became_peered': '2024-09-05T16:34:56.820292+0000', 'last_unstale': '2024-09-05T16:36:45.722157+0000', 'last_undegraded': '2024-09-05T16:34:56.605776+0000', 'last_fullsized': '2024-09-05T16:34:56.559383+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4639, 'log_dups_size': 0, 'ondisk_log_size': 4639, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37967008, 'num_objects': 4635, 'num_object_clones': 0, 'num_object_copies': 18540, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2269, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4635, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4637, 'num_write_kb': 37092, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2368}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:36:48.636 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:36:48.636 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'4703", 'reported_seq': 4778, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:45.403521+0000', 'last_change': '2024-09-05T16:34:56.867900+0000', 'last_active': '2024-09-05T16:36:45.403521+0000', 'last_peered': '2024-09-05T16:36:45.403521+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:36:45.403521+0000', 'last_undegraded': '2024-09-05T16:34:56.590972+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4703, 'log_dups_size': 0, 'ondisk_log_size': 4703, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38500944, 'num_objects': 4700, 'num_object_clones': 0, 'num_object_copies': 18800, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2251, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4700, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4702, 'num_write_kb': 37606, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2450}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:36:48.636 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-05T16:36:48.636 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'4759", 'reported_seq': 8020, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:45.734961+0000', 'last_change': '2024-09-05T16:34:56.800371+0000', 'last_active': '2024-09-05T16:36:45.734961+0000', 'last_peered': '2024-09-05T16:36:45.734961+0000', 'last_clean': '2024-09-05T16:33:53.148717+0000', 'last_became_active': '2024-09-05T16:34:56.791755+0000', 'last_became_peered': '2024-09-05T16:34:56.791755+0000', 'last_unstale': '2024-09-05T16:36:45.734961+0000', 'last_undegraded': '2024-09-05T16:34:56.597522+0000', 'last_fullsized': '2024-09-05T16:34:56.568150+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4759, 'log_dups_size': 0, 'ondisk_log_size': 4759, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38950048, 'num_objects': 4755, 'num_object_clones': 0, 'num_object_copies': 19020, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 692, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4755, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4757, 'num_write_kb': 38045, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1600, 'num_bytes_recovered': 13074432, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 4063}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:36:48.636 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:36:48.636 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'4590", 'reported_seq': 4644, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:44.704437+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:36:44.704437+0000', 'last_peered': '2024-09-05T16:36:44.704437+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:36:44.704437+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4590, 'log_dups_size': 0, 'ondisk_log_size': 4590, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37573792, 'num_objects': 4587, 'num_object_clones': 0, 'num_object_copies': 18348, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2164, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4587, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4588, 'num_write_kb': 36701, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2425}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:36:48.636 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:36:48.636 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'4719", 'reported_seq': 4761, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:44.793023+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:36:44.793023+0000', 'last_peered': '2024-09-05T16:36:44.793023+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:36:44.793023+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4719, 'log_dups_size': 0, 'ondisk_log_size': 4719, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38658048, 'num_objects': 4719, 'num_object_clones': 0, 'num_object_copies': 18876, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4719, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4719, 'num_write_kb': 37752, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2469}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:36:48.636 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:36:48.636 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'4580", 'reported_seq': 4645, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:45.759980+0000', 'last_change': '2024-09-05T16:34:56.903436+0000', 'last_active': '2024-09-05T16:36:45.759980+0000', 'last_peered': '2024-09-05T16:36:45.759980+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:36:45.759980+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4580, 'log_dups_size': 0, 'ondisk_log_size': 4580, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37467296, 'num_objects': 4574, 'num_object_clones': 0, 'num_object_copies': 18296, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2149, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4574, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4578, 'num_write_kb': 36604, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2427}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:36:48.637 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-05T16:36:48.637 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'4696", 'reported_seq': 4760, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:45.760353+0000', 'last_change': '2024-09-05T16:34:56.883319+0000', 'last_active': '2024-09-05T16:36:45.760353+0000', 'last_peered': '2024-09-05T16:36:45.760353+0000', 'last_clean': '2024-09-05T16:33:53.505305+0000', 'last_became_active': '2024-09-05T16:34:56.873389+0000', 'last_became_peered': '2024-09-05T16:34:56.873389+0000', 'last_unstale': '2024-09-05T16:36:45.760353+0000', 'last_undegraded': '2024-09-05T16:34:56.602567+0000', 'last_fullsized': '2024-09-05T16:34:56.559958+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4696, 'log_dups_size': 0, 'ondisk_log_size': 4696, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38425760, 'num_objects': 4691, 'num_object_clones': 0, 'num_object_copies': 18764, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2241, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4691, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4694, 'num_write_kb': 37540, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2452}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:36:48.637 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:36:48.637 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'4711", 'reported_seq': 4767, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:44.802371+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:36:44.802371+0000', 'last_peered': '2024-09-05T16:36:44.802371+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:36:44.802371+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4711, 'log_dups_size': 0, 'ondisk_log_size': 4711, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38521152, 'num_objects': 4703, 'num_object_clones': 0, 'num_object_copies': 18812, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2264, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4703, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4708, 'num_write_kb': 37641, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2442}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:36:48.637 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:36:48.637 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'4722", 'reported_seq': 4798, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:45.574536+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:36:45.574536+0000', 'last_peered': '2024-09-05T16:36:45.574536+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:36:45.574536+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4722, 'log_dups_size': 0, 'ondisk_log_size': 4722, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38663328, 'num_objects': 4720, 'num_object_clones': 0, 'num_object_copies': 18880, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2264, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4720, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4720, 'num_write_kb': 37758, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2458}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:36:48.637 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:36:48.637 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'4783", 'reported_seq': 4839, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:44.729999+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:36:44.729999+0000', 'last_peered': '2024-09-05T16:36:44.729999+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:36:44.729999+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4783, 'log_dups_size': 0, 'ondisk_log_size': 4783, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39137008, 'num_objects': 4778, 'num_object_clones': 0, 'num_object_copies': 19112, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2314, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4778, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4780, 'num_write_kb': 38228, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2467}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:36:48.637 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:36:48.637 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'4638", 'reported_seq': 4674, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:44.816857+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:36:44.816857+0000', 'last_peered': '2024-09-05T16:36:44.816857+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:36:44.816857+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4638, 'log_dups_size': 0, 'ondisk_log_size': 4638, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37984848, 'num_objects': 4637, 'num_object_clones': 0, 'num_object_copies': 18548, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2265, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4637, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4637, 'num_write_kb': 37095, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2373}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:36:48.638 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-05T16:36:48.638 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'4756", 'reported_seq': 4830, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:45.507315+0000', 'last_change': '2024-09-05T16:34:56.798081+0000', 'last_active': '2024-09-05T16:36:45.507315+0000', 'last_peered': '2024-09-05T16:36:45.507315+0000', 'last_clean': '2024-09-05T16:33:45.082039+0000', 'last_became_active': '2024-09-05T16:34:56.792386+0000', 'last_became_peered': '2024-09-05T16:34:56.792386+0000', 'last_unstale': '2024-09-05T16:36:45.507315+0000', 'last_undegraded': '2024-09-05T16:34:56.600595+0000', 'last_fullsized': '2024-09-05T16:34:56.558203+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4756, 'log_dups_size': 0, 'ondisk_log_size': 4756, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38951504, 'num_objects': 4755, 'num_object_clones': 0, 'num_object_copies': 19020, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2295, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4755, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4755, 'num_write_kb': 38039, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2461}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:36:48.638 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:36:48.638 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'4806", 'reported_seq': 4858, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:44.825020+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:36:44.825020+0000', 'last_peered': '2024-09-05T16:36:44.825020+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:36:44.825020+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4806, 'log_dups_size': 0, 'ondisk_log_size': 4806, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39335072, 'num_objects': 4802, 'num_object_clones': 0, 'num_object_copies': 19208, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2359, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4802, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4804, 'num_write_kb': 38428, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2445}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:36:48.638 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:36:48.638 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'4718", 'reported_seq': 4796, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:45.504119+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:36:45.504119+0000', 'last_peered': '2024-09-05T16:36:45.504119+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:36:45.504119+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4718, 'log_dups_size': 0, 'ondisk_log_size': 4718, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38622368, 'num_objects': 4715, 'num_object_clones': 0, 'num_object_copies': 18860, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2322, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4715, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4716, 'num_write_kb': 37725, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2395}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:36:48.638 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:36:48.638 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'4736", 'reported_seq': 4800, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:45.516768+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:36:45.516768+0000', 'last_peered': '2024-09-05T16:36:45.516768+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:36:45.516768+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4736, 'log_dups_size': 0, 'ondisk_log_size': 4736, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38753440, 'num_objects': 4731, 'num_object_clones': 0, 'num_object_copies': 18924, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4731, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4734, 'num_write_kb': 37860, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2494}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:36:48.638 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:36:49.373 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:36:49.373 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 34600/300904 objects degraded (11.499%), 16 pgs degraded, 16 pgs undersized","count":32},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":212,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":75226,"data_bytes":616210624,"bytes_used":2011725824,"bytes_avail":384535330816,"bytes_total":386547056640,"degraded_objects":34600,"degraded_total":300904,"degraded_ratio":0.11498683965650174,"recovering_objects_per_sec":12,"recovering_bytes_per_sec":99629,"recovering_keys_per_sec":0,"num_objects_recovered":146,"num_bytes_recovered":1196032,"num_keys_recovered":0,"write_bytes_sec":2131787,"read_op_per_sec":0,"write_op_per_sec":260},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:36:49.408 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:36:49.409 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:36:49.516 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 147 20 76499 76479 4.06383 2.20312 0.0630913 0.0384352 2024-09-05T16:36:49.986 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:36:49.987 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:36:50.015 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":130,"stamp":"2024-09-05T16:36:48.838231+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":616210624,"num_objects":75226,"num_object_clones":0,"num_object_copies":300904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34600,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":75226,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":75253,"num_write_kb":601906,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1625,"num_bytes_recovered":13074432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75280,"ondisk_log_size":75280,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1964576,"kb_used_data":1062300,"kb_used_omap":14,"kb_used_meta":902193,"kb_avail":375522784,"statfs":{"total":386547056640,"available":384535330816,"internally_reserved":0,"allocated":1087795200,"data_stored":1087063672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":923846199},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,7,3,6,19,17,12,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":25591808,"num_objects":3124,"num_object_clones":0,"num_object_copies":12496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-146,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3124,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3124,"num_write_kb":24992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":146,"num_bytes_recovered":1196032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004857"},"pg_stats":[{"pgid":"1.f","version":"33'4724","reported_seq":4766,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:44.733631+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:36:44.733631+0000","last_peered":"2024-09-05T16:36:44.733631+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:36:44.733631+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4724,"log_dups_size":0,"ondisk_log_size":4724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38699008,"num_objects":4724,"num_object_clones":0,"num_object_copies":18896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4724,"num_write_kb":37792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2462}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'4639","reported_seq":4703,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:45.722157+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:36:45.722157+0000","last_peered":"2024-09-05T16:36:45.722157+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:36:45.722157+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4639,"log_dups_size":0,"ondisk_log_size":4639,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37967008,"num_objects":4635,"num_object_clones":0,"num_object_copies":18540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4635,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":37092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2368}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'4703","reported_seq":4778,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:45.403521+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:36:45.403521+0000","last_peered":"2024-09-05T16:36:45.403521+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:36:45.403521+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4703,"log_dups_size":0,"ondisk_log_size":4703,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38500944,"num_objects":4700,"num_object_clones":0,"num_object_copies":18800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4700,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4702,"num_write_kb":37606,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2450}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'4759","reported_seq":8020,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:45.734961+0000","last_change":"2024-09-05T16:34:56.800371+0000","last_active":"2024-09-05T16:36:45.734961+0000","last_peered":"2024-09-05T16:36:45.734961+0000","last_clean":"2024-09-05T16:33:53.148717+0000","last_became_active":"2024-09-05T16:34:56.791755+0000","last_became_peered":"2024-09-05T16:34:56.791755+0000","last_unstale":"2024-09-05T16:36:45.734961+0000","last_undegraded":"2024-09-05T16:34:56.597522+0000","last_fullsized":"2024-09-05T16:34:56.568150+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4759,"log_dups_size":0,"ondisk_log_size":4759,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38950048,"num_objects":4755,"num_object_clones":0,"num_object_copies":19020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":692,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4755,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4757,"num_write_kb":38045,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1600,"num_bytes_recovered":13074432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4063}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'4590","reported_seq":4644,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:44.704437+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:36:44.704437+0000","last_peered":"2024-09-05T16:36:44.704437+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:36:44.704437+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4590,"log_dups_size":0,"ondisk_log_size":4590,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37573792,"num_objects":4587,"num_object_clones":0,"num_object_copies":18348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4587,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4588,"num_write_kb":36701,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2425}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'4719","reported_seq":4761,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:44.793023+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:36:44.793023+0000","last_peered":"2024-09-05T16:36:44.793023+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:36:44.793023+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4719,"log_dups_size":0,"ondisk_log_size":4719,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38658048,"num_objects":4719,"num_object_clones":0,"num_object_copies":18876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4719,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4719,"num_write_kb":37752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2469}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'4580","reported_seq":4645,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:45.759980+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:36:45.759980+0000","last_peered":"2024-09-05T16:36:45.759980+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:36:45.759980+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4580,"log_dups_size":0,"ondisk_log_size":4580,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37467296,"num_objects":4574,"num_object_clones":0,"num_object_copies":18296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4574,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4578,"num_write_kb":36604,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2427}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'4696","reported_seq":4760,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:45.760353+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:36:45.760353+0000","last_peered":"2024-09-05T16:36:45.760353+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:36:45.760353+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4696,"log_dups_size":0,"ondisk_log_size":4696,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38425760,"num_objects":4691,"num_object_clones":0,"num_object_copies":18764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4694,"num_write_kb":37540,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2452}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'4711","reported_seq":4767,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:44.802371+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:36:44.802371+0000","last_peered":"2024-09-05T16:36:44.802371+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:36:44.802371+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4711,"log_dups_size":0,"ondisk_log_size":4711,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38521152,"num_objects":4703,"num_object_clones":0,"num_object_copies":18812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4703,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4708,"num_write_kb":37641,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2442}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'4722","reported_seq":4798,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:45.574536+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:36:45.574536+0000","last_peered":"2024-09-05T16:36:45.574536+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:36:45.574536+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4722,"log_dups_size":0,"ondisk_log_size":4722,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38663328,"num_objects":4720,"num_object_clones":0,"num_object_copies":18880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4720,"num_write_kb":37758,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2458}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'4783","reported_seq":4839,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:44.729999+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:36:44.729999+0000","last_peered":"2024-09-05T16:36:44.729999+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:36:44.729999+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4783,"log_dups_size":0,"ondisk_log_size":4783,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39137008,"num_objects":4778,"num_object_clones":0,"num_object_copies":19112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4778,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4780,"num_write_kb":38228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2467}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'4638","reported_seq":4674,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:44.816857+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:36:44.816857+0000","last_peered":"2024-09-05T16:36:44.816857+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:36:44.816857+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4638,"log_dups_size":0,"ondisk_log_size":4638,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37984848,"num_objects":4637,"num_object_clones":0,"num_object_copies":18548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4637,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":37095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2373}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'4756","reported_seq":4830,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:45.507315+0000","last_change":"2024-09-05T16:34:56.798081+0000","last_active":"2024-09-05T16:36:45.507315+0000","last_peered":"2024-09-05T16:36:45.507315+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:36:45.507315+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4756,"log_dups_size":0,"ondisk_log_size":4756,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38951504,"num_objects":4755,"num_object_clones":0,"num_object_copies":19020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4755,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4755,"num_write_kb":38039,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2461}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'4806","reported_seq":4858,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:44.825020+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:36:44.825020+0000","last_peered":"2024-09-05T16:36:44.825020+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:36:44.825020+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4806,"log_dups_size":0,"ondisk_log_size":4806,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39335072,"num_objects":4802,"num_object_clones":0,"num_object_copies":19208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4802,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4804,"num_write_kb":38428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2445}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'4718","reported_seq":4796,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:45.504119+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:36:45.504119+0000","last_peered":"2024-09-05T16:36:45.504119+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:36:45.504119+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4718,"log_dups_size":0,"ondisk_log_size":4718,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38622368,"num_objects":4715,"num_object_clones":0,"num_object_copies":18860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4715,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4716,"num_write_kb":37725,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2395}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'4736","reported_seq":4800,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:45.516768+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:36:45.516768+0000","last_peered":"2024-09-05T16:36:45.516768+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:36:45.516768+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4736,"log_dups_size":0,"ondisk_log_size":4736,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38753440,"num_objects":4731,"num_object_clones":0,"num_object_copies":18924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4731,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4734,"num_write_kb":37860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2494}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":616210624,"num_objects":75226,"num_object_clones":0,"num_object_copies":300904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34600,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":75226,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":75253,"num_write_kb":601906,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1625,"num_bytes_recovered":13074432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1090097152,"data_stored":1090097152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75280,"ondisk_log_size":75280,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084322,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":543940,"kb_used_data":299828,"kb_used_omap":4,"kb_used_meta":244091,"kb_avail":93827900,"statfs":{"total":96636764160,"available":96079769600,"internally_reserved":0,"allocated":307023872,"data_stored":306840990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":249949907},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,0,11,6,4,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986199,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":323292,"kb_used_data":161100,"kb_used_omap":4,"kb_used_meta":162171,"kb_avail":94048548,"statfs":{"total":96636764160,"available":96305713152,"internally_reserved":0,"allocated":164966400,"data_stored":164783518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":166063826},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,1,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673005,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":548740,"kb_used_data":300980,"kb_used_omap":1,"kb_used_meta":247742,"kb_avail":93823100,"statfs":{"total":96636764160,"available":96074854400,"internally_reserved":0,"allocated":308203520,"data_stored":308020638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":253688261},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,5,0,0,1,7,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182436,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":548604,"kb_used_data":300392,"kb_used_omap":4,"kb_used_meta":248187,"kb_avail":93823236,"statfs":{"total":96636764160,"available":96074993664,"internally_reserved":0,"allocated":307601408,"data_stored":307418526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":254144205},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,6,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":308674560,"data_stored":308674560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":165482496,"data_stored":165482496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":308482048,"data_stored":308482048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":307458048,"data_stored":307458048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:36:50.516 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 148 20 76797 76777 4.0521 2.32812 0.026218 0.0385471 2024-09-05T16:36:51.516 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 149 20 77127 77107 4.04221 2.57812 0.0607081 0.0386412 2024-09-05T16:36:52.516 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 150 20 77453 77433 4.03224 2.54688 0.0568494 0.038738 2024-09-05T16:36:53.020 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:36:53.516 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 151 20 77747 77727 4.02075 2.29688 0.0792054 0.038843 2024-09-05T16:36:53.566 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:36:53.566 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:36:53.589 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":132,"stamp":"2024-09-05T16:36:52.840427+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":628695232,"num_objects":76750,"num_object_clones":0,"num_object_copies":307000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34520,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76750,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":76777,"num_write_kb":614098,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1705,"num_bytes_recovered":13729792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76804,"ondisk_log_size":76804,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2006440,"kb_used_data":1087528,"kb_used_omap":14,"kb_used_meta":918833,"kb_avail":375480920,"statfs":{"total":386547056640,"available":384492462080,"internally_reserved":0,"allocated":1113628672,"data_stored":1112897144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":940885559},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,9,6,18,22,13,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":25133056,"num_objects":3068,"num_object_clones":0,"num_object_copies":12272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-155,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3068,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3068,"num_write_kb":24544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":155,"num_bytes_recovered":1269760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005130"},"pg_stats":[{"pgid":"1.f","version":"33'4822","reported_seq":4864,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:49.858506+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:36:49.858506+0000","last_peered":"2024-09-05T16:36:49.858506+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:36:49.858506+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4822,"log_dups_size":0,"ondisk_log_size":4822,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39501824,"num_objects":4822,"num_object_clones":0,"num_object_copies":19288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4822,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4822,"num_write_kb":38576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2560}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'4747","reported_seq":4811,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:50.791614+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:36:50.791614+0000","last_peered":"2024-09-05T16:36:50.791614+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:36:50.791614+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4747,"log_dups_size":0,"ondisk_log_size":4747,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38851744,"num_objects":4743,"num_object_clones":0,"num_object_copies":18972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4743,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4745,"num_write_kb":37956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2476}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'4782","reported_seq":4857,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:50.384098+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:36:50.384098+0000","last_peered":"2024-09-05T16:36:50.384098+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:36:50.384098+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4782,"log_dups_size":0,"ondisk_log_size":4782,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39148112,"num_objects":4779,"num_object_clones":0,"num_object_copies":19116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4779,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4781,"num_write_kb":38238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2529}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'4864","reported_seq":8285,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:50.807708+0000","last_change":"2024-09-05T16:34:56.800371+0000","last_active":"2024-09-05T16:36:50.807708+0000","last_peered":"2024-09-05T16:36:50.807708+0000","last_clean":"2024-09-05T16:33:53.148717+0000","last_became_active":"2024-09-05T16:34:56.791755+0000","last_became_peered":"2024-09-05T16:34:56.791755+0000","last_unstale":"2024-09-05T16:36:50.807708+0000","last_undegraded":"2024-09-05T16:34:56.597522+0000","last_fullsized":"2024-09-05T16:34:56.568150+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4864,"log_dups_size":0,"ondisk_log_size":4864,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39810208,"num_objects":4860,"num_object_clones":0,"num_object_copies":19440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":612,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4860,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4862,"num_write_kb":38885,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1680,"num_bytes_recovered":13729792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4248}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'4685","reported_seq":4739,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:49.820456+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:36:49.820456+0000","last_peered":"2024-09-05T16:36:49.820456+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:36:49.820456+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4685,"log_dups_size":0,"ondisk_log_size":4685,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38352032,"num_objects":4682,"num_object_clones":0,"num_object_copies":18728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4682,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4683,"num_write_kb":37461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2520}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'4819","reported_seq":4861,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:49.680826+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:36:49.680826+0000","last_peered":"2024-09-05T16:36:49.680826+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:36:49.680826+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4819,"log_dups_size":0,"ondisk_log_size":4819,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39477248,"num_objects":4819,"num_object_clones":0,"num_object_copies":19276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4819,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4819,"num_write_kb":38552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2569}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'4669","reported_seq":4734,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:50.750394+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:36:50.750394+0000","last_peered":"2024-09-05T16:36:50.750394+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:36:50.750394+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4669,"log_dups_size":0,"ondisk_log_size":4669,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38196384,"num_objects":4663,"num_object_clones":0,"num_object_copies":18652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4667,"num_write_kb":37316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2516}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'4781","reported_seq":4845,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:50.736400+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:36:50.736400+0000","last_peered":"2024-09-05T16:36:50.736400+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:36:50.736400+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4781,"log_dups_size":0,"ondisk_log_size":4781,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39122080,"num_objects":4776,"num_object_clones":0,"num_object_copies":19104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4776,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4779,"num_write_kb":38220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2537}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'4809","reported_seq":4865,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:49.687230+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:36:49.687230+0000","last_peered":"2024-09-05T16:36:49.687230+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:36:49.687230+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4809,"log_dups_size":0,"ondisk_log_size":4809,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39323968,"num_objects":4801,"num_object_clones":0,"num_object_copies":19204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4801,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4806,"num_write_kb":38425,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2540}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'4810","reported_seq":4887,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:50.543780+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:36:50.543780+0000","last_peered":"2024-09-05T16:36:50.543780+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:36:50.543780+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4810,"log_dups_size":0,"ondisk_log_size":4810,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39384224,"num_objects":4808,"num_object_clones":0,"num_object_copies":19232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4808,"num_write_kb":38462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2546}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'4885","reported_seq":4941,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:49.698704+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:36:49.698704+0000","last_peered":"2024-09-05T16:36:49.698704+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:36:49.698704+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4885,"log_dups_size":0,"ondisk_log_size":4885,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39972592,"num_objects":4880,"num_object_clones":0,"num_object_copies":19520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4880,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4882,"num_write_kb":39044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2569}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'4728","reported_seq":4764,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:49.815245+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:36:49.815245+0000","last_peered":"2024-09-05T16:36:49.815245+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:36:49.815245+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4728,"log_dups_size":0,"ondisk_log_size":4728,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38722128,"num_objects":4727,"num_object_clones":0,"num_object_copies":18908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4727,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4727,"num_write_kb":37815,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2463}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'4837","reported_seq":4911,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:50.530811+0000","last_change":"2024-09-05T16:34:56.798081+0000","last_active":"2024-09-05T16:36:50.530811+0000","last_peered":"2024-09-05T16:36:50.530811+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:36:50.530811+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4837,"log_dups_size":0,"ondisk_log_size":4837,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39615056,"num_objects":4836,"num_object_clones":0,"num_object_copies":19344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4836,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4836,"num_write_kb":38687,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2542}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'4915","reported_seq":4967,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:49.820780+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:36:49.820780+0000","last_peered":"2024-09-05T16:36:49.820780+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:36:49.820780+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4915,"log_dups_size":0,"ondisk_log_size":4915,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40228000,"num_objects":4911,"num_object_clones":0,"num_object_copies":19644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4911,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4913,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2554}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'4810","reported_seq":4888,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:50.591760+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:36:50.591760+0000","last_peered":"2024-09-05T16:36:50.591760+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:36:50.591760+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4810,"log_dups_size":0,"ondisk_log_size":4810,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39376032,"num_objects":4807,"num_object_clones":0,"num_object_copies":19228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4807,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4808,"num_write_kb":38461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2487}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'4841","reported_seq":4905,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:50.737874+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:36:50.737874+0000","last_peered":"2024-09-05T16:36:50.737874+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:36:50.737874+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4841,"log_dups_size":0,"ondisk_log_size":4841,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39613600,"num_objects":4836,"num_object_clones":0,"num_object_copies":19344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4836,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4839,"num_write_kb":38700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2599}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":628695232,"num_objects":76750,"num_object_clones":0,"num_object_copies":307000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34520,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76750,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":76777,"num_write_kb":614098,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1705,"num_bytes_recovered":13729792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1115529216,"data_stored":1115529216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76804,"ondisk_log_size":76804,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084323,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":554312,"kb_used_data":306040,"kb_used_omap":4,"kb_used_meta":248251,"kb_avail":93817528,"statfs":{"total":96636764160,"available":96069148672,"internally_reserved":0,"allocated":313384960,"data_stored":313202078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":254209747},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,1,7,7,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986200,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":334008,"kb_used_data":167656,"kb_used_omap":4,"kb_used_meta":166331,"kb_avail":94037832,"statfs":{"total":96636764160,"available":96294739968,"internally_reserved":0,"allocated":171679744,"data_stored":171496862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":170323666},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,6,2,3,10,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673006,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":559080,"kb_used_data":307160,"kb_used_omap":1,"kb_used_meta":251902,"kb_avail":93812760,"statfs":{"total":96636764160,"available":96064266240,"internally_reserved":0,"allocated":314531840,"data_stored":314348958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":257948101},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,6,4,6,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182437,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":559040,"kb_used_data":306672,"kb_used_omap":4,"kb_used_meta":252347,"kb_avail":93812800,"statfs":{"total":96636764160,"available":96064307200,"internally_reserved":0,"allocated":314032128,"data_stored":313849246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":258404045},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,2,1,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":314929152,"data_stored":314929152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":172158976,"data_stored":172158976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":314712064,"data_stored":314712064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":313729024,"data_stored":313729024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:36:53.590 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:36:54.207 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:36:54.208 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:36:54.225 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":132,"stamp":"2024-09-05T16:36:52.840427+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":628695232,"num_objects":76750,"num_object_clones":0,"num_object_copies":307000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34520,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76750,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":76777,"num_write_kb":614098,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1705,"num_bytes_recovered":13729792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76804,"ondisk_log_size":76804,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2006440,"kb_used_data":1087528,"kb_used_omap":14,"kb_used_meta":918833,"kb_avail":375480920,"statfs":{"total":386547056640,"available":384492462080,"internally_reserved":0,"allocated":1113628672,"data_stored":1112897144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":940885559},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,9,6,18,22,13,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":25133056,"num_objects":3068,"num_object_clones":0,"num_object_copies":12272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-155,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3068,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3068,"num_write_kb":24544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":155,"num_bytes_recovered":1269760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005130"},"pg_stats":[{"pgid":"1.f","version":"33'4822","reported_seq":4864,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:49.858506+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:36:49.858506+0000","last_peered":"2024-09-05T16:36:49.858506+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:36:49.858506+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4822,"log_dups_size":0,"ondisk_log_size":4822,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39501824,"num_objects":4822,"num_object_clones":0,"num_object_copies":19288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4822,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4822,"num_write_kb":38576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2560}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'4747","reported_seq":4811,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:50.791614+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:36:50.791614+0000","last_peered":"2024-09-05T16:36:50.791614+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:36:50.791614+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4747,"log_dups_size":0,"ondisk_log_size":4747,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38851744,"num_objects":4743,"num_object_clones":0,"num_object_copies":18972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4743,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4745,"num_write_kb":37956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2476}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'4782","reported_seq":4857,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:50.384098+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:36:50.384098+0000","last_peered":"2024-09-05T16:36:50.384098+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:36:50.384098+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4782,"log_dups_size":0,"ondisk_log_size":4782,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39148112,"num_objects":4779,"num_object_clones":0,"num_object_copies":19116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4779,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4781,"num_write_kb":38238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2529}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'4864","reported_seq":8285,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:50.807708+0000","last_change":"2024-09-05T16:34:56.800371+0000","last_active":"2024-09-05T16:36:50.807708+0000","last_peered":"2024-09-05T16:36:50.807708+0000","last_clean":"2024-09-05T16:33:53.148717+0000","last_became_active":"2024-09-05T16:34:56.791755+0000","last_became_peered":"2024-09-05T16:34:56.791755+0000","last_unstale":"2024-09-05T16:36:50.807708+0000","last_undegraded":"2024-09-05T16:34:56.597522+0000","last_fullsized":"2024-09-05T16:34:56.568150+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4864,"log_dups_size":0,"ondisk_log_size":4864,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39810208,"num_objects":4860,"num_object_clones":0,"num_object_copies":19440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":612,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4860,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4862,"num_write_kb":38885,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1680,"num_bytes_recovered":13729792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4248}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'4685","reported_seq":4739,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:49.820456+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:36:49.820456+0000","last_peered":"2024-09-05T16:36:49.820456+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:36:49.820456+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4685,"log_dups_size":0,"ondisk_log_size":4685,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38352032,"num_objects":4682,"num_object_clones":0,"num_object_copies":18728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4682,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4683,"num_write_kb":37461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2520}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'4819","reported_seq":4861,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:49.680826+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:36:49.680826+0000","last_peered":"2024-09-05T16:36:49.680826+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:36:49.680826+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4819,"log_dups_size":0,"ondisk_log_size":4819,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39477248,"num_objects":4819,"num_object_clones":0,"num_object_copies":19276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4819,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4819,"num_write_kb":38552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2569}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'4669","reported_seq":4734,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:50.750394+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:36:50.750394+0000","last_peered":"2024-09-05T16:36:50.750394+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:36:50.750394+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4669,"log_dups_size":0,"ondisk_log_size":4669,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38196384,"num_objects":4663,"num_object_clones":0,"num_object_copies":18652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4667,"num_write_kb":37316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2516}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'4781","reported_seq":4845,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:50.736400+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:36:50.736400+0000","last_peered":"2024-09-05T16:36:50.736400+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:36:50.736400+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4781,"log_dups_size":0,"ondisk_log_size":4781,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39122080,"num_objects":4776,"num_object_clones":0,"num_object_copies":19104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4776,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4779,"num_write_kb":38220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2537}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'4809","reported_seq":4865,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:49.687230+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:36:49.687230+0000","last_peered":"2024-09-05T16:36:49.687230+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:36:49.687230+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4809,"log_dups_size":0,"ondisk_log_size":4809,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39323968,"num_objects":4801,"num_object_clones":0,"num_object_copies":19204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4801,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4806,"num_write_kb":38425,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2540}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'4810","reported_seq":4887,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:50.543780+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:36:50.543780+0000","last_peered":"2024-09-05T16:36:50.543780+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:36:50.543780+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4810,"log_dups_size":0,"ondisk_log_size":4810,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39384224,"num_objects":4808,"num_object_clones":0,"num_object_copies":19232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4808,"num_write_kb":38462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2546}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'4885","reported_seq":4941,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:49.698704+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:36:49.698704+0000","last_peered":"2024-09-05T16:36:49.698704+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:36:49.698704+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4885,"log_dups_size":0,"ondisk_log_size":4885,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39972592,"num_objects":4880,"num_object_clones":0,"num_object_copies":19520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4880,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4882,"num_write_kb":39044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2569}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'4728","reported_seq":4764,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:49.815245+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:36:49.815245+0000","last_peered":"2024-09-05T16:36:49.815245+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:36:49.815245+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4728,"log_dups_size":0,"ondisk_log_size":4728,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38722128,"num_objects":4727,"num_object_clones":0,"num_object_copies":18908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4727,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4727,"num_write_kb":37815,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2463}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'4837","reported_seq":4911,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:50.530811+0000","last_change":"2024-09-05T16:34:56.798081+0000","last_active":"2024-09-05T16:36:50.530811+0000","last_peered":"2024-09-05T16:36:50.530811+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:36:50.530811+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4837,"log_dups_size":0,"ondisk_log_size":4837,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39615056,"num_objects":4836,"num_object_clones":0,"num_object_copies":19344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4836,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4836,"num_write_kb":38687,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2542}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'4915","reported_seq":4967,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:49.820780+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:36:49.820780+0000","last_peered":"2024-09-05T16:36:49.820780+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:36:49.820780+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4915,"log_dups_size":0,"ondisk_log_size":4915,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40228000,"num_objects":4911,"num_object_clones":0,"num_object_copies":19644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4911,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4913,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2554}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'4810","reported_seq":4888,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:50.591760+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:36:50.591760+0000","last_peered":"2024-09-05T16:36:50.591760+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:36:50.591760+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4810,"log_dups_size":0,"ondisk_log_size":4810,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39376032,"num_objects":4807,"num_object_clones":0,"num_object_copies":19228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4807,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4808,"num_write_kb":38461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2487}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'4841","reported_seq":4905,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:50.737874+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:36:50.737874+0000","last_peered":"2024-09-05T16:36:50.737874+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:36:50.737874+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4841,"log_dups_size":0,"ondisk_log_size":4841,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39613600,"num_objects":4836,"num_object_clones":0,"num_object_copies":19344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4836,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4839,"num_write_kb":38700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2599}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":628695232,"num_objects":76750,"num_object_clones":0,"num_object_copies":307000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34520,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76750,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":76777,"num_write_kb":614098,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1705,"num_bytes_recovered":13729792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1115529216,"data_stored":1115529216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76804,"ondisk_log_size":76804,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084323,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":554312,"kb_used_data":306040,"kb_used_omap":4,"kb_used_meta":248251,"kb_avail":93817528,"statfs":{"total":96636764160,"available":96069148672,"internally_reserved":0,"allocated":313384960,"data_stored":313202078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":254209747},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,1,7,7,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986200,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":334008,"kb_used_data":167656,"kb_used_omap":4,"kb_used_meta":166331,"kb_avail":94037832,"statfs":{"total":96636764160,"available":96294739968,"internally_reserved":0,"allocated":171679744,"data_stored":171496862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":170323666},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,6,2,3,10,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673006,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":559080,"kb_used_data":307160,"kb_used_omap":1,"kb_used_meta":251902,"kb_avail":93812760,"statfs":{"total":96636764160,"available":96064266240,"internally_reserved":0,"allocated":314531840,"data_stored":314348958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":257948101},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,6,4,6,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182437,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":559040,"kb_used_data":306672,"kb_used_omap":4,"kb_used_meta":252347,"kb_avail":93812800,"statfs":{"total":96636764160,"available":96064307200,"internally_reserved":0,"allocated":314032128,"data_stored":313849246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":258404045},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,2,1,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":314929152,"data_stored":314929152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":172158976,"data_stored":172158976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":314712064,"data_stored":314712064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":313729024,"data_stored":313729024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:36:54.226 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-05T16:36:54.226 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'4822", 'reported_seq': 4864, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:49.858506+0000', 'last_change': '2024-09-05T16:34:56.833497+0000', 'last_active': '2024-09-05T16:36:49.858506+0000', 'last_peered': '2024-09-05T16:36:49.858506+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.828289+0000', 'last_became_peered': '2024-09-05T16:34:56.828289+0000', 'last_unstale': '2024-09-05T16:36:49.858506+0000', 'last_undegraded': '2024-09-05T16:34:56.612160+0000', 'last_fullsized': '2024-09-05T16:34:56.557632+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4822, 'log_dups_size': 0, 'ondisk_log_size': 4822, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39501824, 'num_objects': 4822, 'num_object_clones': 0, 'num_object_copies': 19288, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2262, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4822, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4822, 'num_write_kb': 38576, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2560}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:36:54.226 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-05T16:36:54.226 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'4747", 'reported_seq': 4811, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:50.791614+0000', 'last_change': '2024-09-05T16:34:56.825031+0000', 'last_active': '2024-09-05T16:36:50.791614+0000', 'last_peered': '2024-09-05T16:36:50.791614+0000', 'last_clean': '2024-09-05T16:33:53.200762+0000', 'last_became_active': '2024-09-05T16:34:56.820292+0000', 'last_became_peered': '2024-09-05T16:34:56.820292+0000', 'last_unstale': '2024-09-05T16:36:50.791614+0000', 'last_undegraded': '2024-09-05T16:34:56.605776+0000', 'last_fullsized': '2024-09-05T16:34:56.559383+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4747, 'log_dups_size': 0, 'ondisk_log_size': 4747, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38851744, 'num_objects': 4743, 'num_object_clones': 0, 'num_object_copies': 18972, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2269, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4743, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4745, 'num_write_kb': 37956, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2476}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:36:54.226 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:36:54.226 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'4782", 'reported_seq': 4857, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:50.384098+0000', 'last_change': '2024-09-05T16:34:56.867900+0000', 'last_active': '2024-09-05T16:36:50.384098+0000', 'last_peered': '2024-09-05T16:36:50.384098+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:36:50.384098+0000', 'last_undegraded': '2024-09-05T16:34:56.590972+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4782, 'log_dups_size': 0, 'ondisk_log_size': 4782, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39148112, 'num_objects': 4779, 'num_object_clones': 0, 'num_object_copies': 19116, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2251, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4779, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4781, 'num_write_kb': 38238, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2529}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:36:54.227 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-05T16:36:54.227 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'4864", 'reported_seq': 8285, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:50.807708+0000', 'last_change': '2024-09-05T16:34:56.800371+0000', 'last_active': '2024-09-05T16:36:50.807708+0000', 'last_peered': '2024-09-05T16:36:50.807708+0000', 'last_clean': '2024-09-05T16:33:53.148717+0000', 'last_became_active': '2024-09-05T16:34:56.791755+0000', 'last_became_peered': '2024-09-05T16:34:56.791755+0000', 'last_unstale': '2024-09-05T16:36:50.807708+0000', 'last_undegraded': '2024-09-05T16:34:56.597522+0000', 'last_fullsized': '2024-09-05T16:34:56.568150+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4864, 'log_dups_size': 0, 'ondisk_log_size': 4864, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39810208, 'num_objects': 4860, 'num_object_clones': 0, 'num_object_copies': 19440, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 612, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4860, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4862, 'num_write_kb': 38885, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1680, 'num_bytes_recovered': 13729792, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 4248}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:36:54.227 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:36:54.227 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'4685", 'reported_seq': 4739, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:49.820456+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:36:49.820456+0000', 'last_peered': '2024-09-05T16:36:49.820456+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:36:49.820456+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4685, 'log_dups_size': 0, 'ondisk_log_size': 4685, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38352032, 'num_objects': 4682, 'num_object_clones': 0, 'num_object_copies': 18728, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2164, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4682, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4683, 'num_write_kb': 37461, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2520}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:36:54.227 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:36:54.227 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'4819", 'reported_seq': 4861, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:49.680826+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:36:49.680826+0000', 'last_peered': '2024-09-05T16:36:49.680826+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:36:49.680826+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4819, 'log_dups_size': 0, 'ondisk_log_size': 4819, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39477248, 'num_objects': 4819, 'num_object_clones': 0, 'num_object_copies': 19276, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4819, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4819, 'num_write_kb': 38552, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2569}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:36:54.227 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:36:54.227 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'4669", 'reported_seq': 4734, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:50.750394+0000', 'last_change': '2024-09-05T16:34:56.903436+0000', 'last_active': '2024-09-05T16:36:50.750394+0000', 'last_peered': '2024-09-05T16:36:50.750394+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:36:50.750394+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4669, 'log_dups_size': 0, 'ondisk_log_size': 4669, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38196384, 'num_objects': 4663, 'num_object_clones': 0, 'num_object_copies': 18652, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2149, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4663, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4667, 'num_write_kb': 37316, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2516}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:36:54.227 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-05T16:36:54.227 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'4781", 'reported_seq': 4845, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:50.736400+0000', 'last_change': '2024-09-05T16:34:56.883319+0000', 'last_active': '2024-09-05T16:36:50.736400+0000', 'last_peered': '2024-09-05T16:36:50.736400+0000', 'last_clean': '2024-09-05T16:33:53.505305+0000', 'last_became_active': '2024-09-05T16:34:56.873389+0000', 'last_became_peered': '2024-09-05T16:34:56.873389+0000', 'last_unstale': '2024-09-05T16:36:50.736400+0000', 'last_undegraded': '2024-09-05T16:34:56.602567+0000', 'last_fullsized': '2024-09-05T16:34:56.559958+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4781, 'log_dups_size': 0, 'ondisk_log_size': 4781, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39122080, 'num_objects': 4776, 'num_object_clones': 0, 'num_object_copies': 19104, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2241, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4776, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4779, 'num_write_kb': 38220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2537}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:36:54.227 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:36:54.228 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'4809", 'reported_seq': 4865, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:49.687230+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:36:49.687230+0000', 'last_peered': '2024-09-05T16:36:49.687230+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:36:49.687230+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4809, 'log_dups_size': 0, 'ondisk_log_size': 4809, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39323968, 'num_objects': 4801, 'num_object_clones': 0, 'num_object_copies': 19204, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2264, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4801, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4806, 'num_write_kb': 38425, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2540}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:36:54.228 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:36:54.228 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'4810", 'reported_seq': 4887, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:50.543780+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:36:50.543780+0000', 'last_peered': '2024-09-05T16:36:50.543780+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:36:50.543780+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4810, 'log_dups_size': 0, 'ondisk_log_size': 4810, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39384224, 'num_objects': 4808, 'num_object_clones': 0, 'num_object_copies': 19232, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2264, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4808, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4808, 'num_write_kb': 38462, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2546}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:36:54.228 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:36:54.228 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'4885", 'reported_seq': 4941, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:49.698704+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:36:49.698704+0000', 'last_peered': '2024-09-05T16:36:49.698704+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:36:49.698704+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4885, 'log_dups_size': 0, 'ondisk_log_size': 4885, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39972592, 'num_objects': 4880, 'num_object_clones': 0, 'num_object_copies': 19520, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2314, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4880, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4882, 'num_write_kb': 39044, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2569}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:36:54.228 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:36:54.228 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'4728", 'reported_seq': 4764, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:49.815245+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:36:49.815245+0000', 'last_peered': '2024-09-05T16:36:49.815245+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:36:49.815245+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4728, 'log_dups_size': 0, 'ondisk_log_size': 4728, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38722128, 'num_objects': 4727, 'num_object_clones': 0, 'num_object_copies': 18908, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2265, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4727, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4727, 'num_write_kb': 37815, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2463}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:36:54.228 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-05T16:36:54.228 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'4837", 'reported_seq': 4911, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:50.530811+0000', 'last_change': '2024-09-05T16:34:56.798081+0000', 'last_active': '2024-09-05T16:36:50.530811+0000', 'last_peered': '2024-09-05T16:36:50.530811+0000', 'last_clean': '2024-09-05T16:33:45.082039+0000', 'last_became_active': '2024-09-05T16:34:56.792386+0000', 'last_became_peered': '2024-09-05T16:34:56.792386+0000', 'last_unstale': '2024-09-05T16:36:50.530811+0000', 'last_undegraded': '2024-09-05T16:34:56.600595+0000', 'last_fullsized': '2024-09-05T16:34:56.558203+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4837, 'log_dups_size': 0, 'ondisk_log_size': 4837, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39615056, 'num_objects': 4836, 'num_object_clones': 0, 'num_object_copies': 19344, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2295, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4836, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4836, 'num_write_kb': 38687, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2542}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:36:54.228 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:36:54.228 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'4915", 'reported_seq': 4967, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:49.820780+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:36:49.820780+0000', 'last_peered': '2024-09-05T16:36:49.820780+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:36:49.820780+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4915, 'log_dups_size': 0, 'ondisk_log_size': 4915, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40228000, 'num_objects': 4911, 'num_object_clones': 0, 'num_object_copies': 19644, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2359, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4911, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4913, 'num_write_kb': 39300, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2554}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:36:54.229 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:36:54.229 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'4810", 'reported_seq': 4888, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:50.591760+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:36:50.591760+0000', 'last_peered': '2024-09-05T16:36:50.591760+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:36:50.591760+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4810, 'log_dups_size': 0, 'ondisk_log_size': 4810, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39376032, 'num_objects': 4807, 'num_object_clones': 0, 'num_object_copies': 19228, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2322, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4807, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4808, 'num_write_kb': 38461, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2487}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:36:54.229 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:36:54.229 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'4841", 'reported_seq': 4905, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:50.737874+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:36:50.737874+0000', 'last_peered': '2024-09-05T16:36:50.737874+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:36:50.737874+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4841, 'log_dups_size': 0, 'ondisk_log_size': 4841, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39613600, 'num_objects': 4836, 'num_object_clones': 0, 'num_object_copies': 19344, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4836, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4839, 'num_write_kb': 38700, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2599}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:36:54.229 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:36:54.516 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 152 20 78041 78021 4.0094 2.29688 0.0437054 0.0389578 2024-09-05T16:36:54.923 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:36:54.923 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 34520/307000 objects degraded (11.244%), 16 pgs degraded, 16 pgs undersized","count":32},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":218,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":76750,"data_bytes":628695232,"bytes_used":2054594560,"bytes_avail":384492462080,"bytes_total":386547056640,"degraded_objects":34520,"degraded_total":307000,"degraded_ratio":0.11244299674267101,"recovering_objects_per_sec":12,"recovering_bytes_per_sec":105768,"recovering_keys_per_sec":0,"num_objects_recovered":155,"num_bytes_recovered":1269760,"num_keys_recovered":0,"write_bytes_sec":2093526,"read_op_per_sec":0,"write_op_per_sec":255},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:36:54.948 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:36:54.949 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:36:55.516 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 153 20 78326 78306 3.99775 2.22656 0.0590563 0.0390691 2024-09-05T16:36:55.529 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:36:55.530 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:36:55.555 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":133,"stamp":"2024-09-05T16:36:54.841384+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":628695232,"num_objects":76750,"num_object_clones":0,"num_object_copies":307000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34520,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76750,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":76777,"num_write_kb":614098,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1705,"num_bytes_recovered":13729792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76804,"ondisk_log_size":76804,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2018388,"kb_used_data":1093908,"kb_used_omap":14,"kb_used_meta":924401,"kb_avail":375468972,"statfs":{"total":386547056640,"available":384480227328,"internally_reserved":0,"allocated":1120161792,"data_stored":1119430264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":946587191},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,7,5,4,16,14,10,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":25133056,"num_objects":3068,"num_object_clones":0,"num_object_copies":12272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-155,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3068,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3068,"num_write_kb":24544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":155,"num_bytes_recovered":1269760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005091"},"pg_stats":[{"pgid":"1.f","version":"33'4822","reported_seq":4864,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:49.858506+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:36:49.858506+0000","last_peered":"2024-09-05T16:36:49.858506+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:36:49.858506+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4822,"log_dups_size":0,"ondisk_log_size":4822,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39501824,"num_objects":4822,"num_object_clones":0,"num_object_copies":19288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4822,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4822,"num_write_kb":38576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2560}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'4747","reported_seq":4811,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:50.791614+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:36:50.791614+0000","last_peered":"2024-09-05T16:36:50.791614+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:36:50.791614+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4747,"log_dups_size":0,"ondisk_log_size":4747,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38851744,"num_objects":4743,"num_object_clones":0,"num_object_copies":18972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4743,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4745,"num_write_kb":37956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2476}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'4782","reported_seq":4857,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:50.384098+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:36:50.384098+0000","last_peered":"2024-09-05T16:36:50.384098+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:36:50.384098+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4782,"log_dups_size":0,"ondisk_log_size":4782,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39148112,"num_objects":4779,"num_object_clones":0,"num_object_copies":19116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4779,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4781,"num_write_kb":38238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2529}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'4864","reported_seq":8285,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:50.807708+0000","last_change":"2024-09-05T16:34:56.800371+0000","last_active":"2024-09-05T16:36:50.807708+0000","last_peered":"2024-09-05T16:36:50.807708+0000","last_clean":"2024-09-05T16:33:53.148717+0000","last_became_active":"2024-09-05T16:34:56.791755+0000","last_became_peered":"2024-09-05T16:34:56.791755+0000","last_unstale":"2024-09-05T16:36:50.807708+0000","last_undegraded":"2024-09-05T16:34:56.597522+0000","last_fullsized":"2024-09-05T16:34:56.568150+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4864,"log_dups_size":0,"ondisk_log_size":4864,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39810208,"num_objects":4860,"num_object_clones":0,"num_object_copies":19440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":612,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4860,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4862,"num_write_kb":38885,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1680,"num_bytes_recovered":13729792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4248}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'4685","reported_seq":4739,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:49.820456+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:36:49.820456+0000","last_peered":"2024-09-05T16:36:49.820456+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:36:49.820456+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4685,"log_dups_size":0,"ondisk_log_size":4685,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38352032,"num_objects":4682,"num_object_clones":0,"num_object_copies":18728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4682,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4683,"num_write_kb":37461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2520}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'4819","reported_seq":4861,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:49.680826+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:36:49.680826+0000","last_peered":"2024-09-05T16:36:49.680826+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:36:49.680826+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4819,"log_dups_size":0,"ondisk_log_size":4819,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39477248,"num_objects":4819,"num_object_clones":0,"num_object_copies":19276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4819,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4819,"num_write_kb":38552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2569}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'4669","reported_seq":4734,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:50.750394+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:36:50.750394+0000","last_peered":"2024-09-05T16:36:50.750394+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:36:50.750394+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4669,"log_dups_size":0,"ondisk_log_size":4669,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38196384,"num_objects":4663,"num_object_clones":0,"num_object_copies":18652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4667,"num_write_kb":37316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2516}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'4781","reported_seq":4845,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:50.736400+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:36:50.736400+0000","last_peered":"2024-09-05T16:36:50.736400+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:36:50.736400+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4781,"log_dups_size":0,"ondisk_log_size":4781,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39122080,"num_objects":4776,"num_object_clones":0,"num_object_copies":19104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4776,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4779,"num_write_kb":38220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2537}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'4809","reported_seq":4865,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:49.687230+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:36:49.687230+0000","last_peered":"2024-09-05T16:36:49.687230+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:36:49.687230+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4809,"log_dups_size":0,"ondisk_log_size":4809,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39323968,"num_objects":4801,"num_object_clones":0,"num_object_copies":19204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4801,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4806,"num_write_kb":38425,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2540}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'4810","reported_seq":4887,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:50.543780+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:36:50.543780+0000","last_peered":"2024-09-05T16:36:50.543780+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:36:50.543780+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4810,"log_dups_size":0,"ondisk_log_size":4810,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39384224,"num_objects":4808,"num_object_clones":0,"num_object_copies":19232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4808,"num_write_kb":38462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2546}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'4885","reported_seq":4941,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:49.698704+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:36:49.698704+0000","last_peered":"2024-09-05T16:36:49.698704+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:36:49.698704+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4885,"log_dups_size":0,"ondisk_log_size":4885,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39972592,"num_objects":4880,"num_object_clones":0,"num_object_copies":19520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4880,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4882,"num_write_kb":39044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2569}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'4728","reported_seq":4764,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:49.815245+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:36:49.815245+0000","last_peered":"2024-09-05T16:36:49.815245+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:36:49.815245+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4728,"log_dups_size":0,"ondisk_log_size":4728,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38722128,"num_objects":4727,"num_object_clones":0,"num_object_copies":18908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4727,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4727,"num_write_kb":37815,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2463}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'4837","reported_seq":4911,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:50.530811+0000","last_change":"2024-09-05T16:34:56.798081+0000","last_active":"2024-09-05T16:36:50.530811+0000","last_peered":"2024-09-05T16:36:50.530811+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:36:50.530811+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4837,"log_dups_size":0,"ondisk_log_size":4837,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39615056,"num_objects":4836,"num_object_clones":0,"num_object_copies":19344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4836,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4836,"num_write_kb":38687,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2542}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'4915","reported_seq":4967,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:49.820780+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:36:49.820780+0000","last_peered":"2024-09-05T16:36:49.820780+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:36:49.820780+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4915,"log_dups_size":0,"ondisk_log_size":4915,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40228000,"num_objects":4911,"num_object_clones":0,"num_object_copies":19644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4911,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4913,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2554}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'4810","reported_seq":4888,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:50.591760+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:36:50.591760+0000","last_peered":"2024-09-05T16:36:50.591760+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:36:50.591760+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4810,"log_dups_size":0,"ondisk_log_size":4810,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39376032,"num_objects":4807,"num_object_clones":0,"num_object_copies":19228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4807,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4808,"num_write_kb":38461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2487}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'4841","reported_seq":4905,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:50.737874+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:36:50.737874+0000","last_peered":"2024-09-05T16:36:50.737874+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:36:50.737874+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4841,"log_dups_size":0,"ondisk_log_size":4841,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39613600,"num_objects":4836,"num_object_clones":0,"num_object_copies":19344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4836,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4839,"num_write_kb":38700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2599}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":628695232,"num_objects":76750,"num_object_clones":0,"num_object_copies":307000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34520,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76750,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":76777,"num_write_kb":614098,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1705,"num_bytes_recovered":13729792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1122242560,"data_stored":1122242560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76804,"ondisk_log_size":76804,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084323,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":554312,"kb_used_data":306040,"kb_used_omap":4,"kb_used_meta":248251,"kb_avail":93817528,"statfs":{"total":96636764160,"available":96069148672,"internally_reserved":0,"allocated":313384960,"data_stored":313202078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":254209747},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,1,7,7,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986201,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":345956,"kb_used_data":174036,"kb_used_omap":4,"kb_used_meta":171899,"kb_avail":94025884,"statfs":{"total":96636764160,"available":96282505216,"internally_reserved":0,"allocated":178212864,"data_stored":178029982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":176025298},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,6,2,0,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673006,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":559080,"kb_used_data":307160,"kb_used_omap":1,"kb_used_meta":251902,"kb_avail":93812760,"statfs":{"total":96636764160,"available":96064266240,"internally_reserved":0,"allocated":314531840,"data_stored":314348958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":257948101},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,6,4,6,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182437,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":559040,"kb_used_data":306672,"kb_used_omap":4,"kb_used_meta":252347,"kb_avail":93812800,"statfs":{"total":96636764160,"available":96064307200,"internally_reserved":0,"allocated":314032128,"data_stored":313849246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":258404045},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,2,1,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":314929152,"data_stored":314929152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":178872320,"data_stored":178872320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":314712064,"data_stored":314712064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":313729024,"data_stored":313729024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:36:56.516 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 154 20 78654 78634 3.98843 2.5625 0.0353508 0.0391637 2024-09-05T16:36:57.516 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 155 20 78967 78947 3.97848 2.44531 0.0613545 0.0392579 2024-09-05T16:36:58.517 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 156 20 79286 79266 3.96895 2.49219 0.0236228 0.0393553 2024-09-05T16:36:58.557 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:36:59.187 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:36:59.188 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:36:59.213 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":135,"stamp":"2024-09-05T16:36:58.843500+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":641482944,"num_objects":78311,"num_object_clones":0,"num_object_copies":313244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34448,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":78311,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":78338,"num_write_kb":626586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1777,"num_bytes_recovered":14319616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78365,"ondisk_log_size":78365,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2049364,"kb_used_data":1112404,"kb_used_omap":14,"kb_used_meta":936881,"kb_avail":375437996,"statfs":{"total":386547056640,"available":384448507904,"internally_reserved":0,"allocated":1139101696,"data_stored":1138370168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":959366711},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,6,4,12,17,16,4,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":25272320,"num_objects":3085,"num_object_clones":0,"num_object_copies":12340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-152,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3085,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3085,"num_write_kb":24680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":152,"num_bytes_recovered":1245184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005871"},"pg_stats":[{"pgid":"1.f","version":"33'4906","reported_seq":4948,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:54.808906+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:36:54.808906+0000","last_peered":"2024-09-05T16:36:54.808906+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:36:54.808906+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4906,"log_dups_size":0,"ondisk_log_size":4906,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40189952,"num_objects":4906,"num_object_clones":0,"num_object_copies":19624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4906,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4906,"num_write_kb":39248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2644}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'4843","reported_seq":4907,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:55.736924+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:36:55.736924+0000","last_peered":"2024-09-05T16:36:55.736924+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:36:55.736924+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4843,"log_dups_size":0,"ondisk_log_size":4843,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39638176,"num_objects":4839,"num_object_clones":0,"num_object_copies":19356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4839,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4841,"num_write_kb":38724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2572}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'4882","reported_seq":4957,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:55.579215+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:36:55.579215+0000","last_peered":"2024-09-05T16:36:55.579215+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:36:55.579215+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4882,"log_dups_size":0,"ondisk_log_size":4882,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39967312,"num_objects":4879,"num_object_clones":0,"num_object_copies":19516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4879,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4881,"num_write_kb":39038,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2629}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'4972","reported_seq":8537,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:55.767715+0000","last_change":"2024-09-05T16:34:56.800371+0000","last_active":"2024-09-05T16:36:55.767715+0000","last_peered":"2024-09-05T16:36:55.767715+0000","last_clean":"2024-09-05T16:33:53.148717+0000","last_became_active":"2024-09-05T16:34:56.791755+0000","last_became_peered":"2024-09-05T16:34:56.791755+0000","last_unstale":"2024-09-05T16:36:55.767715+0000","last_undegraded":"2024-09-05T16:34:56.597522+0000","last_fullsized":"2024-09-05T16:34:56.568150+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4972,"log_dups_size":0,"ondisk_log_size":4972,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40694944,"num_objects":4968,"num_object_clones":0,"num_object_copies":19872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":540,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4968,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4970,"num_write_kb":39749,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1752,"num_bytes_recovered":14319616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4428}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'4791","reported_seq":4845,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:54.845432+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:36:54.845432+0000","last_peered":"2024-09-05T16:36:54.845432+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:36:54.845432+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4791,"log_dups_size":0,"ondisk_log_size":4791,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39220384,"num_objects":4788,"num_object_clones":0,"num_object_copies":19152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4789,"num_write_kb":38309,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2626}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'4904","reported_seq":4946,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:54.876380+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:36:54.876380+0000","last_peered":"2024-09-05T16:36:54.876380+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:36:54.876380+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4904,"log_dups_size":0,"ondisk_log_size":4904,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40173568,"num_objects":4904,"num_object_clones":0,"num_object_copies":19616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4904,"num_write_kb":39232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2654}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'4770","reported_seq":4835,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:55.657580+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:36:55.657580+0000","last_peered":"2024-09-05T16:36:55.657580+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:36:55.657580+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4770,"log_dups_size":0,"ondisk_log_size":4770,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39023776,"num_objects":4764,"num_object_clones":0,"num_object_copies":19056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4764,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4768,"num_write_kb":38124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2617}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'4882","reported_seq":4946,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:55.771404+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:36:55.771404+0000","last_peered":"2024-09-05T16:36:55.771404+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:36:55.771404+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4882,"log_dups_size":0,"ondisk_log_size":4882,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39949472,"num_objects":4877,"num_object_clones":0,"num_object_copies":19508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4877,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4880,"num_write_kb":39028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2638}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'4918","reported_seq":4974,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:54.881125+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:36:54.881125+0000","last_peered":"2024-09-05T16:36:54.881125+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:36:54.881125+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4918,"log_dups_size":0,"ondisk_log_size":4918,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40216896,"num_objects":4910,"num_object_clones":0,"num_object_copies":19640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4915,"num_write_kb":39297,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2649}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'4903","reported_seq":4980,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:55.656958+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:36:55.656958+0000","last_peered":"2024-09-05T16:36:55.656958+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:36:55.656958+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4903,"log_dups_size":0,"ondisk_log_size":4903,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40146080,"num_objects":4901,"num_object_clones":0,"num_object_copies":19604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4901,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4901,"num_write_kb":39206,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2639}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'4987","reported_seq":5043,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:54.907177+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:36:54.907177+0000","last_peered":"2024-09-05T16:36:54.907177+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:36:54.907177+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4987,"log_dups_size":0,"ondisk_log_size":4987,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40808176,"num_objects":4982,"num_object_clones":0,"num_object_copies":19928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4982,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4984,"num_write_kb":39860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2671}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'4811","reported_seq":4846,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:54.834149+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:36:54.834149+0000","last_peered":"2024-09-05T16:36:54.834149+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:36:54.834149+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4811,"log_dups_size":0,"ondisk_log_size":4811,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39402064,"num_objects":4810,"num_object_clones":0,"num_object_copies":19240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4810,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4810,"num_write_kb":38479,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2546}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'4934","reported_seq":5008,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:55.626943+0000","last_change":"2024-09-05T16:34:56.798081+0000","last_active":"2024-09-05T16:36:55.626943+0000","last_peered":"2024-09-05T16:36:55.626943+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:36:55.626943+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4934,"log_dups_size":0,"ondisk_log_size":4934,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40409680,"num_objects":4933,"num_object_clones":0,"num_object_copies":19732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4933,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4933,"num_write_kb":39463,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2639}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'5019","reported_seq":5071,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:54.862344+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:36:54.862344+0000","last_peered":"2024-09-05T16:36:54.862344+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:36:54.862344+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5019,"log_dups_size":0,"ondisk_log_size":5019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41079968,"num_objects":5015,"num_object_clones":0,"num_object_copies":20060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5015,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5017,"num_write_kb":40132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2658}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'4915","reported_seq":4993,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:55.537302+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:36:55.537302+0000","last_peered":"2024-09-05T16:36:55.537302+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:36:55.537302+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4915,"log_dups_size":0,"ondisk_log_size":4915,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40236192,"num_objects":4912,"num_object_clones":0,"num_object_copies":19648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4912,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4913,"num_write_kb":39301,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2592}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'4928","reported_seq":4992,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:55.663753+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:36:55.663753+0000","last_peered":"2024-09-05T16:36:55.663753+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:36:55.663753+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4928,"log_dups_size":0,"ondisk_log_size":4928,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40326304,"num_objects":4923,"num_object_clones":0,"num_object_copies":19692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4926,"num_write_kb":39396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2686}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":641482944,"num_objects":78311,"num_object_clones":0,"num_object_copies":313244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34448,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":78311,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":78338,"num_write_kb":626586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1777,"num_bytes_recovered":14319616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1141207040,"data_stored":1141207040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78365,"ondisk_log_size":78365,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084324,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":564692,"kb_used_data":312260,"kb_used_omap":4,"kb_used_meta":252411,"kb_avail":93807148,"statfs":{"total":96636764160,"available":96058519552,"internally_reserved":0,"allocated":319754240,"data_stored":319571358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":258469587},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,5,8,9,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986201,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":345956,"kb_used_data":174036,"kb_used_omap":4,"kb_used_meta":171899,"kb_avail":94025884,"statfs":{"total":96636764160,"available":96282505216,"internally_reserved":0,"allocated":178212864,"data_stored":178029982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":176025298},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,6,2,0,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673007,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":569340,"kb_used_data":313260,"kb_used_omap":1,"kb_used_meta":256062,"kb_avail":93802500,"statfs":{"total":96636764160,"available":96053760000,"internally_reserved":0,"allocated":320778240,"data_stored":320595358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":262207941},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,1,4,2,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182438,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":569376,"kb_used_data":312848,"kb_used_omap":4,"kb_used_meta":256507,"kb_avail":93802464,"statfs":{"total":96636764160,"available":96053723136,"internally_reserved":0,"allocated":320356352,"data_stored":320173470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":262663885},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,6,4,3,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":321220608,"data_stored":321220608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":178872320,"data_stored":178872320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":321024000,"data_stored":321024000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":320090112,"data_stored":320090112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:36:59.215 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:36:59.517 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 157 20 79557 79537 3.95715 2.11719 0.0649247 0.0394597 2024-09-05T16:36:59.801 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:36:59.802 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:36:59.821 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":135,"stamp":"2024-09-05T16:36:58.843500+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":641482944,"num_objects":78311,"num_object_clones":0,"num_object_copies":313244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34448,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":78311,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":78338,"num_write_kb":626586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1777,"num_bytes_recovered":14319616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78365,"ondisk_log_size":78365,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2049364,"kb_used_data":1112404,"kb_used_omap":14,"kb_used_meta":936881,"kb_avail":375437996,"statfs":{"total":386547056640,"available":384448507904,"internally_reserved":0,"allocated":1139101696,"data_stored":1138370168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":959366711},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,6,4,12,17,16,4,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":25272320,"num_objects":3085,"num_object_clones":0,"num_object_copies":12340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-152,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3085,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3085,"num_write_kb":24680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":152,"num_bytes_recovered":1245184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005871"},"pg_stats":[{"pgid":"1.f","version":"33'4906","reported_seq":4948,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:54.808906+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:36:54.808906+0000","last_peered":"2024-09-05T16:36:54.808906+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:36:54.808906+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4906,"log_dups_size":0,"ondisk_log_size":4906,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40189952,"num_objects":4906,"num_object_clones":0,"num_object_copies":19624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4906,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4906,"num_write_kb":39248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2644}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'4843","reported_seq":4907,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:55.736924+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:36:55.736924+0000","last_peered":"2024-09-05T16:36:55.736924+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:36:55.736924+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4843,"log_dups_size":0,"ondisk_log_size":4843,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39638176,"num_objects":4839,"num_object_clones":0,"num_object_copies":19356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4839,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4841,"num_write_kb":38724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2572}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'4882","reported_seq":4957,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:55.579215+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:36:55.579215+0000","last_peered":"2024-09-05T16:36:55.579215+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:36:55.579215+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4882,"log_dups_size":0,"ondisk_log_size":4882,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39967312,"num_objects":4879,"num_object_clones":0,"num_object_copies":19516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4879,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4881,"num_write_kb":39038,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2629}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'4972","reported_seq":8537,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:55.767715+0000","last_change":"2024-09-05T16:34:56.800371+0000","last_active":"2024-09-05T16:36:55.767715+0000","last_peered":"2024-09-05T16:36:55.767715+0000","last_clean":"2024-09-05T16:33:53.148717+0000","last_became_active":"2024-09-05T16:34:56.791755+0000","last_became_peered":"2024-09-05T16:34:56.791755+0000","last_unstale":"2024-09-05T16:36:55.767715+0000","last_undegraded":"2024-09-05T16:34:56.597522+0000","last_fullsized":"2024-09-05T16:34:56.568150+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4972,"log_dups_size":0,"ondisk_log_size":4972,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40694944,"num_objects":4968,"num_object_clones":0,"num_object_copies":19872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":540,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4968,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4970,"num_write_kb":39749,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1752,"num_bytes_recovered":14319616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4428}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'4791","reported_seq":4845,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:54.845432+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:36:54.845432+0000","last_peered":"2024-09-05T16:36:54.845432+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:36:54.845432+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4791,"log_dups_size":0,"ondisk_log_size":4791,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39220384,"num_objects":4788,"num_object_clones":0,"num_object_copies":19152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4789,"num_write_kb":38309,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2626}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'4904","reported_seq":4946,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:54.876380+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:36:54.876380+0000","last_peered":"2024-09-05T16:36:54.876380+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:36:54.876380+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4904,"log_dups_size":0,"ondisk_log_size":4904,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40173568,"num_objects":4904,"num_object_clones":0,"num_object_copies":19616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4904,"num_write_kb":39232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2654}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'4770","reported_seq":4835,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:55.657580+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:36:55.657580+0000","last_peered":"2024-09-05T16:36:55.657580+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:36:55.657580+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4770,"log_dups_size":0,"ondisk_log_size":4770,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39023776,"num_objects":4764,"num_object_clones":0,"num_object_copies":19056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4764,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4768,"num_write_kb":38124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2617}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'4882","reported_seq":4946,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:55.771404+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:36:55.771404+0000","last_peered":"2024-09-05T16:36:55.771404+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:36:55.771404+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4882,"log_dups_size":0,"ondisk_log_size":4882,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39949472,"num_objects":4877,"num_object_clones":0,"num_object_copies":19508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4877,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4880,"num_write_kb":39028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2638}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'4918","reported_seq":4974,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:54.881125+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:36:54.881125+0000","last_peered":"2024-09-05T16:36:54.881125+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:36:54.881125+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4918,"log_dups_size":0,"ondisk_log_size":4918,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40216896,"num_objects":4910,"num_object_clones":0,"num_object_copies":19640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4915,"num_write_kb":39297,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2649}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'4903","reported_seq":4980,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:55.656958+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:36:55.656958+0000","last_peered":"2024-09-05T16:36:55.656958+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:36:55.656958+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4903,"log_dups_size":0,"ondisk_log_size":4903,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40146080,"num_objects":4901,"num_object_clones":0,"num_object_copies":19604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4901,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4901,"num_write_kb":39206,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2639}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'4987","reported_seq":5043,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:54.907177+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:36:54.907177+0000","last_peered":"2024-09-05T16:36:54.907177+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:36:54.907177+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4987,"log_dups_size":0,"ondisk_log_size":4987,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40808176,"num_objects":4982,"num_object_clones":0,"num_object_copies":19928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4982,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4984,"num_write_kb":39860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2671}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'4811","reported_seq":4846,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:54.834149+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:36:54.834149+0000","last_peered":"2024-09-05T16:36:54.834149+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:36:54.834149+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4811,"log_dups_size":0,"ondisk_log_size":4811,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39402064,"num_objects":4810,"num_object_clones":0,"num_object_copies":19240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4810,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4810,"num_write_kb":38479,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2546}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'4934","reported_seq":5008,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:55.626943+0000","last_change":"2024-09-05T16:34:56.798081+0000","last_active":"2024-09-05T16:36:55.626943+0000","last_peered":"2024-09-05T16:36:55.626943+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:36:55.626943+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4934,"log_dups_size":0,"ondisk_log_size":4934,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40409680,"num_objects":4933,"num_object_clones":0,"num_object_copies":19732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4933,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4933,"num_write_kb":39463,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2639}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'5019","reported_seq":5071,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:54.862344+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:36:54.862344+0000","last_peered":"2024-09-05T16:36:54.862344+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:36:54.862344+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5019,"log_dups_size":0,"ondisk_log_size":5019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41079968,"num_objects":5015,"num_object_clones":0,"num_object_copies":20060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5015,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5017,"num_write_kb":40132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2658}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'4915","reported_seq":4993,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:55.537302+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:36:55.537302+0000","last_peered":"2024-09-05T16:36:55.537302+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:36:55.537302+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4915,"log_dups_size":0,"ondisk_log_size":4915,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40236192,"num_objects":4912,"num_object_clones":0,"num_object_copies":19648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4912,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4913,"num_write_kb":39301,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2592}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'4928","reported_seq":4992,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:55.663753+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:36:55.663753+0000","last_peered":"2024-09-05T16:36:55.663753+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:36:55.663753+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4928,"log_dups_size":0,"ondisk_log_size":4928,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40326304,"num_objects":4923,"num_object_clones":0,"num_object_copies":19692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4926,"num_write_kb":39396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2686}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":641482944,"num_objects":78311,"num_object_clones":0,"num_object_copies":313244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34448,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":78311,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":78338,"num_write_kb":626586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1777,"num_bytes_recovered":14319616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1141207040,"data_stored":1141207040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78365,"ondisk_log_size":78365,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084324,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":564692,"kb_used_data":312260,"kb_used_omap":4,"kb_used_meta":252411,"kb_avail":93807148,"statfs":{"total":96636764160,"available":96058519552,"internally_reserved":0,"allocated":319754240,"data_stored":319571358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":258469587},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,5,8,9,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986201,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":345956,"kb_used_data":174036,"kb_used_omap":4,"kb_used_meta":171899,"kb_avail":94025884,"statfs":{"total":96636764160,"available":96282505216,"internally_reserved":0,"allocated":178212864,"data_stored":178029982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":176025298},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,6,2,0,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673007,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":569340,"kb_used_data":313260,"kb_used_omap":1,"kb_used_meta":256062,"kb_avail":93802500,"statfs":{"total":96636764160,"available":96053760000,"internally_reserved":0,"allocated":320778240,"data_stored":320595358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":262207941},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,1,4,2,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182438,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":569376,"kb_used_data":312848,"kb_used_omap":4,"kb_used_meta":256507,"kb_avail":93802464,"statfs":{"total":96636764160,"available":96053723136,"internally_reserved":0,"allocated":320356352,"data_stored":320173470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":262663885},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,6,4,3,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":321220608,"data_stored":321220608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":178872320,"data_stored":178872320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":321024000,"data_stored":321024000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":320090112,"data_stored":320090112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:36:59.822 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-05T16:36:59.822 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'4906", 'reported_seq': 4948, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:54.808906+0000', 'last_change': '2024-09-05T16:34:56.833497+0000', 'last_active': '2024-09-05T16:36:54.808906+0000', 'last_peered': '2024-09-05T16:36:54.808906+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.828289+0000', 'last_became_peered': '2024-09-05T16:34:56.828289+0000', 'last_unstale': '2024-09-05T16:36:54.808906+0000', 'last_undegraded': '2024-09-05T16:34:56.612160+0000', 'last_fullsized': '2024-09-05T16:34:56.557632+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4906, 'log_dups_size': 0, 'ondisk_log_size': 4906, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40189952, 'num_objects': 4906, 'num_object_clones': 0, 'num_object_copies': 19624, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2262, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4906, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4906, 'num_write_kb': 39248, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2644}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:36:59.822 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-05T16:36:59.822 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'4843", 'reported_seq': 4907, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:55.736924+0000', 'last_change': '2024-09-05T16:34:56.825031+0000', 'last_active': '2024-09-05T16:36:55.736924+0000', 'last_peered': '2024-09-05T16:36:55.736924+0000', 'last_clean': '2024-09-05T16:33:53.200762+0000', 'last_became_active': '2024-09-05T16:34:56.820292+0000', 'last_became_peered': '2024-09-05T16:34:56.820292+0000', 'last_unstale': '2024-09-05T16:36:55.736924+0000', 'last_undegraded': '2024-09-05T16:34:56.605776+0000', 'last_fullsized': '2024-09-05T16:34:56.559383+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4843, 'log_dups_size': 0, 'ondisk_log_size': 4843, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39638176, 'num_objects': 4839, 'num_object_clones': 0, 'num_object_copies': 19356, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2269, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4839, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4841, 'num_write_kb': 38724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2572}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:36:59.822 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:36:59.823 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'4882", 'reported_seq': 4957, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:55.579215+0000', 'last_change': '2024-09-05T16:34:56.867900+0000', 'last_active': '2024-09-05T16:36:55.579215+0000', 'last_peered': '2024-09-05T16:36:55.579215+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:36:55.579215+0000', 'last_undegraded': '2024-09-05T16:34:56.590972+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4882, 'log_dups_size': 0, 'ondisk_log_size': 4882, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39967312, 'num_objects': 4879, 'num_object_clones': 0, 'num_object_copies': 19516, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2251, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4879, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4881, 'num_write_kb': 39038, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2629}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:36:59.823 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-05T16:36:59.823 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'4972", 'reported_seq': 8537, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:55.767715+0000', 'last_change': '2024-09-05T16:34:56.800371+0000', 'last_active': '2024-09-05T16:36:55.767715+0000', 'last_peered': '2024-09-05T16:36:55.767715+0000', 'last_clean': '2024-09-05T16:33:53.148717+0000', 'last_became_active': '2024-09-05T16:34:56.791755+0000', 'last_became_peered': '2024-09-05T16:34:56.791755+0000', 'last_unstale': '2024-09-05T16:36:55.767715+0000', 'last_undegraded': '2024-09-05T16:34:56.597522+0000', 'last_fullsized': '2024-09-05T16:34:56.568150+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4972, 'log_dups_size': 0, 'ondisk_log_size': 4972, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40694944, 'num_objects': 4968, 'num_object_clones': 0, 'num_object_copies': 19872, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 540, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4968, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4970, 'num_write_kb': 39749, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1752, 'num_bytes_recovered': 14319616, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 4428}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:36:59.823 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:36:59.823 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'4791", 'reported_seq': 4845, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:54.845432+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:36:54.845432+0000', 'last_peered': '2024-09-05T16:36:54.845432+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:36:54.845432+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4791, 'log_dups_size': 0, 'ondisk_log_size': 4791, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39220384, 'num_objects': 4788, 'num_object_clones': 0, 'num_object_copies': 19152, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2164, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4788, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4789, 'num_write_kb': 38309, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2626}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:36:59.823 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:36:59.823 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'4904", 'reported_seq': 4946, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:54.876380+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:36:54.876380+0000', 'last_peered': '2024-09-05T16:36:54.876380+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:36:54.876380+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4904, 'log_dups_size': 0, 'ondisk_log_size': 4904, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40173568, 'num_objects': 4904, 'num_object_clones': 0, 'num_object_copies': 19616, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4904, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4904, 'num_write_kb': 39232, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2654}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:36:59.823 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:36:59.823 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'4770", 'reported_seq': 4835, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:55.657580+0000', 'last_change': '2024-09-05T16:34:56.903436+0000', 'last_active': '2024-09-05T16:36:55.657580+0000', 'last_peered': '2024-09-05T16:36:55.657580+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:36:55.657580+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4770, 'log_dups_size': 0, 'ondisk_log_size': 4770, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39023776, 'num_objects': 4764, 'num_object_clones': 0, 'num_object_copies': 19056, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2149, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4764, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4768, 'num_write_kb': 38124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2617}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:36:59.824 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-05T16:36:59.824 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'4882", 'reported_seq': 4946, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:55.771404+0000', 'last_change': '2024-09-05T16:34:56.883319+0000', 'last_active': '2024-09-05T16:36:55.771404+0000', 'last_peered': '2024-09-05T16:36:55.771404+0000', 'last_clean': '2024-09-05T16:33:53.505305+0000', 'last_became_active': '2024-09-05T16:34:56.873389+0000', 'last_became_peered': '2024-09-05T16:34:56.873389+0000', 'last_unstale': '2024-09-05T16:36:55.771404+0000', 'last_undegraded': '2024-09-05T16:34:56.602567+0000', 'last_fullsized': '2024-09-05T16:34:56.559958+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4882, 'log_dups_size': 0, 'ondisk_log_size': 4882, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39949472, 'num_objects': 4877, 'num_object_clones': 0, 'num_object_copies': 19508, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2241, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4877, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4880, 'num_write_kb': 39028, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2638}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:36:59.824 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:36:59.824 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'4918", 'reported_seq': 4974, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:54.881125+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:36:54.881125+0000', 'last_peered': '2024-09-05T16:36:54.881125+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:36:54.881125+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4918, 'log_dups_size': 0, 'ondisk_log_size': 4918, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40216896, 'num_objects': 4910, 'num_object_clones': 0, 'num_object_copies': 19640, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2264, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4910, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4915, 'num_write_kb': 39297, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2649}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:36:59.824 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:36:59.824 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'4903", 'reported_seq': 4980, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:55.656958+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:36:55.656958+0000', 'last_peered': '2024-09-05T16:36:55.656958+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:36:55.656958+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4903, 'log_dups_size': 0, 'ondisk_log_size': 4903, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40146080, 'num_objects': 4901, 'num_object_clones': 0, 'num_object_copies': 19604, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2264, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4901, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4901, 'num_write_kb': 39206, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2639}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:36:59.824 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:36:59.824 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'4987", 'reported_seq': 5043, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:54.907177+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:36:54.907177+0000', 'last_peered': '2024-09-05T16:36:54.907177+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:36:54.907177+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4987, 'log_dups_size': 0, 'ondisk_log_size': 4987, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40808176, 'num_objects': 4982, 'num_object_clones': 0, 'num_object_copies': 19928, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2314, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4982, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4984, 'num_write_kb': 39860, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2671}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:36:59.825 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:36:59.825 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'4811", 'reported_seq': 4846, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:54.834149+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:36:54.834149+0000', 'last_peered': '2024-09-05T16:36:54.834149+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:36:54.834149+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4811, 'log_dups_size': 0, 'ondisk_log_size': 4811, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39402064, 'num_objects': 4810, 'num_object_clones': 0, 'num_object_copies': 19240, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2265, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4810, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4810, 'num_write_kb': 38479, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2546}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:36:59.825 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-05T16:36:59.825 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'4934", 'reported_seq': 5008, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:55.626943+0000', 'last_change': '2024-09-05T16:34:56.798081+0000', 'last_active': '2024-09-05T16:36:55.626943+0000', 'last_peered': '2024-09-05T16:36:55.626943+0000', 'last_clean': '2024-09-05T16:33:45.082039+0000', 'last_became_active': '2024-09-05T16:34:56.792386+0000', 'last_became_peered': '2024-09-05T16:34:56.792386+0000', 'last_unstale': '2024-09-05T16:36:55.626943+0000', 'last_undegraded': '2024-09-05T16:34:56.600595+0000', 'last_fullsized': '2024-09-05T16:34:56.558203+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4934, 'log_dups_size': 0, 'ondisk_log_size': 4934, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40409680, 'num_objects': 4933, 'num_object_clones': 0, 'num_object_copies': 19732, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2295, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4933, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4933, 'num_write_kb': 39463, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2639}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:36:59.825 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:36:59.825 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'5019", 'reported_seq': 5071, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:54.862344+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:36:54.862344+0000', 'last_peered': '2024-09-05T16:36:54.862344+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:36:54.862344+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5019, 'log_dups_size': 0, 'ondisk_log_size': 5019, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41079968, 'num_objects': 5015, 'num_object_clones': 0, 'num_object_copies': 20060, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2359, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5015, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5017, 'num_write_kb': 40132, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2658}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:36:59.825 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:36:59.825 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'4915", 'reported_seq': 4993, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:55.537302+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:36:55.537302+0000', 'last_peered': '2024-09-05T16:36:55.537302+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:36:55.537302+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4915, 'log_dups_size': 0, 'ondisk_log_size': 4915, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40236192, 'num_objects': 4912, 'num_object_clones': 0, 'num_object_copies': 19648, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2322, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4912, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4913, 'num_write_kb': 39301, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2592}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:36:59.825 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:36:59.825 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'4928", 'reported_seq': 4992, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:55.663753+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:36:55.663753+0000', 'last_peered': '2024-09-05T16:36:55.663753+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:36:55.663753+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4928, 'log_dups_size': 0, 'ondisk_log_size': 4928, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40326304, 'num_objects': 4923, 'num_object_clones': 0, 'num_object_copies': 19692, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4923, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4926, 'num_write_kb': 39396, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2686}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:36:59.826 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:37:00.517 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 158 20 79833 79813 3.94575 2.15625 0.0677921 0.0395862 2024-09-05T16:37:00.542 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:37:00.542 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 34448/313244 objects degraded (10.997%), 16 pgs degraded, 16 pgs undersized","count":32},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":223,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":78311,"data_bytes":641482944,"bytes_used":2098548736,"bytes_avail":384448507904,"bytes_total":386547056640,"degraded_objects":34448,"degraded_total":313244,"degraded_ratio":0.10997177918810895,"recovering_objects_per_sec":12,"recovering_bytes_per_sec":103714,"recovering_keys_per_sec":0,"num_objects_recovered":152,"num_bytes_recovered":1245184,"num_keys_recovered":0,"write_bytes_sec":2104996,"read_op_per_sec":0,"write_op_per_sec":256},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:37:00.565 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:37:00.565 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:37:01.297 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:37:01.298 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:37:01.328 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":136,"stamp":"2024-09-05T16:37:00.844344+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":650019008,"num_objects":79353,"num_object_clones":0,"num_object_copies":317412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34448,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":79353,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":79380,"num_write_kb":634922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1777,"num_bytes_recovered":14319616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79407,"ondisk_log_size":79407,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2063928,"kb_used_data":1131128,"kb_used_omap":14,"kb_used_meta":932721,"kb_avail":375423432,"statfs":{"total":386547056640,"available":384433594368,"internally_reserved":0,"allocated":1158275072,"data_stored":1157543544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":955106871},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,9,7,6,16,8,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":33808384,"num_objects":4127,"num_object_clones":0,"num_object_copies":16508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-152,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4127,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4127,"num_write_kb":33016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":152,"num_bytes_recovered":1245184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006112"},"pg_stats":[{"pgid":"1.f","version":"33'5005","reported_seq":5047,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:59.877030+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:36:59.877030+0000","last_peered":"2024-09-05T16:36:59.877030+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:36:59.877030+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5005,"log_dups_size":0,"ondisk_log_size":5005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41000960,"num_objects":5005,"num_object_clones":0,"num_object_copies":20020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5005,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5005,"num_write_kb":40040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2743}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'4843","reported_seq":4907,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:55.736924+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:36:55.736924+0000","last_peered":"2024-09-05T16:36:55.736924+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:36:55.736924+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4843,"log_dups_size":0,"ondisk_log_size":4843,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39638176,"num_objects":4839,"num_object_clones":0,"num_object_copies":19356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4839,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4841,"num_write_kb":38724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2572}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'4977","reported_seq":5052,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:00.619010+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:37:00.619010+0000","last_peered":"2024-09-05T16:37:00.619010+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:37:00.619010+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4977,"log_dups_size":0,"ondisk_log_size":4977,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40745552,"num_objects":4974,"num_object_clones":0,"num_object_copies":19896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4974,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4976,"num_write_kb":39798,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2724}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'4972","reported_seq":8537,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:55.767715+0000","last_change":"2024-09-05T16:34:56.800371+0000","last_active":"2024-09-05T16:36:55.767715+0000","last_peered":"2024-09-05T16:36:55.767715+0000","last_clean":"2024-09-05T16:33:53.148717+0000","last_became_active":"2024-09-05T16:34:56.791755+0000","last_became_peered":"2024-09-05T16:34:56.791755+0000","last_unstale":"2024-09-05T16:36:55.767715+0000","last_undegraded":"2024-09-05T16:34:56.597522+0000","last_fullsized":"2024-09-05T16:34:56.568150+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4972,"log_dups_size":0,"ondisk_log_size":4972,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40694944,"num_objects":4968,"num_object_clones":0,"num_object_copies":19872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":540,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4968,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4970,"num_write_kb":39749,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1752,"num_bytes_recovered":14319616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4428}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'4901","reported_seq":4955,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:59.934992+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:36:59.934992+0000","last_peered":"2024-09-05T16:36:59.934992+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:36:59.934992+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4901,"log_dups_size":0,"ondisk_log_size":4901,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40121504,"num_objects":4898,"num_object_clones":0,"num_object_copies":19592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4898,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4899,"num_write_kb":39189,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2736}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'4994","reported_seq":5036,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:59.941180+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:36:59.941180+0000","last_peered":"2024-09-05T16:36:59.941180+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:36:59.941180+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4994,"log_dups_size":0,"ondisk_log_size":4994,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40910848,"num_objects":4994,"num_object_clones":0,"num_object_copies":19976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4994,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4994,"num_write_kb":39952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2744}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'4770","reported_seq":4835,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:55.657580+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:36:55.657580+0000","last_peered":"2024-09-05T16:36:55.657580+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:36:55.657580+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4770,"log_dups_size":0,"ondisk_log_size":4770,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39023776,"num_objects":4764,"num_object_clones":0,"num_object_copies":19056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4764,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4768,"num_write_kb":38124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2617}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'4882","reported_seq":4946,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:55.771404+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:36:55.771404+0000","last_peered":"2024-09-05T16:36:55.771404+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:36:55.771404+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4882,"log_dups_size":0,"ondisk_log_size":4882,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39949472,"num_objects":4877,"num_object_clones":0,"num_object_copies":19508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4877,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4880,"num_write_kb":39028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2638}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'5004","reported_seq":5060,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:59.939983+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:36:59.939983+0000","last_peered":"2024-09-05T16:36:59.939983+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:36:59.939983+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5004,"log_dups_size":0,"ondisk_log_size":5004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40921408,"num_objects":4996,"num_object_clones":0,"num_object_copies":19984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4996,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5001,"num_write_kb":39985,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2735}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'4986","reported_seq":5063,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:00.645302+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:37:00.645302+0000","last_peered":"2024-09-05T16:37:00.645302+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:37:00.645302+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4986,"log_dups_size":0,"ondisk_log_size":4986,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40826016,"num_objects":4984,"num_object_clones":0,"num_object_copies":19936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4984,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4984,"num_write_kb":39870,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2722}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'5088","reported_seq":5144,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:59.740969+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:36:59.740969+0000","last_peered":"2024-09-05T16:36:59.740969+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:36:59.740969+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5088,"log_dups_size":0,"ondisk_log_size":5088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41635568,"num_objects":5083,"num_object_clones":0,"num_object_copies":20332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5083,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5085,"num_write_kb":40668,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2772}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'4910","reported_seq":4946,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:59.656920+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:36:59.656920+0000","last_peered":"2024-09-05T16:36:59.656920+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:36:59.656920+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4910,"log_dups_size":0,"ondisk_log_size":4910,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40213072,"num_objects":4909,"num_object_clones":0,"num_object_copies":19636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4909,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4909,"num_write_kb":39271,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2645}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'5025","reported_seq":5099,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:00.663907+0000","last_change":"2024-09-05T16:34:56.798081+0000","last_active":"2024-09-05T16:37:00.663907+0000","last_peered":"2024-09-05T16:37:00.663907+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:37:00.663907+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5025,"log_dups_size":0,"ondisk_log_size":5025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41155152,"num_objects":5024,"num_object_clones":0,"num_object_copies":20096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5024,"num_write_kb":40191,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2730}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'5113","reported_seq":5165,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:59.917362+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:36:59.917362+0000","last_peered":"2024-09-05T16:36:59.917362+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:36:59.917362+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5113,"log_dups_size":0,"ondisk_log_size":5113,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41850016,"num_objects":5109,"num_object_clones":0,"num_object_copies":20436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5109,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5111,"num_write_kb":40884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2752}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'5009","reported_seq":5087,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:00.521563+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:37:00.521563+0000","last_peered":"2024-09-05T16:37:00.521563+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:37:00.521563+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5009,"log_dups_size":0,"ondisk_log_size":5009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41006240,"num_objects":5006,"num_object_clones":0,"num_object_copies":20024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5006,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5007,"num_write_kb":40053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2686}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'4928","reported_seq":4992,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:55.663753+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:36:55.663753+0000","last_peered":"2024-09-05T16:36:55.663753+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:36:55.663753+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4928,"log_dups_size":0,"ondisk_log_size":4928,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40326304,"num_objects":4923,"num_object_clones":0,"num_object_copies":19692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4926,"num_write_kb":39396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2686}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":650019008,"num_objects":79353,"num_object_clones":0,"num_object_copies":317412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34448,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":79353,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":79380,"num_write_kb":634922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1777,"num_bytes_recovered":14319616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1160237056,"data_stored":1160237056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79407,"ondisk_log_size":79407,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084325,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":575068,"kb_used_data":318476,"kb_used_omap":4,"kb_used_meta":256571,"kb_avail":93796772,"statfs":{"total":96636764160,"available":96047894528,"internally_reserved":0,"allocated":326119424,"data_stored":325936542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":262729427},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,0,7,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986202,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":356648,"kb_used_data":180568,"kb_used_omap":4,"kb_used_meta":176059,"kb_avail":94015192,"statfs":{"total":96636764160,"available":96271556608,"internally_reserved":0,"allocated":184901632,"data_stored":184718750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":180285138},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,1,5,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673008,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":562836,"kb_used_data":319236,"kb_used_omap":1,"kb_used_meta":243582,"kb_avail":93809004,"statfs":{"total":96636764160,"available":96060420096,"internally_reserved":0,"allocated":326897664,"data_stored":326714782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":249428421},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,5,0,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182438,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":569376,"kb_used_data":312848,"kb_used_omap":4,"kb_used_meta":256507,"kb_avail":93802464,"statfs":{"total":96636764160,"available":96053723136,"internally_reserved":0,"allocated":320356352,"data_stored":320173470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":262663885},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,6,4,3,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":321220608,"data_stored":321220608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":185421824,"data_stored":185421824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":327233536,"data_stored":327233536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":326361088,"data_stored":326361088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:37:01.517 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 159 20 80127 80107 3.93538 2.29688 0.0811497 0.0396938 2024-09-05T16:37:02.517 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout:2024-09-05T16:37:02.525167+0000 min lat: 0.00357551 max lat: 2.29705 avg lat: 0.0397823 2024-09-05T16:37:02.517 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-05T16:37:02.517 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 160 20 80450 80430 3.92656 2.52344 0.0777067 0.0397823 2024-09-05T16:37:03.517 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 161 20 80772 80752 3.91779 2.51562 0.0384593 0.0398708 2024-09-05T16:37:04.330 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:37:04.517 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 162 20 81075 81055 3.90822 2.36719 0.0366479 0.0399634 2024-09-05T16:37:04.906 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:37:04.906 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:37:04.929 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":138,"stamp":"2024-09-05T16:37:04.845971+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":653861056,"num_objects":79822,"num_object_clones":0,"num_object_copies":319288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34370,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":79822,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":79849,"num_write_kb":638674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1855,"num_bytes_recovered":14958592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79876,"ondisk_log_size":79876,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2086032,"kb_used_data":1143500,"kb_used_omap":14,"kb_used_meta":942449,"kb_avail":375401328,"statfs":{"total":386547056640,"available":384410959872,"internally_reserved":0,"allocated":1170944000,"data_stored":1170212472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":965068343},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,6,3,15,15,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":25165824,"num_objects":3072,"num_object_clones":0,"num_object_copies":12288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-150,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3072,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3072,"num_write_kb":24576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":150,"num_bytes_recovered":1228800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005543"},"pg_stats":[{"pgid":"1.f","version":"33'5005","reported_seq":5047,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:59.877030+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:36:59.877030+0000","last_peered":"2024-09-05T16:36:59.877030+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:36:59.877030+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5005,"log_dups_size":0,"ondisk_log_size":5005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41000960,"num_objects":5005,"num_object_clones":0,"num_object_copies":20020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5005,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5005,"num_write_kb":40040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2743}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'4944","reported_seq":5008,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:00.848106+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:37:00.848106+0000","last_peered":"2024-09-05T16:37:00.848106+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:37:00.848106+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4944,"log_dups_size":0,"ondisk_log_size":4944,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40465568,"num_objects":4940,"num_object_clones":0,"num_object_copies":19760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4940,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4942,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2673}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'4977","reported_seq":5052,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:00.619010+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:37:00.619010+0000","last_peered":"2024-09-05T16:37:00.619010+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:37:00.619010+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4977,"log_dups_size":0,"ondisk_log_size":4977,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40745552,"num_objects":4974,"num_object_clones":0,"num_object_copies":19896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4974,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4976,"num_write_kb":39798,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2724}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'5053","reported_seq":8774,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:00.817720+0000","last_change":"2024-09-05T16:34:56.800371+0000","last_active":"2024-09-05T16:37:00.817720+0000","last_peered":"2024-09-05T16:37:00.817720+0000","last_clean":"2024-09-05T16:33:53.148717+0000","last_became_active":"2024-09-05T16:34:56.791755+0000","last_became_peered":"2024-09-05T16:34:56.791755+0000","last_unstale":"2024-09-05T16:37:00.817720+0000","last_undegraded":"2024-09-05T16:34:56.597522+0000","last_fullsized":"2024-09-05T16:34:56.568150+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5053,"log_dups_size":0,"ondisk_log_size":5053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41358496,"num_objects":5049,"num_object_clones":0,"num_object_copies":20196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":462,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5049,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5051,"num_write_kb":40397,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1830,"num_bytes_recovered":14958592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4587}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'4901","reported_seq":4955,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:59.934992+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:36:59.934992+0000","last_peered":"2024-09-05T16:36:59.934992+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:36:59.934992+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4901,"log_dups_size":0,"ondisk_log_size":4901,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40121504,"num_objects":4898,"num_object_clones":0,"num_object_copies":19592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4898,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4899,"num_write_kb":39189,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2736}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'4994","reported_seq":5036,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:59.941180+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:36:59.941180+0000","last_peered":"2024-09-05T16:36:59.941180+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:36:59.941180+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4994,"log_dups_size":0,"ondisk_log_size":4994,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40910848,"num_objects":4994,"num_object_clones":0,"num_object_copies":19976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4994,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4994,"num_write_kb":39952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2744}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'4867","reported_seq":4932,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:00.798673+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:37:00.798673+0000","last_peered":"2024-09-05T16:37:00.798673+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:37:00.798673+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4867,"log_dups_size":0,"ondisk_log_size":4867,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39818400,"num_objects":4861,"num_object_clones":0,"num_object_copies":19444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4861,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4865,"num_write_kb":38900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2714}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'4982","reported_seq":5046,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:00.817451+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:37:00.817451+0000","last_peered":"2024-09-05T16:37:00.817451+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:37:00.817451+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4982,"log_dups_size":0,"ondisk_log_size":4982,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40768672,"num_objects":4977,"num_object_clones":0,"num_object_copies":19908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4977,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4980,"num_write_kb":39828,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2738}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'5004","reported_seq":5060,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:59.939983+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:36:59.939983+0000","last_peered":"2024-09-05T16:36:59.939983+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:36:59.939983+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5004,"log_dups_size":0,"ondisk_log_size":5004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40921408,"num_objects":4996,"num_object_clones":0,"num_object_copies":19984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4996,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5001,"num_write_kb":39985,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2735}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'4986","reported_seq":5063,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:00.645302+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:37:00.645302+0000","last_peered":"2024-09-05T16:37:00.645302+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:37:00.645302+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4986,"log_dups_size":0,"ondisk_log_size":4986,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40826016,"num_objects":4984,"num_object_clones":0,"num_object_copies":19936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4984,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4984,"num_write_kb":39870,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2722}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'5088","reported_seq":5144,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:59.740969+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:36:59.740969+0000","last_peered":"2024-09-05T16:36:59.740969+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:36:59.740969+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5088,"log_dups_size":0,"ondisk_log_size":5088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41635568,"num_objects":5083,"num_object_clones":0,"num_object_copies":20332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5083,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5085,"num_write_kb":40668,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2772}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'4910","reported_seq":4946,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:59.656920+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:36:59.656920+0000","last_peered":"2024-09-05T16:36:59.656920+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:36:59.656920+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4910,"log_dups_size":0,"ondisk_log_size":4910,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40213072,"num_objects":4909,"num_object_clones":0,"num_object_copies":19636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4909,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4909,"num_write_kb":39271,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2645}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'5025","reported_seq":5099,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:00.663907+0000","last_change":"2024-09-05T16:34:56.798081+0000","last_active":"2024-09-05T16:37:00.663907+0000","last_peered":"2024-09-05T16:37:00.663907+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:37:00.663907+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5025,"log_dups_size":0,"ondisk_log_size":5025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41155152,"num_objects":5024,"num_object_clones":0,"num_object_copies":20096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5024,"num_write_kb":40191,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2730}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'5113","reported_seq":5165,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:59.917362+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:36:59.917362+0000","last_peered":"2024-09-05T16:36:59.917362+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:36:59.917362+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5113,"log_dups_size":0,"ondisk_log_size":5113,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41850016,"num_objects":5109,"num_object_clones":0,"num_object_copies":20436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5109,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5111,"num_write_kb":40884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2752}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'5009","reported_seq":5087,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:00.521563+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:37:00.521563+0000","last_peered":"2024-09-05T16:37:00.521563+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:37:00.521563+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5009,"log_dups_size":0,"ondisk_log_size":5009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41006240,"num_objects":5006,"num_object_clones":0,"num_object_copies":20024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5006,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5007,"num_write_kb":40053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2686}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'5018","reported_seq":5082,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:00.776099+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:37:00.776099+0000","last_peered":"2024-09-05T16:37:00.776099+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:37:00.776099+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5018,"log_dups_size":0,"ondisk_log_size":5018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41063584,"num_objects":5013,"num_object_clones":0,"num_object_copies":20052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5013,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5016,"num_write_kb":40116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2776}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":653861056,"num_objects":79822,"num_object_clones":0,"num_object_copies":319288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34370,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":79822,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":79849,"num_write_kb":638674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1855,"num_bytes_recovered":14958592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1173061632,"data_stored":1173061632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79876,"ondisk_log_size":79876,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084325,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":575068,"kb_used_data":318476,"kb_used_omap":4,"kb_used_meta":256571,"kb_avail":93796772,"statfs":{"total":96636764160,"available":96047894528,"internally_reserved":0,"allocated":326119424,"data_stored":325936542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":262729427},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,0,7,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986203,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":385160,"kb_used_data":186868,"kb_used_omap":4,"kb_used_meta":198267,"kb_avail":93986680,"statfs":{"total":96636764160,"available":96242360320,"internally_reserved":0,"allocated":191352832,"data_stored":191169950,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":203026130},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2,7,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673008,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":562836,"kb_used_data":319236,"kb_used_omap":1,"kb_used_meta":243582,"kb_avail":93809004,"statfs":{"total":96636764160,"available":96060420096,"internally_reserved":0,"allocated":326897664,"data_stored":326714782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":249428421},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,5,0,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182439,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":562968,"kb_used_data":318920,"kb_used_omap":4,"kb_used_meta":244027,"kb_avail":93808872,"statfs":{"total":96636764160,"available":96060284928,"internally_reserved":0,"allocated":326574080,"data_stored":326391198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":249884365},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,7,2,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327426048,"data_stored":327426048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":192040960,"data_stored":192040960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":327233536,"data_stored":327233536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":326361088,"data_stored":326361088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:37:04.930 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:37:05.499 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:37:05.499 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:37:05.517 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 163 20 81356 81336 3.89771 2.19531 0.12147 0.0400748 2024-09-05T16:37:05.521 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":138,"stamp":"2024-09-05T16:37:04.845971+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":653861056,"num_objects":79822,"num_object_clones":0,"num_object_copies":319288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34370,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":79822,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":79849,"num_write_kb":638674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1855,"num_bytes_recovered":14958592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79876,"ondisk_log_size":79876,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2086032,"kb_used_data":1143500,"kb_used_omap":14,"kb_used_meta":942449,"kb_avail":375401328,"statfs":{"total":386547056640,"available":384410959872,"internally_reserved":0,"allocated":1170944000,"data_stored":1170212472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":965068343},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,6,3,15,15,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":25165824,"num_objects":3072,"num_object_clones":0,"num_object_copies":12288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-150,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3072,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3072,"num_write_kb":24576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":150,"num_bytes_recovered":1228800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005543"},"pg_stats":[{"pgid":"1.f","version":"33'5005","reported_seq":5047,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:59.877030+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:36:59.877030+0000","last_peered":"2024-09-05T16:36:59.877030+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:36:59.877030+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5005,"log_dups_size":0,"ondisk_log_size":5005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41000960,"num_objects":5005,"num_object_clones":0,"num_object_copies":20020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5005,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5005,"num_write_kb":40040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2743}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'4944","reported_seq":5008,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:00.848106+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:37:00.848106+0000","last_peered":"2024-09-05T16:37:00.848106+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:37:00.848106+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4944,"log_dups_size":0,"ondisk_log_size":4944,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40465568,"num_objects":4940,"num_object_clones":0,"num_object_copies":19760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4940,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4942,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2673}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'4977","reported_seq":5052,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:00.619010+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:37:00.619010+0000","last_peered":"2024-09-05T16:37:00.619010+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:37:00.619010+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4977,"log_dups_size":0,"ondisk_log_size":4977,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40745552,"num_objects":4974,"num_object_clones":0,"num_object_copies":19896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4974,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4976,"num_write_kb":39798,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2724}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'5053","reported_seq":8774,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:00.817720+0000","last_change":"2024-09-05T16:34:56.800371+0000","last_active":"2024-09-05T16:37:00.817720+0000","last_peered":"2024-09-05T16:37:00.817720+0000","last_clean":"2024-09-05T16:33:53.148717+0000","last_became_active":"2024-09-05T16:34:56.791755+0000","last_became_peered":"2024-09-05T16:34:56.791755+0000","last_unstale":"2024-09-05T16:37:00.817720+0000","last_undegraded":"2024-09-05T16:34:56.597522+0000","last_fullsized":"2024-09-05T16:34:56.568150+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5053,"log_dups_size":0,"ondisk_log_size":5053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41358496,"num_objects":5049,"num_object_clones":0,"num_object_copies":20196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":462,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5049,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5051,"num_write_kb":40397,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1830,"num_bytes_recovered":14958592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4587}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'4901","reported_seq":4955,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:59.934992+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:36:59.934992+0000","last_peered":"2024-09-05T16:36:59.934992+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:36:59.934992+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4901,"log_dups_size":0,"ondisk_log_size":4901,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40121504,"num_objects":4898,"num_object_clones":0,"num_object_copies":19592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4898,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4899,"num_write_kb":39189,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2736}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'4994","reported_seq":5036,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:59.941180+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:36:59.941180+0000","last_peered":"2024-09-05T16:36:59.941180+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:36:59.941180+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4994,"log_dups_size":0,"ondisk_log_size":4994,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40910848,"num_objects":4994,"num_object_clones":0,"num_object_copies":19976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4994,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4994,"num_write_kb":39952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2744}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'4867","reported_seq":4932,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:00.798673+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:37:00.798673+0000","last_peered":"2024-09-05T16:37:00.798673+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:37:00.798673+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4867,"log_dups_size":0,"ondisk_log_size":4867,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39818400,"num_objects":4861,"num_object_clones":0,"num_object_copies":19444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4861,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4865,"num_write_kb":38900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2714}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'4982","reported_seq":5046,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:00.817451+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:37:00.817451+0000","last_peered":"2024-09-05T16:37:00.817451+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:37:00.817451+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4982,"log_dups_size":0,"ondisk_log_size":4982,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40768672,"num_objects":4977,"num_object_clones":0,"num_object_copies":19908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4977,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4980,"num_write_kb":39828,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2738}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'5004","reported_seq":5060,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:59.939983+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:36:59.939983+0000","last_peered":"2024-09-05T16:36:59.939983+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:36:59.939983+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5004,"log_dups_size":0,"ondisk_log_size":5004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40921408,"num_objects":4996,"num_object_clones":0,"num_object_copies":19984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4996,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5001,"num_write_kb":39985,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2735}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'4986","reported_seq":5063,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:00.645302+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:37:00.645302+0000","last_peered":"2024-09-05T16:37:00.645302+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:37:00.645302+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4986,"log_dups_size":0,"ondisk_log_size":4986,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40826016,"num_objects":4984,"num_object_clones":0,"num_object_copies":19936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4984,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4984,"num_write_kb":39870,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2722}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'5088","reported_seq":5144,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:59.740969+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:36:59.740969+0000","last_peered":"2024-09-05T16:36:59.740969+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:36:59.740969+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5088,"log_dups_size":0,"ondisk_log_size":5088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41635568,"num_objects":5083,"num_object_clones":0,"num_object_copies":20332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5083,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5085,"num_write_kb":40668,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2772}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'4910","reported_seq":4946,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:59.656920+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:36:59.656920+0000","last_peered":"2024-09-05T16:36:59.656920+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:36:59.656920+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4910,"log_dups_size":0,"ondisk_log_size":4910,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40213072,"num_objects":4909,"num_object_clones":0,"num_object_copies":19636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4909,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4909,"num_write_kb":39271,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2645}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'5025","reported_seq":5099,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:00.663907+0000","last_change":"2024-09-05T16:34:56.798081+0000","last_active":"2024-09-05T16:37:00.663907+0000","last_peered":"2024-09-05T16:37:00.663907+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:37:00.663907+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5025,"log_dups_size":0,"ondisk_log_size":5025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41155152,"num_objects":5024,"num_object_clones":0,"num_object_copies":20096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5024,"num_write_kb":40191,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2730}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'5113","reported_seq":5165,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:36:59.917362+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:36:59.917362+0000","last_peered":"2024-09-05T16:36:59.917362+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:36:59.917362+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5113,"log_dups_size":0,"ondisk_log_size":5113,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41850016,"num_objects":5109,"num_object_clones":0,"num_object_copies":20436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5109,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5111,"num_write_kb":40884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2752}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'5009","reported_seq":5087,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:00.521563+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:37:00.521563+0000","last_peered":"2024-09-05T16:37:00.521563+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:37:00.521563+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5009,"log_dups_size":0,"ondisk_log_size":5009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41006240,"num_objects":5006,"num_object_clones":0,"num_object_copies":20024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5006,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5007,"num_write_kb":40053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2686}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'5018","reported_seq":5082,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:00.776099+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:37:00.776099+0000","last_peered":"2024-09-05T16:37:00.776099+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:37:00.776099+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5018,"log_dups_size":0,"ondisk_log_size":5018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41063584,"num_objects":5013,"num_object_clones":0,"num_object_copies":20052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5013,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5016,"num_write_kb":40116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2776}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":653861056,"num_objects":79822,"num_object_clones":0,"num_object_copies":319288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34370,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":79822,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":79849,"num_write_kb":638674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1855,"num_bytes_recovered":14958592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1173061632,"data_stored":1173061632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79876,"ondisk_log_size":79876,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084325,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":575068,"kb_used_data":318476,"kb_used_omap":4,"kb_used_meta":256571,"kb_avail":93796772,"statfs":{"total":96636764160,"available":96047894528,"internally_reserved":0,"allocated":326119424,"data_stored":325936542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":262729427},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,0,7,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986203,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":385160,"kb_used_data":186868,"kb_used_omap":4,"kb_used_meta":198267,"kb_avail":93986680,"statfs":{"total":96636764160,"available":96242360320,"internally_reserved":0,"allocated":191352832,"data_stored":191169950,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":203026130},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2,7,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673008,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":562836,"kb_used_data":319236,"kb_used_omap":1,"kb_used_meta":243582,"kb_avail":93809004,"statfs":{"total":96636764160,"available":96060420096,"internally_reserved":0,"allocated":326897664,"data_stored":326714782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":249428421},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,5,0,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182439,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":562968,"kb_used_data":318920,"kb_used_omap":4,"kb_used_meta":244027,"kb_avail":93808872,"statfs":{"total":96636764160,"available":96060284928,"internally_reserved":0,"allocated":326574080,"data_stored":326391198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":249884365},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,7,2,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327426048,"data_stored":327426048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":192040960,"data_stored":192040960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":327233536,"data_stored":327233536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":326361088,"data_stored":326361088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:37:05.522 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-05T16:37:05.522 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'5005", 'reported_seq': 5047, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:59.877030+0000', 'last_change': '2024-09-05T16:34:56.833497+0000', 'last_active': '2024-09-05T16:36:59.877030+0000', 'last_peered': '2024-09-05T16:36:59.877030+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.828289+0000', 'last_became_peered': '2024-09-05T16:34:56.828289+0000', 'last_unstale': '2024-09-05T16:36:59.877030+0000', 'last_undegraded': '2024-09-05T16:34:56.612160+0000', 'last_fullsized': '2024-09-05T16:34:56.557632+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5005, 'log_dups_size': 0, 'ondisk_log_size': 5005, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41000960, 'num_objects': 5005, 'num_object_clones': 0, 'num_object_copies': 20020, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2262, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5005, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5005, 'num_write_kb': 40040, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2743}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:37:05.522 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-05T16:37:05.522 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'4944", 'reported_seq': 5008, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:00.848106+0000', 'last_change': '2024-09-05T16:34:56.825031+0000', 'last_active': '2024-09-05T16:37:00.848106+0000', 'last_peered': '2024-09-05T16:37:00.848106+0000', 'last_clean': '2024-09-05T16:33:53.200762+0000', 'last_became_active': '2024-09-05T16:34:56.820292+0000', 'last_became_peered': '2024-09-05T16:34:56.820292+0000', 'last_unstale': '2024-09-05T16:37:00.848106+0000', 'last_undegraded': '2024-09-05T16:34:56.605776+0000', 'last_fullsized': '2024-09-05T16:34:56.559383+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4944, 'log_dups_size': 0, 'ondisk_log_size': 4944, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40465568, 'num_objects': 4940, 'num_object_clones': 0, 'num_object_copies': 19760, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2269, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4940, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4942, 'num_write_kb': 39532, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2673}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:37:05.523 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:37:05.523 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'4977", 'reported_seq': 5052, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:00.619010+0000', 'last_change': '2024-09-05T16:34:56.867900+0000', 'last_active': '2024-09-05T16:37:00.619010+0000', 'last_peered': '2024-09-05T16:37:00.619010+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:37:00.619010+0000', 'last_undegraded': '2024-09-05T16:34:56.590972+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4977, 'log_dups_size': 0, 'ondisk_log_size': 4977, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40745552, 'num_objects': 4974, 'num_object_clones': 0, 'num_object_copies': 19896, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2251, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4974, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4976, 'num_write_kb': 39798, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2724}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:37:05.523 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-05T16:37:05.523 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'5053", 'reported_seq': 8774, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:00.817720+0000', 'last_change': '2024-09-05T16:34:56.800371+0000', 'last_active': '2024-09-05T16:37:00.817720+0000', 'last_peered': '2024-09-05T16:37:00.817720+0000', 'last_clean': '2024-09-05T16:33:53.148717+0000', 'last_became_active': '2024-09-05T16:34:56.791755+0000', 'last_became_peered': '2024-09-05T16:34:56.791755+0000', 'last_unstale': '2024-09-05T16:37:00.817720+0000', 'last_undegraded': '2024-09-05T16:34:56.597522+0000', 'last_fullsized': '2024-09-05T16:34:56.568150+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5053, 'log_dups_size': 0, 'ondisk_log_size': 5053, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41358496, 'num_objects': 5049, 'num_object_clones': 0, 'num_object_copies': 20196, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 462, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5049, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5051, 'num_write_kb': 40397, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1830, 'num_bytes_recovered': 14958592, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 4587}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:37:05.523 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:37:05.523 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'4901", 'reported_seq': 4955, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:59.934992+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:36:59.934992+0000', 'last_peered': '2024-09-05T16:36:59.934992+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:36:59.934992+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4901, 'log_dups_size': 0, 'ondisk_log_size': 4901, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40121504, 'num_objects': 4898, 'num_object_clones': 0, 'num_object_copies': 19592, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2164, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4898, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4899, 'num_write_kb': 39189, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2736}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:37:05.523 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:37:05.523 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'4994", 'reported_seq': 5036, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:59.941180+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:36:59.941180+0000', 'last_peered': '2024-09-05T16:36:59.941180+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:36:59.941180+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4994, 'log_dups_size': 0, 'ondisk_log_size': 4994, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40910848, 'num_objects': 4994, 'num_object_clones': 0, 'num_object_copies': 19976, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4994, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4994, 'num_write_kb': 39952, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2744}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:37:05.523 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:37:05.523 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'4867", 'reported_seq': 4932, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:00.798673+0000', 'last_change': '2024-09-05T16:34:56.903436+0000', 'last_active': '2024-09-05T16:37:00.798673+0000', 'last_peered': '2024-09-05T16:37:00.798673+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:37:00.798673+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4867, 'log_dups_size': 0, 'ondisk_log_size': 4867, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39818400, 'num_objects': 4861, 'num_object_clones': 0, 'num_object_copies': 19444, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2149, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4861, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4865, 'num_write_kb': 38900, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2714}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:37:05.524 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-05T16:37:05.524 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'4982", 'reported_seq': 5046, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:00.817451+0000', 'last_change': '2024-09-05T16:34:56.883319+0000', 'last_active': '2024-09-05T16:37:00.817451+0000', 'last_peered': '2024-09-05T16:37:00.817451+0000', 'last_clean': '2024-09-05T16:33:53.505305+0000', 'last_became_active': '2024-09-05T16:34:56.873389+0000', 'last_became_peered': '2024-09-05T16:34:56.873389+0000', 'last_unstale': '2024-09-05T16:37:00.817451+0000', 'last_undegraded': '2024-09-05T16:34:56.602567+0000', 'last_fullsized': '2024-09-05T16:34:56.559958+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4982, 'log_dups_size': 0, 'ondisk_log_size': 4982, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40768672, 'num_objects': 4977, 'num_object_clones': 0, 'num_object_copies': 19908, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2241, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4977, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4980, 'num_write_kb': 39828, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2738}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:37:05.524 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:37:05.524 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'5004", 'reported_seq': 5060, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:59.939983+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:36:59.939983+0000', 'last_peered': '2024-09-05T16:36:59.939983+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:36:59.939983+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5004, 'log_dups_size': 0, 'ondisk_log_size': 5004, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40921408, 'num_objects': 4996, 'num_object_clones': 0, 'num_object_copies': 19984, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2264, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4996, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5001, 'num_write_kb': 39985, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2735}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:37:05.524 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:37:05.524 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'4986", 'reported_seq': 5063, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:00.645302+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:37:00.645302+0000', 'last_peered': '2024-09-05T16:37:00.645302+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:37:00.645302+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4986, 'log_dups_size': 0, 'ondisk_log_size': 4986, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40826016, 'num_objects': 4984, 'num_object_clones': 0, 'num_object_copies': 19936, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2264, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4984, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4984, 'num_write_kb': 39870, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2722}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:37:05.524 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:37:05.524 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'5088", 'reported_seq': 5144, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:59.740969+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:36:59.740969+0000', 'last_peered': '2024-09-05T16:36:59.740969+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:36:59.740969+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5088, 'log_dups_size': 0, 'ondisk_log_size': 5088, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41635568, 'num_objects': 5083, 'num_object_clones': 0, 'num_object_copies': 20332, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2314, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5083, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5085, 'num_write_kb': 40668, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2772}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:37:05.524 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:37:05.524 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'4910", 'reported_seq': 4946, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:59.656920+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:36:59.656920+0000', 'last_peered': '2024-09-05T16:36:59.656920+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:36:59.656920+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4910, 'log_dups_size': 0, 'ondisk_log_size': 4910, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40213072, 'num_objects': 4909, 'num_object_clones': 0, 'num_object_copies': 19636, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2265, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4909, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4909, 'num_write_kb': 39271, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2645}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:37:05.524 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-05T16:37:05.524 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'5025", 'reported_seq': 5099, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:00.663907+0000', 'last_change': '2024-09-05T16:34:56.798081+0000', 'last_active': '2024-09-05T16:37:00.663907+0000', 'last_peered': '2024-09-05T16:37:00.663907+0000', 'last_clean': '2024-09-05T16:33:45.082039+0000', 'last_became_active': '2024-09-05T16:34:56.792386+0000', 'last_became_peered': '2024-09-05T16:34:56.792386+0000', 'last_unstale': '2024-09-05T16:37:00.663907+0000', 'last_undegraded': '2024-09-05T16:34:56.600595+0000', 'last_fullsized': '2024-09-05T16:34:56.558203+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5025, 'log_dups_size': 0, 'ondisk_log_size': 5025, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41155152, 'num_objects': 5024, 'num_object_clones': 0, 'num_object_copies': 20096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2295, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5024, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5024, 'num_write_kb': 40191, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2730}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:37:05.525 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:37:05.525 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'5113", 'reported_seq': 5165, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:36:59.917362+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:36:59.917362+0000', 'last_peered': '2024-09-05T16:36:59.917362+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:36:59.917362+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5113, 'log_dups_size': 0, 'ondisk_log_size': 5113, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41850016, 'num_objects': 5109, 'num_object_clones': 0, 'num_object_copies': 20436, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2359, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5109, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5111, 'num_write_kb': 40884, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2752}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:37:05.525 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:37:05.525 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'5009", 'reported_seq': 5087, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:00.521563+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:37:00.521563+0000', 'last_peered': '2024-09-05T16:37:00.521563+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:37:00.521563+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5009, 'log_dups_size': 0, 'ondisk_log_size': 5009, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41006240, 'num_objects': 5006, 'num_object_clones': 0, 'num_object_copies': 20024, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2322, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5006, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5007, 'num_write_kb': 40053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2686}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:37:05.525 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:37:05.525 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'5018", 'reported_seq': 5082, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:00.776099+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:37:00.776099+0000', 'last_peered': '2024-09-05T16:37:00.776099+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:37:00.776099+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5018, 'log_dups_size': 0, 'ondisk_log_size': 5018, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41063584, 'num_objects': 5013, 'num_object_clones': 0, 'num_object_copies': 20052, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5013, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5016, 'num_write_kb': 40116, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2776}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:37:05.525 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:37:06.303 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:37:06.303 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 34370/319288 objects degraded (10.765%), 16 pgs degraded, 16 pgs undersized","count":32},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":229,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":79822,"data_bytes":653861056,"bytes_used":2136096768,"bytes_avail":384410959872,"bytes_total":386547056640,"degraded_objects":34370,"degraded_total":319288,"degraded_ratio":0.1076457618200496,"recovering_objects_per_sec":12,"recovering_bytes_per_sec":102352,"recovering_keys_per_sec":0,"num_objects_recovered":150,"num_bytes_recovered":1228800,"num_keys_recovered":0,"write_bytes_sec":2096183,"read_op_per_sec":0,"write_op_per_sec":255},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:37:06.329 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:37:06.329 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:37:06.517 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 164 20 81640 81620 3.88748 2.21875 0.0660849 0.040179 2024-09-05T16:37:06.898 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:37:06.898 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:37:06.925 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":139,"stamp":"2024-09-05T16:37:06.846589+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":666476736,"num_objects":81362,"num_object_clones":0,"num_object_copies":325448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81362,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":81389,"num_write_kb":650994,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1934,"num_bytes_recovered":15605760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81416,"ondisk_log_size":81416,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2099988,"kb_used_data":1161612,"kb_used_omap":14,"kb_used_meta":938289,"kb_avail":375387372,"statfs":{"total":386547056640,"available":384396668928,"internally_reserved":0,"allocated":1189490688,"data_stored":1188759160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":960808503},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,7,16,14,15,12,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":37781504,"num_objects":4612,"num_object_clones":0,"num_object_copies":18448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-229,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4612,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4612,"num_write_kb":36896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":229,"num_bytes_recovered":1875968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005205"},"pg_stats":[{"pgid":"1.f","version":"33'5086","reported_seq":5128,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:04.961028+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:37:04.961028+0000","last_peered":"2024-09-05T16:37:04.961028+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:37:04.961028+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5086,"log_dups_size":0,"ondisk_log_size":5086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41664512,"num_objects":5086,"num_object_clones":0,"num_object_copies":20344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5086,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5086,"num_write_kb":40688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2824}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'5036","reported_seq":5100,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:05.773717+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:37:05.773717+0000","last_peered":"2024-09-05T16:37:05.773717+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:37:05.773717+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5036,"log_dups_size":0,"ondisk_log_size":5036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41219232,"num_objects":5032,"num_object_clones":0,"num_object_copies":20128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5032,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5034,"num_write_kb":40268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2765}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'5048","reported_seq":5123,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:05.548299+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:37:05.548299+0000","last_peered":"2024-09-05T16:37:05.548299+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:37:05.548299+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5048,"log_dups_size":0,"ondisk_log_size":5048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41327184,"num_objects":5045,"num_object_clones":0,"num_object_copies":20180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5045,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5047,"num_write_kb":40366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2795}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'5131","reported_seq":9009,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:05.885046+0000","last_change":"2024-09-05T16:34:56.800371+0000","last_active":"2024-09-05T16:37:05.885046+0000","last_peered":"2024-09-05T16:37:05.885046+0000","last_clean":"2024-09-05T16:33:53.148717+0000","last_became_active":"2024-09-05T16:34:56.791755+0000","last_became_peered":"2024-09-05T16:34:56.791755+0000","last_unstale":"2024-09-05T16:37:05.885046+0000","last_undegraded":"2024-09-05T16:34:56.597522+0000","last_fullsized":"2024-09-05T16:34:56.568150+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5131,"log_dups_size":0,"ondisk_log_size":5131,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41997472,"num_objects":5127,"num_object_clones":0,"num_object_copies":20508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":383,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5127,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5129,"num_write_kb":41021,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1909,"num_bytes_recovered":15605760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4744}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'5006","reported_seq":5059,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:05.018818+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:37:05.018818+0000","last_peered":"2024-09-05T16:37:05.018818+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:37:05.018818+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5006,"log_dups_size":0,"ondisk_log_size":5006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40981664,"num_objects":5003,"num_object_clones":0,"num_object_copies":20012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5003,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5004,"num_write_kb":40029,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2841}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'5089","reported_seq":5130,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:05.000582+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:37:05.000582+0000","last_peered":"2024-09-05T16:37:05.000582+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:37:05.000582+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5089,"log_dups_size":0,"ondisk_log_size":5089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41689088,"num_objects":5089,"num_object_clones":0,"num_object_copies":20356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5089,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5089,"num_write_kb":40712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2839}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'4983","reported_seq":5047,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:05.814514+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:37:05.814514+0000","last_peered":"2024-09-05T16:37:05.814514+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:37:05.814514+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4983,"log_dups_size":0,"ondisk_log_size":4983,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40768672,"num_objects":4977,"num_object_clones":0,"num_object_copies":19908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4977,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4981,"num_write_kb":39828,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2830}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'5072","reported_seq":5135,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:05.707098+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:37:05.707098+0000","last_peered":"2024-09-05T16:37:05.707098+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:37:05.707098+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5072,"log_dups_size":0,"ondisk_log_size":5072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41505952,"num_objects":5067,"num_object_clones":0,"num_object_copies":20268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5067,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5070,"num_write_kb":40548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2828}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'5117","reported_seq":5172,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:05.012449+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:37:05.012449+0000","last_peered":"2024-09-05T16:37:05.012449+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:37:05.012449+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5117,"log_dups_size":0,"ondisk_log_size":5117,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41847104,"num_objects":5109,"num_object_clones":0,"num_object_copies":20436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5109,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5114,"num_write_kb":40889,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2848}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'5096","reported_seq":5173,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:05.723729+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:37:05.723729+0000","last_peered":"2024-09-05T16:37:05.723729+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:37:05.723729+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5096,"log_dups_size":0,"ondisk_log_size":5096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41727136,"num_objects":5094,"num_object_clones":0,"num_object_copies":20376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5094,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5094,"num_write_kb":40750,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2832}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'5184","reported_seq":5240,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:04.884537+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:37:04.884537+0000","last_peered":"2024-09-05T16:37:04.884537+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:37:04.884537+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5184,"log_dups_size":0,"ondisk_log_size":5184,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42422000,"num_objects":5179,"num_object_clones":0,"num_object_copies":20716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5179,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5181,"num_write_kb":41436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2868}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'5013","reported_seq":5049,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:05.061663+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:37:05.061663+0000","last_peered":"2024-09-05T16:37:05.061663+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:37:05.061663+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5013,"log_dups_size":0,"ondisk_log_size":5013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41056848,"num_objects":5012,"num_object_clones":0,"num_object_copies":20048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5012,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5012,"num_write_kb":40095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2748}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'5124","reported_seq":5198,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:05.594517+0000","last_change":"2024-09-05T16:34:56.798081+0000","last_active":"2024-09-05T16:37:05.594517+0000","last_peered":"2024-09-05T16:37:05.594517+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:37:05.594517+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5124,"log_dups_size":0,"ondisk_log_size":5124,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41966160,"num_objects":5123,"num_object_clones":0,"num_object_copies":20492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5123,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5123,"num_write_kb":40983,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2829}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'5204","reported_seq":5256,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:04.923771+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:37:04.923771+0000","last_peered":"2024-09-05T16:37:04.923771+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:37:04.923771+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5204,"log_dups_size":0,"ondisk_log_size":5204,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42595488,"num_objects":5200,"num_object_clones":0,"num_object_copies":20800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5200,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5202,"num_write_kb":41612,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2843}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'5128","reported_seq":5206,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:05.669707+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:37:05.669707+0000","last_peered":"2024-09-05T16:37:05.669707+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:37:05.669707+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5128,"log_dups_size":0,"ondisk_log_size":5128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41981088,"num_objects":5125,"num_object_clones":0,"num_object_copies":20500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5125,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5126,"num_write_kb":41005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2805}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'5099","reported_seq":5163,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:05.741208+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:37:05.741208+0000","last_peered":"2024-09-05T16:37:05.741208+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:37:05.741208+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5099,"log_dups_size":0,"ondisk_log_size":5099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41727136,"num_objects":5094,"num_object_clones":0,"num_object_copies":20376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5094,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5097,"num_write_kb":40764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2857}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":666476736,"num_objects":81362,"num_object_clones":0,"num_object_copies":325448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81362,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":81389,"num_write_kb":650994,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1934,"num_bytes_recovered":15605760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1191944192,"data_stored":1191944192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81416,"ondisk_log_size":81416,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084326,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":568576,"kb_used_data":324464,"kb_used_omap":4,"kb_used_meta":244091,"kb_avail":93803264,"statfs":{"total":96636764160,"available":96054542336,"internally_reserved":0,"allocated":332251136,"data_stored":332068254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":249949907},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,6,0,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986203,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":385160,"kb_used_data":186868,"kb_used_omap":4,"kb_used_meta":198267,"kb_avail":93986680,"statfs":{"total":96636764160,"available":96242360320,"internally_reserved":0,"allocated":191352832,"data_stored":191169950,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":203026130},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2,7,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673009,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":572980,"kb_used_data":325220,"kb_used_omap":1,"kb_used_meta":247742,"kb_avail":93798860,"statfs":{"total":96636764160,"available":96050032640,"internally_reserved":0,"allocated":333025280,"data_stored":332842398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":253688261},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,10,1,7,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182440,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":573272,"kb_used_data":325060,"kb_used_omap":4,"kb_used_meta":248187,"kb_avail":93798568,"statfs":{"total":96636764160,"available":96049733632,"internally_reserved":0,"allocated":332861440,"data_stored":332678558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":254144205},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,5,3,0,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":333656064,"data_stored":333656064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":192040960,"data_stored":192040960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":333504512,"data_stored":333504512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":332742656,"data_stored":332742656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:37:07.517 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 165 20 81946 81926 3.8784 2.39062 0.0801161 0.0402778 2024-09-05T16:37:08.517 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 166 20 82258 82238 3.86972 2.4375 0.0889672 0.040365 2024-09-05T16:37:09.517 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 167 20 82578 82558 3.86152 2.5 0.0596029 0.0404514 2024-09-05T16:37:09.928 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:37:10.517 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 168 20 82872 82852 3.85221 2.29688 0.0655791 0.0405476 2024-09-05T16:37:10.526 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:37:10.526 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:37:10.555 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":140,"stamp":"2024-09-05T16:37:08.849557+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":666476736,"num_objects":81362,"num_object_clones":0,"num_object_copies":325448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81362,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":81389,"num_write_kb":650994,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1934,"num_bytes_recovered":15605760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81416,"ondisk_log_size":81416,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2099988,"kb_used_data":1161612,"kb_used_omap":14,"kb_used_meta":938289,"kb_avail":375387372,"statfs":{"total":386547056640,"available":384396668928,"internally_reserved":0,"allocated":1189490688,"data_stored":1188759160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":960808503},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,7,16,14,15,12,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":24993792,"num_objects":3051,"num_object_clones":0,"num_object_copies":12204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-157,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3051,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3051,"num_write_kb":24408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":157,"num_bytes_recovered":1286144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007279"},"pg_stats":[{"pgid":"1.f","version":"33'5086","reported_seq":5128,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:04.961028+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:37:04.961028+0000","last_peered":"2024-09-05T16:37:04.961028+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:37:04.961028+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5086,"log_dups_size":0,"ondisk_log_size":5086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41664512,"num_objects":5086,"num_object_clones":0,"num_object_copies":20344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5086,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5086,"num_write_kb":40688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2824}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'5036","reported_seq":5100,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:05.773717+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:37:05.773717+0000","last_peered":"2024-09-05T16:37:05.773717+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:37:05.773717+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5036,"log_dups_size":0,"ondisk_log_size":5036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41219232,"num_objects":5032,"num_object_clones":0,"num_object_copies":20128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5032,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5034,"num_write_kb":40268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2765}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'5048","reported_seq":5123,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:05.548299+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:37:05.548299+0000","last_peered":"2024-09-05T16:37:05.548299+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:37:05.548299+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5048,"log_dups_size":0,"ondisk_log_size":5048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41327184,"num_objects":5045,"num_object_clones":0,"num_object_copies":20180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5045,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5047,"num_write_kb":40366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2795}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'5131","reported_seq":9009,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:05.885046+0000","last_change":"2024-09-05T16:34:56.800371+0000","last_active":"2024-09-05T16:37:05.885046+0000","last_peered":"2024-09-05T16:37:05.885046+0000","last_clean":"2024-09-05T16:33:53.148717+0000","last_became_active":"2024-09-05T16:34:56.791755+0000","last_became_peered":"2024-09-05T16:34:56.791755+0000","last_unstale":"2024-09-05T16:37:05.885046+0000","last_undegraded":"2024-09-05T16:34:56.597522+0000","last_fullsized":"2024-09-05T16:34:56.568150+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5131,"log_dups_size":0,"ondisk_log_size":5131,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41997472,"num_objects":5127,"num_object_clones":0,"num_object_copies":20508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":383,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5127,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5129,"num_write_kb":41021,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1909,"num_bytes_recovered":15605760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4744}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'5006","reported_seq":5059,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:05.018818+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:37:05.018818+0000","last_peered":"2024-09-05T16:37:05.018818+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:37:05.018818+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5006,"log_dups_size":0,"ondisk_log_size":5006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40981664,"num_objects":5003,"num_object_clones":0,"num_object_copies":20012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5003,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5004,"num_write_kb":40029,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2841}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'5089","reported_seq":5130,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:05.000582+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:37:05.000582+0000","last_peered":"2024-09-05T16:37:05.000582+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:37:05.000582+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5089,"log_dups_size":0,"ondisk_log_size":5089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41689088,"num_objects":5089,"num_object_clones":0,"num_object_copies":20356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5089,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5089,"num_write_kb":40712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2839}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'4983","reported_seq":5047,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:05.814514+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:37:05.814514+0000","last_peered":"2024-09-05T16:37:05.814514+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:37:05.814514+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4983,"log_dups_size":0,"ondisk_log_size":4983,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40768672,"num_objects":4977,"num_object_clones":0,"num_object_copies":19908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4977,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4981,"num_write_kb":39828,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2830}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'5072","reported_seq":5135,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:05.707098+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:37:05.707098+0000","last_peered":"2024-09-05T16:37:05.707098+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:37:05.707098+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5072,"log_dups_size":0,"ondisk_log_size":5072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41505952,"num_objects":5067,"num_object_clones":0,"num_object_copies":20268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5067,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5070,"num_write_kb":40548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2828}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'5117","reported_seq":5172,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:05.012449+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:37:05.012449+0000","last_peered":"2024-09-05T16:37:05.012449+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:37:05.012449+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5117,"log_dups_size":0,"ondisk_log_size":5117,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41847104,"num_objects":5109,"num_object_clones":0,"num_object_copies":20436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5109,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5114,"num_write_kb":40889,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2848}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'5096","reported_seq":5173,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:05.723729+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:37:05.723729+0000","last_peered":"2024-09-05T16:37:05.723729+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:37:05.723729+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5096,"log_dups_size":0,"ondisk_log_size":5096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41727136,"num_objects":5094,"num_object_clones":0,"num_object_copies":20376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5094,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5094,"num_write_kb":40750,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2832}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'5184","reported_seq":5240,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:04.884537+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:37:04.884537+0000","last_peered":"2024-09-05T16:37:04.884537+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:37:04.884537+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5184,"log_dups_size":0,"ondisk_log_size":5184,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42422000,"num_objects":5179,"num_object_clones":0,"num_object_copies":20716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5179,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5181,"num_write_kb":41436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2868}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'5013","reported_seq":5049,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:05.061663+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:37:05.061663+0000","last_peered":"2024-09-05T16:37:05.061663+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:37:05.061663+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5013,"log_dups_size":0,"ondisk_log_size":5013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41056848,"num_objects":5012,"num_object_clones":0,"num_object_copies":20048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5012,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5012,"num_write_kb":40095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2748}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'5124","reported_seq":5198,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:05.594517+0000","last_change":"2024-09-05T16:34:56.798081+0000","last_active":"2024-09-05T16:37:05.594517+0000","last_peered":"2024-09-05T16:37:05.594517+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:37:05.594517+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5124,"log_dups_size":0,"ondisk_log_size":5124,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41966160,"num_objects":5123,"num_object_clones":0,"num_object_copies":20492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5123,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5123,"num_write_kb":40983,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2829}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'5204","reported_seq":5256,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:04.923771+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:37:04.923771+0000","last_peered":"2024-09-05T16:37:04.923771+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:37:04.923771+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5204,"log_dups_size":0,"ondisk_log_size":5204,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42595488,"num_objects":5200,"num_object_clones":0,"num_object_copies":20800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5200,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5202,"num_write_kb":41612,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2843}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'5128","reported_seq":5206,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:05.669707+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:37:05.669707+0000","last_peered":"2024-09-05T16:37:05.669707+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:37:05.669707+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5128,"log_dups_size":0,"ondisk_log_size":5128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41981088,"num_objects":5125,"num_object_clones":0,"num_object_copies":20500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5125,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5126,"num_write_kb":41005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2805}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'5099","reported_seq":5163,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:05.741208+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:37:05.741208+0000","last_peered":"2024-09-05T16:37:05.741208+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:37:05.741208+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5099,"log_dups_size":0,"ondisk_log_size":5099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41727136,"num_objects":5094,"num_object_clones":0,"num_object_copies":20376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5094,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5097,"num_write_kb":40764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2857}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":666476736,"num_objects":81362,"num_object_clones":0,"num_object_copies":325448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81362,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":81389,"num_write_kb":650994,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1934,"num_bytes_recovered":15605760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1191944192,"data_stored":1191944192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81416,"ondisk_log_size":81416,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084326,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":568576,"kb_used_data":324464,"kb_used_omap":4,"kb_used_meta":244091,"kb_avail":93803264,"statfs":{"total":96636764160,"available":96054542336,"internally_reserved":0,"allocated":332251136,"data_stored":332068254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":249949907},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,6,0,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986203,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":385160,"kb_used_data":186868,"kb_used_omap":4,"kb_used_meta":198267,"kb_avail":93986680,"statfs":{"total":96636764160,"available":96242360320,"internally_reserved":0,"allocated":191352832,"data_stored":191169950,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":203026130},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2,7,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673009,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":572980,"kb_used_data":325220,"kb_used_omap":1,"kb_used_meta":247742,"kb_avail":93798860,"statfs":{"total":96636764160,"available":96050032640,"internally_reserved":0,"allocated":333025280,"data_stored":332842398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":253688261},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,10,1,7,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182440,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":573272,"kb_used_data":325060,"kb_used_omap":4,"kb_used_meta":248187,"kb_avail":93798568,"statfs":{"total":96636764160,"available":96049733632,"internally_reserved":0,"allocated":332861440,"data_stored":332678558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":254144205},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,5,3,0,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":333656064,"data_stored":333656064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":192040960,"data_stored":192040960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":333504512,"data_stored":333504512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":332742656,"data_stored":332742656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:37:10.556 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:37:11.133 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:37:11.133 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:37:11.153 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":141,"stamp":"2024-09-05T16:37:10.850226+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":675061952,"num_objects":82410,"num_object_clones":0,"num_object_copies":329640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82410,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":82437,"num_write_kb":659378,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1934,"num_bytes_recovered":15605760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82464,"ondisk_log_size":82464,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2149044,"kb_used_data":1180144,"kb_used_omap":14,"kb_used_meta":968817,"kb_avail":375338316,"statfs":{"total":386547056640,"available":384346435584,"internally_reserved":0,"allocated":1208467456,"data_stored":1207735928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":992069175},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,7,6,6,12,19,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":33579008,"num_objects":4099,"num_object_clones":0,"num_object_copies":16396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-157,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4099,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4099,"num_write_kb":32792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":157,"num_bytes_recovered":1286144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006726"},"pg_stats":[{"pgid":"1.f","version":"33'5189","reported_seq":5231,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:10.039900+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:37:10.039900+0000","last_peered":"2024-09-05T16:37:10.039900+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:37:10.039900+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5189,"log_dups_size":0,"ondisk_log_size":5189,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42508288,"num_objects":5189,"num_object_clones":0,"num_object_copies":20756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5189,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5189,"num_write_kb":41512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2927}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'5036","reported_seq":5100,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:05.773717+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:37:05.773717+0000","last_peered":"2024-09-05T16:37:05.773717+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:37:05.773717+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5036,"log_dups_size":0,"ondisk_log_size":5036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41219232,"num_objects":5032,"num_object_clones":0,"num_object_copies":20128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5032,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5034,"num_write_kb":40268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2765}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'5154","reported_seq":5229,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:10.770075+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:37:10.770075+0000","last_peered":"2024-09-05T16:37:10.770075+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:37:10.770075+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5154,"log_dups_size":0,"ondisk_log_size":5154,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42195536,"num_objects":5151,"num_object_clones":0,"num_object_copies":20604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5151,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5153,"num_write_kb":41214,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2901}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'5131","reported_seq":9009,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:05.885046+0000","last_change":"2024-09-05T16:34:56.800371+0000","last_active":"2024-09-05T16:37:05.885046+0000","last_peered":"2024-09-05T16:37:05.885046+0000","last_clean":"2024-09-05T16:33:53.148717+0000","last_became_active":"2024-09-05T16:34:56.791755+0000","last_became_peered":"2024-09-05T16:34:56.791755+0000","last_unstale":"2024-09-05T16:37:05.885046+0000","last_undegraded":"2024-09-05T16:34:56.597522+0000","last_fullsized":"2024-09-05T16:34:56.568150+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5131,"log_dups_size":0,"ondisk_log_size":5131,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41997472,"num_objects":5127,"num_object_clones":0,"num_object_copies":20508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":383,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5127,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5129,"num_write_kb":41021,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1909,"num_bytes_recovered":15605760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4744}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'5091","reported_seq":5145,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:09.841416+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:37:09.841416+0000","last_peered":"2024-09-05T16:37:09.841416+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:37:09.841416+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5091,"log_dups_size":0,"ondisk_log_size":5091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41677984,"num_objects":5088,"num_object_clones":0,"num_object_copies":20352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5088,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5089,"num_write_kb":40709,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2926}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'5175","reported_seq":5217,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:10.077101+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:37:10.077101+0000","last_peered":"2024-09-05T16:37:10.077101+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:37:10.077101+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5175,"log_dups_size":0,"ondisk_log_size":5175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42393600,"num_objects":5175,"num_object_clones":0,"num_object_copies":20700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5175,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5175,"num_write_kb":41400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2925}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'4983","reported_seq":5047,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:05.814514+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:37:05.814514+0000","last_peered":"2024-09-05T16:37:05.814514+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:37:05.814514+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4983,"log_dups_size":0,"ondisk_log_size":4983,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40768672,"num_objects":4977,"num_object_clones":0,"num_object_copies":19908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4977,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4981,"num_write_kb":39828,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2830}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'5072","reported_seq":5135,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:05.707098+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:37:05.707098+0000","last_peered":"2024-09-05T16:37:05.707098+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:37:05.707098+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5072,"log_dups_size":0,"ondisk_log_size":5072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41505952,"num_objects":5067,"num_object_clones":0,"num_object_copies":20268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5067,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5070,"num_write_kb":40548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2828}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'5216","reported_seq":5272,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:10.013692+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:37:10.013692+0000","last_peered":"2024-09-05T16:37:10.013692+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:37:10.013692+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5216,"log_dups_size":0,"ondisk_log_size":5216,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42658112,"num_objects":5208,"num_object_clones":0,"num_object_copies":20832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5208,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5213,"num_write_kb":41681,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2947}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'5204","reported_seq":5280,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:10.825114+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:37:10.825114+0000","last_peered":"2024-09-05T16:37:10.825114+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:37:10.825114+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5204,"log_dups_size":0,"ondisk_log_size":5204,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42611872,"num_objects":5202,"num_object_clones":0,"num_object_copies":20808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5202,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5202,"num_write_kb":41614,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2940}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'5277","reported_seq":5331,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:10.062047+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:37:10.062047+0000","last_peered":"2024-09-05T16:37:10.062047+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:37:10.062047+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5277,"log_dups_size":0,"ondisk_log_size":5277,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43183856,"num_objects":5272,"num_object_clones":0,"num_object_copies":21088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5272,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5274,"num_write_kb":42180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2961}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'5115","reported_seq":5151,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:10.098818+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:37:10.098818+0000","last_peered":"2024-09-05T16:37:10.098818+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:37:10.098818+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5115,"log_dups_size":0,"ondisk_log_size":5115,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41892432,"num_objects":5114,"num_object_clones":0,"num_object_copies":20456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5114,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5114,"num_write_kb":40911,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2850}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'5206","reported_seq":5280,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:10.756923+0000","last_change":"2024-09-05T16:34:56.798081+0000","last_active":"2024-09-05T16:37:10.756923+0000","last_peered":"2024-09-05T16:37:10.756923+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:37:10.756923+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5206,"log_dups_size":0,"ondisk_log_size":5206,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42637904,"num_objects":5205,"num_object_clones":0,"num_object_copies":20820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5205,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5205,"num_write_kb":41639,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2911}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'5296","reported_seq":5348,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:10.017197+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:37:10.017197+0000","last_peered":"2024-09-05T16:37:10.017197+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:37:10.017197+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5296,"log_dups_size":0,"ondisk_log_size":5296,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43349152,"num_objects":5292,"num_object_clones":0,"num_object_copies":21168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5294,"num_write_kb":42348,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2935}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'5220","reported_seq":5298,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:10.612866+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:37:10.612866+0000","last_peered":"2024-09-05T16:37:10.612866+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:37:10.612866+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5220,"log_dups_size":0,"ondisk_log_size":5220,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42734752,"num_objects":5217,"num_object_clones":0,"num_object_copies":20868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5217,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5218,"num_write_kb":41741,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2897}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'5099","reported_seq":5163,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:05.741208+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:37:05.741208+0000","last_peered":"2024-09-05T16:37:05.741208+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:37:05.741208+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5099,"log_dups_size":0,"ondisk_log_size":5099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41727136,"num_objects":5094,"num_object_clones":0,"num_object_copies":20376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5094,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5097,"num_write_kb":40764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2857}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":675061952,"num_objects":82410,"num_object_clones":0,"num_object_copies":329640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82410,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":82437,"num_write_kb":659378,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1934,"num_bytes_recovered":15605760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1211002880,"data_stored":1211002880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82464,"ondisk_log_size":82464,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084327,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":578844,"kb_used_data":330572,"kb_used_omap":4,"kb_used_meta":248251,"kb_avail":93792996,"statfs":{"total":96636764160,"available":96044027904,"internally_reserved":0,"allocated":338505728,"data_stored":338322846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":254209747},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,2,4,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986204,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":395620,"kb_used_data":193172,"kb_used_omap":4,"kb_used_meta":202427,"kb_avail":93976220,"statfs":{"total":96636764160,"available":96231649280,"internally_reserved":0,"allocated":197808128,"data_stored":197625246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":207285970},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2,4,8],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673010,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":601308,"kb_used_data":331340,"kb_used_omap":1,"kb_used_meta":269950,"kb_avail":93770532,"statfs":{"total":96636764160,"available":96021024768,"internally_reserved":0,"allocated":339292160,"data_stored":339109278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":276429253},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,2,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182440,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":573272,"kb_used_data":325060,"kb_used_omap":4,"kb_used_meta":248187,"kb_avail":93798568,"statfs":{"total":96636764160,"available":96049733632,"internally_reserved":0,"allocated":332861440,"data_stored":332678558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":254144205},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,5,3,0,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":333656064,"data_stored":333656064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":198598656,"data_stored":198598656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":339779584,"data_stored":339779584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":338968576,"data_stored":338968576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:37:11.154 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-05T16:37:11.154 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'5189", 'reported_seq': 5231, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:10.039900+0000', 'last_change': '2024-09-05T16:34:56.833497+0000', 'last_active': '2024-09-05T16:37:10.039900+0000', 'last_peered': '2024-09-05T16:37:10.039900+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.828289+0000', 'last_became_peered': '2024-09-05T16:34:56.828289+0000', 'last_unstale': '2024-09-05T16:37:10.039900+0000', 'last_undegraded': '2024-09-05T16:34:56.612160+0000', 'last_fullsized': '2024-09-05T16:34:56.557632+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5189, 'log_dups_size': 0, 'ondisk_log_size': 5189, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42508288, 'num_objects': 5189, 'num_object_clones': 0, 'num_object_copies': 20756, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2262, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5189, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5189, 'num_write_kb': 41512, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2927}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:37:11.154 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-05T16:37:11.154 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'5036", 'reported_seq': 5100, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:05.773717+0000', 'last_change': '2024-09-05T16:34:56.825031+0000', 'last_active': '2024-09-05T16:37:05.773717+0000', 'last_peered': '2024-09-05T16:37:05.773717+0000', 'last_clean': '2024-09-05T16:33:53.200762+0000', 'last_became_active': '2024-09-05T16:34:56.820292+0000', 'last_became_peered': '2024-09-05T16:34:56.820292+0000', 'last_unstale': '2024-09-05T16:37:05.773717+0000', 'last_undegraded': '2024-09-05T16:34:56.605776+0000', 'last_fullsized': '2024-09-05T16:34:56.559383+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5036, 'log_dups_size': 0, 'ondisk_log_size': 5036, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41219232, 'num_objects': 5032, 'num_object_clones': 0, 'num_object_copies': 20128, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2269, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5032, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5034, 'num_write_kb': 40268, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2765}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:37:11.154 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:37:11.154 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'5154", 'reported_seq': 5229, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:10.770075+0000', 'last_change': '2024-09-05T16:34:56.867900+0000', 'last_active': '2024-09-05T16:37:10.770075+0000', 'last_peered': '2024-09-05T16:37:10.770075+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:37:10.770075+0000', 'last_undegraded': '2024-09-05T16:34:56.590972+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5154, 'log_dups_size': 0, 'ondisk_log_size': 5154, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42195536, 'num_objects': 5151, 'num_object_clones': 0, 'num_object_copies': 20604, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2251, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5151, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5153, 'num_write_kb': 41214, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2901}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:37:11.154 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-05T16:37:11.154 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'5131", 'reported_seq': 9009, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:05.885046+0000', 'last_change': '2024-09-05T16:34:56.800371+0000', 'last_active': '2024-09-05T16:37:05.885046+0000', 'last_peered': '2024-09-05T16:37:05.885046+0000', 'last_clean': '2024-09-05T16:33:53.148717+0000', 'last_became_active': '2024-09-05T16:34:56.791755+0000', 'last_became_peered': '2024-09-05T16:34:56.791755+0000', 'last_unstale': '2024-09-05T16:37:05.885046+0000', 'last_undegraded': '2024-09-05T16:34:56.597522+0000', 'last_fullsized': '2024-09-05T16:34:56.568150+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5131, 'log_dups_size': 0, 'ondisk_log_size': 5131, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41997472, 'num_objects': 5127, 'num_object_clones': 0, 'num_object_copies': 20508, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 383, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5127, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5129, 'num_write_kb': 41021, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1909, 'num_bytes_recovered': 15605760, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 4744}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:37:11.155 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:37:11.155 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'5091", 'reported_seq': 5145, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:09.841416+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:37:09.841416+0000', 'last_peered': '2024-09-05T16:37:09.841416+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:37:09.841416+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5091, 'log_dups_size': 0, 'ondisk_log_size': 5091, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41677984, 'num_objects': 5088, 'num_object_clones': 0, 'num_object_copies': 20352, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2164, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5088, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5089, 'num_write_kb': 40709, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2926}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:37:11.155 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:37:11.155 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'5175", 'reported_seq': 5217, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:10.077101+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:37:10.077101+0000', 'last_peered': '2024-09-05T16:37:10.077101+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:37:10.077101+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5175, 'log_dups_size': 0, 'ondisk_log_size': 5175, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42393600, 'num_objects': 5175, 'num_object_clones': 0, 'num_object_copies': 20700, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5175, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5175, 'num_write_kb': 41400, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2925}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:37:11.155 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:37:11.155 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'4983", 'reported_seq': 5047, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:05.814514+0000', 'last_change': '2024-09-05T16:34:56.903436+0000', 'last_active': '2024-09-05T16:37:05.814514+0000', 'last_peered': '2024-09-05T16:37:05.814514+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:37:05.814514+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 4983, 'log_dups_size': 0, 'ondisk_log_size': 4983, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40768672, 'num_objects': 4977, 'num_object_clones': 0, 'num_object_copies': 19908, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2149, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4977, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4981, 'num_write_kb': 39828, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2830}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:37:11.155 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-05T16:37:11.155 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'5072", 'reported_seq': 5135, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:05.707098+0000', 'last_change': '2024-09-05T16:34:56.883319+0000', 'last_active': '2024-09-05T16:37:05.707098+0000', 'last_peered': '2024-09-05T16:37:05.707098+0000', 'last_clean': '2024-09-05T16:33:53.505305+0000', 'last_became_active': '2024-09-05T16:34:56.873389+0000', 'last_became_peered': '2024-09-05T16:34:56.873389+0000', 'last_unstale': '2024-09-05T16:37:05.707098+0000', 'last_undegraded': '2024-09-05T16:34:56.602567+0000', 'last_fullsized': '2024-09-05T16:34:56.559958+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5072, 'log_dups_size': 0, 'ondisk_log_size': 5072, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41505952, 'num_objects': 5067, 'num_object_clones': 0, 'num_object_copies': 20268, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2241, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5067, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5070, 'num_write_kb': 40548, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2828}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:37:11.155 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:37:11.155 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'5216", 'reported_seq': 5272, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:10.013692+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:37:10.013692+0000', 'last_peered': '2024-09-05T16:37:10.013692+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:37:10.013692+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5216, 'log_dups_size': 0, 'ondisk_log_size': 5216, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42658112, 'num_objects': 5208, 'num_object_clones': 0, 'num_object_copies': 20832, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2264, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5208, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5213, 'num_write_kb': 41681, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2947}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:37:11.156 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:37:11.156 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'5204", 'reported_seq': 5280, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:10.825114+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:37:10.825114+0000', 'last_peered': '2024-09-05T16:37:10.825114+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:37:10.825114+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5204, 'log_dups_size': 0, 'ondisk_log_size': 5204, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42611872, 'num_objects': 5202, 'num_object_clones': 0, 'num_object_copies': 20808, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2264, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5202, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5202, 'num_write_kb': 41614, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2940}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:37:11.156 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:37:11.156 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'5277", 'reported_seq': 5331, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:10.062047+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:37:10.062047+0000', 'last_peered': '2024-09-05T16:37:10.062047+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:37:10.062047+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5277, 'log_dups_size': 0, 'ondisk_log_size': 5277, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43183856, 'num_objects': 5272, 'num_object_clones': 0, 'num_object_copies': 21088, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2314, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5272, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5274, 'num_write_kb': 42180, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2961}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:37:11.156 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:37:11.156 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'5115", 'reported_seq': 5151, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:10.098818+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:37:10.098818+0000', 'last_peered': '2024-09-05T16:37:10.098818+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:37:10.098818+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5115, 'log_dups_size': 0, 'ondisk_log_size': 5115, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41892432, 'num_objects': 5114, 'num_object_clones': 0, 'num_object_copies': 20456, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2265, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5114, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5114, 'num_write_kb': 40911, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2850}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:37:11.156 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-05T16:37:11.156 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'5206", 'reported_seq': 5280, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:10.756923+0000', 'last_change': '2024-09-05T16:34:56.798081+0000', 'last_active': '2024-09-05T16:37:10.756923+0000', 'last_peered': '2024-09-05T16:37:10.756923+0000', 'last_clean': '2024-09-05T16:33:45.082039+0000', 'last_became_active': '2024-09-05T16:34:56.792386+0000', 'last_became_peered': '2024-09-05T16:34:56.792386+0000', 'last_unstale': '2024-09-05T16:37:10.756923+0000', 'last_undegraded': '2024-09-05T16:34:56.600595+0000', 'last_fullsized': '2024-09-05T16:34:56.558203+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5206, 'log_dups_size': 0, 'ondisk_log_size': 5206, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42637904, 'num_objects': 5205, 'num_object_clones': 0, 'num_object_copies': 20820, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2295, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5205, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5205, 'num_write_kb': 41639, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2911}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:37:11.156 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:37:11.156 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'5296", 'reported_seq': 5348, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:10.017197+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:37:10.017197+0000', 'last_peered': '2024-09-05T16:37:10.017197+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:37:10.017197+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5296, 'log_dups_size': 0, 'ondisk_log_size': 5296, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43349152, 'num_objects': 5292, 'num_object_clones': 0, 'num_object_copies': 21168, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2359, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5292, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5294, 'num_write_kb': 42348, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2935}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:37:11.156 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:37:11.157 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'5220", 'reported_seq': 5298, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:10.612866+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:37:10.612866+0000', 'last_peered': '2024-09-05T16:37:10.612866+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:37:10.612866+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5220, 'log_dups_size': 0, 'ondisk_log_size': 5220, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42734752, 'num_objects': 5217, 'num_object_clones': 0, 'num_object_copies': 20868, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2322, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5217, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5218, 'num_write_kb': 41741, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2897}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:37:11.157 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:37:11.157 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'5099", 'reported_seq': 5163, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:05.741208+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:37:05.741208+0000', 'last_peered': '2024-09-05T16:37:05.741208+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:37:05.741208+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5099, 'log_dups_size': 0, 'ondisk_log_size': 5099, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41727136, 'num_objects': 5094, 'num_object_clones': 0, 'num_object_copies': 20376, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5094, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5097, 'num_write_kb': 40764, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2857}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:37:11.157 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:37:11.517 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 169 20 83140 83120 3.8418 2.09375 0.138875 0.0406511 2024-09-05T16:37:11.871 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:37:11.871 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 34291/329640 objects degraded (10.403%), 16 pgs degraded, 16 pgs undersized","count":32},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":235,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":82410,"data_bytes":675061952,"bytes_used":2200621056,"bytes_avail":384346435584,"bytes_total":386547056640,"degraded_objects":34291,"degraded_total":329640,"degraded_ratio":0.1040256036888727,"recovering_objects_per_sec":13,"recovering_bytes_per_sec":107118,"recovering_keys_per_sec":0,"num_objects_recovered":157,"num_bytes_recovered":1286144,"num_keys_recovered":0,"write_bytes_sec":2796683,"read_op_per_sec":0,"write_op_per_sec":341},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:37:11.913 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:37:11.914 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:37:12.518 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 170 20 83423 83403 3.83221 2.21094 0.0770122 0.0407575 2024-09-05T16:37:12.544 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:37:12.546 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:37:12.570 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":141,"stamp":"2024-09-05T16:37:10.850226+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":675061952,"num_objects":82410,"num_object_clones":0,"num_object_copies":329640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82410,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":82437,"num_write_kb":659378,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1934,"num_bytes_recovered":15605760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82464,"ondisk_log_size":82464,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2149044,"kb_used_data":1180144,"kb_used_omap":14,"kb_used_meta":968817,"kb_avail":375338316,"statfs":{"total":386547056640,"available":384346435584,"internally_reserved":0,"allocated":1208467456,"data_stored":1207735928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":992069175},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,7,6,6,12,19,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":33579008,"num_objects":4099,"num_object_clones":0,"num_object_copies":16396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-157,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4099,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4099,"num_write_kb":32792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":157,"num_bytes_recovered":1286144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006726"},"pg_stats":[{"pgid":"1.f","version":"33'5189","reported_seq":5231,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:10.039900+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:37:10.039900+0000","last_peered":"2024-09-05T16:37:10.039900+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:37:10.039900+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5189,"log_dups_size":0,"ondisk_log_size":5189,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42508288,"num_objects":5189,"num_object_clones":0,"num_object_copies":20756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5189,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5189,"num_write_kb":41512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2927}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'5036","reported_seq":5100,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:05.773717+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:37:05.773717+0000","last_peered":"2024-09-05T16:37:05.773717+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:37:05.773717+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5036,"log_dups_size":0,"ondisk_log_size":5036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41219232,"num_objects":5032,"num_object_clones":0,"num_object_copies":20128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5032,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5034,"num_write_kb":40268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2765}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'5154","reported_seq":5229,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:10.770075+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:37:10.770075+0000","last_peered":"2024-09-05T16:37:10.770075+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:37:10.770075+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5154,"log_dups_size":0,"ondisk_log_size":5154,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42195536,"num_objects":5151,"num_object_clones":0,"num_object_copies":20604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5151,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5153,"num_write_kb":41214,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2901}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'5131","reported_seq":9009,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:05.885046+0000","last_change":"2024-09-05T16:34:56.800371+0000","last_active":"2024-09-05T16:37:05.885046+0000","last_peered":"2024-09-05T16:37:05.885046+0000","last_clean":"2024-09-05T16:33:53.148717+0000","last_became_active":"2024-09-05T16:34:56.791755+0000","last_became_peered":"2024-09-05T16:34:56.791755+0000","last_unstale":"2024-09-05T16:37:05.885046+0000","last_undegraded":"2024-09-05T16:34:56.597522+0000","last_fullsized":"2024-09-05T16:34:56.568150+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5131,"log_dups_size":0,"ondisk_log_size":5131,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41997472,"num_objects":5127,"num_object_clones":0,"num_object_copies":20508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":383,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5127,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5129,"num_write_kb":41021,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1909,"num_bytes_recovered":15605760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4744}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'5091","reported_seq":5145,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:09.841416+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:37:09.841416+0000","last_peered":"2024-09-05T16:37:09.841416+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:37:09.841416+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5091,"log_dups_size":0,"ondisk_log_size":5091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41677984,"num_objects":5088,"num_object_clones":0,"num_object_copies":20352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5088,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5089,"num_write_kb":40709,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2926}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'5175","reported_seq":5217,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:10.077101+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:37:10.077101+0000","last_peered":"2024-09-05T16:37:10.077101+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:37:10.077101+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5175,"log_dups_size":0,"ondisk_log_size":5175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42393600,"num_objects":5175,"num_object_clones":0,"num_object_copies":20700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5175,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5175,"num_write_kb":41400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2925}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'4983","reported_seq":5047,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:05.814514+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:37:05.814514+0000","last_peered":"2024-09-05T16:37:05.814514+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:37:05.814514+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":4983,"log_dups_size":0,"ondisk_log_size":4983,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40768672,"num_objects":4977,"num_object_clones":0,"num_object_copies":19908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4977,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4981,"num_write_kb":39828,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2830}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'5072","reported_seq":5135,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:05.707098+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:37:05.707098+0000","last_peered":"2024-09-05T16:37:05.707098+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:37:05.707098+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5072,"log_dups_size":0,"ondisk_log_size":5072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41505952,"num_objects":5067,"num_object_clones":0,"num_object_copies":20268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5067,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5070,"num_write_kb":40548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2828}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'5216","reported_seq":5272,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:10.013692+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:37:10.013692+0000","last_peered":"2024-09-05T16:37:10.013692+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:37:10.013692+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5216,"log_dups_size":0,"ondisk_log_size":5216,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42658112,"num_objects":5208,"num_object_clones":0,"num_object_copies":20832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5208,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5213,"num_write_kb":41681,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2947}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'5204","reported_seq":5280,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:10.825114+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:37:10.825114+0000","last_peered":"2024-09-05T16:37:10.825114+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:37:10.825114+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5204,"log_dups_size":0,"ondisk_log_size":5204,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42611872,"num_objects":5202,"num_object_clones":0,"num_object_copies":20808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5202,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5202,"num_write_kb":41614,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2940}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'5277","reported_seq":5331,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:10.062047+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:37:10.062047+0000","last_peered":"2024-09-05T16:37:10.062047+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:37:10.062047+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5277,"log_dups_size":0,"ondisk_log_size":5277,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43183856,"num_objects":5272,"num_object_clones":0,"num_object_copies":21088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5272,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5274,"num_write_kb":42180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2961}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'5115","reported_seq":5151,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:10.098818+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:37:10.098818+0000","last_peered":"2024-09-05T16:37:10.098818+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:37:10.098818+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5115,"log_dups_size":0,"ondisk_log_size":5115,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41892432,"num_objects":5114,"num_object_clones":0,"num_object_copies":20456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5114,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5114,"num_write_kb":40911,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2850}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'5206","reported_seq":5280,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:10.756923+0000","last_change":"2024-09-05T16:34:56.798081+0000","last_active":"2024-09-05T16:37:10.756923+0000","last_peered":"2024-09-05T16:37:10.756923+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:37:10.756923+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5206,"log_dups_size":0,"ondisk_log_size":5206,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42637904,"num_objects":5205,"num_object_clones":0,"num_object_copies":20820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5205,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5205,"num_write_kb":41639,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2911}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'5296","reported_seq":5348,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:10.017197+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:37:10.017197+0000","last_peered":"2024-09-05T16:37:10.017197+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:37:10.017197+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5296,"log_dups_size":0,"ondisk_log_size":5296,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43349152,"num_objects":5292,"num_object_clones":0,"num_object_copies":21168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5294,"num_write_kb":42348,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2935}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'5220","reported_seq":5298,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:10.612866+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:37:10.612866+0000","last_peered":"2024-09-05T16:37:10.612866+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:37:10.612866+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5220,"log_dups_size":0,"ondisk_log_size":5220,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42734752,"num_objects":5217,"num_object_clones":0,"num_object_copies":20868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5217,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5218,"num_write_kb":41741,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2897}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'5099","reported_seq":5163,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:05.741208+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:37:05.741208+0000","last_peered":"2024-09-05T16:37:05.741208+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:37:05.741208+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5099,"log_dups_size":0,"ondisk_log_size":5099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41727136,"num_objects":5094,"num_object_clones":0,"num_object_copies":20376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5094,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5097,"num_write_kb":40764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2857}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":675061952,"num_objects":82410,"num_object_clones":0,"num_object_copies":329640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82410,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":82437,"num_write_kb":659378,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1934,"num_bytes_recovered":15605760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1211002880,"data_stored":1211002880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82464,"ondisk_log_size":82464,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084327,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":578844,"kb_used_data":330572,"kb_used_omap":4,"kb_used_meta":248251,"kb_avail":93792996,"statfs":{"total":96636764160,"available":96044027904,"internally_reserved":0,"allocated":338505728,"data_stored":338322846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":254209747},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,2,4,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986204,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":395620,"kb_used_data":193172,"kb_used_omap":4,"kb_used_meta":202427,"kb_avail":93976220,"statfs":{"total":96636764160,"available":96231649280,"internally_reserved":0,"allocated":197808128,"data_stored":197625246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":207285970},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2,4,8],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673010,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":601308,"kb_used_data":331340,"kb_used_omap":1,"kb_used_meta":269950,"kb_avail":93770532,"statfs":{"total":96636764160,"available":96021024768,"internally_reserved":0,"allocated":339292160,"data_stored":339109278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":276429253},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,2,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182440,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":573272,"kb_used_data":325060,"kb_used_omap":4,"kb_used_meta":248187,"kb_avail":93798568,"statfs":{"total":96636764160,"available":96049733632,"internally_reserved":0,"allocated":332861440,"data_stored":332678558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":254144205},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,5,3,0,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":333656064,"data_stored":333656064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":198598656,"data_stored":198598656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":339779584,"data_stored":339779584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":338968576,"data_stored":338968576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:37:13.518 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 171 20 83735 83715 3.82405 2.4375 0.1275 0.0408457 2024-09-05T16:37:14.518 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 172 20 84053 84033 3.81626 2.48438 0.0403104 0.0409318 2024-09-05T16:37:15.518 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 173 20 84359 84339 3.80802 2.39062 0.10198 0.0410155 2024-09-05T16:37:15.573 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:37:16.203 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:37:16.203 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:37:16.231 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":143,"stamp":"2024-09-05T16:37:14.852009+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":678985920,"num_objects":82889,"num_object_clones":0,"num_object_copies":331556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34210,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82889,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":82916,"num_write_kb":663210,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2015,"num_bytes_recovered":16269312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82943,"ondisk_log_size":82943,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2153036,"kb_used_data":1192460,"kb_used_omap":14,"kb_used_meta":960497,"kb_avail":375334324,"statfs":{"total":386547056640,"available":384342347776,"internally_reserved":0,"allocated":1221079040,"data_stored":1220347512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":983549495},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,1,6,10,11,11,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":25124864,"num_objects":3067,"num_object_clones":0,"num_object_copies":12268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3067,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3067,"num_write_kb":24536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":160,"num_bytes_recovered":1310720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006845"},"pg_stats":[{"pgid":"1.f","version":"33'5189","reported_seq":5231,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:10.039900+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:37:10.039900+0000","last_peered":"2024-09-05T16:37:10.039900+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:37:10.039900+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5189,"log_dups_size":0,"ondisk_log_size":5189,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42508288,"num_objects":5189,"num_object_clones":0,"num_object_copies":20756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5189,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5189,"num_write_kb":41512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2927}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'5135","reported_seq":5199,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:10.937308+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:37:10.937308+0000","last_peered":"2024-09-05T16:37:10.937308+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:37:10.937308+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5135,"log_dups_size":0,"ondisk_log_size":5135,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42030240,"num_objects":5131,"num_object_clones":0,"num_object_copies":20524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5131,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5133,"num_write_kb":41060,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2864}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'5154","reported_seq":5229,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:10.770075+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:37:10.770075+0000","last_peered":"2024-09-05T16:37:10.770075+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:37:10.770075+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5154,"log_dups_size":0,"ondisk_log_size":5154,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42195536,"num_objects":5151,"num_object_clones":0,"num_object_copies":20604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5151,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5153,"num_write_kb":41214,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2901}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'5230","reported_seq":9269,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:10.920388+0000","last_change":"2024-09-05T16:34:56.800371+0000","last_active":"2024-09-05T16:37:10.920388+0000","last_peered":"2024-09-05T16:37:10.920388+0000","last_clean":"2024-09-05T16:33:53.148717+0000","last_became_active":"2024-09-05T16:34:56.791755+0000","last_became_peered":"2024-09-05T16:34:56.791755+0000","last_unstale":"2024-09-05T16:37:10.920388+0000","last_undegraded":"2024-09-05T16:34:56.597522+0000","last_fullsized":"2024-09-05T16:34:56.568150+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5230,"log_dups_size":0,"ondisk_log_size":5230,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42808480,"num_objects":5226,"num_object_clones":0,"num_object_copies":20904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":302,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5226,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5228,"num_write_kb":41813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1990,"num_bytes_recovered":16269312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4924}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'5091","reported_seq":5145,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:09.841416+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:37:09.841416+0000","last_peered":"2024-09-05T16:37:09.841416+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:37:09.841416+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5091,"log_dups_size":0,"ondisk_log_size":5091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41677984,"num_objects":5088,"num_object_clones":0,"num_object_copies":20352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5088,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5089,"num_write_kb":40709,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2926}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'5175","reported_seq":5217,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:10.077101+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:37:10.077101+0000","last_peered":"2024-09-05T16:37:10.077101+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:37:10.077101+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5175,"log_dups_size":0,"ondisk_log_size":5175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42393600,"num_objects":5175,"num_object_clones":0,"num_object_copies":20700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5175,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5175,"num_write_kb":41400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2925}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'5083","reported_seq":5148,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:10.818422+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:37:10.818422+0000","last_peered":"2024-09-05T16:37:10.818422+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:37:10.818422+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5083,"log_dups_size":0,"ondisk_log_size":5083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41587872,"num_objects":5077,"num_object_clones":0,"num_object_copies":20308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5077,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5081,"num_write_kb":40628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2930}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'5159","reported_seq":5223,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:10.831239+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:37:10.831239+0000","last_peered":"2024-09-05T16:37:10.831239+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:37:10.831239+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5159,"log_dups_size":0,"ondisk_log_size":5159,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42218656,"num_objects":5154,"num_object_clones":0,"num_object_copies":20616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5154,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5157,"num_write_kb":41244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2915}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'5216","reported_seq":5272,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:10.013692+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:37:10.013692+0000","last_peered":"2024-09-05T16:37:10.013692+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:37:10.013692+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5216,"log_dups_size":0,"ondisk_log_size":5216,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42658112,"num_objects":5208,"num_object_clones":0,"num_object_copies":20832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5208,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5213,"num_write_kb":41681,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2947}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'5204","reported_seq":5280,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:10.825114+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:37:10.825114+0000","last_peered":"2024-09-05T16:37:10.825114+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:37:10.825114+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5204,"log_dups_size":0,"ondisk_log_size":5204,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42611872,"num_objects":5202,"num_object_clones":0,"num_object_copies":20808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5202,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5202,"num_write_kb":41614,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2940}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'5277","reported_seq":5331,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:10.062047+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:37:10.062047+0000","last_peered":"2024-09-05T16:37:10.062047+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:37:10.062047+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5277,"log_dups_size":0,"ondisk_log_size":5277,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43183856,"num_objects":5272,"num_object_clones":0,"num_object_copies":21088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5272,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5274,"num_write_kb":42180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2961}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'5115","reported_seq":5151,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:10.098818+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:37:10.098818+0000","last_peered":"2024-09-05T16:37:10.098818+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:37:10.098818+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5115,"log_dups_size":0,"ondisk_log_size":5115,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41892432,"num_objects":5114,"num_object_clones":0,"num_object_copies":20456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5114,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5114,"num_write_kb":40911,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2850}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'5206","reported_seq":5280,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:10.756923+0000","last_change":"2024-09-05T16:34:56.798081+0000","last_active":"2024-09-05T16:37:10.756923+0000","last_peered":"2024-09-05T16:37:10.756923+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:37:10.756923+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5206,"log_dups_size":0,"ondisk_log_size":5206,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42637904,"num_objects":5205,"num_object_clones":0,"num_object_copies":20820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5205,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5205,"num_write_kb":41639,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2911}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'5296","reported_seq":5348,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:10.017197+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:37:10.017197+0000","last_peered":"2024-09-05T16:37:10.017197+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:37:10.017197+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5296,"log_dups_size":0,"ondisk_log_size":5296,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43349152,"num_objects":5292,"num_object_clones":0,"num_object_copies":21168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5294,"num_write_kb":42348,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2935}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'5220","reported_seq":5298,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:10.612866+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:37:10.612866+0000","last_peered":"2024-09-05T16:37:10.612866+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:37:10.612866+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5220,"log_dups_size":0,"ondisk_log_size":5220,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42734752,"num_objects":5217,"num_object_clones":0,"num_object_copies":20868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5217,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5218,"num_write_kb":41741,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2897}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'5193","reported_seq":5257,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:10.903041+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:37:10.903041+0000","last_peered":"2024-09-05T16:37:10.903041+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:37:10.903041+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5193,"log_dups_size":0,"ondisk_log_size":5193,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42497184,"num_objects":5188,"num_object_clones":0,"num_object_copies":20752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5188,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5191,"num_write_kb":41516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2951}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":678985920,"num_objects":82889,"num_object_clones":0,"num_object_copies":331556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34210,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82889,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":82916,"num_write_kb":663210,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2015,"num_bytes_recovered":16269312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1223647232,"data_stored":1223647232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82943,"ondisk_log_size":82943,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084327,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":578844,"kb_used_data":330572,"kb_used_omap":4,"kb_used_meta":248251,"kb_avail":93792996,"statfs":{"total":96636764160,"available":96044027904,"internally_reserved":0,"allocated":338505728,"data_stored":338322846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":254209747},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,2,4,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986205,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":389368,"kb_used_data":199400,"kb_used_omap":4,"kb_used_meta":189947,"kb_avail":93982472,"statfs":{"total":96636764160,"available":96238051328,"internally_reserved":0,"allocated":204185600,"data_stored":204002718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":194506450},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,1,2,3,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673010,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":601308,"kb_used_data":331340,"kb_used_omap":1,"kb_used_meta":269950,"kb_avail":93770532,"statfs":{"total":96636764160,"available":96021024768,"internally_reserved":0,"allocated":339292160,"data_stored":339109278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":276429253},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,2,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182441,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":583516,"kb_used_data":331148,"kb_used_omap":4,"kb_used_meta":252347,"kb_avail":93788324,"statfs":{"total":96636764160,"available":96039243776,"internally_reserved":0,"allocated":339095552,"data_stored":338912670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":258404045},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,4,3,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":339918848,"data_stored":339918848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":204980224,"data_stored":204980224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":339779584,"data_stored":339779584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":338968576,"data_stored":338968576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:37:16.232 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:37:16.520 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 174 20 84631 84611 3.79829 2.125 0.0927762 0.041123 2024-09-05T16:37:16.864 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:37:16.864 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:37:16.887 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":144,"stamp":"2024-09-05T16:37:16.852685+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":691232960,"num_objects":84384,"num_object_clones":0,"num_object_copies":337536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84384,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":84411,"num_write_kb":675170,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2091,"num_bytes_recovered":16891904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":84438,"ondisk_log_size":84438,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2219320,"kb_used_data":1210168,"kb_used_omap":14,"kb_used_meta":1009073,"kb_avail":375268040,"statfs":{"total":386547056640,"available":384274472960,"internally_reserved":0,"allocated":1239212032,"data_stored":1238480504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1033291319},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[10,0,10,8,8,12,9,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":37371904,"num_objects":4562,"num_object_clones":0,"num_object_copies":18248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4562,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4562,"num_write_kb":36496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":236,"num_bytes_recovered":1933312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006713"},"pg_stats":[{"pgid":"1.f","version":"33'5293","reported_seq":5335,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:15.115464+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:37:15.115464+0000","last_peered":"2024-09-05T16:37:15.115464+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:37:15.115464+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5293,"log_dups_size":0,"ondisk_log_size":5293,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43360256,"num_objects":5293,"num_object_clones":0,"num_object_copies":21172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5293,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5293,"num_write_kb":42344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3031}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'5228","reported_seq":5292,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:15.934989+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:37:15.934989+0000","last_peered":"2024-09-05T16:37:15.934989+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:37:15.934989+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5228,"log_dups_size":0,"ondisk_log_size":5228,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42792096,"num_objects":5224,"num_object_clones":0,"num_object_copies":20896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5224,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5226,"num_write_kb":41804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2957}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'5239","reported_seq":5314,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:15.708451+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:37:15.708451+0000","last_peered":"2024-09-05T16:37:15.708451+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:37:15.708451+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5239,"log_dups_size":0,"ondisk_log_size":5239,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42891856,"num_objects":5236,"num_object_clones":0,"num_object_copies":20944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5238,"num_write_kb":41894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2986}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'5338","reported_seq":9529,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:15.973460+0000","last_change":"2024-09-05T16:34:56.800371+0000","last_active":"2024-09-05T16:37:15.973460+0000","last_peered":"2024-09-05T16:37:15.973460+0000","last_clean":"2024-09-05T16:33:53.148717+0000","last_became_active":"2024-09-05T16:34:56.791755+0000","last_became_peered":"2024-09-05T16:34:56.791755+0000","last_unstale":"2024-09-05T16:37:15.973460+0000","last_undegraded":"2024-09-05T16:34:56.597522+0000","last_fullsized":"2024-09-05T16:34:56.568150+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5338,"log_dups_size":0,"ondisk_log_size":5338,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43693216,"num_objects":5334,"num_object_clones":0,"num_object_copies":21336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5334,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5336,"num_write_kb":42677,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2066,"num_bytes_recovered":16891904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":5108}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'5187","reported_seq":5241,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:15.089518+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:37:15.089518+0000","last_peered":"2024-09-05T16:37:15.089518+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:37:15.089518+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5187,"log_dups_size":0,"ondisk_log_size":5187,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42464416,"num_objects":5184,"num_object_clones":0,"num_object_copies":20736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5184,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5185,"num_write_kb":41477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3022}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'5259","reported_seq":5301,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:14.974047+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:37:14.974047+0000","last_peered":"2024-09-05T16:37:14.974047+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:37:14.974047+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5259,"log_dups_size":0,"ondisk_log_size":5259,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43081728,"num_objects":5259,"num_object_clones":0,"num_object_copies":21036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5259,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5259,"num_write_kb":42072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3009}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'5168","reported_seq":5233,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:15.674587+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:37:15.674587+0000","last_peered":"2024-09-05T16:37:15.674587+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:37:15.674587+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5168,"log_dups_size":0,"ondisk_log_size":5168,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42284192,"num_objects":5162,"num_object_clones":0,"num_object_copies":20648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5162,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5166,"num_write_kb":41308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3015}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'5253","reported_seq":5316,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:15.718859+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:37:15.718859+0000","last_peered":"2024-09-05T16:37:15.718859+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:37:15.718859+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5253,"log_dups_size":0,"ondisk_log_size":5253,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42988704,"num_objects":5248,"num_object_clones":0,"num_object_copies":20992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5248,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5251,"num_write_kb":41996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3009}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'5309","reported_seq":5365,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:14.997196+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:37:14.997196+0000","last_peered":"2024-09-05T16:37:14.997196+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:37:14.997196+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5309,"log_dups_size":0,"ondisk_log_size":5309,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43419968,"num_objects":5301,"num_object_clones":0,"num_object_copies":21204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5301,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5306,"num_write_kb":42425,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3040}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'5305","reported_seq":5382,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:15.826782+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:37:15.826782+0000","last_peered":"2024-09-05T16:37:15.826782+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:37:15.826782+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5305,"log_dups_size":0,"ondisk_log_size":5305,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43439264,"num_objects":5303,"num_object_clones":0,"num_object_copies":21212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5303,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5303,"num_write_kb":42422,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3041}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'5373","reported_seq":5428,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:15.140976+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:37:15.140976+0000","last_peered":"2024-09-05T16:37:15.140976+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:37:15.140976+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5373,"log_dups_size":0,"ondisk_log_size":5373,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43970288,"num_objects":5368,"num_object_clones":0,"num_object_copies":21472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5368,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5370,"num_write_kb":42948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3057}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'5202","reported_seq":5238,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:15.106495+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:37:15.106495+0000","last_peered":"2024-09-05T16:37:15.106495+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:37:15.106495+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5202,"log_dups_size":0,"ondisk_log_size":5202,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42605136,"num_objects":5201,"num_object_clones":0,"num_object_copies":20804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5201,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5201,"num_write_kb":41607,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2937}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'5301","reported_seq":5375,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:15.659004+0000","last_change":"2024-09-05T16:34:56.798081+0000","last_active":"2024-09-05T16:37:15.659004+0000","last_peered":"2024-09-05T16:37:15.659004+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:37:15.659004+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5301,"log_dups_size":0,"ondisk_log_size":5301,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43416144,"num_objects":5300,"num_object_clones":0,"num_object_copies":21200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5300,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5300,"num_write_kb":42399,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3006}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'5398","reported_seq":5450,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:14.973998+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:37:14.973998+0000","last_peered":"2024-09-05T16:37:14.973998+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:37:14.973998+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5398,"log_dups_size":0,"ondisk_log_size":5398,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44184736,"num_objects":5394,"num_object_clones":0,"num_object_copies":21576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5394,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5396,"num_write_kb":43164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3037}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'5319","reported_seq":5397,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:15.624380+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:37:15.624380+0000","last_peered":"2024-09-05T16:37:15.624380+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:37:15.624380+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5319,"log_dups_size":0,"ondisk_log_size":5319,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43545760,"num_objects":5316,"num_object_clones":0,"num_object_copies":21264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5316,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5317,"num_write_kb":42533,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2996}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'5266","reported_seq":5330,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:15.785999+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:37:15.785999+0000","last_peered":"2024-09-05T16:37:15.785999+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:37:15.785999+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5266,"log_dups_size":0,"ondisk_log_size":5266,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43095200,"num_objects":5261,"num_object_clones":0,"num_object_copies":21044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5261,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5264,"num_write_kb":42100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3024}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":691232960,"num_objects":84384,"num_object_clones":0,"num_object_copies":337536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84384,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":84411,"num_write_kb":675170,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2091,"num_bytes_recovered":16891904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1241919488,"data_stored":1241919488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":84438,"ondisk_log_size":84438,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084328,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":606912,"kb_used_data":336432,"kb_used_omap":4,"kb_used_meta":270459,"kb_avail":93764928,"statfs":{"total":96636764160,"available":96015286272,"internally_reserved":0,"allocated":344506368,"data_stored":344323486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":276950739},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,6,3,5,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986205,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":389368,"kb_used_data":199400,"kb_used_omap":4,"kb_used_meta":189947,"kb_avail":93982472,"statfs":{"total":96636764160,"available":96238051328,"internally_reserved":0,"allocated":204185600,"data_stored":204002718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":194506450},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,1,2,3,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673011,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":611396,"kb_used_data":337268,"kb_used_omap":1,"kb_used_meta":274110,"kb_avail":93760444,"statfs":{"total":96636764160,"available":96010694656,"internally_reserved":0,"allocated":345362432,"data_stored":345179550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":280689093},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[7,0,2,1,2,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182442,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":611644,"kb_used_data":337068,"kb_used_omap":4,"kb_used_meta":274555,"kb_avail":93760196,"statfs":{"total":96636764160,"available":96010440704,"internally_reserved":0,"allocated":345157632,"data_stored":344974750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":281145037},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,0,1,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":346034176,"data_stored":346034176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":204980224,"data_stored":204980224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":345858048,"data_stored":345858048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":345047040,"data_stored":345047040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:37:16.888 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-05T16:37:16.889 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'5293", 'reported_seq': 5335, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:15.115464+0000', 'last_change': '2024-09-05T16:34:56.833497+0000', 'last_active': '2024-09-05T16:37:15.115464+0000', 'last_peered': '2024-09-05T16:37:15.115464+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.828289+0000', 'last_became_peered': '2024-09-05T16:34:56.828289+0000', 'last_unstale': '2024-09-05T16:37:15.115464+0000', 'last_undegraded': '2024-09-05T16:34:56.612160+0000', 'last_fullsized': '2024-09-05T16:34:56.557632+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5293, 'log_dups_size': 0, 'ondisk_log_size': 5293, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43360256, 'num_objects': 5293, 'num_object_clones': 0, 'num_object_copies': 21172, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2262, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5293, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5293, 'num_write_kb': 42344, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3031}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:37:16.889 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-05T16:37:16.889 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'5228", 'reported_seq': 5292, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:15.934989+0000', 'last_change': '2024-09-05T16:34:56.825031+0000', 'last_active': '2024-09-05T16:37:15.934989+0000', 'last_peered': '2024-09-05T16:37:15.934989+0000', 'last_clean': '2024-09-05T16:33:53.200762+0000', 'last_became_active': '2024-09-05T16:34:56.820292+0000', 'last_became_peered': '2024-09-05T16:34:56.820292+0000', 'last_unstale': '2024-09-05T16:37:15.934989+0000', 'last_undegraded': '2024-09-05T16:34:56.605776+0000', 'last_fullsized': '2024-09-05T16:34:56.559383+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5228, 'log_dups_size': 0, 'ondisk_log_size': 5228, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42792096, 'num_objects': 5224, 'num_object_clones': 0, 'num_object_copies': 20896, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2269, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5224, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5226, 'num_write_kb': 41804, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2957}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:37:16.889 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:37:16.889 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'5239", 'reported_seq': 5314, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:15.708451+0000', 'last_change': '2024-09-05T16:34:56.867900+0000', 'last_active': '2024-09-05T16:37:15.708451+0000', 'last_peered': '2024-09-05T16:37:15.708451+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:37:15.708451+0000', 'last_undegraded': '2024-09-05T16:34:56.590972+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5239, 'log_dups_size': 0, 'ondisk_log_size': 5239, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42891856, 'num_objects': 5236, 'num_object_clones': 0, 'num_object_copies': 20944, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2251, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5236, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5238, 'num_write_kb': 41894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2986}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:37:16.889 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-05T16:37:16.889 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'5338", 'reported_seq': 9529, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:15.973460+0000', 'last_change': '2024-09-05T16:34:56.800371+0000', 'last_active': '2024-09-05T16:37:15.973460+0000', 'last_peered': '2024-09-05T16:37:15.973460+0000', 'last_clean': '2024-09-05T16:33:53.148717+0000', 'last_became_active': '2024-09-05T16:34:56.791755+0000', 'last_became_peered': '2024-09-05T16:34:56.791755+0000', 'last_unstale': '2024-09-05T16:37:15.973460+0000', 'last_undegraded': '2024-09-05T16:34:56.597522+0000', 'last_fullsized': '2024-09-05T16:34:56.568150+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5338, 'log_dups_size': 0, 'ondisk_log_size': 5338, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43693216, 'num_objects': 5334, 'num_object_clones': 0, 'num_object_copies': 21336, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 226, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5334, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5336, 'num_write_kb': 42677, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2066, 'num_bytes_recovered': 16891904, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 5108}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:37:16.889 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:37:16.889 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'5187", 'reported_seq': 5241, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:15.089518+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:37:15.089518+0000', 'last_peered': '2024-09-05T16:37:15.089518+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:37:15.089518+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5187, 'log_dups_size': 0, 'ondisk_log_size': 5187, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42464416, 'num_objects': 5184, 'num_object_clones': 0, 'num_object_copies': 20736, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2164, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5184, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5185, 'num_write_kb': 41477, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3022}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:37:16.890 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:37:16.890 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'5259", 'reported_seq': 5301, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:14.974047+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:37:14.974047+0000', 'last_peered': '2024-09-05T16:37:14.974047+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:37:14.974047+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5259, 'log_dups_size': 0, 'ondisk_log_size': 5259, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43081728, 'num_objects': 5259, 'num_object_clones': 0, 'num_object_copies': 21036, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5259, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5259, 'num_write_kb': 42072, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3009}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:37:16.890 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:37:16.890 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'5168", 'reported_seq': 5233, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:15.674587+0000', 'last_change': '2024-09-05T16:34:56.903436+0000', 'last_active': '2024-09-05T16:37:15.674587+0000', 'last_peered': '2024-09-05T16:37:15.674587+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:37:15.674587+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5168, 'log_dups_size': 0, 'ondisk_log_size': 5168, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42284192, 'num_objects': 5162, 'num_object_clones': 0, 'num_object_copies': 20648, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2149, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5162, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5166, 'num_write_kb': 41308, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3015}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:37:16.890 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-05T16:37:16.890 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'5253", 'reported_seq': 5316, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:15.718859+0000', 'last_change': '2024-09-05T16:34:56.883319+0000', 'last_active': '2024-09-05T16:37:15.718859+0000', 'last_peered': '2024-09-05T16:37:15.718859+0000', 'last_clean': '2024-09-05T16:33:53.505305+0000', 'last_became_active': '2024-09-05T16:34:56.873389+0000', 'last_became_peered': '2024-09-05T16:34:56.873389+0000', 'last_unstale': '2024-09-05T16:37:15.718859+0000', 'last_undegraded': '2024-09-05T16:34:56.602567+0000', 'last_fullsized': '2024-09-05T16:34:56.559958+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5253, 'log_dups_size': 0, 'ondisk_log_size': 5253, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42988704, 'num_objects': 5248, 'num_object_clones': 0, 'num_object_copies': 20992, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2241, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5248, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5251, 'num_write_kb': 41996, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3009}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:37:16.890 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:37:16.890 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'5309", 'reported_seq': 5365, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:14.997196+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:37:14.997196+0000', 'last_peered': '2024-09-05T16:37:14.997196+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:37:14.997196+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5309, 'log_dups_size': 0, 'ondisk_log_size': 5309, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43419968, 'num_objects': 5301, 'num_object_clones': 0, 'num_object_copies': 21204, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2264, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5301, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5306, 'num_write_kb': 42425, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3040}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:37:16.890 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:37:16.890 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'5305", 'reported_seq': 5382, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:15.826782+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:37:15.826782+0000', 'last_peered': '2024-09-05T16:37:15.826782+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:37:15.826782+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5305, 'log_dups_size': 0, 'ondisk_log_size': 5305, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43439264, 'num_objects': 5303, 'num_object_clones': 0, 'num_object_copies': 21212, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2264, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5303, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5303, 'num_write_kb': 42422, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3041}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:37:16.891 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:37:16.891 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'5373", 'reported_seq': 5428, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:15.140976+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:37:15.140976+0000', 'last_peered': '2024-09-05T16:37:15.140976+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:37:15.140976+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5373, 'log_dups_size': 0, 'ondisk_log_size': 5373, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43970288, 'num_objects': 5368, 'num_object_clones': 0, 'num_object_copies': 21472, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2314, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5368, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5370, 'num_write_kb': 42948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3057}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:37:16.891 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:37:16.891 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'5202", 'reported_seq': 5238, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:15.106495+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:37:15.106495+0000', 'last_peered': '2024-09-05T16:37:15.106495+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:37:15.106495+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5202, 'log_dups_size': 0, 'ondisk_log_size': 5202, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42605136, 'num_objects': 5201, 'num_object_clones': 0, 'num_object_copies': 20804, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2265, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5201, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5201, 'num_write_kb': 41607, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2937}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:37:16.891 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-05T16:37:16.891 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'5301", 'reported_seq': 5375, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:15.659004+0000', 'last_change': '2024-09-05T16:34:56.798081+0000', 'last_active': '2024-09-05T16:37:15.659004+0000', 'last_peered': '2024-09-05T16:37:15.659004+0000', 'last_clean': '2024-09-05T16:33:45.082039+0000', 'last_became_active': '2024-09-05T16:34:56.792386+0000', 'last_became_peered': '2024-09-05T16:34:56.792386+0000', 'last_unstale': '2024-09-05T16:37:15.659004+0000', 'last_undegraded': '2024-09-05T16:34:56.600595+0000', 'last_fullsized': '2024-09-05T16:34:56.558203+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5301, 'log_dups_size': 0, 'ondisk_log_size': 5301, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43416144, 'num_objects': 5300, 'num_object_clones': 0, 'num_object_copies': 21200, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2295, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5300, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5300, 'num_write_kb': 42399, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3006}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:37:16.891 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:37:16.891 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'5398", 'reported_seq': 5450, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:14.973998+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:37:14.973998+0000', 'last_peered': '2024-09-05T16:37:14.973998+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:37:14.973998+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5398, 'log_dups_size': 0, 'ondisk_log_size': 5398, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44184736, 'num_objects': 5394, 'num_object_clones': 0, 'num_object_copies': 21576, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2359, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5394, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5396, 'num_write_kb': 43164, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3037}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:37:16.891 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:37:16.891 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'5319", 'reported_seq': 5397, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:15.624380+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:37:15.624380+0000', 'last_peered': '2024-09-05T16:37:15.624380+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:37:15.624380+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5319, 'log_dups_size': 0, 'ondisk_log_size': 5319, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43545760, 'num_objects': 5316, 'num_object_clones': 0, 'num_object_copies': 21264, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2322, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5316, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5317, 'num_write_kb': 42533, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2996}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:37:16.892 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:37:16.892 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'5266", 'reported_seq': 5330, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:15.785999+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:37:15.785999+0000', 'last_peered': '2024-09-05T16:37:15.785999+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:37:15.785999+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5266, 'log_dups_size': 0, 'ondisk_log_size': 5266, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43095200, 'num_objects': 5261, 'num_object_clones': 0, 'num_object_copies': 21044, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5261, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5264, 'num_write_kb': 42100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3024}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:37:16.892 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:37:17.520 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 175 20 84902 84882 3.78869 2.11719 0.0691276 0.0412272 2024-09-05T16:37:17.646 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:37:17.646 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 34134/337536 objects degraded (10.113%), 16 pgs degraded, 16 pgs undersized","count":32},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":240,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":84384,"data_bytes":691232960,"bytes_used":2272583680,"bytes_avail":384274472960,"bytes_total":386547056640,"degraded_objects":34134,"degraded_total":337536,"degraded_ratio":0.10112699089874858,"recovering_objects_per_sec":19,"recovering_bytes_per_sec":161019,"recovering_keys_per_sec":0,"num_objects_recovered":236,"num_bytes_recovered":1933312,"num_keys_recovered":0,"write_bytes_sec":3112583,"read_op_per_sec":0,"write_op_per_sec":379},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:37:17.673 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:37:17.673 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:37:18.274 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:37:18.275 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:37:18.305 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":144,"stamp":"2024-09-05T16:37:16.852685+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":691232960,"num_objects":84384,"num_object_clones":0,"num_object_copies":337536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84384,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":84411,"num_write_kb":675170,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2091,"num_bytes_recovered":16891904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":84438,"ondisk_log_size":84438,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2219320,"kb_used_data":1210168,"kb_used_omap":14,"kb_used_meta":1009073,"kb_avail":375268040,"statfs":{"total":386547056640,"available":384274472960,"internally_reserved":0,"allocated":1239212032,"data_stored":1238480504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1033291319},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[10,0,10,8,8,12,9,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":37371904,"num_objects":4562,"num_object_clones":0,"num_object_copies":18248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4562,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4562,"num_write_kb":36496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":236,"num_bytes_recovered":1933312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006713"},"pg_stats":[{"pgid":"1.f","version":"33'5293","reported_seq":5335,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:15.115464+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:37:15.115464+0000","last_peered":"2024-09-05T16:37:15.115464+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:37:15.115464+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5293,"log_dups_size":0,"ondisk_log_size":5293,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43360256,"num_objects":5293,"num_object_clones":0,"num_object_copies":21172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5293,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5293,"num_write_kb":42344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3031}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'5228","reported_seq":5292,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:15.934989+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:37:15.934989+0000","last_peered":"2024-09-05T16:37:15.934989+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:37:15.934989+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5228,"log_dups_size":0,"ondisk_log_size":5228,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42792096,"num_objects":5224,"num_object_clones":0,"num_object_copies":20896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5224,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5226,"num_write_kb":41804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2957}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'5239","reported_seq":5314,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:15.708451+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:37:15.708451+0000","last_peered":"2024-09-05T16:37:15.708451+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:37:15.708451+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5239,"log_dups_size":0,"ondisk_log_size":5239,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42891856,"num_objects":5236,"num_object_clones":0,"num_object_copies":20944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5238,"num_write_kb":41894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2986}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'5338","reported_seq":9529,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:15.973460+0000","last_change":"2024-09-05T16:34:56.800371+0000","last_active":"2024-09-05T16:37:15.973460+0000","last_peered":"2024-09-05T16:37:15.973460+0000","last_clean":"2024-09-05T16:33:53.148717+0000","last_became_active":"2024-09-05T16:34:56.791755+0000","last_became_peered":"2024-09-05T16:34:56.791755+0000","last_unstale":"2024-09-05T16:37:15.973460+0000","last_undegraded":"2024-09-05T16:34:56.597522+0000","last_fullsized":"2024-09-05T16:34:56.568150+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5338,"log_dups_size":0,"ondisk_log_size":5338,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43693216,"num_objects":5334,"num_object_clones":0,"num_object_copies":21336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5334,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5336,"num_write_kb":42677,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2066,"num_bytes_recovered":16891904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":5108}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'5187","reported_seq":5241,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:15.089518+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:37:15.089518+0000","last_peered":"2024-09-05T16:37:15.089518+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:37:15.089518+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5187,"log_dups_size":0,"ondisk_log_size":5187,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42464416,"num_objects":5184,"num_object_clones":0,"num_object_copies":20736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5184,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5185,"num_write_kb":41477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3022}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'5259","reported_seq":5301,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:14.974047+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:37:14.974047+0000","last_peered":"2024-09-05T16:37:14.974047+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:37:14.974047+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5259,"log_dups_size":0,"ondisk_log_size":5259,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43081728,"num_objects":5259,"num_object_clones":0,"num_object_copies":21036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5259,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5259,"num_write_kb":42072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3009}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'5168","reported_seq":5233,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:15.674587+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:37:15.674587+0000","last_peered":"2024-09-05T16:37:15.674587+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:37:15.674587+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5168,"log_dups_size":0,"ondisk_log_size":5168,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42284192,"num_objects":5162,"num_object_clones":0,"num_object_copies":20648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5162,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5166,"num_write_kb":41308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3015}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'5253","reported_seq":5316,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:15.718859+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:37:15.718859+0000","last_peered":"2024-09-05T16:37:15.718859+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:37:15.718859+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5253,"log_dups_size":0,"ondisk_log_size":5253,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42988704,"num_objects":5248,"num_object_clones":0,"num_object_copies":20992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5248,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5251,"num_write_kb":41996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3009}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'5309","reported_seq":5365,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:14.997196+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:37:14.997196+0000","last_peered":"2024-09-05T16:37:14.997196+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:37:14.997196+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5309,"log_dups_size":0,"ondisk_log_size":5309,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43419968,"num_objects":5301,"num_object_clones":0,"num_object_copies":21204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5301,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5306,"num_write_kb":42425,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3040}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'5305","reported_seq":5382,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:15.826782+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:37:15.826782+0000","last_peered":"2024-09-05T16:37:15.826782+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:37:15.826782+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5305,"log_dups_size":0,"ondisk_log_size":5305,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43439264,"num_objects":5303,"num_object_clones":0,"num_object_copies":21212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5303,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5303,"num_write_kb":42422,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3041}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'5373","reported_seq":5428,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:15.140976+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:37:15.140976+0000","last_peered":"2024-09-05T16:37:15.140976+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:37:15.140976+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5373,"log_dups_size":0,"ondisk_log_size":5373,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43970288,"num_objects":5368,"num_object_clones":0,"num_object_copies":21472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5368,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5370,"num_write_kb":42948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3057}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'5202","reported_seq":5238,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:15.106495+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:37:15.106495+0000","last_peered":"2024-09-05T16:37:15.106495+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:37:15.106495+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5202,"log_dups_size":0,"ondisk_log_size":5202,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42605136,"num_objects":5201,"num_object_clones":0,"num_object_copies":20804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5201,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5201,"num_write_kb":41607,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2937}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'5301","reported_seq":5375,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:15.659004+0000","last_change":"2024-09-05T16:34:56.798081+0000","last_active":"2024-09-05T16:37:15.659004+0000","last_peered":"2024-09-05T16:37:15.659004+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:37:15.659004+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5301,"log_dups_size":0,"ondisk_log_size":5301,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43416144,"num_objects":5300,"num_object_clones":0,"num_object_copies":21200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5300,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5300,"num_write_kb":42399,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3006}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'5398","reported_seq":5450,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:14.973998+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:37:14.973998+0000","last_peered":"2024-09-05T16:37:14.973998+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:37:14.973998+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5398,"log_dups_size":0,"ondisk_log_size":5398,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44184736,"num_objects":5394,"num_object_clones":0,"num_object_copies":21576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5394,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5396,"num_write_kb":43164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3037}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'5319","reported_seq":5397,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:15.624380+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:37:15.624380+0000","last_peered":"2024-09-05T16:37:15.624380+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:37:15.624380+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5319,"log_dups_size":0,"ondisk_log_size":5319,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43545760,"num_objects":5316,"num_object_clones":0,"num_object_copies":21264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5316,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5317,"num_write_kb":42533,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2996}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'5266","reported_seq":5330,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:15.785999+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:37:15.785999+0000","last_peered":"2024-09-05T16:37:15.785999+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:37:15.785999+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5266,"log_dups_size":0,"ondisk_log_size":5266,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43095200,"num_objects":5261,"num_object_clones":0,"num_object_copies":21044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5261,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5264,"num_write_kb":42100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3024}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":691232960,"num_objects":84384,"num_object_clones":0,"num_object_copies":337536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84384,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":84411,"num_write_kb":675170,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2091,"num_bytes_recovered":16891904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1241919488,"data_stored":1241919488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":84438,"ondisk_log_size":84438,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084328,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":606912,"kb_used_data":336432,"kb_used_omap":4,"kb_used_meta":270459,"kb_avail":93764928,"statfs":{"total":96636764160,"available":96015286272,"internally_reserved":0,"allocated":344506368,"data_stored":344323486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":276950739},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,6,3,5,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986205,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":389368,"kb_used_data":199400,"kb_used_omap":4,"kb_used_meta":189947,"kb_avail":93982472,"statfs":{"total":96636764160,"available":96238051328,"internally_reserved":0,"allocated":204185600,"data_stored":204002718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":194506450},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,1,2,3,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673011,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":611396,"kb_used_data":337268,"kb_used_omap":1,"kb_used_meta":274110,"kb_avail":93760444,"statfs":{"total":96636764160,"available":96010694656,"internally_reserved":0,"allocated":345362432,"data_stored":345179550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":280689093},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[7,0,2,1,2,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182442,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":611644,"kb_used_data":337068,"kb_used_omap":4,"kb_used_meta":274555,"kb_avail":93760196,"statfs":{"total":96636764160,"available":96010440704,"internally_reserved":0,"allocated":345157632,"data_stored":344974750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":281145037},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,0,1,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":346034176,"data_stored":346034176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":204980224,"data_stored":204980224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":345858048,"data_stored":345858048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":345047040,"data_stored":345047040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:37:18.521 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 176 20 85186 85166 3.77977 2.21875 0.0443379 0.0413263 2024-09-05T16:37:19.523 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 177 20 85494 85474 3.77195 2.40625 0.0328597 0.0414098 2024-09-05T16:37:20.525 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 178 20 85808 85788 3.76451 2.45312 0.0422755 0.041493 2024-09-05T16:37:21.307 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:37:21.525 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 179 20 86112 86092 3.75674 2.375 0.0607628 0.0415736 2024-09-05T16:37:21.946 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:37:21.947 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:37:21.984 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":146,"stamp":"2024-09-05T16:37:20.854415+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":699490496,"num_objects":85392,"num_object_clones":0,"num_object_copies":341568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":85392,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":85419,"num_write_kb":683234,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2091,"num_bytes_recovered":16891904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85446,"ondisk_log_size":85446,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2269052,"kb_used_data":1227964,"kb_used_omap":14,"kb_used_meta":1041009,"kb_avail":375218308,"statfs":{"total":386547056640,"available":384223547392,"internally_reserved":0,"allocated":1257435136,"data_stored":1256703608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1065993783},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,1,5,4,7,24,10,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":33013760,"num_objects":4030,"num_object_clones":0,"num_object_copies":16120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-157,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4030,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4030,"num_write_kb":32240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":157,"num_bytes_recovered":1286144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004857"},"pg_stats":[{"pgid":"1.f","version":"33'5394","reported_seq":5436,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:20.219077+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:37:20.219077+0000","last_peered":"2024-09-05T16:37:20.219077+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:37:20.219077+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5394,"log_dups_size":0,"ondisk_log_size":5394,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44187648,"num_objects":5394,"num_object_clones":0,"num_object_copies":21576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5394,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5394,"num_write_kb":43152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3132}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'5228","reported_seq":5292,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:15.934989+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:37:15.934989+0000","last_peered":"2024-09-05T16:37:15.934989+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:37:15.934989+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5228,"log_dups_size":0,"ondisk_log_size":5228,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42792096,"num_objects":5224,"num_object_clones":0,"num_object_copies":20896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5224,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5226,"num_write_kb":41804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2957}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'5340","reported_seq":5415,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:20.623971+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:37:20.623971+0000","last_peered":"2024-09-05T16:37:20.623971+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:37:20.623971+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5340,"log_dups_size":0,"ondisk_log_size":5340,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43719248,"num_objects":5337,"num_object_clones":0,"num_object_copies":21348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5337,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5339,"num_write_kb":42702,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3087}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'5338","reported_seq":9529,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:15.973460+0000","last_change":"2024-09-05T16:34:56.800371+0000","last_active":"2024-09-05T16:37:15.973460+0000","last_peered":"2024-09-05T16:37:15.973460+0000","last_clean":"2024-09-05T16:33:53.148717+0000","last_became_active":"2024-09-05T16:34:56.791755+0000","last_became_peered":"2024-09-05T16:34:56.791755+0000","last_unstale":"2024-09-05T16:37:15.973460+0000","last_undegraded":"2024-09-05T16:34:56.597522+0000","last_fullsized":"2024-09-05T16:34:56.568150+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5338,"log_dups_size":0,"ondisk_log_size":5338,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43693216,"num_objects":5334,"num_object_clones":0,"num_object_copies":21336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5334,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5336,"num_write_kb":42677,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2066,"num_bytes_recovered":16891904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":5108}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'5271","reported_seq":5325,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:20.078889+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:37:20.078889+0000","last_peered":"2024-09-05T16:37:20.078889+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:37:20.078889+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5271,"log_dups_size":0,"ondisk_log_size":5271,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43152544,"num_objects":5268,"num_object_clones":0,"num_object_copies":21072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5268,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5269,"num_write_kb":42149,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3106}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'5344","reported_seq":5386,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:19.918822+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:37:19.918822+0000","last_peered":"2024-09-05T16:37:19.918822+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:37:19.918822+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5344,"log_dups_size":0,"ondisk_log_size":5344,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43778048,"num_objects":5344,"num_object_clones":0,"num_object_copies":21376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5344,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5344,"num_write_kb":42752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3094}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'5168","reported_seq":5233,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:15.674587+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:37:15.674587+0000","last_peered":"2024-09-05T16:37:15.674587+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:37:15.674587+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5168,"log_dups_size":0,"ondisk_log_size":5168,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42284192,"num_objects":5162,"num_object_clones":0,"num_object_copies":20648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5162,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5166,"num_write_kb":41308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3015}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'5253","reported_seq":5316,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:15.718859+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:37:15.718859+0000","last_peered":"2024-09-05T16:37:15.718859+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:37:15.718859+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5253,"log_dups_size":0,"ondisk_log_size":5253,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42988704,"num_objects":5248,"num_object_clones":0,"num_object_copies":20992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5248,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5251,"num_write_kb":41996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3009}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'5392","reported_seq":5448,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:20.161258+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:37:20.161258+0000","last_peered":"2024-09-05T16:37:20.161258+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:37:20.161258+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5392,"log_dups_size":0,"ondisk_log_size":5392,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44099904,"num_objects":5384,"num_object_clones":0,"num_object_copies":21536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5384,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5389,"num_write_kb":43089,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3123}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'5386","reported_seq":5463,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:20.700081+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:37:20.700081+0000","last_peered":"2024-09-05T16:37:20.700081+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:37:20.700081+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5386,"log_dups_size":0,"ondisk_log_size":5386,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44102816,"num_objects":5384,"num_object_clones":0,"num_object_copies":21536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5384,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5384,"num_write_kb":43070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3122}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'5465","reported_seq":5521,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:20.158468+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:37:20.158468+0000","last_peered":"2024-09-05T16:37:20.158468+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:37:20.158468+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5465,"log_dups_size":0,"ondisk_log_size":5465,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44723952,"num_objects":5460,"num_object_clones":0,"num_object_copies":21840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5460,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5462,"num_write_kb":43684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3149}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'5300","reported_seq":5336,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:20.172796+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:37:20.172796+0000","last_peered":"2024-09-05T16:37:20.172796+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:37:20.172796+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5300,"log_dups_size":0,"ondisk_log_size":5300,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43407952,"num_objects":5299,"num_object_clones":0,"num_object_copies":21196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5299,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5299,"num_write_kb":42391,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3035}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'5392","reported_seq":5465,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:20.803484+0000","last_change":"2024-09-05T16:34:56.798081+0000","last_active":"2024-09-05T16:37:20.803484+0000","last_peered":"2024-09-05T16:37:20.803484+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:37:20.803484+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5392,"log_dups_size":0,"ondisk_log_size":5392,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44161616,"num_objects":5391,"num_object_clones":0,"num_object_copies":21564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5391,"num_write_kb":43127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3097}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'5500","reported_seq":5552,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:20.146119+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:37:20.146119+0000","last_peered":"2024-09-05T16:37:20.146119+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:37:20.146119+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5500,"log_dups_size":0,"ondisk_log_size":5500,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45020320,"num_objects":5496,"num_object_clones":0,"num_object_copies":21984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5496,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5498,"num_write_kb":43980,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3139}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'5409","reported_seq":5487,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:20.737581+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:37:20.737581+0000","last_peered":"2024-09-05T16:37:20.737581+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:37:20.737581+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5409,"log_dups_size":0,"ondisk_log_size":5409,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44283040,"num_objects":5406,"num_object_clones":0,"num_object_copies":21624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5406,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5407,"num_write_kb":43253,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3086}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'5266","reported_seq":5330,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:15.785999+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:37:15.785999+0000","last_peered":"2024-09-05T16:37:15.785999+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:37:15.785999+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5266,"log_dups_size":0,"ondisk_log_size":5266,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43095200,"num_objects":5261,"num_object_clones":0,"num_object_copies":21044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5261,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5264,"num_write_kb":42100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3024}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":699490496,"num_objects":85392,"num_object_clones":0,"num_object_copies":341568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":85392,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":85419,"num_write_kb":683234,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2091,"num_bytes_recovered":16891904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1260261376,"data_stored":1260261376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85446,"ondisk_log_size":85446,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084329,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":635016,"kb_used_data":342328,"kb_used_omap":4,"kb_used_meta":292667,"kb_avail":93736824,"statfs":{"total":96636764160,"available":95986507776,"internally_reserved":0,"allocated":350543872,"data_stored":350360990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":299691731},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,0,0,2,12,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986206,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":399632,"kb_used_data":205504,"kb_used_omap":4,"kb_used_meta":194107,"kb_avail":93972208,"statfs":{"total":96636764160,"available":96227540992,"internally_reserved":0,"allocated":210436096,"data_stored":210253214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":198766290},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,0,7,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673012,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":622760,"kb_used_data":343064,"kb_used_omap":1,"kb_used_meta":279678,"kb_avail":93749080,"statfs":{"total":96636764160,"available":95999057920,"internally_reserved":0,"allocated":351297536,"data_stored":351114654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":286390725},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,4,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182442,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":611644,"kb_used_data":337068,"kb_used_omap":4,"kb_used_meta":274555,"kb_avail":93760196,"statfs":{"total":96636764160,"available":96010440704,"internally_reserved":0,"allocated":345157632,"data_stored":344974750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":281145037},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,0,1,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":346034176,"data_stored":346034176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":211251200,"data_stored":211251200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":351846400,"data_stored":351846400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":351129600,"data_stored":351129600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:37:21.985 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:37:22.525 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout:2024-09-05T16:37:22.533602+0000 min lat: 0.00357551 max lat: 2.29705 avg lat: 0.0416778 2024-09-05T16:37:22.525 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-05T16:37:22.525 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 180 20 86393 86373 3.74807 2.19531 0.0966006 0.0416778 2024-09-05T16:37:22.530 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:37:22.532 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:37:22.556 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":146,"stamp":"2024-09-05T16:37:20.854415+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":699490496,"num_objects":85392,"num_object_clones":0,"num_object_copies":341568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":85392,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":85419,"num_write_kb":683234,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2091,"num_bytes_recovered":16891904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85446,"ondisk_log_size":85446,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2269052,"kb_used_data":1227964,"kb_used_omap":14,"kb_used_meta":1041009,"kb_avail":375218308,"statfs":{"total":386547056640,"available":384223547392,"internally_reserved":0,"allocated":1257435136,"data_stored":1256703608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1065993783},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,1,5,4,7,24,10,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":33013760,"num_objects":4030,"num_object_clones":0,"num_object_copies":16120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-157,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4030,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4030,"num_write_kb":32240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":157,"num_bytes_recovered":1286144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004857"},"pg_stats":[{"pgid":"1.f","version":"33'5394","reported_seq":5436,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:20.219077+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:37:20.219077+0000","last_peered":"2024-09-05T16:37:20.219077+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:37:20.219077+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5394,"log_dups_size":0,"ondisk_log_size":5394,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44187648,"num_objects":5394,"num_object_clones":0,"num_object_copies":21576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5394,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5394,"num_write_kb":43152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3132}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'5228","reported_seq":5292,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:15.934989+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:37:15.934989+0000","last_peered":"2024-09-05T16:37:15.934989+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:37:15.934989+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5228,"log_dups_size":0,"ondisk_log_size":5228,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42792096,"num_objects":5224,"num_object_clones":0,"num_object_copies":20896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5224,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5226,"num_write_kb":41804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2957}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'5340","reported_seq":5415,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:20.623971+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:37:20.623971+0000","last_peered":"2024-09-05T16:37:20.623971+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:37:20.623971+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5340,"log_dups_size":0,"ondisk_log_size":5340,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43719248,"num_objects":5337,"num_object_clones":0,"num_object_copies":21348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5337,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5339,"num_write_kb":42702,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3087}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'5338","reported_seq":9529,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:15.973460+0000","last_change":"2024-09-05T16:34:56.800371+0000","last_active":"2024-09-05T16:37:15.973460+0000","last_peered":"2024-09-05T16:37:15.973460+0000","last_clean":"2024-09-05T16:33:53.148717+0000","last_became_active":"2024-09-05T16:34:56.791755+0000","last_became_peered":"2024-09-05T16:34:56.791755+0000","last_unstale":"2024-09-05T16:37:15.973460+0000","last_undegraded":"2024-09-05T16:34:56.597522+0000","last_fullsized":"2024-09-05T16:34:56.568150+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5338,"log_dups_size":0,"ondisk_log_size":5338,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43693216,"num_objects":5334,"num_object_clones":0,"num_object_copies":21336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5334,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5336,"num_write_kb":42677,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2066,"num_bytes_recovered":16891904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":5108}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'5271","reported_seq":5325,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:20.078889+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:37:20.078889+0000","last_peered":"2024-09-05T16:37:20.078889+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:37:20.078889+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5271,"log_dups_size":0,"ondisk_log_size":5271,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43152544,"num_objects":5268,"num_object_clones":0,"num_object_copies":21072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5268,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5269,"num_write_kb":42149,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3106}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'5344","reported_seq":5386,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:19.918822+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:37:19.918822+0000","last_peered":"2024-09-05T16:37:19.918822+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:37:19.918822+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5344,"log_dups_size":0,"ondisk_log_size":5344,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43778048,"num_objects":5344,"num_object_clones":0,"num_object_copies":21376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5344,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5344,"num_write_kb":42752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3094}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'5168","reported_seq":5233,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:15.674587+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:37:15.674587+0000","last_peered":"2024-09-05T16:37:15.674587+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:37:15.674587+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5168,"log_dups_size":0,"ondisk_log_size":5168,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42284192,"num_objects":5162,"num_object_clones":0,"num_object_copies":20648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5162,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5166,"num_write_kb":41308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3015}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'5253","reported_seq":5316,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:15.718859+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:37:15.718859+0000","last_peered":"2024-09-05T16:37:15.718859+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:37:15.718859+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5253,"log_dups_size":0,"ondisk_log_size":5253,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42988704,"num_objects":5248,"num_object_clones":0,"num_object_copies":20992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5248,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5251,"num_write_kb":41996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3009}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'5392","reported_seq":5448,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:20.161258+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:37:20.161258+0000","last_peered":"2024-09-05T16:37:20.161258+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:37:20.161258+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5392,"log_dups_size":0,"ondisk_log_size":5392,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44099904,"num_objects":5384,"num_object_clones":0,"num_object_copies":21536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5384,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5389,"num_write_kb":43089,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3123}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'5386","reported_seq":5463,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:20.700081+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:37:20.700081+0000","last_peered":"2024-09-05T16:37:20.700081+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:37:20.700081+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5386,"log_dups_size":0,"ondisk_log_size":5386,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44102816,"num_objects":5384,"num_object_clones":0,"num_object_copies":21536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5384,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5384,"num_write_kb":43070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3122}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'5465","reported_seq":5521,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:20.158468+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:37:20.158468+0000","last_peered":"2024-09-05T16:37:20.158468+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:37:20.158468+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5465,"log_dups_size":0,"ondisk_log_size":5465,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44723952,"num_objects":5460,"num_object_clones":0,"num_object_copies":21840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5460,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5462,"num_write_kb":43684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3149}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'5300","reported_seq":5336,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:20.172796+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:37:20.172796+0000","last_peered":"2024-09-05T16:37:20.172796+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:37:20.172796+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5300,"log_dups_size":0,"ondisk_log_size":5300,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43407952,"num_objects":5299,"num_object_clones":0,"num_object_copies":21196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5299,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5299,"num_write_kb":42391,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3035}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'5392","reported_seq":5465,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:20.803484+0000","last_change":"2024-09-05T16:34:56.798081+0000","last_active":"2024-09-05T16:37:20.803484+0000","last_peered":"2024-09-05T16:37:20.803484+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:37:20.803484+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5392,"log_dups_size":0,"ondisk_log_size":5392,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44161616,"num_objects":5391,"num_object_clones":0,"num_object_copies":21564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5391,"num_write_kb":43127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3097}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'5500","reported_seq":5552,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:20.146119+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:37:20.146119+0000","last_peered":"2024-09-05T16:37:20.146119+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:37:20.146119+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5500,"log_dups_size":0,"ondisk_log_size":5500,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45020320,"num_objects":5496,"num_object_clones":0,"num_object_copies":21984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5496,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5498,"num_write_kb":43980,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3139}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'5409","reported_seq":5487,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:20.737581+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:37:20.737581+0000","last_peered":"2024-09-05T16:37:20.737581+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:37:20.737581+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5409,"log_dups_size":0,"ondisk_log_size":5409,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44283040,"num_objects":5406,"num_object_clones":0,"num_object_copies":21624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5406,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5407,"num_write_kb":43253,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3086}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'5266","reported_seq":5330,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:15.785999+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:37:15.785999+0000","last_peered":"2024-09-05T16:37:15.785999+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:37:15.785999+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5266,"log_dups_size":0,"ondisk_log_size":5266,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43095200,"num_objects":5261,"num_object_clones":0,"num_object_copies":21044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5261,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5264,"num_write_kb":42100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3024}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":699490496,"num_objects":85392,"num_object_clones":0,"num_object_copies":341568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":85392,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":85419,"num_write_kb":683234,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2091,"num_bytes_recovered":16891904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1260261376,"data_stored":1260261376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85446,"ondisk_log_size":85446,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084329,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":635016,"kb_used_data":342328,"kb_used_omap":4,"kb_used_meta":292667,"kb_avail":93736824,"statfs":{"total":96636764160,"available":95986507776,"internally_reserved":0,"allocated":350543872,"data_stored":350360990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":299691731},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,0,0,2,12,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986206,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":399632,"kb_used_data":205504,"kb_used_omap":4,"kb_used_meta":194107,"kb_avail":93972208,"statfs":{"total":96636764160,"available":96227540992,"internally_reserved":0,"allocated":210436096,"data_stored":210253214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":198766290},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,0,7,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673012,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":622760,"kb_used_data":343064,"kb_used_omap":1,"kb_used_meta":279678,"kb_avail":93749080,"statfs":{"total":96636764160,"available":95999057920,"internally_reserved":0,"allocated":351297536,"data_stored":351114654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":286390725},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,4,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182442,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":611644,"kb_used_data":337068,"kb_used_omap":4,"kb_used_meta":274555,"kb_avail":93760196,"statfs":{"total":96636764160,"available":96010440704,"internally_reserved":0,"allocated":345157632,"data_stored":344974750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":281145037},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,0,1,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":346034176,"data_stored":346034176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":211251200,"data_stored":211251200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":351846400,"data_stored":351846400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":351129600,"data_stored":351129600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:37:22.557 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-05T16:37:22.557 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'5394", 'reported_seq': 5436, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:20.219077+0000', 'last_change': '2024-09-05T16:34:56.833497+0000', 'last_active': '2024-09-05T16:37:20.219077+0000', 'last_peered': '2024-09-05T16:37:20.219077+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.828289+0000', 'last_became_peered': '2024-09-05T16:34:56.828289+0000', 'last_unstale': '2024-09-05T16:37:20.219077+0000', 'last_undegraded': '2024-09-05T16:34:56.612160+0000', 'last_fullsized': '2024-09-05T16:34:56.557632+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5394, 'log_dups_size': 0, 'ondisk_log_size': 5394, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44187648, 'num_objects': 5394, 'num_object_clones': 0, 'num_object_copies': 21576, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2262, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5394, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5394, 'num_write_kb': 43152, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3132}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:37:22.557 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-05T16:37:22.557 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'5228", 'reported_seq': 5292, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:15.934989+0000', 'last_change': '2024-09-05T16:34:56.825031+0000', 'last_active': '2024-09-05T16:37:15.934989+0000', 'last_peered': '2024-09-05T16:37:15.934989+0000', 'last_clean': '2024-09-05T16:33:53.200762+0000', 'last_became_active': '2024-09-05T16:34:56.820292+0000', 'last_became_peered': '2024-09-05T16:34:56.820292+0000', 'last_unstale': '2024-09-05T16:37:15.934989+0000', 'last_undegraded': '2024-09-05T16:34:56.605776+0000', 'last_fullsized': '2024-09-05T16:34:56.559383+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5228, 'log_dups_size': 0, 'ondisk_log_size': 5228, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42792096, 'num_objects': 5224, 'num_object_clones': 0, 'num_object_copies': 20896, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2269, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5224, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5226, 'num_write_kb': 41804, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2957}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:37:22.557 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:37:22.557 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'5340", 'reported_seq': 5415, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:20.623971+0000', 'last_change': '2024-09-05T16:34:56.867900+0000', 'last_active': '2024-09-05T16:37:20.623971+0000', 'last_peered': '2024-09-05T16:37:20.623971+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:37:20.623971+0000', 'last_undegraded': '2024-09-05T16:34:56.590972+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5340, 'log_dups_size': 0, 'ondisk_log_size': 5340, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43719248, 'num_objects': 5337, 'num_object_clones': 0, 'num_object_copies': 21348, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2251, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5337, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5339, 'num_write_kb': 42702, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3087}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:37:22.557 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-05T16:37:22.557 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'5338", 'reported_seq': 9529, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:15.973460+0000', 'last_change': '2024-09-05T16:34:56.800371+0000', 'last_active': '2024-09-05T16:37:15.973460+0000', 'last_peered': '2024-09-05T16:37:15.973460+0000', 'last_clean': '2024-09-05T16:33:53.148717+0000', 'last_became_active': '2024-09-05T16:34:56.791755+0000', 'last_became_peered': '2024-09-05T16:34:56.791755+0000', 'last_unstale': '2024-09-05T16:37:15.973460+0000', 'last_undegraded': '2024-09-05T16:34:56.597522+0000', 'last_fullsized': '2024-09-05T16:34:56.568150+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5338, 'log_dups_size': 0, 'ondisk_log_size': 5338, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43693216, 'num_objects': 5334, 'num_object_clones': 0, 'num_object_copies': 21336, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 226, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5334, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5336, 'num_write_kb': 42677, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2066, 'num_bytes_recovered': 16891904, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 5108}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:37:22.557 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:37:22.558 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'5271", 'reported_seq': 5325, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:20.078889+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:37:20.078889+0000', 'last_peered': '2024-09-05T16:37:20.078889+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:37:20.078889+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5271, 'log_dups_size': 0, 'ondisk_log_size': 5271, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43152544, 'num_objects': 5268, 'num_object_clones': 0, 'num_object_copies': 21072, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2164, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5268, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5269, 'num_write_kb': 42149, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3106}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:37:22.558 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:37:22.558 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'5344", 'reported_seq': 5386, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:19.918822+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:37:19.918822+0000', 'last_peered': '2024-09-05T16:37:19.918822+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:37:19.918822+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5344, 'log_dups_size': 0, 'ondisk_log_size': 5344, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43778048, 'num_objects': 5344, 'num_object_clones': 0, 'num_object_copies': 21376, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5344, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5344, 'num_write_kb': 42752, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3094}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:37:22.558 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:37:22.558 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'5168", 'reported_seq': 5233, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:15.674587+0000', 'last_change': '2024-09-05T16:34:56.903436+0000', 'last_active': '2024-09-05T16:37:15.674587+0000', 'last_peered': '2024-09-05T16:37:15.674587+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:37:15.674587+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5168, 'log_dups_size': 0, 'ondisk_log_size': 5168, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42284192, 'num_objects': 5162, 'num_object_clones': 0, 'num_object_copies': 20648, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2149, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5162, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5166, 'num_write_kb': 41308, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3015}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:37:22.558 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-05T16:37:22.558 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'5253", 'reported_seq': 5316, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:15.718859+0000', 'last_change': '2024-09-05T16:34:56.883319+0000', 'last_active': '2024-09-05T16:37:15.718859+0000', 'last_peered': '2024-09-05T16:37:15.718859+0000', 'last_clean': '2024-09-05T16:33:53.505305+0000', 'last_became_active': '2024-09-05T16:34:56.873389+0000', 'last_became_peered': '2024-09-05T16:34:56.873389+0000', 'last_unstale': '2024-09-05T16:37:15.718859+0000', 'last_undegraded': '2024-09-05T16:34:56.602567+0000', 'last_fullsized': '2024-09-05T16:34:56.559958+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5253, 'log_dups_size': 0, 'ondisk_log_size': 5253, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42988704, 'num_objects': 5248, 'num_object_clones': 0, 'num_object_copies': 20992, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2241, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5248, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5251, 'num_write_kb': 41996, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3009}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:37:22.558 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:37:22.558 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'5392", 'reported_seq': 5448, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:20.161258+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:37:20.161258+0000', 'last_peered': '2024-09-05T16:37:20.161258+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:37:20.161258+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5392, 'log_dups_size': 0, 'ondisk_log_size': 5392, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44099904, 'num_objects': 5384, 'num_object_clones': 0, 'num_object_copies': 21536, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2264, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5384, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5389, 'num_write_kb': 43089, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3123}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:37:22.558 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:37:22.558 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'5386", 'reported_seq': 5463, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:20.700081+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:37:20.700081+0000', 'last_peered': '2024-09-05T16:37:20.700081+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:37:20.700081+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5386, 'log_dups_size': 0, 'ondisk_log_size': 5386, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44102816, 'num_objects': 5384, 'num_object_clones': 0, 'num_object_copies': 21536, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2264, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5384, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5384, 'num_write_kb': 43070, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3122}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:37:22.559 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:37:22.559 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'5465", 'reported_seq': 5521, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:20.158468+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:37:20.158468+0000', 'last_peered': '2024-09-05T16:37:20.158468+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:37:20.158468+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5465, 'log_dups_size': 0, 'ondisk_log_size': 5465, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44723952, 'num_objects': 5460, 'num_object_clones': 0, 'num_object_copies': 21840, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2314, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5460, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5462, 'num_write_kb': 43684, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3149}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:37:22.559 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:37:22.559 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'5300", 'reported_seq': 5336, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:20.172796+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:37:20.172796+0000', 'last_peered': '2024-09-05T16:37:20.172796+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:37:20.172796+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5300, 'log_dups_size': 0, 'ondisk_log_size': 5300, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43407952, 'num_objects': 5299, 'num_object_clones': 0, 'num_object_copies': 21196, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2265, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5299, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5299, 'num_write_kb': 42391, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3035}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:37:22.559 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-05T16:37:22.559 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'5392", 'reported_seq': 5465, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:20.803484+0000', 'last_change': '2024-09-05T16:34:56.798081+0000', 'last_active': '2024-09-05T16:37:20.803484+0000', 'last_peered': '2024-09-05T16:37:20.803484+0000', 'last_clean': '2024-09-05T16:33:45.082039+0000', 'last_became_active': '2024-09-05T16:34:56.792386+0000', 'last_became_peered': '2024-09-05T16:34:56.792386+0000', 'last_unstale': '2024-09-05T16:37:20.803484+0000', 'last_undegraded': '2024-09-05T16:34:56.600595+0000', 'last_fullsized': '2024-09-05T16:34:56.558203+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5392, 'log_dups_size': 0, 'ondisk_log_size': 5392, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44161616, 'num_objects': 5391, 'num_object_clones': 0, 'num_object_copies': 21564, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2295, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5391, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5391, 'num_write_kb': 43127, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3097}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:37:22.560 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:37:22.560 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'5500", 'reported_seq': 5552, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:20.146119+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:37:20.146119+0000', 'last_peered': '2024-09-05T16:37:20.146119+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:37:20.146119+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5500, 'log_dups_size': 0, 'ondisk_log_size': 5500, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45020320, 'num_objects': 5496, 'num_object_clones': 0, 'num_object_copies': 21984, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2359, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5496, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5498, 'num_write_kb': 43980, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3139}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:37:22.560 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:37:22.560 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'5409", 'reported_seq': 5487, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:20.737581+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:37:20.737581+0000', 'last_peered': '2024-09-05T16:37:20.737581+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:37:20.737581+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5409, 'log_dups_size': 0, 'ondisk_log_size': 5409, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44283040, 'num_objects': 5406, 'num_object_clones': 0, 'num_object_copies': 21624, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2322, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5406, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5407, 'num_write_kb': 43253, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3086}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:37:22.560 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:37:22.560 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'5266", 'reported_seq': 5330, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:15.785999+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:37:15.785999+0000', 'last_peered': '2024-09-05T16:37:15.785999+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:37:15.785999+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5266, 'log_dups_size': 0, 'ondisk_log_size': 5266, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43095200, 'num_objects': 5261, 'num_object_clones': 0, 'num_object_copies': 21044, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5261, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5264, 'num_write_kb': 42100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3024}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:37:22.560 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:37:23.265 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:37:23.266 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 34134/341568 objects degraded (9.993%), 16 pgs degraded, 16 pgs undersized","count":32},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":246,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":85392,"data_bytes":699490496,"bytes_used":2323509248,"bytes_avail":384223547392,"bytes_total":386547056640,"degraded_objects":34134,"degraded_total":341568,"degraded_ratio":0.099933249016301295,"recovering_objects_per_sec":13,"recovering_bytes_per_sec":107135,"recovering_keys_per_sec":0,"num_objects_recovered":157,"num_bytes_recovered":1286144,"num_keys_recovered":0,"write_bytes_sec":2750033,"read_op_per_sec":0,"write_op_per_sec":335},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:37:23.289 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:37:23.290 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:37:23.525 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 181 20 86656 86636 3.73871 2.05469 0.0459639 0.0417783 2024-09-05T16:37:23.896 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:37:23.897 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:37:23.940 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":147,"stamp":"2024-09-05T16:37:22.855286+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":703160512,"num_objects":85840,"num_object_clones":0,"num_object_copies":343360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":85840,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":85867,"num_write_kb":686818,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2171,"num_bytes_recovered":17547264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85894,"ondisk_log_size":85894,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2281584,"kb_used_data":1234928,"kb_used_omap":14,"kb_used_meta":1046577,"kb_avail":375205776,"statfs":{"total":386547056640,"available":384210714624,"internally_reserved":0,"allocated":1264566272,"data_stored":1263834744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1071695415},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,1,2,6,11,23,10,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":28098560,"num_objects":3430,"num_object_clones":0,"num_object_copies":13720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-237,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3430,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3430,"num_write_kb":27440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":237,"num_bytes_recovered":1941504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005060"},"pg_stats":[{"pgid":"1.f","version":"33'5394","reported_seq":5436,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:20.219077+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:37:20.219077+0000","last_peered":"2024-09-05T16:37:20.219077+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:37:20.219077+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5394,"log_dups_size":0,"ondisk_log_size":5394,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44187648,"num_objects":5394,"num_object_clones":0,"num_object_copies":21576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5394,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5394,"num_write_kb":43152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3132}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'5324","reported_seq":5388,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:20.900895+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:37:20.900895+0000","last_peered":"2024-09-05T16:37:20.900895+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:37:20.900895+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5324,"log_dups_size":0,"ondisk_log_size":5324,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43578528,"num_objects":5320,"num_object_clones":0,"num_object_copies":21280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5320,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5322,"num_write_kb":42572,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3053}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'5340","reported_seq":5415,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:20.623971+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:37:20.623971+0000","last_peered":"2024-09-05T16:37:20.623971+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:37:20.623971+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5340,"log_dups_size":0,"ondisk_log_size":5340,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43719248,"num_objects":5337,"num_object_clones":0,"num_object_copies":21348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5337,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5339,"num_write_kb":42702,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3087}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'5432","reported_seq":9785,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:20.961949+0000","last_change":"2024-09-05T16:34:56.800371+0000","last_active":"2024-09-05T16:37:20.961949+0000","last_peered":"2024-09-05T16:37:20.961949+0000","last_clean":"2024-09-05T16:33:53.148717+0000","last_became_active":"2024-09-05T16:34:56.791755+0000","last_became_peered":"2024-09-05T16:34:56.791755+0000","last_unstale":"2024-09-05T16:37:20.961949+0000","last_undegraded":"2024-09-05T16:34:56.597522+0000","last_fullsized":"2024-09-05T16:34:56.568150+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5432,"log_dups_size":0,"ondisk_log_size":5432,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44463264,"num_objects":5428,"num_object_clones":0,"num_object_copies":21712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":146,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5428,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5430,"num_write_kb":43429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2146,"num_bytes_recovered":17547264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":5282}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'5271","reported_seq":5325,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:20.078889+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:37:20.078889+0000","last_peered":"2024-09-05T16:37:20.078889+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:37:20.078889+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5271,"log_dups_size":0,"ondisk_log_size":5271,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43152544,"num_objects":5268,"num_object_clones":0,"num_object_copies":21072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5268,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5269,"num_write_kb":42149,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3106}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'5344","reported_seq":5386,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:19.918822+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:37:19.918822+0000","last_peered":"2024-09-05T16:37:19.918822+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:37:19.918822+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5344,"log_dups_size":0,"ondisk_log_size":5344,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43778048,"num_objects":5344,"num_object_clones":0,"num_object_copies":21376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5344,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5344,"num_write_kb":42752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3094}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'5247","reported_seq":5312,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:20.853074+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:37:20.853074+0000","last_peered":"2024-09-05T16:37:20.853074+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:37:20.853074+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5247,"log_dups_size":0,"ondisk_log_size":5247,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42931360,"num_objects":5241,"num_object_clones":0,"num_object_copies":20964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5245,"num_write_kb":41940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3094}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'5330","reported_seq":5394,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:20.874242+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:37:20.874242+0000","last_peered":"2024-09-05T16:37:20.874242+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:37:20.874242+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5330,"log_dups_size":0,"ondisk_log_size":5330,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43619488,"num_objects":5325,"num_object_clones":0,"num_object_copies":21300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5325,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5328,"num_write_kb":42612,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3086}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'5392","reported_seq":5448,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:20.161258+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:37:20.161258+0000","last_peered":"2024-09-05T16:37:20.161258+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:37:20.161258+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5392,"log_dups_size":0,"ondisk_log_size":5392,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44099904,"num_objects":5384,"num_object_clones":0,"num_object_copies":21536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5384,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5389,"num_write_kb":43089,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3123}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'5386","reported_seq":5463,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:20.700081+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:37:20.700081+0000","last_peered":"2024-09-05T16:37:20.700081+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:37:20.700081+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5386,"log_dups_size":0,"ondisk_log_size":5386,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44102816,"num_objects":5384,"num_object_clones":0,"num_object_copies":21536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5384,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5384,"num_write_kb":43070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3122}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'5465","reported_seq":5521,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:20.158468+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:37:20.158468+0000","last_peered":"2024-09-05T16:37:20.158468+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:37:20.158468+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5465,"log_dups_size":0,"ondisk_log_size":5465,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44723952,"num_objects":5460,"num_object_clones":0,"num_object_copies":21840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5460,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5462,"num_write_kb":43684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3149}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'5300","reported_seq":5336,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:20.172796+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:37:20.172796+0000","last_peered":"2024-09-05T16:37:20.172796+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:37:20.172796+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5300,"log_dups_size":0,"ondisk_log_size":5300,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43407952,"num_objects":5299,"num_object_clones":0,"num_object_copies":21196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5299,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5299,"num_write_kb":42391,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3035}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'5392","reported_seq":5465,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:20.803484+0000","last_change":"2024-09-05T16:34:56.798081+0000","last_active":"2024-09-05T16:37:20.803484+0000","last_peered":"2024-09-05T16:37:20.803484+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:37:20.803484+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5392,"log_dups_size":0,"ondisk_log_size":5392,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44161616,"num_objects":5391,"num_object_clones":0,"num_object_copies":21564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5391,"num_write_kb":43127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3097}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'5500","reported_seq":5552,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:20.146119+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:37:20.146119+0000","last_peered":"2024-09-05T16:37:20.146119+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:37:20.146119+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5500,"log_dups_size":0,"ondisk_log_size":5500,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45020320,"num_objects":5496,"num_object_clones":0,"num_object_copies":21984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5496,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5498,"num_write_kb":43980,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3139}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'5409","reported_seq":5487,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:20.737581+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:37:20.737581+0000","last_peered":"2024-09-05T16:37:20.737581+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:37:20.737581+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5409,"log_dups_size":0,"ondisk_log_size":5409,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44283040,"num_objects":5406,"num_object_clones":0,"num_object_copies":21624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5406,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5407,"num_write_kb":43253,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3086}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'5368","reported_seq":5432,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:20.862998+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:37:20.862998+0000","last_peered":"2024-09-05T16:37:20.862998+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:37:20.862998+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5368,"log_dups_size":0,"ondisk_log_size":5368,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43930784,"num_objects":5363,"num_object_clones":0,"num_object_copies":21452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5363,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5366,"num_write_kb":42916,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3126}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":703160512,"num_objects":85840,"num_object_clones":0,"num_object_copies":343360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":85840,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":85867,"num_write_kb":686818,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2171,"num_bytes_recovered":17547264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1266241536,"data_stored":1266241536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85894,"ondisk_log_size":85894,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084329,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":635016,"kb_used_data":342328,"kb_used_omap":4,"kb_used_meta":292667,"kb_avail":93736824,"statfs":{"total":96636764160,"available":95986507776,"internally_reserved":0,"allocated":350543872,"data_stored":350360990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":299691731},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,0,0,2,12,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986206,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":399632,"kb_used_data":205504,"kb_used_omap":4,"kb_used_meta":194107,"kb_avail":93972208,"statfs":{"total":96636764160,"available":96227540992,"internally_reserved":0,"allocated":210436096,"data_stored":210253214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":198766290},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,0,7,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673012,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":622760,"kb_used_data":343064,"kb_used_omap":1,"kb_used_meta":279678,"kb_avail":93749080,"statfs":{"total":96636764160,"available":95999057920,"internally_reserved":0,"allocated":351297536,"data_stored":351114654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":286390725},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,4,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182443,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":624176,"kb_used_data":344032,"kb_used_omap":4,"kb_used_meta":280123,"kb_avail":93747664,"statfs":{"total":96636764160,"available":95997607936,"internally_reserved":0,"allocated":352288768,"data_stored":352105886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":286846669},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,5,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":352014336,"data_stored":352014336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":211251200,"data_stored":211251200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":351846400,"data_stored":351846400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":351129600,"data_stored":351129600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:37:24.526 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 182 20 86944 86924 3.73053 2.25 0.0677696 0.0418691 2024-09-05T16:37:25.526 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 183 20 87250 87230 3.72321 2.39062 0.072746 0.0419542 2024-09-05T16:37:26.525 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 184 20 87550 87530 3.71571 2.34375 0.111671 0.0420369 2024-09-05T16:37:26.942 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:37:27.525 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 185 20 87843 87823 3.708 2.28906 0.0496693 0.0421255 2024-09-05T16:37:27.560 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:37:27.561 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:37:27.589 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":149,"stamp":"2024-09-05T16:37:26.858987+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":715145408,"num_objects":87303,"num_object_clones":0,"num_object_copies":349212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33980,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":87303,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":87330,"num_write_kb":698522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2245,"num_bytes_recovered":18153472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87357,"ondisk_log_size":87357,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2305084,"kb_used_data":1258424,"kb_used_omap":14,"kb_used_meta":1046577,"kb_avail":375182276,"statfs":{"total":386547056640,"available":384186650624,"internally_reserved":0,"allocated":1288626176,"data_stored":1287894648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1071695415},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,2,4,7,8,19,15,10],"upper_bound":256},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":36159488,"num_objects":4414,"num_object_clones":0,"num_object_copies":17656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4414,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4414,"num_write_kb":35312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":230,"num_bytes_recovered":1884160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006977"},"pg_stats":[{"pgid":"1.f","version":"33'5482","reported_seq":5523,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:25.319669+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:37:25.319669+0000","last_peered":"2024-09-05T16:37:25.319669+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:37:25.319669+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5482,"log_dups_size":0,"ondisk_log_size":5482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44908544,"num_objects":5482,"num_object_clones":0,"num_object_copies":21928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5482,"num_write_kb":43856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3220}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'5423","reported_seq":5487,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:25.863478+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:37:25.863478+0000","last_peered":"2024-09-05T16:37:25.863478+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:37:25.863478+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5423,"log_dups_size":0,"ondisk_log_size":5423,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44389536,"num_objects":5419,"num_object_clones":0,"num_object_copies":21676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5419,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5421,"num_write_kb":43364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3152}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'5419","reported_seq":5494,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:25.731501+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:37:25.731501+0000","last_peered":"2024-09-05T16:37:25.731501+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:37:25.731501+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5419,"log_dups_size":0,"ondisk_log_size":5419,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44366416,"num_objects":5416,"num_object_clones":0,"num_object_copies":21664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5416,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5418,"num_write_kb":43334,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3166}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'5531","reported_seq":10032,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:25.969590+0000","last_change":"2024-09-05T16:34:56.800371+0000","last_active":"2024-09-05T16:37:25.969590+0000","last_peered":"2024-09-05T16:37:25.969590+0000","last_clean":"2024-09-05T16:33:53.148717+0000","last_became_active":"2024-09-05T16:34:56.791755+0000","last_became_peered":"2024-09-05T16:34:56.791755+0000","last_unstale":"2024-09-05T16:37:25.969590+0000","last_undegraded":"2024-09-05T16:34:56.597522+0000","last_fullsized":"2024-09-05T16:34:56.568150+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5531,"log_dups_size":0,"ondisk_log_size":5531,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45274272,"num_objects":5527,"num_object_clones":0,"num_object_copies":22108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":72,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5527,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5529,"num_write_kb":44221,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2220,"num_bytes_recovered":18153472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":5455}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'5359","reported_seq":5413,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:25.191849+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:37:25.191849+0000","last_peered":"2024-09-05T16:37:25.191849+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:37:25.191849+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5359,"log_dups_size":0,"ondisk_log_size":5359,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43873440,"num_objects":5356,"num_object_clones":0,"num_object_copies":21424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5356,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5357,"num_write_kb":42853,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3194}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'5442","reported_seq":5484,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:25.269286+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:37:25.269286+0000","last_peered":"2024-09-05T16:37:25.269286+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:37:25.269286+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5442,"log_dups_size":0,"ondisk_log_size":5442,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44580864,"num_objects":5442,"num_object_clones":0,"num_object_copies":21768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5442,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5442,"num_write_kb":43536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3192}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'5340","reported_seq":5405,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:25.930945+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:37:25.930945+0000","last_peered":"2024-09-05T16:37:25.930945+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:37:25.930945+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5340,"log_dups_size":0,"ondisk_log_size":5340,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43693216,"num_objects":5334,"num_object_clones":0,"num_object_copies":21336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5334,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5338,"num_write_kb":42684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3187}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'5435","reported_seq":5499,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:25.929637+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:37:25.929637+0000","last_peered":"2024-09-05T16:37:25.929637+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:37:25.929637+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5435,"log_dups_size":0,"ondisk_log_size":5435,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44479648,"num_objects":5430,"num_object_clones":0,"num_object_copies":21720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5430,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5433,"num_write_kb":43452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3191}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'5484","reported_seq":5540,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:25.195934+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:37:25.195934+0000","last_peered":"2024-09-05T16:37:25.195934+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:37:25.195934+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5484,"log_dups_size":0,"ondisk_log_size":5484,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44853568,"num_objects":5476,"num_object_clones":0,"num_object_copies":21904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5476,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5481,"num_write_kb":43825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3215}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'5473","reported_seq":5550,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:25.808708+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:37:25.808708+0000","last_peered":"2024-09-05T16:37:25.808708+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:37:25.808708+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5473,"log_dups_size":0,"ondisk_log_size":5473,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44815520,"num_objects":5471,"num_object_clones":0,"num_object_copies":21884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5471,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5471,"num_write_kb":43766,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3209}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'5562","reported_seq":5618,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:25.170088+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:37:25.170088+0000","last_peered":"2024-09-05T16:37:25.170088+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:37:25.170088+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5562,"log_dups_size":0,"ondisk_log_size":5562,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45518576,"num_objects":5557,"num_object_clones":0,"num_object_copies":22228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5557,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5559,"num_write_kb":44460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3246}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'5398","reported_seq":5434,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:25.234082+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:37:25.234082+0000","last_peered":"2024-09-05T16:37:25.234082+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:37:25.234082+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5398,"log_dups_size":0,"ondisk_log_size":5398,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44210768,"num_objects":5397,"num_object_clones":0,"num_object_copies":21588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5397,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5397,"num_write_kb":43175,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3133}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'5483","reported_seq":5557,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:25.769452+0000","last_change":"2024-09-05T16:34:56.798081+0000","last_active":"2024-09-05T16:37:25.769452+0000","last_peered":"2024-09-05T16:37:25.769452+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:37:25.769452+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5483,"log_dups_size":0,"ondisk_log_size":5483,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44907088,"num_objects":5482,"num_object_clones":0,"num_object_copies":21928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5482,"num_write_kb":43855,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3188}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'5581","reported_seq":5633,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:25.308215+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:37:25.308215+0000","last_peered":"2024-09-05T16:37:25.308215+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:37:25.308215+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5581,"log_dups_size":0,"ondisk_log_size":5581,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45683872,"num_objects":5577,"num_object_clones":0,"num_object_copies":22308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5577,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5579,"num_write_kb":44628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3220}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'5485","reported_seq":5563,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:25.835817+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:37:25.835817+0000","last_peered":"2024-09-05T16:37:25.835817+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:37:25.835817+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5485,"log_dups_size":0,"ondisk_log_size":5485,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44905632,"num_objects":5482,"num_object_clones":0,"num_object_copies":21928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5483,"num_write_kb":43861,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3162}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'5460","reported_seq":5524,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:25.895634+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:37:25.895634+0000","last_peered":"2024-09-05T16:37:25.895634+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:37:25.895634+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5460,"log_dups_size":0,"ondisk_log_size":5460,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44684448,"num_objects":5455,"num_object_clones":0,"num_object_copies":21820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5455,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5458,"num_write_kb":43652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3218}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":715145408,"num_objects":87303,"num_object_clones":0,"num_object_copies":349212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33980,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":87303,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":87330,"num_write_kb":698522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2245,"num_bytes_recovered":18153472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1290342400,"data_stored":1290342400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87357,"ondisk_log_size":87357,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084330,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":628380,"kb_used_data":348172,"kb_used_omap":4,"kb_used_meta":280187,"kb_avail":93743460,"statfs":{"total":96636764160,"available":95993303040,"internally_reserved":0,"allocated":356528128,"data_stored":356345246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":286912211},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,2,0,0,3,2,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986207,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":409984,"kb_used_data":211692,"kb_used_omap":4,"kb_used_meta":198267,"kb_avail":93961856,"statfs":{"total":96636764160,"available":96216940544,"internally_reserved":0,"allocated":216772608,"data_stored":216589726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":203026130},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,7,0,4,5,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673013,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":632660,"kb_used_data":348804,"kb_used_omap":1,"kb_used_meta":283838,"kb_avail":93739180,"statfs":{"total":96636764160,"available":95988920320,"internally_reserved":0,"allocated":357175296,"data_stored":356992414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":290650565},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,4,6,8,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182444,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":634060,"kb_used_data":349756,"kb_used_omap":4,"kb_used_meta":284283,"kb_avail":93737780,"statfs":{"total":96636764160,"available":95987486720,"internally_reserved":0,"allocated":358150144,"data_stored":357967262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":291106509},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,4,6],"upper_bound":64},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":357949440,"data_stored":357949440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":217501696,"data_stored":217501696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":357797888,"data_stored":357797888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":357093376,"data_stored":357093376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:37:27.590 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:37:28.159 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:37:28.160 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:37:28.180 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":149,"stamp":"2024-09-05T16:37:26.858987+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":715145408,"num_objects":87303,"num_object_clones":0,"num_object_copies":349212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33980,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":87303,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":87330,"num_write_kb":698522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2245,"num_bytes_recovered":18153472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87357,"ondisk_log_size":87357,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2305084,"kb_used_data":1258424,"kb_used_omap":14,"kb_used_meta":1046577,"kb_avail":375182276,"statfs":{"total":386547056640,"available":384186650624,"internally_reserved":0,"allocated":1288626176,"data_stored":1287894648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1071695415},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,2,4,7,8,19,15,10],"upper_bound":256},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":36159488,"num_objects":4414,"num_object_clones":0,"num_object_copies":17656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4414,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4414,"num_write_kb":35312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":230,"num_bytes_recovered":1884160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006977"},"pg_stats":[{"pgid":"1.f","version":"33'5482","reported_seq":5523,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:25.319669+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:37:25.319669+0000","last_peered":"2024-09-05T16:37:25.319669+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:37:25.319669+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5482,"log_dups_size":0,"ondisk_log_size":5482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44908544,"num_objects":5482,"num_object_clones":0,"num_object_copies":21928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5482,"num_write_kb":43856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3220}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'5423","reported_seq":5487,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:25.863478+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:37:25.863478+0000","last_peered":"2024-09-05T16:37:25.863478+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:37:25.863478+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5423,"log_dups_size":0,"ondisk_log_size":5423,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44389536,"num_objects":5419,"num_object_clones":0,"num_object_copies":21676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5419,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5421,"num_write_kb":43364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3152}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'5419","reported_seq":5494,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:25.731501+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:37:25.731501+0000","last_peered":"2024-09-05T16:37:25.731501+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:37:25.731501+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5419,"log_dups_size":0,"ondisk_log_size":5419,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44366416,"num_objects":5416,"num_object_clones":0,"num_object_copies":21664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5416,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5418,"num_write_kb":43334,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3166}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'5531","reported_seq":10032,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:25.969590+0000","last_change":"2024-09-05T16:34:56.800371+0000","last_active":"2024-09-05T16:37:25.969590+0000","last_peered":"2024-09-05T16:37:25.969590+0000","last_clean":"2024-09-05T16:33:53.148717+0000","last_became_active":"2024-09-05T16:34:56.791755+0000","last_became_peered":"2024-09-05T16:34:56.791755+0000","last_unstale":"2024-09-05T16:37:25.969590+0000","last_undegraded":"2024-09-05T16:34:56.597522+0000","last_fullsized":"2024-09-05T16:34:56.568150+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5531,"log_dups_size":0,"ondisk_log_size":5531,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45274272,"num_objects":5527,"num_object_clones":0,"num_object_copies":22108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":72,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5527,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5529,"num_write_kb":44221,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2220,"num_bytes_recovered":18153472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":5455}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'5359","reported_seq":5413,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:25.191849+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:37:25.191849+0000","last_peered":"2024-09-05T16:37:25.191849+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:37:25.191849+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5359,"log_dups_size":0,"ondisk_log_size":5359,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43873440,"num_objects":5356,"num_object_clones":0,"num_object_copies":21424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5356,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5357,"num_write_kb":42853,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3194}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'5442","reported_seq":5484,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:25.269286+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:37:25.269286+0000","last_peered":"2024-09-05T16:37:25.269286+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:37:25.269286+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5442,"log_dups_size":0,"ondisk_log_size":5442,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44580864,"num_objects":5442,"num_object_clones":0,"num_object_copies":21768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5442,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5442,"num_write_kb":43536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3192}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'5340","reported_seq":5405,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:25.930945+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:37:25.930945+0000","last_peered":"2024-09-05T16:37:25.930945+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:37:25.930945+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5340,"log_dups_size":0,"ondisk_log_size":5340,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43693216,"num_objects":5334,"num_object_clones":0,"num_object_copies":21336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5334,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5338,"num_write_kb":42684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3187}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'5435","reported_seq":5499,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:25.929637+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:37:25.929637+0000","last_peered":"2024-09-05T16:37:25.929637+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:37:25.929637+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5435,"log_dups_size":0,"ondisk_log_size":5435,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44479648,"num_objects":5430,"num_object_clones":0,"num_object_copies":21720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5430,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5433,"num_write_kb":43452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3191}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'5484","reported_seq":5540,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:25.195934+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:37:25.195934+0000","last_peered":"2024-09-05T16:37:25.195934+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:37:25.195934+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5484,"log_dups_size":0,"ondisk_log_size":5484,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44853568,"num_objects":5476,"num_object_clones":0,"num_object_copies":21904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5476,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5481,"num_write_kb":43825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3215}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'5473","reported_seq":5550,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:25.808708+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:37:25.808708+0000","last_peered":"2024-09-05T16:37:25.808708+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:37:25.808708+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5473,"log_dups_size":0,"ondisk_log_size":5473,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44815520,"num_objects":5471,"num_object_clones":0,"num_object_copies":21884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5471,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5471,"num_write_kb":43766,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3209}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'5562","reported_seq":5618,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:25.170088+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:37:25.170088+0000","last_peered":"2024-09-05T16:37:25.170088+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:37:25.170088+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5562,"log_dups_size":0,"ondisk_log_size":5562,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45518576,"num_objects":5557,"num_object_clones":0,"num_object_copies":22228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5557,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5559,"num_write_kb":44460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3246}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'5398","reported_seq":5434,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:25.234082+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:37:25.234082+0000","last_peered":"2024-09-05T16:37:25.234082+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:37:25.234082+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5398,"log_dups_size":0,"ondisk_log_size":5398,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44210768,"num_objects":5397,"num_object_clones":0,"num_object_copies":21588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5397,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5397,"num_write_kb":43175,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3133}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'5483","reported_seq":5557,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:25.769452+0000","last_change":"2024-09-05T16:34:56.798081+0000","last_active":"2024-09-05T16:37:25.769452+0000","last_peered":"2024-09-05T16:37:25.769452+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:37:25.769452+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5483,"log_dups_size":0,"ondisk_log_size":5483,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44907088,"num_objects":5482,"num_object_clones":0,"num_object_copies":21928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5482,"num_write_kb":43855,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3188}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'5581","reported_seq":5633,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:25.308215+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:37:25.308215+0000","last_peered":"2024-09-05T16:37:25.308215+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:37:25.308215+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5581,"log_dups_size":0,"ondisk_log_size":5581,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45683872,"num_objects":5577,"num_object_clones":0,"num_object_copies":22308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5577,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5579,"num_write_kb":44628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3220}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'5485","reported_seq":5563,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:25.835817+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:37:25.835817+0000","last_peered":"2024-09-05T16:37:25.835817+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:37:25.835817+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5485,"log_dups_size":0,"ondisk_log_size":5485,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44905632,"num_objects":5482,"num_object_clones":0,"num_object_copies":21928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5483,"num_write_kb":43861,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3162}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'5460","reported_seq":5524,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:25.895634+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:37:25.895634+0000","last_peered":"2024-09-05T16:37:25.895634+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:37:25.895634+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5460,"log_dups_size":0,"ondisk_log_size":5460,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44684448,"num_objects":5455,"num_object_clones":0,"num_object_copies":21820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5455,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5458,"num_write_kb":43652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3218}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":715145408,"num_objects":87303,"num_object_clones":0,"num_object_copies":349212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33980,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":87303,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":87330,"num_write_kb":698522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2245,"num_bytes_recovered":18153472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1290342400,"data_stored":1290342400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87357,"ondisk_log_size":87357,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084330,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":628380,"kb_used_data":348172,"kb_used_omap":4,"kb_used_meta":280187,"kb_avail":93743460,"statfs":{"total":96636764160,"available":95993303040,"internally_reserved":0,"allocated":356528128,"data_stored":356345246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":286912211},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,2,0,0,3,2,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986207,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":409984,"kb_used_data":211692,"kb_used_omap":4,"kb_used_meta":198267,"kb_avail":93961856,"statfs":{"total":96636764160,"available":96216940544,"internally_reserved":0,"allocated":216772608,"data_stored":216589726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":203026130},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,7,0,4,5,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673013,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":632660,"kb_used_data":348804,"kb_used_omap":1,"kb_used_meta":283838,"kb_avail":93739180,"statfs":{"total":96636764160,"available":95988920320,"internally_reserved":0,"allocated":357175296,"data_stored":356992414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":290650565},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,4,6,8,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182444,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":634060,"kb_used_data":349756,"kb_used_omap":4,"kb_used_meta":284283,"kb_avail":93737780,"statfs":{"total":96636764160,"available":95987486720,"internally_reserved":0,"allocated":358150144,"data_stored":357967262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":291106509},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,4,6],"upper_bound":64},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":357949440,"data_stored":357949440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":217501696,"data_stored":217501696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":357797888,"data_stored":357797888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":357093376,"data_stored":357093376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:37:28.182 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-05T16:37:28.183 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'5482", 'reported_seq': 5523, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:25.319669+0000', 'last_change': '2024-09-05T16:34:56.833497+0000', 'last_active': '2024-09-05T16:37:25.319669+0000', 'last_peered': '2024-09-05T16:37:25.319669+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.828289+0000', 'last_became_peered': '2024-09-05T16:34:56.828289+0000', 'last_unstale': '2024-09-05T16:37:25.319669+0000', 'last_undegraded': '2024-09-05T16:34:56.612160+0000', 'last_fullsized': '2024-09-05T16:34:56.557632+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5482, 'log_dups_size': 0, 'ondisk_log_size': 5482, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44908544, 'num_objects': 5482, 'num_object_clones': 0, 'num_object_copies': 21928, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2262, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5482, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5482, 'num_write_kb': 43856, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3220}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:37:28.184 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-05T16:37:28.184 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'5423", 'reported_seq': 5487, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:25.863478+0000', 'last_change': '2024-09-05T16:34:56.825031+0000', 'last_active': '2024-09-05T16:37:25.863478+0000', 'last_peered': '2024-09-05T16:37:25.863478+0000', 'last_clean': '2024-09-05T16:33:53.200762+0000', 'last_became_active': '2024-09-05T16:34:56.820292+0000', 'last_became_peered': '2024-09-05T16:34:56.820292+0000', 'last_unstale': '2024-09-05T16:37:25.863478+0000', 'last_undegraded': '2024-09-05T16:34:56.605776+0000', 'last_fullsized': '2024-09-05T16:34:56.559383+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5423, 'log_dups_size': 0, 'ondisk_log_size': 5423, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44389536, 'num_objects': 5419, 'num_object_clones': 0, 'num_object_copies': 21676, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2269, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5419, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5421, 'num_write_kb': 43364, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3152}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:37:28.184 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:37:28.185 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'5419", 'reported_seq': 5494, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:25.731501+0000', 'last_change': '2024-09-05T16:34:56.867900+0000', 'last_active': '2024-09-05T16:37:25.731501+0000', 'last_peered': '2024-09-05T16:37:25.731501+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:37:25.731501+0000', 'last_undegraded': '2024-09-05T16:34:56.590972+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5419, 'log_dups_size': 0, 'ondisk_log_size': 5419, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44366416, 'num_objects': 5416, 'num_object_clones': 0, 'num_object_copies': 21664, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2251, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5416, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5418, 'num_write_kb': 43334, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3166}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:37:28.185 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-05T16:37:28.185 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'5531", 'reported_seq': 10032, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:25.969590+0000', 'last_change': '2024-09-05T16:34:56.800371+0000', 'last_active': '2024-09-05T16:37:25.969590+0000', 'last_peered': '2024-09-05T16:37:25.969590+0000', 'last_clean': '2024-09-05T16:33:53.148717+0000', 'last_became_active': '2024-09-05T16:34:56.791755+0000', 'last_became_peered': '2024-09-05T16:34:56.791755+0000', 'last_unstale': '2024-09-05T16:37:25.969590+0000', 'last_undegraded': '2024-09-05T16:34:56.597522+0000', 'last_fullsized': '2024-09-05T16:34:56.568150+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5531, 'log_dups_size': 0, 'ondisk_log_size': 5531, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45274272, 'num_objects': 5527, 'num_object_clones': 0, 'num_object_copies': 22108, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 72, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5527, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5529, 'num_write_kb': 44221, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2220, 'num_bytes_recovered': 18153472, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 5455}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:37:28.185 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:37:28.186 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'5359", 'reported_seq': 5413, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:25.191849+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:37:25.191849+0000', 'last_peered': '2024-09-05T16:37:25.191849+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:37:25.191849+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5359, 'log_dups_size': 0, 'ondisk_log_size': 5359, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43873440, 'num_objects': 5356, 'num_object_clones': 0, 'num_object_copies': 21424, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2164, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5356, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5357, 'num_write_kb': 42853, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3194}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:37:28.186 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:37:28.186 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'5442", 'reported_seq': 5484, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:25.269286+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:37:25.269286+0000', 'last_peered': '2024-09-05T16:37:25.269286+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:37:25.269286+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5442, 'log_dups_size': 0, 'ondisk_log_size': 5442, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44580864, 'num_objects': 5442, 'num_object_clones': 0, 'num_object_copies': 21768, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5442, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5442, 'num_write_kb': 43536, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3192}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:37:28.186 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:37:28.186 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'5340", 'reported_seq': 5405, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:25.930945+0000', 'last_change': '2024-09-05T16:34:56.903436+0000', 'last_active': '2024-09-05T16:37:25.930945+0000', 'last_peered': '2024-09-05T16:37:25.930945+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:37:25.930945+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5340, 'log_dups_size': 0, 'ondisk_log_size': 5340, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43693216, 'num_objects': 5334, 'num_object_clones': 0, 'num_object_copies': 21336, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2149, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5334, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5338, 'num_write_kb': 42684, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3187}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:37:28.187 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-05T16:37:28.187 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'5435", 'reported_seq': 5499, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:25.929637+0000', 'last_change': '2024-09-05T16:34:56.883319+0000', 'last_active': '2024-09-05T16:37:25.929637+0000', 'last_peered': '2024-09-05T16:37:25.929637+0000', 'last_clean': '2024-09-05T16:33:53.505305+0000', 'last_became_active': '2024-09-05T16:34:56.873389+0000', 'last_became_peered': '2024-09-05T16:34:56.873389+0000', 'last_unstale': '2024-09-05T16:37:25.929637+0000', 'last_undegraded': '2024-09-05T16:34:56.602567+0000', 'last_fullsized': '2024-09-05T16:34:56.559958+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5435, 'log_dups_size': 0, 'ondisk_log_size': 5435, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44479648, 'num_objects': 5430, 'num_object_clones': 0, 'num_object_copies': 21720, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2241, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5430, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5433, 'num_write_kb': 43452, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3191}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:37:28.187 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:37:28.187 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'5484", 'reported_seq': 5540, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:25.195934+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:37:25.195934+0000', 'last_peered': '2024-09-05T16:37:25.195934+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:37:25.195934+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5484, 'log_dups_size': 0, 'ondisk_log_size': 5484, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44853568, 'num_objects': 5476, 'num_object_clones': 0, 'num_object_copies': 21904, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2264, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5476, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5481, 'num_write_kb': 43825, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3215}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:37:28.188 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:37:28.188 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'5473", 'reported_seq': 5550, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:25.808708+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:37:25.808708+0000', 'last_peered': '2024-09-05T16:37:25.808708+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:37:25.808708+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5473, 'log_dups_size': 0, 'ondisk_log_size': 5473, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44815520, 'num_objects': 5471, 'num_object_clones': 0, 'num_object_copies': 21884, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2264, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5471, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5471, 'num_write_kb': 43766, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3209}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:37:28.188 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:37:28.188 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'5562", 'reported_seq': 5618, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:25.170088+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:37:25.170088+0000', 'last_peered': '2024-09-05T16:37:25.170088+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:37:25.170088+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5562, 'log_dups_size': 0, 'ondisk_log_size': 5562, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45518576, 'num_objects': 5557, 'num_object_clones': 0, 'num_object_copies': 22228, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2314, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5557, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5559, 'num_write_kb': 44460, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3246}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:37:28.188 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:37:28.189 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'5398", 'reported_seq': 5434, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:25.234082+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:37:25.234082+0000', 'last_peered': '2024-09-05T16:37:25.234082+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:37:25.234082+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5398, 'log_dups_size': 0, 'ondisk_log_size': 5398, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44210768, 'num_objects': 5397, 'num_object_clones': 0, 'num_object_copies': 21588, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2265, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5397, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5397, 'num_write_kb': 43175, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3133}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:37:28.189 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-05T16:37:28.189 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'5483", 'reported_seq': 5557, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:25.769452+0000', 'last_change': '2024-09-05T16:34:56.798081+0000', 'last_active': '2024-09-05T16:37:25.769452+0000', 'last_peered': '2024-09-05T16:37:25.769452+0000', 'last_clean': '2024-09-05T16:33:45.082039+0000', 'last_became_active': '2024-09-05T16:34:56.792386+0000', 'last_became_peered': '2024-09-05T16:34:56.792386+0000', 'last_unstale': '2024-09-05T16:37:25.769452+0000', 'last_undegraded': '2024-09-05T16:34:56.600595+0000', 'last_fullsized': '2024-09-05T16:34:56.558203+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5483, 'log_dups_size': 0, 'ondisk_log_size': 5483, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44907088, 'num_objects': 5482, 'num_object_clones': 0, 'num_object_copies': 21928, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2295, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5482, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5482, 'num_write_kb': 43855, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3188}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:37:28.189 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:37:28.190 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'5581", 'reported_seq': 5633, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:25.308215+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:37:25.308215+0000', 'last_peered': '2024-09-05T16:37:25.308215+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:37:25.308215+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5581, 'log_dups_size': 0, 'ondisk_log_size': 5581, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45683872, 'num_objects': 5577, 'num_object_clones': 0, 'num_object_copies': 22308, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2359, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5577, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5579, 'num_write_kb': 44628, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3220}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:37:28.190 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:37:28.190 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'5485", 'reported_seq': 5563, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:25.835817+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:37:25.835817+0000', 'last_peered': '2024-09-05T16:37:25.835817+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:37:25.835817+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5485, 'log_dups_size': 0, 'ondisk_log_size': 5485, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44905632, 'num_objects': 5482, 'num_object_clones': 0, 'num_object_copies': 21928, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2322, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5482, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5483, 'num_write_kb': 43861, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3162}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:37:28.190 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:37:28.190 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'5460", 'reported_seq': 5524, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:25.895634+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:37:25.895634+0000', 'last_peered': '2024-09-05T16:37:25.895634+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:37:25.895634+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5460, 'log_dups_size': 0, 'ondisk_log_size': 5460, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44684448, 'num_objects': 5455, 'num_object_clones': 0, 'num_object_copies': 21820, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5455, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5458, 'num_write_kb': 43652, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3218}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:37:28.191 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:37:28.526 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 186 20 88113 88093 3.69941 2.10938 0.0783524 0.0422223 2024-09-05T16:37:28.915 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:37:28.915 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 33980/349212 objects degraded (9.730%), 16 pgs degraded, 16 pgs undersized","count":32},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":252,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":87303,"data_bytes":715145408,"bytes_used":2360406016,"bytes_avail":384186650624,"bytes_total":386547056640,"degraded_objects":33980,"degraded_total":349212,"degraded_ratio":0.097304789067958725,"recovering_objects_per_sec":19,"recovering_bytes_per_sec":156922,"recovering_keys_per_sec":0,"num_objects_recovered":230,"num_bytes_recovered":1884160,"num_keys_recovered":0,"write_bytes_sec":3011539,"read_op_per_sec":0,"write_op_per_sec":367},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:37:28.946 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:37:28.946 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:37:29.526 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 187 20 88381 88361 3.69082 2.09375 0.161454 0.0423196 2024-09-05T16:37:29.574 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:37:29.574 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:37:29.601 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":150,"stamp":"2024-09-05T16:37:28.859796+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":715145408,"num_objects":87303,"num_object_clones":0,"num_object_copies":349212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33980,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":87303,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":87330,"num_write_kb":698522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2245,"num_bytes_recovered":18153472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87357,"ondisk_log_size":87357,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2305084,"kb_used_data":1258424,"kb_used_omap":14,"kb_used_meta":1046577,"kb_avail":375182276,"statfs":{"total":386547056640,"available":384186650624,"internally_reserved":0,"allocated":1288626176,"data_stored":1287894648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1071695415},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,2,4,7,8,19,15,10],"upper_bound":256},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":23912448,"num_objects":2919,"num_object_clones":0,"num_object_copies":11676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2919,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2919,"num_write_kb":23352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":154,"num_bytes_recovered":1261568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007111"},"pg_stats":[{"pgid":"1.f","version":"33'5482","reported_seq":5523,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:25.319669+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:37:25.319669+0000","last_peered":"2024-09-05T16:37:25.319669+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:37:25.319669+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5482,"log_dups_size":0,"ondisk_log_size":5482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44908544,"num_objects":5482,"num_object_clones":0,"num_object_copies":21928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5482,"num_write_kb":43856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3220}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'5423","reported_seq":5487,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:25.863478+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:37:25.863478+0000","last_peered":"2024-09-05T16:37:25.863478+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:37:25.863478+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5423,"log_dups_size":0,"ondisk_log_size":5423,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44389536,"num_objects":5419,"num_object_clones":0,"num_object_copies":21676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5419,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5421,"num_write_kb":43364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3152}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'5419","reported_seq":5494,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:25.731501+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:37:25.731501+0000","last_peered":"2024-09-05T16:37:25.731501+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:37:25.731501+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5419,"log_dups_size":0,"ondisk_log_size":5419,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44366416,"num_objects":5416,"num_object_clones":0,"num_object_copies":21664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5416,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5418,"num_write_kb":43334,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3166}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'5531","reported_seq":10032,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:25.969590+0000","last_change":"2024-09-05T16:34:56.800371+0000","last_active":"2024-09-05T16:37:25.969590+0000","last_peered":"2024-09-05T16:37:25.969590+0000","last_clean":"2024-09-05T16:33:53.148717+0000","last_became_active":"2024-09-05T16:34:56.791755+0000","last_became_peered":"2024-09-05T16:34:56.791755+0000","last_unstale":"2024-09-05T16:37:25.969590+0000","last_undegraded":"2024-09-05T16:34:56.597522+0000","last_fullsized":"2024-09-05T16:34:56.568150+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5531,"log_dups_size":0,"ondisk_log_size":5531,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45274272,"num_objects":5527,"num_object_clones":0,"num_object_copies":22108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":72,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5527,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5529,"num_write_kb":44221,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2220,"num_bytes_recovered":18153472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":5455}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'5359","reported_seq":5413,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:25.191849+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:37:25.191849+0000","last_peered":"2024-09-05T16:37:25.191849+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:37:25.191849+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5359,"log_dups_size":0,"ondisk_log_size":5359,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43873440,"num_objects":5356,"num_object_clones":0,"num_object_copies":21424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5356,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5357,"num_write_kb":42853,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3194}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'5442","reported_seq":5484,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:25.269286+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:37:25.269286+0000","last_peered":"2024-09-05T16:37:25.269286+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:37:25.269286+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5442,"log_dups_size":0,"ondisk_log_size":5442,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44580864,"num_objects":5442,"num_object_clones":0,"num_object_copies":21768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5442,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5442,"num_write_kb":43536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3192}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'5340","reported_seq":5405,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:25.930945+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:37:25.930945+0000","last_peered":"2024-09-05T16:37:25.930945+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:37:25.930945+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5340,"log_dups_size":0,"ondisk_log_size":5340,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43693216,"num_objects":5334,"num_object_clones":0,"num_object_copies":21336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5334,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5338,"num_write_kb":42684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3187}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'5435","reported_seq":5499,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:25.929637+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:37:25.929637+0000","last_peered":"2024-09-05T16:37:25.929637+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:37:25.929637+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5435,"log_dups_size":0,"ondisk_log_size":5435,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44479648,"num_objects":5430,"num_object_clones":0,"num_object_copies":21720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5430,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5433,"num_write_kb":43452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3191}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'5484","reported_seq":5540,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:25.195934+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:37:25.195934+0000","last_peered":"2024-09-05T16:37:25.195934+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:37:25.195934+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5484,"log_dups_size":0,"ondisk_log_size":5484,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44853568,"num_objects":5476,"num_object_clones":0,"num_object_copies":21904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5476,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5481,"num_write_kb":43825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3215}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'5473","reported_seq":5550,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:25.808708+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:37:25.808708+0000","last_peered":"2024-09-05T16:37:25.808708+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:37:25.808708+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5473,"log_dups_size":0,"ondisk_log_size":5473,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44815520,"num_objects":5471,"num_object_clones":0,"num_object_copies":21884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5471,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5471,"num_write_kb":43766,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3209}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'5562","reported_seq":5618,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:25.170088+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:37:25.170088+0000","last_peered":"2024-09-05T16:37:25.170088+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:37:25.170088+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5562,"log_dups_size":0,"ondisk_log_size":5562,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45518576,"num_objects":5557,"num_object_clones":0,"num_object_copies":22228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5557,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5559,"num_write_kb":44460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3246}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'5398","reported_seq":5434,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:25.234082+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:37:25.234082+0000","last_peered":"2024-09-05T16:37:25.234082+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:37:25.234082+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5398,"log_dups_size":0,"ondisk_log_size":5398,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44210768,"num_objects":5397,"num_object_clones":0,"num_object_copies":21588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5397,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5397,"num_write_kb":43175,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3133}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'5483","reported_seq":5557,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:25.769452+0000","last_change":"2024-09-05T16:34:56.798081+0000","last_active":"2024-09-05T16:37:25.769452+0000","last_peered":"2024-09-05T16:37:25.769452+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:37:25.769452+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5483,"log_dups_size":0,"ondisk_log_size":5483,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44907088,"num_objects":5482,"num_object_clones":0,"num_object_copies":21928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5482,"num_write_kb":43855,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3188}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'5581","reported_seq":5633,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:25.308215+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:37:25.308215+0000","last_peered":"2024-09-05T16:37:25.308215+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:37:25.308215+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5581,"log_dups_size":0,"ondisk_log_size":5581,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45683872,"num_objects":5577,"num_object_clones":0,"num_object_copies":22308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5577,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5579,"num_write_kb":44628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3220}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'5485","reported_seq":5563,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:25.835817+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:37:25.835817+0000","last_peered":"2024-09-05T16:37:25.835817+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:37:25.835817+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5485,"log_dups_size":0,"ondisk_log_size":5485,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44905632,"num_objects":5482,"num_object_clones":0,"num_object_copies":21928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5483,"num_write_kb":43861,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3162}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'5460","reported_seq":5524,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:25.895634+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:37:25.895634+0000","last_peered":"2024-09-05T16:37:25.895634+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:37:25.895634+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5460,"log_dups_size":0,"ondisk_log_size":5460,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44684448,"num_objects":5455,"num_object_clones":0,"num_object_copies":21820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5455,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5458,"num_write_kb":43652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3218}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":715145408,"num_objects":87303,"num_object_clones":0,"num_object_copies":349212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33980,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":87303,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":87330,"num_write_kb":698522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2245,"num_bytes_recovered":18153472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1290342400,"data_stored":1290342400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87357,"ondisk_log_size":87357,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084330,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":628380,"kb_used_data":348172,"kb_used_omap":4,"kb_used_meta":280187,"kb_avail":93743460,"statfs":{"total":96636764160,"available":95993303040,"internally_reserved":0,"allocated":356528128,"data_stored":356345246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":286912211},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,2,0,0,3,2,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986207,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":409984,"kb_used_data":211692,"kb_used_omap":4,"kb_used_meta":198267,"kb_avail":93961856,"statfs":{"total":96636764160,"available":96216940544,"internally_reserved":0,"allocated":216772608,"data_stored":216589726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":203026130},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,7,0,4,5,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673013,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":632660,"kb_used_data":348804,"kb_used_omap":1,"kb_used_meta":283838,"kb_avail":93739180,"statfs":{"total":96636764160,"available":95988920320,"internally_reserved":0,"allocated":357175296,"data_stored":356992414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":290650565},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,4,6,8,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182444,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":634060,"kb_used_data":349756,"kb_used_omap":4,"kb_used_meta":284283,"kb_avail":93737780,"statfs":{"total":96636764160,"available":95987486720,"internally_reserved":0,"allocated":358150144,"data_stored":357967262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":291106509},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,4,6],"upper_bound":64},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":357949440,"data_stored":357949440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":217501696,"data_stored":217501696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":357797888,"data_stored":357797888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":357093376,"data_stored":357093376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:37:30.526 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 188 20 88679 88659 3.68357 2.32812 0.0520613 0.042404 2024-09-05T16:37:31.526 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 189 20 88949 88929 3.67524 2.10938 0.0201498 0.0424489 2024-09-05T16:37:32.526 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 190 20 89152 89132 3.66425 1.58594 0.0635633 0.0426241 2024-09-05T16:37:32.603 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:37:33.216 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:37:33.216 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:37:33.228 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":154,"stamp":"2024-09-05T16:37:32.861160+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":725999808,"num_objects":88628,"num_object_clones":0,"num_object_copies":354512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":88628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":88655,"num_write_kb":709122,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2248,"num_bytes_recovered":18178048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88682,"ondisk_log_size":88682,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2489336,"kb_used_data":1281656,"kb_used_omap":14,"kb_used_meta":1207601,"kb_avail":374998024,"statfs":{"total":386547056640,"available":383997976576,"internally_reserved":0,"allocated":1312415744,"data_stored":1311684216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1236583991},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,3,4,14,10,11,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":22839296,"num_objects":2788,"num_object_clones":0,"num_object_copies":11152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-78,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2788,"num_write_kb":22304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":77,"num_bytes_recovered":630784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.005089"},"pg_stats":[{"pgid":"1.f","version":"33'5577","reported_seq":5619,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:30.217676+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:37:30.217676+0000","last_peered":"2024-09-05T16:37:30.217676+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:37:30.217676+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5577,"log_dups_size":0,"ondisk_log_size":5577,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45686784,"num_objects":5577,"num_object_clones":0,"num_object_copies":22308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5577,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5577,"num_write_kb":44616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'5511","reported_seq":5577,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:30.984028+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:37:30.984028+0000","last_peered":"2024-09-05T16:37:30.984028+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:37:30.984028+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5511,"log_dups_size":0,"ondisk_log_size":5511,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45110432,"num_objects":5507,"num_object_clones":0,"num_object_copies":22028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5507,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5509,"num_write_kb":44068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3240}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'5506","reported_seq":5581,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:30.772908+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:37:30.772908+0000","last_peered":"2024-09-05T16:37:30.772908+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:37:30.772908+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5506,"log_dups_size":0,"ondisk_log_size":5506,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45079120,"num_objects":5503,"num_object_clones":0,"num_object_copies":22012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5505,"num_write_kb":44030,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3253}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'5531","reported_seq":10032,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:25.969590+0000","last_change":"2024-09-05T16:34:56.800371+0000","last_active":"2024-09-05T16:37:25.969590+0000","last_peered":"2024-09-05T16:37:25.969590+0000","last_clean":"2024-09-05T16:33:53.148717+0000","last_became_active":"2024-09-05T16:34:56.791755+0000","last_became_peered":"2024-09-05T16:34:56.791755+0000","last_unstale":"2024-09-05T16:37:25.969590+0000","last_undegraded":"2024-09-05T16:34:56.597522+0000","last_fullsized":"2024-09-05T16:34:56.568150+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5531,"log_dups_size":0,"ondisk_log_size":5531,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45274272,"num_objects":5527,"num_object_clones":0,"num_object_copies":22108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":72,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5527,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5529,"num_write_kb":44221,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2220,"num_bytes_recovered":18153472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":5455}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'5448","reported_seq":5502,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:30.191361+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:37:30.191361+0000","last_peered":"2024-09-05T16:37:30.191361+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:37:30.191361+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5448,"log_dups_size":0,"ondisk_log_size":5448,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44602528,"num_objects":5445,"num_object_clones":0,"num_object_copies":21780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5445,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5446,"num_write_kb":43565,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3283}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'5532","reported_seq":5573,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:30.320874+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:37:30.320874+0000","last_peered":"2024-09-05T16:37:30.320874+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:37:30.320874+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5532,"log_dups_size":0,"ondisk_log_size":5532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45318144,"num_objects":5532,"num_object_clones":0,"num_object_copies":22128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5532,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5532,"num_write_kb":44256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3282}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'5440","reported_seq":5507,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:31.001895+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:37:31.001895+0000","last_peered":"2024-09-05T16:37:31.001895+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:37:31.001895+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5440,"log_dups_size":0,"ondisk_log_size":5440,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44512416,"num_objects":5434,"num_object_clones":0,"num_object_copies":21736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5434,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5438,"num_write_kb":43484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3287}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'5532","reported_seq":5598,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:30.989821+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:37:30.989821+0000","last_peered":"2024-09-05T16:37:30.989821+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:37:30.989821+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5532,"log_dups_size":0,"ondisk_log_size":5532,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45274272,"num_objects":5527,"num_object_clones":0,"num_object_copies":22108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5527,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5530,"num_write_kb":44228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3288}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'5584","reported_seq":5640,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:30.270377+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:37:30.270377+0000","last_peered":"2024-09-05T16:37:30.270377+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:37:30.270377+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5584,"log_dups_size":0,"ondisk_log_size":5584,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45672768,"num_objects":5576,"num_object_clones":0,"num_object_copies":22304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5576,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5581,"num_write_kb":44625,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'5557","reported_seq":5634,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:30.821652+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:37:30.821652+0000","last_peered":"2024-09-05T16:37:30.821652+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:37:30.821652+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5557,"log_dups_size":0,"ondisk_log_size":5557,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45503648,"num_objects":5555,"num_object_clones":0,"num_object_copies":22220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5555,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5555,"num_write_kb":44438,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3293}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'5651","reported_seq":5707,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:30.329842+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:37:30.329842+0000","last_peered":"2024-09-05T16:37:30.329842+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:37:30.329842+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5651,"log_dups_size":0,"ondisk_log_size":5651,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46247664,"num_objects":5646,"num_object_clones":0,"num_object_copies":22584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5646,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5648,"num_write_kb":45172,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'5483","reported_seq":5519,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:30.333787+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:37:30.333787+0000","last_peered":"2024-09-05T16:37:30.333787+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:37:30.333787+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5483,"log_dups_size":0,"ondisk_log_size":5483,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44907088,"num_objects":5482,"num_object_clones":0,"num_object_copies":21928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5482,"num_write_kb":43855,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3218}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'5570","reported_seq":5651,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:30.839043+0000","last_change":"2024-09-05T16:37:30.634570+0000","last_active":"2024-09-05T16:37:30.839043+0000","last_peered":"2024-09-05T16:37:30.839043+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:37:30.839043+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5570,"log_dups_size":0,"ondisk_log_size":5570,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45619792,"num_objects":5569,"num_object_clones":0,"num_object_copies":22276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5569,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5569,"num_write_kb":44551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":24576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3278}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'5664","reported_seq":5716,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:30.280518+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:37:30.280518+0000","last_peered":"2024-09-05T16:37:30.280518+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:37:30.280518+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5664,"log_dups_size":0,"ondisk_log_size":5664,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46363808,"num_objects":5660,"num_object_clones":0,"num_object_copies":22640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5660,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5662,"num_write_kb":45292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3303}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'5555","reported_seq":5633,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:30.853574+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:37:30.853574+0000","last_peered":"2024-09-05T16:37:30.853574+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:37:30.853574+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5555,"log_dups_size":0,"ondisk_log_size":5555,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45479072,"num_objects":5552,"num_object_clones":0,"num_object_copies":22208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5552,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5553,"num_write_kb":44421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3232}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'5541","reported_seq":5607,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:30.986369+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:37:30.986369+0000","last_peered":"2024-09-05T16:37:30.986369+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:37:30.986369+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5541,"log_dups_size":0,"ondisk_log_size":5541,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45348000,"num_objects":5536,"num_object_clones":0,"num_object_copies":22144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5536,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5539,"num_write_kb":44300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3299}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":725999808,"num_objects":88628,"num_object_clones":0,"num_object_copies":354512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":88628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":88655,"num_write_kb":709122,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2248,"num_bytes_recovered":18178048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1314160640,"data_stored":1314160640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88682,"ondisk_log_size":88682,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084331,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":674368,"kb_used_data":353904,"kb_used_omap":4,"kb_used_meta":320443,"kb_avail":93697472,"statfs":{"total":96636764160,"available":95946211328,"internally_reserved":0,"allocated":362397696,"data_stored":362214814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":328134355},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,3,5,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986208,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":420092,"kb_used_data":217644,"kb_used_omap":4,"kb_used_meta":202427,"kb_avail":93951748,"statfs":{"total":96636764160,"available":96206589952,"internally_reserved":0,"allocated":222867456,"data_stored":222684574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":207285970},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,7,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673014,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":696828,"kb_used_data":354668,"kb_used_omap":1,"kb_used_meta":342142,"kb_avail":93675012,"statfs":{"total":96636764160,"available":95923212288,"internally_reserved":0,"allocated":363180032,"data_stored":362997150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":350353861},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,0,2,1,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182445,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":698048,"kb_used_data":355440,"kb_used_omap":4,"kb_used_meta":342587,"kb_avail":93673792,"statfs":{"total":96636764160,"available":95921963008,"internally_reserved":0,"allocated":363970560,"data_stored":363787678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":350809805},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2,2,1,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":363782144,"data_stored":363782144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":223694848,"data_stored":223694848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":363646976,"data_stored":363646976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":363036672,"data_stored":363036672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:37:33.229 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:37:33.526 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 191 20 89402 89382 3.65529 1.95312 0.0409228 0.0427301 2024-09-05T16:37:33.821 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:37:33.821 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:37:33.847 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":154,"stamp":"2024-09-05T16:37:32.861160+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":725999808,"num_objects":88628,"num_object_clones":0,"num_object_copies":354512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":88628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":88655,"num_write_kb":709122,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2248,"num_bytes_recovered":18178048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88682,"ondisk_log_size":88682,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2489336,"kb_used_data":1281656,"kb_used_omap":14,"kb_used_meta":1207601,"kb_avail":374998024,"statfs":{"total":386547056640,"available":383997976576,"internally_reserved":0,"allocated":1312415744,"data_stored":1311684216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1236583991},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,3,4,14,10,11,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":22839296,"num_objects":2788,"num_object_clones":0,"num_object_copies":11152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-78,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2788,"num_write_kb":22304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":77,"num_bytes_recovered":630784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.005089"},"pg_stats":[{"pgid":"1.f","version":"33'5577","reported_seq":5619,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:30.217676+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:37:30.217676+0000","last_peered":"2024-09-05T16:37:30.217676+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:37:30.217676+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5577,"log_dups_size":0,"ondisk_log_size":5577,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45686784,"num_objects":5577,"num_object_clones":0,"num_object_copies":22308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5577,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5577,"num_write_kb":44616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'5511","reported_seq":5577,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:30.984028+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:37:30.984028+0000","last_peered":"2024-09-05T16:37:30.984028+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:37:30.984028+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5511,"log_dups_size":0,"ondisk_log_size":5511,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45110432,"num_objects":5507,"num_object_clones":0,"num_object_copies":22028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5507,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5509,"num_write_kb":44068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3240}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'5506","reported_seq":5581,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:30.772908+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:37:30.772908+0000","last_peered":"2024-09-05T16:37:30.772908+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:37:30.772908+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5506,"log_dups_size":0,"ondisk_log_size":5506,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45079120,"num_objects":5503,"num_object_clones":0,"num_object_copies":22012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5505,"num_write_kb":44030,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3253}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'5531","reported_seq":10032,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:25.969590+0000","last_change":"2024-09-05T16:34:56.800371+0000","last_active":"2024-09-05T16:37:25.969590+0000","last_peered":"2024-09-05T16:37:25.969590+0000","last_clean":"2024-09-05T16:33:53.148717+0000","last_became_active":"2024-09-05T16:34:56.791755+0000","last_became_peered":"2024-09-05T16:34:56.791755+0000","last_unstale":"2024-09-05T16:37:25.969590+0000","last_undegraded":"2024-09-05T16:34:56.597522+0000","last_fullsized":"2024-09-05T16:34:56.568150+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5531,"log_dups_size":0,"ondisk_log_size":5531,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45274272,"num_objects":5527,"num_object_clones":0,"num_object_copies":22108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":72,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5527,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5529,"num_write_kb":44221,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2220,"num_bytes_recovered":18153472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":5455}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'5448","reported_seq":5502,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:30.191361+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:37:30.191361+0000","last_peered":"2024-09-05T16:37:30.191361+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:37:30.191361+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5448,"log_dups_size":0,"ondisk_log_size":5448,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44602528,"num_objects":5445,"num_object_clones":0,"num_object_copies":21780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5445,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5446,"num_write_kb":43565,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3283}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'5532","reported_seq":5573,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:30.320874+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:37:30.320874+0000","last_peered":"2024-09-05T16:37:30.320874+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:37:30.320874+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5532,"log_dups_size":0,"ondisk_log_size":5532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45318144,"num_objects":5532,"num_object_clones":0,"num_object_copies":22128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5532,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5532,"num_write_kb":44256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3282}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'5440","reported_seq":5507,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:31.001895+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:37:31.001895+0000","last_peered":"2024-09-05T16:37:31.001895+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:37:31.001895+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5440,"log_dups_size":0,"ondisk_log_size":5440,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44512416,"num_objects":5434,"num_object_clones":0,"num_object_copies":21736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5434,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5438,"num_write_kb":43484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3287}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'5532","reported_seq":5598,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:30.989821+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:37:30.989821+0000","last_peered":"2024-09-05T16:37:30.989821+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:37:30.989821+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5532,"log_dups_size":0,"ondisk_log_size":5532,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45274272,"num_objects":5527,"num_object_clones":0,"num_object_copies":22108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5527,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5530,"num_write_kb":44228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3288}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'5584","reported_seq":5640,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:30.270377+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:37:30.270377+0000","last_peered":"2024-09-05T16:37:30.270377+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:37:30.270377+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5584,"log_dups_size":0,"ondisk_log_size":5584,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45672768,"num_objects":5576,"num_object_clones":0,"num_object_copies":22304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5576,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5581,"num_write_kb":44625,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'5557","reported_seq":5634,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:30.821652+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:37:30.821652+0000","last_peered":"2024-09-05T16:37:30.821652+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:37:30.821652+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5557,"log_dups_size":0,"ondisk_log_size":5557,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45503648,"num_objects":5555,"num_object_clones":0,"num_object_copies":22220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5555,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5555,"num_write_kb":44438,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3293}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'5651","reported_seq":5707,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:30.329842+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:37:30.329842+0000","last_peered":"2024-09-05T16:37:30.329842+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:37:30.329842+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5651,"log_dups_size":0,"ondisk_log_size":5651,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46247664,"num_objects":5646,"num_object_clones":0,"num_object_copies":22584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5646,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5648,"num_write_kb":45172,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'5483","reported_seq":5519,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:30.333787+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:37:30.333787+0000","last_peered":"2024-09-05T16:37:30.333787+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:37:30.333787+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5483,"log_dups_size":0,"ondisk_log_size":5483,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44907088,"num_objects":5482,"num_object_clones":0,"num_object_copies":21928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5482,"num_write_kb":43855,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3218}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'5570","reported_seq":5651,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:30.839043+0000","last_change":"2024-09-05T16:37:30.634570+0000","last_active":"2024-09-05T16:37:30.839043+0000","last_peered":"2024-09-05T16:37:30.839043+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:37:30.839043+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5570,"log_dups_size":0,"ondisk_log_size":5570,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45619792,"num_objects":5569,"num_object_clones":0,"num_object_copies":22276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5569,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5569,"num_write_kb":44551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":24576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3278}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'5664","reported_seq":5716,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:30.280518+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:37:30.280518+0000","last_peered":"2024-09-05T16:37:30.280518+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:37:30.280518+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5664,"log_dups_size":0,"ondisk_log_size":5664,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46363808,"num_objects":5660,"num_object_clones":0,"num_object_copies":22640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5660,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5662,"num_write_kb":45292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3303}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'5555","reported_seq":5633,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:30.853574+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:37:30.853574+0000","last_peered":"2024-09-05T16:37:30.853574+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:37:30.853574+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5555,"log_dups_size":0,"ondisk_log_size":5555,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45479072,"num_objects":5552,"num_object_clones":0,"num_object_copies":22208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5552,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5553,"num_write_kb":44421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3232}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'5541","reported_seq":5607,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:30.986369+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:37:30.986369+0000","last_peered":"2024-09-05T16:37:30.986369+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:37:30.986369+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5541,"log_dups_size":0,"ondisk_log_size":5541,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45348000,"num_objects":5536,"num_object_clones":0,"num_object_copies":22144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5536,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5539,"num_write_kb":44300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3299}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":725999808,"num_objects":88628,"num_object_clones":0,"num_object_copies":354512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":88628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":88655,"num_write_kb":709122,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2248,"num_bytes_recovered":18178048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1314160640,"data_stored":1314160640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88682,"ondisk_log_size":88682,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084331,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":674368,"kb_used_data":353904,"kb_used_omap":4,"kb_used_meta":320443,"kb_avail":93697472,"statfs":{"total":96636764160,"available":95946211328,"internally_reserved":0,"allocated":362397696,"data_stored":362214814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":328134355},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,3,5,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986208,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":420092,"kb_used_data":217644,"kb_used_omap":4,"kb_used_meta":202427,"kb_avail":93951748,"statfs":{"total":96636764160,"available":96206589952,"internally_reserved":0,"allocated":222867456,"data_stored":222684574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":207285970},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,7,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673014,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":696828,"kb_used_data":354668,"kb_used_omap":1,"kb_used_meta":342142,"kb_avail":93675012,"statfs":{"total":96636764160,"available":95923212288,"internally_reserved":0,"allocated":363180032,"data_stored":362997150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":350353861},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,0,2,1,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182445,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":698048,"kb_used_data":355440,"kb_used_omap":4,"kb_used_meta":342587,"kb_avail":93673792,"statfs":{"total":96636764160,"available":95921963008,"internally_reserved":0,"allocated":363970560,"data_stored":363787678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":350809805},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2,2,1,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":363782144,"data_stored":363782144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":223694848,"data_stored":223694848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":363646976,"data_stored":363646976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":363036672,"data_stored":363036672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:37:33.848 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-05T16:37:33.848 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'5577", 'reported_seq': 5619, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:30.217676+0000', 'last_change': '2024-09-05T16:34:56.833497+0000', 'last_active': '2024-09-05T16:37:30.217676+0000', 'last_peered': '2024-09-05T16:37:30.217676+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.828289+0000', 'last_became_peered': '2024-09-05T16:34:56.828289+0000', 'last_unstale': '2024-09-05T16:37:30.217676+0000', 'last_undegraded': '2024-09-05T16:34:56.612160+0000', 'last_fullsized': '2024-09-05T16:34:56.557632+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5577, 'log_dups_size': 0, 'ondisk_log_size': 5577, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45686784, 'num_objects': 5577, 'num_object_clones': 0, 'num_object_copies': 22308, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2262, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5577, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5577, 'num_write_kb': 44616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:37:33.848 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-05T16:37:33.848 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'5511", 'reported_seq': 5577, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:30.984028+0000', 'last_change': '2024-09-05T16:34:56.825031+0000', 'last_active': '2024-09-05T16:37:30.984028+0000', 'last_peered': '2024-09-05T16:37:30.984028+0000', 'last_clean': '2024-09-05T16:33:53.200762+0000', 'last_became_active': '2024-09-05T16:34:56.820292+0000', 'last_became_peered': '2024-09-05T16:34:56.820292+0000', 'last_unstale': '2024-09-05T16:37:30.984028+0000', 'last_undegraded': '2024-09-05T16:34:56.605776+0000', 'last_fullsized': '2024-09-05T16:34:56.559383+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5511, 'log_dups_size': 0, 'ondisk_log_size': 5511, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45110432, 'num_objects': 5507, 'num_object_clones': 0, 'num_object_copies': 22028, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2269, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5507, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5509, 'num_write_kb': 44068, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3240}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:37:33.849 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:37:33.849 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'5506", 'reported_seq': 5581, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:30.772908+0000', 'last_change': '2024-09-05T16:34:56.867900+0000', 'last_active': '2024-09-05T16:37:30.772908+0000', 'last_peered': '2024-09-05T16:37:30.772908+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:37:30.772908+0000', 'last_undegraded': '2024-09-05T16:34:56.590972+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5506, 'log_dups_size': 0, 'ondisk_log_size': 5506, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45079120, 'num_objects': 5503, 'num_object_clones': 0, 'num_object_copies': 22012, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2251, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5503, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5505, 'num_write_kb': 44030, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3253}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:37:33.849 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-05T16:37:33.849 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'5531", 'reported_seq': 10032, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:25.969590+0000', 'last_change': '2024-09-05T16:34:56.800371+0000', 'last_active': '2024-09-05T16:37:25.969590+0000', 'last_peered': '2024-09-05T16:37:25.969590+0000', 'last_clean': '2024-09-05T16:33:53.148717+0000', 'last_became_active': '2024-09-05T16:34:56.791755+0000', 'last_became_peered': '2024-09-05T16:34:56.791755+0000', 'last_unstale': '2024-09-05T16:37:25.969590+0000', 'last_undegraded': '2024-09-05T16:34:56.597522+0000', 'last_fullsized': '2024-09-05T16:34:56.568150+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5531, 'log_dups_size': 0, 'ondisk_log_size': 5531, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45274272, 'num_objects': 5527, 'num_object_clones': 0, 'num_object_copies': 22108, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 72, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5527, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5529, 'num_write_kb': 44221, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2220, 'num_bytes_recovered': 18153472, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 5455}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:37:33.849 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:37:33.849 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'5448", 'reported_seq': 5502, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:30.191361+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:37:30.191361+0000', 'last_peered': '2024-09-05T16:37:30.191361+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:37:30.191361+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5448, 'log_dups_size': 0, 'ondisk_log_size': 5448, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44602528, 'num_objects': 5445, 'num_object_clones': 0, 'num_object_copies': 21780, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2164, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5445, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5446, 'num_write_kb': 43565, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3283}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:37:33.849 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:37:33.849 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'5532", 'reported_seq': 5573, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:30.320874+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:37:30.320874+0000', 'last_peered': '2024-09-05T16:37:30.320874+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:37:30.320874+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5532, 'log_dups_size': 0, 'ondisk_log_size': 5532, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45318144, 'num_objects': 5532, 'num_object_clones': 0, 'num_object_copies': 22128, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5532, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5532, 'num_write_kb': 44256, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3282}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:37:33.849 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:37:33.849 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'5440", 'reported_seq': 5507, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:31.001895+0000', 'last_change': '2024-09-05T16:34:56.903436+0000', 'last_active': '2024-09-05T16:37:31.001895+0000', 'last_peered': '2024-09-05T16:37:31.001895+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:37:31.001895+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5440, 'log_dups_size': 0, 'ondisk_log_size': 5440, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44512416, 'num_objects': 5434, 'num_object_clones': 0, 'num_object_copies': 21736, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2149, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5434, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5438, 'num_write_kb': 43484, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3287}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:37:33.850 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-05T16:37:33.850 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'5532", 'reported_seq': 5598, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:30.989821+0000', 'last_change': '2024-09-05T16:34:56.883319+0000', 'last_active': '2024-09-05T16:37:30.989821+0000', 'last_peered': '2024-09-05T16:37:30.989821+0000', 'last_clean': '2024-09-05T16:33:53.505305+0000', 'last_became_active': '2024-09-05T16:34:56.873389+0000', 'last_became_peered': '2024-09-05T16:34:56.873389+0000', 'last_unstale': '2024-09-05T16:37:30.989821+0000', 'last_undegraded': '2024-09-05T16:34:56.602567+0000', 'last_fullsized': '2024-09-05T16:34:56.559958+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5532, 'log_dups_size': 0, 'ondisk_log_size': 5532, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45274272, 'num_objects': 5527, 'num_object_clones': 0, 'num_object_copies': 22108, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2241, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5527, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5530, 'num_write_kb': 44228, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3288}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:37:33.850 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:37:33.850 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'5584", 'reported_seq': 5640, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:30.270377+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:37:30.270377+0000', 'last_peered': '2024-09-05T16:37:30.270377+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:37:30.270377+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5584, 'log_dups_size': 0, 'ondisk_log_size': 5584, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45672768, 'num_objects': 5576, 'num_object_clones': 0, 'num_object_copies': 22304, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2264, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5576, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5581, 'num_write_kb': 44625, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:37:33.850 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:37:33.850 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'5557", 'reported_seq': 5634, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:30.821652+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:37:30.821652+0000', 'last_peered': '2024-09-05T16:37:30.821652+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:37:30.821652+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5557, 'log_dups_size': 0, 'ondisk_log_size': 5557, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45503648, 'num_objects': 5555, 'num_object_clones': 0, 'num_object_copies': 22220, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2264, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5555, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5555, 'num_write_kb': 44438, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3293}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:37:33.850 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:37:33.850 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'5651", 'reported_seq': 5707, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:30.329842+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:37:30.329842+0000', 'last_peered': '2024-09-05T16:37:30.329842+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:37:30.329842+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5651, 'log_dups_size': 0, 'ondisk_log_size': 5651, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46247664, 'num_objects': 5646, 'num_object_clones': 0, 'num_object_copies': 22584, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2314, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5646, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5648, 'num_write_kb': 45172, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:37:33.850 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:37:33.850 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'5483", 'reported_seq': 5519, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:30.333787+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:37:30.333787+0000', 'last_peered': '2024-09-05T16:37:30.333787+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:37:30.333787+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5483, 'log_dups_size': 0, 'ondisk_log_size': 5483, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44907088, 'num_objects': 5482, 'num_object_clones': 0, 'num_object_copies': 21928, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2265, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5482, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5482, 'num_write_kb': 43855, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3218}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:37:33.851 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-05T16:37:33.851 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'5570", 'reported_seq': 5651, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:30.839043+0000', 'last_change': '2024-09-05T16:37:30.634570+0000', 'last_active': '2024-09-05T16:37:30.839043+0000', 'last_peered': '2024-09-05T16:37:30.839043+0000', 'last_clean': '2024-09-05T16:33:45.082039+0000', 'last_became_active': '2024-09-05T16:34:56.792386+0000', 'last_became_peered': '2024-09-05T16:34:56.792386+0000', 'last_unstale': '2024-09-05T16:37:30.839043+0000', 'last_undegraded': '2024-09-05T16:34:56.600595+0000', 'last_fullsized': '2024-09-05T16:34:56.558203+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5570, 'log_dups_size': 0, 'ondisk_log_size': 5570, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45619792, 'num_objects': 5569, 'num_object_clones': 0, 'num_object_copies': 22276, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2291, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5569, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5569, 'num_write_kb': 44551, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 24576, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3278}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:37:33.851 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:37:33.851 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'5664", 'reported_seq': 5716, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:30.280518+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:37:30.280518+0000', 'last_peered': '2024-09-05T16:37:30.280518+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:37:30.280518+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5664, 'log_dups_size': 0, 'ondisk_log_size': 5664, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46363808, 'num_objects': 5660, 'num_object_clones': 0, 'num_object_copies': 22640, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2359, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5660, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5662, 'num_write_kb': 45292, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3303}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:37:33.851 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:37:33.851 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'5555", 'reported_seq': 5633, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:30.853574+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:37:30.853574+0000', 'last_peered': '2024-09-05T16:37:30.853574+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:37:30.853574+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5555, 'log_dups_size': 0, 'ondisk_log_size': 5555, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45479072, 'num_objects': 5552, 'num_object_clones': 0, 'num_object_copies': 22208, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2322, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5552, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5553, 'num_write_kb': 44421, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3232}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:37:33.851 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:37:33.851 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'5541", 'reported_seq': 5607, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:30.986369+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:37:30.986369+0000', 'last_peered': '2024-09-05T16:37:30.986369+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:37:30.986369+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5541, 'log_dups_size': 0, 'ondisk_log_size': 5541, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45348000, 'num_objects': 5536, 'num_object_clones': 0, 'num_object_copies': 22144, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5536, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5539, 'num_write_kb': 44300, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3299}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:37:33.851 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:37:34.526 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 192 20 89649 89629 3.6463 1.92969 0.0774109 0.0428367 2024-09-05T16:37:34.596 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:37:34.596 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 33976/354512 objects degraded (9.584%), 16 pgs degraded, 16 pgs undersized","count":32},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":257,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+recovering+undersized+degraded+remapped","count":2}],"num_pgs":16,"num_pools":1,"num_objects":88628,"data_bytes":725999808,"bytes_used":2549080064,"bytes_avail":383997976576,"bytes_total":386547056640,"degraded_objects":33976,"degraded_total":354512,"degraded_ratio":0.095838786839373566,"recovering_objects_per_sec":9,"recovering_bytes_per_sec":78797,"recovering_keys_per_sec":0,"num_objects_recovered":77,"num_bytes_recovered":630784,"num_keys_recovered":0,"write_bytes_sec":2853097,"read_op_per_sec":0,"write_op_per_sec":348},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:37:34.624 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:37:34.624 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:37:35.247 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:37:35.248 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:37:35.270 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":155,"stamp":"2024-09-05T16:37:34.861906+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":725999808,"num_objects":88628,"num_object_clones":0,"num_object_copies":354512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":88628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":88655,"num_write_kb":709122,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2248,"num_bytes_recovered":18178048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88682,"ondisk_log_size":88682,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2489336,"kb_used_data":1281656,"kb_used_omap":14,"kb_used_meta":1207601,"kb_avail":374998024,"statfs":{"total":386547056640,"available":383997976576,"internally_reserved":0,"allocated":1312415744,"data_stored":1311684216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1236583991},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,3,4,14,10,11,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":10854400,"num_objects":1325,"num_object_clones":0,"num_object_copies":5300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1325,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1325,"num_write_kb":10600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":24576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002919"},"pg_stats":[{"pgid":"1.f","version":"33'5577","reported_seq":5619,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:30.217676+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:37:30.217676+0000","last_peered":"2024-09-05T16:37:30.217676+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:37:30.217676+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5577,"log_dups_size":0,"ondisk_log_size":5577,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45686784,"num_objects":5577,"num_object_clones":0,"num_object_copies":22308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5577,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5577,"num_write_kb":44616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'5511","reported_seq":5577,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:30.984028+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:37:30.984028+0000","last_peered":"2024-09-05T16:37:30.984028+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:37:30.984028+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5511,"log_dups_size":0,"ondisk_log_size":5511,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45110432,"num_objects":5507,"num_object_clones":0,"num_object_copies":22028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5507,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5509,"num_write_kb":44068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3240}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'5506","reported_seq":5581,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:30.772908+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:37:30.772908+0000","last_peered":"2024-09-05T16:37:30.772908+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:37:30.772908+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5506,"log_dups_size":0,"ondisk_log_size":5506,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45079120,"num_objects":5503,"num_object_clones":0,"num_object_copies":22012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5505,"num_write_kb":44030,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3253}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'5531","reported_seq":10032,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:25.969590+0000","last_change":"2024-09-05T16:34:56.800371+0000","last_active":"2024-09-05T16:37:25.969590+0000","last_peered":"2024-09-05T16:37:25.969590+0000","last_clean":"2024-09-05T16:33:53.148717+0000","last_became_active":"2024-09-05T16:34:56.791755+0000","last_became_peered":"2024-09-05T16:34:56.791755+0000","last_unstale":"2024-09-05T16:37:25.969590+0000","last_undegraded":"2024-09-05T16:34:56.597522+0000","last_fullsized":"2024-09-05T16:34:56.568150+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5531,"log_dups_size":0,"ondisk_log_size":5531,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45274272,"num_objects":5527,"num_object_clones":0,"num_object_copies":22108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":72,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5527,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5529,"num_write_kb":44221,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2220,"num_bytes_recovered":18153472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":5455}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'5448","reported_seq":5502,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:30.191361+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:37:30.191361+0000","last_peered":"2024-09-05T16:37:30.191361+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:37:30.191361+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5448,"log_dups_size":0,"ondisk_log_size":5448,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44602528,"num_objects":5445,"num_object_clones":0,"num_object_copies":21780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5445,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5446,"num_write_kb":43565,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3283}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'5532","reported_seq":5573,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:30.320874+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:37:30.320874+0000","last_peered":"2024-09-05T16:37:30.320874+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:37:30.320874+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5532,"log_dups_size":0,"ondisk_log_size":5532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45318144,"num_objects":5532,"num_object_clones":0,"num_object_copies":22128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5532,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5532,"num_write_kb":44256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3282}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'5440","reported_seq":5507,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:31.001895+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:37:31.001895+0000","last_peered":"2024-09-05T16:37:31.001895+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:37:31.001895+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5440,"log_dups_size":0,"ondisk_log_size":5440,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44512416,"num_objects":5434,"num_object_clones":0,"num_object_copies":21736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5434,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5438,"num_write_kb":43484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3287}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'5532","reported_seq":5598,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:30.989821+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:37:30.989821+0000","last_peered":"2024-09-05T16:37:30.989821+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:37:30.989821+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5532,"log_dups_size":0,"ondisk_log_size":5532,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45274272,"num_objects":5527,"num_object_clones":0,"num_object_copies":22108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5527,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5530,"num_write_kb":44228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3288}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'5584","reported_seq":5640,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:30.270377+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:37:30.270377+0000","last_peered":"2024-09-05T16:37:30.270377+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:37:30.270377+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5584,"log_dups_size":0,"ondisk_log_size":5584,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45672768,"num_objects":5576,"num_object_clones":0,"num_object_copies":22304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5576,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5581,"num_write_kb":44625,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'5557","reported_seq":5634,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:30.821652+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:37:30.821652+0000","last_peered":"2024-09-05T16:37:30.821652+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:37:30.821652+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5557,"log_dups_size":0,"ondisk_log_size":5557,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45503648,"num_objects":5555,"num_object_clones":0,"num_object_copies":22220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5555,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5555,"num_write_kb":44438,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3293}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'5651","reported_seq":5707,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:30.329842+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:37:30.329842+0000","last_peered":"2024-09-05T16:37:30.329842+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:37:30.329842+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5651,"log_dups_size":0,"ondisk_log_size":5651,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46247664,"num_objects":5646,"num_object_clones":0,"num_object_copies":22584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5646,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5648,"num_write_kb":45172,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'5483","reported_seq":5519,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:30.333787+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:37:30.333787+0000","last_peered":"2024-09-05T16:37:30.333787+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:37:30.333787+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5483,"log_dups_size":0,"ondisk_log_size":5483,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44907088,"num_objects":5482,"num_object_clones":0,"num_object_copies":21928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5482,"num_write_kb":43855,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3218}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'5570","reported_seq":5651,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:30.839043+0000","last_change":"2024-09-05T16:37:30.634570+0000","last_active":"2024-09-05T16:37:30.839043+0000","last_peered":"2024-09-05T16:37:30.839043+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:37:30.839043+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5570,"log_dups_size":0,"ondisk_log_size":5570,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45619792,"num_objects":5569,"num_object_clones":0,"num_object_copies":22276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5569,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5569,"num_write_kb":44551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":24576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3278}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'5664","reported_seq":5716,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:30.280518+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:37:30.280518+0000","last_peered":"2024-09-05T16:37:30.280518+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:37:30.280518+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5664,"log_dups_size":0,"ondisk_log_size":5664,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46363808,"num_objects":5660,"num_object_clones":0,"num_object_copies":22640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5660,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5662,"num_write_kb":45292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3303}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'5555","reported_seq":5633,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:30.853574+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:37:30.853574+0000","last_peered":"2024-09-05T16:37:30.853574+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:37:30.853574+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5555,"log_dups_size":0,"ondisk_log_size":5555,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45479072,"num_objects":5552,"num_object_clones":0,"num_object_copies":22208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5552,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5553,"num_write_kb":44421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3232}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'5541","reported_seq":5607,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:30.986369+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:37:30.986369+0000","last_peered":"2024-09-05T16:37:30.986369+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:37:30.986369+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5541,"log_dups_size":0,"ondisk_log_size":5541,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45348000,"num_objects":5536,"num_object_clones":0,"num_object_copies":22144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5536,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5539,"num_write_kb":44300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3299}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":725999808,"num_objects":88628,"num_object_clones":0,"num_object_copies":354512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":88628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":88655,"num_write_kb":709122,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2248,"num_bytes_recovered":18178048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1314160640,"data_stored":1314160640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88682,"ondisk_log_size":88682,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084331,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":674368,"kb_used_data":353904,"kb_used_omap":4,"kb_used_meta":320443,"kb_avail":93697472,"statfs":{"total":96636764160,"available":95946211328,"internally_reserved":0,"allocated":362397696,"data_stored":362214814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":328134355},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,3,5,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986208,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":420092,"kb_used_data":217644,"kb_used_omap":4,"kb_used_meta":202427,"kb_avail":93951748,"statfs":{"total":96636764160,"available":96206589952,"internally_reserved":0,"allocated":222867456,"data_stored":222684574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":207285970},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,7,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673014,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":696828,"kb_used_data":354668,"kb_used_omap":1,"kb_used_meta":342142,"kb_avail":93675012,"statfs":{"total":96636764160,"available":95923212288,"internally_reserved":0,"allocated":363180032,"data_stored":362997150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":350353861},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,0,2,1,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182445,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":698048,"kb_used_data":355440,"kb_used_omap":4,"kb_used_meta":342587,"kb_avail":93673792,"statfs":{"total":96636764160,"available":95921963008,"internally_reserved":0,"allocated":363970560,"data_stored":363787678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":350809805},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2,2,1,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":363782144,"data_stored":363782144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":223694848,"data_stored":223694848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":363646976,"data_stored":363646976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":363036672,"data_stored":363036672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:37:35.529 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 193 20 89916 89896 3.63816 2.08594 0.0668667 0.0429349 2024-09-05T16:37:36.529 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 194 20 90201 90181 3.63089 2.22656 0.0359082 0.043023 2024-09-05T16:37:37.529 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 195 20 90482 90462 3.62353 2.19531 0.147014 0.0431079 2024-09-05T16:37:38.273 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:37:38.529 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 196 20 90764 90744 3.61628 2.20312 0.179443 0.0431952 2024-09-05T16:37:38.908 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:37:38.909 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:37:38.939 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":157,"stamp":"2024-09-05T16:37:38.863400+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":736919744,"num_objects":89961,"num_object_clones":0,"num_object_copies":359844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33827,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89961,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":89988,"num_write_kb":719786,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2397,"num_bytes_recovered":19398656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90015,"ondisk_log_size":90015,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2545968,"kb_used_data":1302188,"kb_used_omap":14,"kb_used_meta":1243697,"kb_avail":374941392,"statfs":{"total":386547056640,"available":383939985408,"internally_reserved":0,"allocated":1333440512,"data_stored":1332672348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1273546295},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,10,5,12,11,9,5,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":16605184,"num_objects":2027,"num_object_clones":0,"num_object_copies":8108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-153,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2027,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2027,"num_write_kb":16216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":152,"num_bytes_recovered":1245184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002964"},"pg_stats":[{"pgid":"1.f","version":"35'5666","reported_seq":5712,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:35.225440+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:37:35.225440+0000","last_peered":"2024-09-05T16:37:35.225440+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:37:35.225440+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5666,"log_dups_size":0,"ondisk_log_size":5666,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46415872,"num_objects":5666,"num_object_clones":0,"num_object_copies":22664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5666,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5666,"num_write_kb":45328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3404}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'5594","reported_seq":5662,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:35.694281+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:37:35.694281+0000","last_peered":"2024-09-05T16:37:35.694281+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:37:35.694281+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5594,"log_dups_size":0,"ondisk_log_size":5594,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45790368,"num_objects":5590,"num_object_clones":0,"num_object_copies":22360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5590,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5592,"num_write_kb":44732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3323}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'5594","reported_seq":5672,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:35.891240+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:37:35.891240+0000","last_peered":"2024-09-05T16:37:35.891240+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:37:35.891240+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5594,"log_dups_size":0,"ondisk_log_size":5594,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45800016,"num_objects":5591,"num_object_clones":0,"num_object_copies":22364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5591,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5593,"num_write_kb":44734,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3341}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'5684","reported_seq":10363,"reported_epoch":35,"state":"active+clean+scrubbing","last_fresh":"2024-09-05T16:37:34.883851+0000","last_change":"2024-09-05T16:37:32.894678+0000","last_active":"2024-09-05T16:37:34.883851+0000","last_peered":"2024-09-05T16:37:34.883851+0000","last_clean":"2024-09-05T16:37:34.883851+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:37:34.883851+0000","last_undegraded":"2024-09-05T16:37:34.883851+0000","last_fullsized":"2024-09-05T16:37:34.883851+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":1899,"log_size":5684,"log_dups_size":0,"ondisk_log_size":5684,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 2s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46527648,"num_objects":5680,"num_object_clones":0,"num_object_copies":22720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5680,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5682,"num_write_kb":45445,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2292,"num_bytes_recovered":18743296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'5544","reported_seq":5602,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:35.216300+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:37:35.216300+0000","last_peered":"2024-09-05T16:37:35.216300+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:37:35.216300+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5544,"log_dups_size":0,"ondisk_log_size":5544,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45388960,"num_objects":5541,"num_object_clones":0,"num_object_copies":22164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5541,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5542,"num_write_kb":44333,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3379}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'5603","reported_seq":5649,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:35.292300+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:37:35.292300+0000","last_peered":"2024-09-05T16:37:35.292300+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:37:35.292300+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5603,"log_dups_size":0,"ondisk_log_size":5603,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45899776,"num_objects":5603,"num_object_clones":0,"num_object_copies":22412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5603,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5603,"num_write_kb":44824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3353}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'5507","reported_seq":5576,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:35.991768+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:37:35.991768+0000","last_peered":"2024-09-05T16:37:35.991768+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:37:35.991768+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5507,"log_dups_size":0,"ondisk_log_size":5507,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45061280,"num_objects":5501,"num_object_clones":0,"num_object_copies":22004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5501,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5505,"num_write_kb":44020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3354}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'5600","reported_seq":5668,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:35.892108+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:37:35.892108+0000","last_peered":"2024-09-05T16:37:35.892108+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:37:35.892108+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5600,"log_dups_size":0,"ondisk_log_size":5600,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45831328,"num_objects":5595,"num_object_clones":0,"num_object_copies":22380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5595,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5598,"num_write_kb":44772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3356}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'5661","reported_seq":5721,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:35.294232+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:37:35.294232+0000","last_peered":"2024-09-05T16:37:35.294232+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:37:35.294232+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5661,"log_dups_size":0,"ondisk_log_size":5661,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46303552,"num_objects":5653,"num_object_clones":0,"num_object_copies":22612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5653,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5658,"num_write_kb":45241,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3392}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'5635","reported_seq":5716,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:35.811956+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:37:35.811956+0000","last_peered":"2024-09-05T16:37:35.811956+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:37:35.811956+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5635,"log_dups_size":0,"ondisk_log_size":5635,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46142624,"num_objects":5633,"num_object_clones":0,"num_object_copies":22532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5633,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5633,"num_write_kb":45062,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3371}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'5729","reported_seq":5788,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:35.391296+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:37:35.391296+0000","last_peered":"2024-09-05T16:37:35.391296+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:37:35.391296+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5729,"log_dups_size":0,"ondisk_log_size":5729,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46886640,"num_objects":5724,"num_object_clones":0,"num_object_copies":22896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5726,"num_write_kb":45796,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3413}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'5564","reported_seq":5603,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:35.371066+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:37:35.371066+0000","last_peered":"2024-09-05T16:37:35.371066+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:37:35.371066+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5564,"log_dups_size":0,"ondisk_log_size":5564,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45570640,"num_objects":5563,"num_object_clones":0,"num_object_copies":22252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5563,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5563,"num_write_kb":44503,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3299}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'5651","reported_seq":5890,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:35.894444+0000","last_change":"2024-09-05T16:37:30.634570+0000","last_active":"2024-09-05T16:37:35.894444+0000","last_peered":"2024-09-05T16:37:35.894444+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:37:35.894444+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5651,"log_dups_size":0,"ondisk_log_size":5651,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46283344,"num_objects":5650,"num_object_clones":0,"num_object_copies":22600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2214,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5650,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5650,"num_write_kb":45199,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3436}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'5748","reported_seq":5804,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:35.288152+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:37:35.288152+0000","last_peered":"2024-09-05T16:37:35.288152+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:37:35.288152+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5748,"log_dups_size":0,"ondisk_log_size":5748,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47051936,"num_objects":5744,"num_object_clones":0,"num_object_copies":22976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5744,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5746,"num_write_kb":45964,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3387}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'5630","reported_seq":5710,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:35.871260+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:37:35.871260+0000","last_peered":"2024-09-05T16:37:35.871260+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:37:35.871260+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5630,"log_dups_size":0,"ondisk_log_size":5630,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46093472,"num_objects":5627,"num_object_clones":0,"num_object_copies":22508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5627,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5628,"num_write_kb":45021,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3307}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'5605","reported_seq":5673,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:36.027495+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:37:36.027495+0000","last_peered":"2024-09-05T16:37:36.027495+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:37:36.027495+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5605,"log_dups_size":0,"ondisk_log_size":5605,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45872288,"num_objects":5600,"num_object_clones":0,"num_object_copies":22400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5600,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5603,"num_write_kb":44812,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3363}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":736919744,"num_objects":89961,"num_object_clones":0,"num_object_copies":359844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33827,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89961,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":89988,"num_write_kb":719786,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2397,"num_bytes_recovered":19398656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1334935552,"data_stored":1334935552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90015,"ondisk_log_size":90015,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084332,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":701704,"kb_used_data":359032,"kb_used_omap":4,"kb_used_meta":342651,"kb_avail":93670136,"statfs":{"total":96636764160,"available":95918219264,"internally_reserved":0,"allocated":367648768,"data_stored":367456727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":350875347},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,6,6,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986209,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":431132,"kb_used_data":223112,"kb_used_omap":4,"kb_used_meta":207995,"kb_avail":93940708,"statfs":{"total":96636764160,"available":96195284992,"internally_reserved":0,"allocated":228466688,"data_stored":228274647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":212987602},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,2,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673015,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":706024,"kb_used_data":359704,"kb_used_omap":1,"kb_used_meta":346302,"kb_avail":93665816,"statfs":{"total":96636764160,"available":95913795584,"internally_reserved":0,"allocated":368336896,"data_stored":368144855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":354613701},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,4,4,2,3,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182446,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":707108,"kb_used_data":360340,"kb_used_omap":4,"kb_used_meta":346747,"kb_avail":93664732,"statfs":{"total":96636764160,"available":95912685568,"internally_reserved":0,"allocated":368988160,"data_stored":368796119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":355069645},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,1,1,1,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":368877568,"data_stored":368877568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":229236736,"data_stored":229236736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":368726016,"data_stored":368726016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":368095232,"data_stored":368095232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:37:38.940 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:37:39.532 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 197 20 91016 90996 3.60789 1.96875 0.0746129 0.0432963 2024-09-05T16:37:39.567 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:37:39.567 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:37:39.594 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":157,"stamp":"2024-09-05T16:37:38.863400+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":736919744,"num_objects":89961,"num_object_clones":0,"num_object_copies":359844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33827,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89961,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":89988,"num_write_kb":719786,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2397,"num_bytes_recovered":19398656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90015,"ondisk_log_size":90015,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2545968,"kb_used_data":1302188,"kb_used_omap":14,"kb_used_meta":1243697,"kb_avail":374941392,"statfs":{"total":386547056640,"available":383939985408,"internally_reserved":0,"allocated":1333440512,"data_stored":1332672348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1273546295},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,10,5,12,11,9,5,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":16605184,"num_objects":2027,"num_object_clones":0,"num_object_copies":8108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-153,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2027,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2027,"num_write_kb":16216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":152,"num_bytes_recovered":1245184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002964"},"pg_stats":[{"pgid":"1.f","version":"35'5666","reported_seq":5712,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:35.225440+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:37:35.225440+0000","last_peered":"2024-09-05T16:37:35.225440+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:37:35.225440+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5666,"log_dups_size":0,"ondisk_log_size":5666,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46415872,"num_objects":5666,"num_object_clones":0,"num_object_copies":22664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5666,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5666,"num_write_kb":45328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3404}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'5594","reported_seq":5662,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:35.694281+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:37:35.694281+0000","last_peered":"2024-09-05T16:37:35.694281+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:37:35.694281+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5594,"log_dups_size":0,"ondisk_log_size":5594,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45790368,"num_objects":5590,"num_object_clones":0,"num_object_copies":22360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5590,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5592,"num_write_kb":44732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3323}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'5594","reported_seq":5672,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:35.891240+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:37:35.891240+0000","last_peered":"2024-09-05T16:37:35.891240+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:37:35.891240+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5594,"log_dups_size":0,"ondisk_log_size":5594,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45800016,"num_objects":5591,"num_object_clones":0,"num_object_copies":22364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5591,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5593,"num_write_kb":44734,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3341}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'5684","reported_seq":10363,"reported_epoch":35,"state":"active+clean+scrubbing","last_fresh":"2024-09-05T16:37:34.883851+0000","last_change":"2024-09-05T16:37:32.894678+0000","last_active":"2024-09-05T16:37:34.883851+0000","last_peered":"2024-09-05T16:37:34.883851+0000","last_clean":"2024-09-05T16:37:34.883851+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:37:34.883851+0000","last_undegraded":"2024-09-05T16:37:34.883851+0000","last_fullsized":"2024-09-05T16:37:34.883851+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":1899,"log_size":5684,"log_dups_size":0,"ondisk_log_size":5684,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 2s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46527648,"num_objects":5680,"num_object_clones":0,"num_object_copies":22720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5680,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5682,"num_write_kb":45445,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2292,"num_bytes_recovered":18743296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'5544","reported_seq":5602,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:35.216300+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:37:35.216300+0000","last_peered":"2024-09-05T16:37:35.216300+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:37:35.216300+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5544,"log_dups_size":0,"ondisk_log_size":5544,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45388960,"num_objects":5541,"num_object_clones":0,"num_object_copies":22164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5541,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5542,"num_write_kb":44333,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3379}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'5603","reported_seq":5649,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:35.292300+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:37:35.292300+0000","last_peered":"2024-09-05T16:37:35.292300+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:37:35.292300+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5603,"log_dups_size":0,"ondisk_log_size":5603,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45899776,"num_objects":5603,"num_object_clones":0,"num_object_copies":22412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5603,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5603,"num_write_kb":44824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3353}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'5507","reported_seq":5576,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:35.991768+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:37:35.991768+0000","last_peered":"2024-09-05T16:37:35.991768+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:37:35.991768+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5507,"log_dups_size":0,"ondisk_log_size":5507,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45061280,"num_objects":5501,"num_object_clones":0,"num_object_copies":22004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5501,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5505,"num_write_kb":44020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3354}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'5600","reported_seq":5668,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:35.892108+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:37:35.892108+0000","last_peered":"2024-09-05T16:37:35.892108+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:37:35.892108+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5600,"log_dups_size":0,"ondisk_log_size":5600,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45831328,"num_objects":5595,"num_object_clones":0,"num_object_copies":22380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5595,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5598,"num_write_kb":44772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3356}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'5661","reported_seq":5721,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:35.294232+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:37:35.294232+0000","last_peered":"2024-09-05T16:37:35.294232+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:37:35.294232+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5661,"log_dups_size":0,"ondisk_log_size":5661,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46303552,"num_objects":5653,"num_object_clones":0,"num_object_copies":22612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5653,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5658,"num_write_kb":45241,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3392}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'5635","reported_seq":5716,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:35.811956+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:37:35.811956+0000","last_peered":"2024-09-05T16:37:35.811956+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:37:35.811956+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5635,"log_dups_size":0,"ondisk_log_size":5635,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46142624,"num_objects":5633,"num_object_clones":0,"num_object_copies":22532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5633,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5633,"num_write_kb":45062,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3371}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'5729","reported_seq":5788,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:35.391296+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:37:35.391296+0000","last_peered":"2024-09-05T16:37:35.391296+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:37:35.391296+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5729,"log_dups_size":0,"ondisk_log_size":5729,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46886640,"num_objects":5724,"num_object_clones":0,"num_object_copies":22896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5726,"num_write_kb":45796,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3413}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'5564","reported_seq":5603,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:35.371066+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:37:35.371066+0000","last_peered":"2024-09-05T16:37:35.371066+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:37:35.371066+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5564,"log_dups_size":0,"ondisk_log_size":5564,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45570640,"num_objects":5563,"num_object_clones":0,"num_object_copies":22252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5563,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5563,"num_write_kb":44503,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3299}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'5651","reported_seq":5890,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:35.894444+0000","last_change":"2024-09-05T16:37:30.634570+0000","last_active":"2024-09-05T16:37:35.894444+0000","last_peered":"2024-09-05T16:37:35.894444+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:37:35.894444+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5651,"log_dups_size":0,"ondisk_log_size":5651,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46283344,"num_objects":5650,"num_object_clones":0,"num_object_copies":22600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2214,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5650,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5650,"num_write_kb":45199,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3436}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'5748","reported_seq":5804,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:35.288152+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:37:35.288152+0000","last_peered":"2024-09-05T16:37:35.288152+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:37:35.288152+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5748,"log_dups_size":0,"ondisk_log_size":5748,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47051936,"num_objects":5744,"num_object_clones":0,"num_object_copies":22976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5744,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5746,"num_write_kb":45964,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3387}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'5630","reported_seq":5710,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:35.871260+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:37:35.871260+0000","last_peered":"2024-09-05T16:37:35.871260+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:37:35.871260+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5630,"log_dups_size":0,"ondisk_log_size":5630,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46093472,"num_objects":5627,"num_object_clones":0,"num_object_copies":22508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5627,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5628,"num_write_kb":45021,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3307}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'5605","reported_seq":5673,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:36.027495+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:37:36.027495+0000","last_peered":"2024-09-05T16:37:36.027495+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:37:36.027495+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5605,"log_dups_size":0,"ondisk_log_size":5605,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45872288,"num_objects":5600,"num_object_clones":0,"num_object_copies":22400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5600,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5603,"num_write_kb":44812,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3363}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":736919744,"num_objects":89961,"num_object_clones":0,"num_object_copies":359844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33827,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89961,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":89988,"num_write_kb":719786,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2397,"num_bytes_recovered":19398656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1334935552,"data_stored":1334935552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90015,"ondisk_log_size":90015,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084332,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":701704,"kb_used_data":359032,"kb_used_omap":4,"kb_used_meta":342651,"kb_avail":93670136,"statfs":{"total":96636764160,"available":95918219264,"internally_reserved":0,"allocated":367648768,"data_stored":367456727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":350875347},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,6,6,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986209,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":431132,"kb_used_data":223112,"kb_used_omap":4,"kb_used_meta":207995,"kb_avail":93940708,"statfs":{"total":96636764160,"available":96195284992,"internally_reserved":0,"allocated":228466688,"data_stored":228274647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":212987602},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,2,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673015,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":706024,"kb_used_data":359704,"kb_used_omap":1,"kb_used_meta":346302,"kb_avail":93665816,"statfs":{"total":96636764160,"available":95913795584,"internally_reserved":0,"allocated":368336896,"data_stored":368144855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":354613701},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,4,4,2,3,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182446,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":707108,"kb_used_data":360340,"kb_used_omap":4,"kb_used_meta":346747,"kb_avail":93664732,"statfs":{"total":96636764160,"available":95912685568,"internally_reserved":0,"allocated":368988160,"data_stored":368796119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":355069645},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,1,1,1,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":368877568,"data_stored":368877568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":229236736,"data_stored":229236736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":368726016,"data_stored":368726016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":368095232,"data_stored":368095232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:37:39.595 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-05T16:37:39.596 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'5666", 'reported_seq': 5712, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:35.225440+0000', 'last_change': '2024-09-05T16:34:56.833497+0000', 'last_active': '2024-09-05T16:37:35.225440+0000', 'last_peered': '2024-09-05T16:37:35.225440+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.828289+0000', 'last_became_peered': '2024-09-05T16:34:56.828289+0000', 'last_unstale': '2024-09-05T16:37:35.225440+0000', 'last_undegraded': '2024-09-05T16:34:56.612160+0000', 'last_fullsized': '2024-09-05T16:34:56.557632+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5666, 'log_dups_size': 0, 'ondisk_log_size': 5666, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46415872, 'num_objects': 5666, 'num_object_clones': 0, 'num_object_copies': 22664, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2262, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5666, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5666, 'num_write_kb': 45328, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3404}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:37:39.596 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-05T16:37:39.596 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "35'5594", 'reported_seq': 5662, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:35.694281+0000', 'last_change': '2024-09-05T16:34:56.825031+0000', 'last_active': '2024-09-05T16:37:35.694281+0000', 'last_peered': '2024-09-05T16:37:35.694281+0000', 'last_clean': '2024-09-05T16:33:53.200762+0000', 'last_became_active': '2024-09-05T16:34:56.820292+0000', 'last_became_peered': '2024-09-05T16:34:56.820292+0000', 'last_unstale': '2024-09-05T16:37:35.694281+0000', 'last_undegraded': '2024-09-05T16:34:56.605776+0000', 'last_fullsized': '2024-09-05T16:34:56.559383+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5594, 'log_dups_size': 0, 'ondisk_log_size': 5594, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45790368, 'num_objects': 5590, 'num_object_clones': 0, 'num_object_copies': 22360, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2269, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5590, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5592, 'num_write_kb': 44732, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3323}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:37:39.596 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:37:39.596 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'5594", 'reported_seq': 5672, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:35.891240+0000', 'last_change': '2024-09-05T16:34:56.867900+0000', 'last_active': '2024-09-05T16:37:35.891240+0000', 'last_peered': '2024-09-05T16:37:35.891240+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:37:35.891240+0000', 'last_undegraded': '2024-09-05T16:34:56.590972+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5594, 'log_dups_size': 0, 'ondisk_log_size': 5594, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45800016, 'num_objects': 5591, 'num_object_clones': 0, 'num_object_copies': 22364, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2251, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5591, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5593, 'num_write_kb': 44734, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3341}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:37:39.596 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-05T16:37:39.596 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'5684", 'reported_seq': 10363, 'reported_epoch': 35, 'state': 'active+clean+scrubbing', 'last_fresh': '2024-09-05T16:37:34.883851+0000', 'last_change': '2024-09-05T16:37:32.894678+0000', 'last_active': '2024-09-05T16:37:34.883851+0000', 'last_peered': '2024-09-05T16:37:34.883851+0000', 'last_clean': '2024-09-05T16:37:34.883851+0000', 'last_became_active': '2024-09-05T16:37:31.984892+0000', 'last_became_peered': '2024-09-05T16:37:31.984892+0000', 'last_unstale': '2024-09-05T16:37:34.883851+0000', 'last_undegraded': '2024-09-05T16:37:34.883851+0000', 'last_fullsized': '2024-09-05T16:37:34.883851+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 35, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 1899, 'log_size': 5684, 'log_dups_size': 0, 'ondisk_log_size': 5684, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'scrubbing for 2s', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46527648, 'num_objects': 5680, 'num_object_clones': 0, 'num_object_copies': 22720, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5680, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5682, 'num_write_kb': 45445, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2292, 'num_bytes_recovered': 18743296, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [1, 0, 2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-05T16:37:39.596 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:37:39.597 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'5544", 'reported_seq': 5602, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:35.216300+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:37:35.216300+0000', 'last_peered': '2024-09-05T16:37:35.216300+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:37:35.216300+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5544, 'log_dups_size': 0, 'ondisk_log_size': 5544, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45388960, 'num_objects': 5541, 'num_object_clones': 0, 'num_object_copies': 22164, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2164, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5541, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5542, 'num_write_kb': 44333, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3379}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:37:39.597 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:37:39.597 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'5603", 'reported_seq': 5649, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:35.292300+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:37:35.292300+0000', 'last_peered': '2024-09-05T16:37:35.292300+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:37:35.292300+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5603, 'log_dups_size': 0, 'ondisk_log_size': 5603, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45899776, 'num_objects': 5603, 'num_object_clones': 0, 'num_object_copies': 22412, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5603, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5603, 'num_write_kb': 44824, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3353}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:37:39.597 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:37:39.597 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'5507", 'reported_seq': 5576, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:35.991768+0000', 'last_change': '2024-09-05T16:34:56.903436+0000', 'last_active': '2024-09-05T16:37:35.991768+0000', 'last_peered': '2024-09-05T16:37:35.991768+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:37:35.991768+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5507, 'log_dups_size': 0, 'ondisk_log_size': 5507, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45061280, 'num_objects': 5501, 'num_object_clones': 0, 'num_object_copies': 22004, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2149, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5501, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5505, 'num_write_kb': 44020, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3354}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:37:39.597 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-05T16:37:39.597 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "35'5600", 'reported_seq': 5668, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:35.892108+0000', 'last_change': '2024-09-05T16:34:56.883319+0000', 'last_active': '2024-09-05T16:37:35.892108+0000', 'last_peered': '2024-09-05T16:37:35.892108+0000', 'last_clean': '2024-09-05T16:33:53.505305+0000', 'last_became_active': '2024-09-05T16:34:56.873389+0000', 'last_became_peered': '2024-09-05T16:34:56.873389+0000', 'last_unstale': '2024-09-05T16:37:35.892108+0000', 'last_undegraded': '2024-09-05T16:34:56.602567+0000', 'last_fullsized': '2024-09-05T16:34:56.559958+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5600, 'log_dups_size': 0, 'ondisk_log_size': 5600, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45831328, 'num_objects': 5595, 'num_object_clones': 0, 'num_object_copies': 22380, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2241, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5595, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5598, 'num_write_kb': 44772, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3356}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:37:39.597 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:37:39.597 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'5661", 'reported_seq': 5721, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:35.294232+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:37:35.294232+0000', 'last_peered': '2024-09-05T16:37:35.294232+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:37:35.294232+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5661, 'log_dups_size': 0, 'ondisk_log_size': 5661, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46303552, 'num_objects': 5653, 'num_object_clones': 0, 'num_object_copies': 22612, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2264, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5653, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5658, 'num_write_kb': 45241, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3392}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:37:39.597 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:37:39.597 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'5635", 'reported_seq': 5716, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:35.811956+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:37:35.811956+0000', 'last_peered': '2024-09-05T16:37:35.811956+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:37:35.811956+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5635, 'log_dups_size': 0, 'ondisk_log_size': 5635, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46142624, 'num_objects': 5633, 'num_object_clones': 0, 'num_object_copies': 22532, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2264, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5633, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5633, 'num_write_kb': 45062, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3371}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:37:39.598 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:37:39.598 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'5729", 'reported_seq': 5788, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:35.391296+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:37:35.391296+0000', 'last_peered': '2024-09-05T16:37:35.391296+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:37:35.391296+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5729, 'log_dups_size': 0, 'ondisk_log_size': 5729, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46886640, 'num_objects': 5724, 'num_object_clones': 0, 'num_object_copies': 22896, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2314, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5724, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5726, 'num_write_kb': 45796, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3413}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:37:39.598 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:37:39.598 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'5564", 'reported_seq': 5603, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:35.371066+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:37:35.371066+0000', 'last_peered': '2024-09-05T16:37:35.371066+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:37:35.371066+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5564, 'log_dups_size': 0, 'ondisk_log_size': 5564, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45570640, 'num_objects': 5563, 'num_object_clones': 0, 'num_object_copies': 22252, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2265, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5563, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5563, 'num_write_kb': 44503, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3299}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:37:39.598 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-05T16:37:39.598 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'5651", 'reported_seq': 5890, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:35.894444+0000', 'last_change': '2024-09-05T16:37:30.634570+0000', 'last_active': '2024-09-05T16:37:35.894444+0000', 'last_peered': '2024-09-05T16:37:35.894444+0000', 'last_clean': '2024-09-05T16:33:45.082039+0000', 'last_became_active': '2024-09-05T16:34:56.792386+0000', 'last_became_peered': '2024-09-05T16:34:56.792386+0000', 'last_unstale': '2024-09-05T16:37:35.894444+0000', 'last_undegraded': '2024-09-05T16:34:56.600595+0000', 'last_fullsized': '2024-09-05T16:34:56.558203+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5651, 'log_dups_size': 0, 'ondisk_log_size': 5651, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46283344, 'num_objects': 5650, 'num_object_clones': 0, 'num_object_copies': 22600, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2214, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5650, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5650, 'num_write_kb': 45199, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 81, 'num_bytes_recovered': 655360, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3436}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:37:39.598 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:37:39.598 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'5748", 'reported_seq': 5804, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:35.288152+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:37:35.288152+0000', 'last_peered': '2024-09-05T16:37:35.288152+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:37:35.288152+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5748, 'log_dups_size': 0, 'ondisk_log_size': 5748, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47051936, 'num_objects': 5744, 'num_object_clones': 0, 'num_object_copies': 22976, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2359, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5744, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5746, 'num_write_kb': 45964, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3387}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:37:39.598 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:37:39.598 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'5630", 'reported_seq': 5710, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:35.871260+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:37:35.871260+0000', 'last_peered': '2024-09-05T16:37:35.871260+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:37:35.871260+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5630, 'log_dups_size': 0, 'ondisk_log_size': 5630, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46093472, 'num_objects': 5627, 'num_object_clones': 0, 'num_object_copies': 22508, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2322, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5627, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5628, 'num_write_kb': 45021, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3307}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:37:39.599 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:37:39.599 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'5605", 'reported_seq': 5673, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:36.027495+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:37:36.027495+0000', 'last_peered': '2024-09-05T16:37:36.027495+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:37:36.027495+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5605, 'log_dups_size': 0, 'ondisk_log_size': 5605, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45872288, 'num_objects': 5600, 'num_object_clones': 0, 'num_object_copies': 22400, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5600, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5603, 'num_write_kb': 44812, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3363}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:37:39.599 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:37:40.334 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:37:40.334 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 33827/359844 objects degraded (9.400%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":263,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+clean+scrubbing","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":89961,"data_bytes":736919744,"bytes_used":2607071232,"bytes_avail":383939985408,"bytes_total":386547056640,"degraded_objects":33827,"degraded_total":359844,"degraded_ratio":0.094004624226053532,"recovering_objects_per_sec":18,"recovering_bytes_per_sec":155590,"recovering_keys_per_sec":0,"num_objects_recovered":152,"num_bytes_recovered":1245184,"num_keys_recovered":0,"write_bytes_sec":2074879,"read_op_per_sec":0,"write_op_per_sec":253},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:37:40.365 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:37:40.366 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:37:40.531 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 198 20 91269 91249 3.59965 1.97656 0.0774205 0.0433929 2024-09-05T16:37:40.984 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:37:40.985 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:37:41.012 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":158,"stamp":"2024-09-05T16:37:40.864015+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":742607904,"num_objects":90655,"num_object_clones":0,"num_object_copies":362620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33827,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":90655,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":84986,"num_write_kb":679781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":105,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90711,"ondisk_log_size":90711,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2584388,"kb_used_data":1314244,"kb_used_omap":14,"kb_used_meta":1270065,"kb_avail":374902972,"statfs":{"total":386547056640,"available":383900643328,"internally_reserved":0,"allocated":1345785856,"data_stored":1345017692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1300547127},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,2,6,5,13,14,19,8,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19606368,"num_objects":2393,"num_object_clones":0,"num_object_copies":9572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2393,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-3303,"num_write_kb":-26413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-2143,"num_bytes_recovered":-17522688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.914454"},"pg_stats":[{"pgid":"1.f","version":"35'5767","reported_seq":5810,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:40.328921+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:37:40.328921+0000","last_peered":"2024-09-05T16:37:40.328921+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:37:40.328921+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5767,"log_dups_size":0,"ondisk_log_size":5767,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47243264,"num_objects":5767,"num_object_clones":0,"num_object_copies":23068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5767,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5767,"num_write_kb":46136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3505}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'5594","reported_seq":5662,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:35.694281+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:37:35.694281+0000","last_peered":"2024-09-05T16:37:35.694281+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:37:35.694281+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5594,"log_dups_size":0,"ondisk_log_size":5594,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45790368,"num_objects":5590,"num_object_clones":0,"num_object_copies":22360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5590,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5592,"num_write_kb":44732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3323}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'5594","reported_seq":5672,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:35.891240+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:37:35.891240+0000","last_peered":"2024-09-05T16:37:35.891240+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:37:35.891240+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5594,"log_dups_size":0,"ondisk_log_size":5594,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45800016,"num_objects":5591,"num_object_clones":0,"num_object_copies":22364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5591,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5593,"num_write_kb":44734,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3341}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'5770","reported_seq":10493,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-05T16:37:39.846148+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:37:39.846148+0000","last_peered":"2024-09-05T16:37:39.846148+0000","last_clean":"2024-09-05T16:37:39.846148+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:37:39.846148+0000","last_undegraded":"2024-09-05T16:37:39.846148+0000","last_fullsized":"2024-09-05T16:37:39.846148+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":5770,"log_dups_size":0,"ondisk_log_size":5770,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47218688,"num_objects":5764,"num_object_clones":0,"num_object_copies":23056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5764,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":70,"num_write_kb":560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'5626","reported_seq":5684,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:40.137708+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:37:40.137708+0000","last_peered":"2024-09-05T16:37:40.137708+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:37:40.137708+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5626,"log_dups_size":0,"ondisk_log_size":5626,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46060704,"num_objects":5623,"num_object_clones":0,"num_object_copies":22492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5623,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5624,"num_write_kb":44989,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3461}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'5681","reported_seq":5727,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:40.283125+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:37:40.283125+0000","last_peered":"2024-09-05T16:37:40.283125+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:37:40.283125+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5681,"log_dups_size":0,"ondisk_log_size":5681,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46538752,"num_objects":5681,"num_object_clones":0,"num_object_copies":22724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5681,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5681,"num_write_kb":45448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3431}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'5507","reported_seq":5576,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:35.991768+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:37:35.991768+0000","last_peered":"2024-09-05T16:37:35.991768+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:37:35.991768+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5507,"log_dups_size":0,"ondisk_log_size":5507,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45061280,"num_objects":5501,"num_object_clones":0,"num_object_copies":22004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5501,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5505,"num_write_kb":44020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3354}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'5600","reported_seq":5668,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:35.892108+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:37:35.892108+0000","last_peered":"2024-09-05T16:37:35.892108+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:37:35.892108+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5600,"log_dups_size":0,"ondisk_log_size":5600,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45831328,"num_objects":5595,"num_object_clones":0,"num_object_copies":22380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5595,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5598,"num_write_kb":44772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3356}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'5743","reported_seq":5803,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:40.170906+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:37:40.170906+0000","last_peered":"2024-09-05T16:37:40.170906+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:37:40.170906+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5743,"log_dups_size":0,"ondisk_log_size":5743,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46975296,"num_objects":5735,"num_object_clones":0,"num_object_copies":22940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5735,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5740,"num_write_kb":45897,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3474}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'5635","reported_seq":5716,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:35.811956+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:37:35.811956+0000","last_peered":"2024-09-05T16:37:35.811956+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:37:35.811956+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5635,"log_dups_size":0,"ondisk_log_size":5635,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46142624,"num_objects":5633,"num_object_clones":0,"num_object_copies":22532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5633,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5633,"num_write_kb":45062,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3371}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'5824","reported_seq":5884,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:40.333727+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:37:40.333727+0000","last_peered":"2024-09-05T16:37:40.333727+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:37:40.333727+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5824,"log_dups_size":0,"ondisk_log_size":5824,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47664880,"num_objects":5819,"num_object_clones":0,"num_object_copies":23276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5819,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5821,"num_write_kb":46556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3508}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'5654","reported_seq":5694,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:40.220060+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:37:40.220060+0000","last_peered":"2024-09-05T16:37:40.220060+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:37:40.220060+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5654,"log_dups_size":0,"ondisk_log_size":5654,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46307920,"num_objects":5653,"num_object_clones":0,"num_object_copies":22612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5653,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5653,"num_write_kb":45223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3389}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'5651","reported_seq":5890,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:35.894444+0000","last_change":"2024-09-05T16:37:30.634570+0000","last_active":"2024-09-05T16:37:35.894444+0000","last_peered":"2024-09-05T16:37:35.894444+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:37:35.894444+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5651,"log_dups_size":0,"ondisk_log_size":5651,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46283344,"num_objects":5650,"num_object_clones":0,"num_object_copies":22600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2214,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5650,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5650,"num_write_kb":45199,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3436}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'5830","reported_seq":5886,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:40.315773+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:37:40.315773+0000","last_peered":"2024-09-05T16:37:40.315773+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:37:40.315773+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5830,"log_dups_size":0,"ondisk_log_size":5830,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47723680,"num_objects":5826,"num_object_clones":0,"num_object_copies":23304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5826,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5828,"num_write_kb":46620,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3469}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'5630","reported_seq":5710,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:35.871260+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:37:35.871260+0000","last_peered":"2024-09-05T16:37:35.871260+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:37:35.871260+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5630,"log_dups_size":0,"ondisk_log_size":5630,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46093472,"num_objects":5627,"num_object_clones":0,"num_object_copies":22508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5627,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5628,"num_write_kb":45021,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3307}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'5605","reported_seq":5673,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:36.027495+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:37:36.027495+0000","last_peered":"2024-09-05T16:37:36.027495+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:37:36.027495+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5605,"log_dups_size":0,"ondisk_log_size":5605,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45872288,"num_objects":5600,"num_object_clones":0,"num_object_copies":22400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5600,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5603,"num_write_kb":44812,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3363}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":742607904,"num_objects":90655,"num_object_clones":0,"num_object_copies":362620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33827,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":90655,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":84986,"num_write_kb":679781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":105,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1346293760,"data_stored":1346293760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90711,"ondisk_log_size":90711,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084333,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":711340,"kb_used_data":364508,"kb_used_omap":4,"kb_used_meta":346811,"kb_avail":93660500,"statfs":{"total":96636764160,"available":95908352000,"internally_reserved":0,"allocated":373256192,"data_stored":373064151,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":355135187},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,0,7,6,8,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986210,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":459916,"kb_used_data":229692,"kb_used_omap":4,"kb_used_meta":230203,"kb_avail":93911924,"statfs":{"total":96636764160,"available":96165810176,"internally_reserved":0,"allocated":235204608,"data_stored":235012567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":235728594},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,1,5,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673015,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":706024,"kb_used_data":359704,"kb_used_omap":1,"kb_used_meta":346302,"kb_avail":93665816,"statfs":{"total":96636764160,"available":95913795584,"internally_reserved":0,"allocated":368336896,"data_stored":368144855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":354613701},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,4,4,2,3,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182446,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":707108,"kb_used_data":360340,"kb_used_omap":4,"kb_used_meta":346747,"kb_avail":93664732,"statfs":{"total":96636764160,"available":95912685568,"internally_reserved":0,"allocated":368988160,"data_stored":368796119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":355069645},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,1,1,1,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":368877568,"data_stored":368877568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":235012096,"data_stored":235012096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":368726016,"data_stored":368726016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":373678080,"data_stored":373678080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:37:41.531 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 199 20 91548 91528 3.59251 2.17969 0.0446036 0.0434833 2024-09-05T16:37:42.531 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout:2024-09-05T16:37:42.539636+0000 min lat: 0.00357551 max lat: 2.29705 avg lat: 0.0435601 2024-09-05T16:37:42.531 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-05T16:37:42.531 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 200 20 91843 91823 3.58607 2.30469 0.0479712 0.0435601 2024-09-05T16:37:43.532 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 201 20 92135 92115 3.57958 2.28125 0.0357951 0.0436378 2024-09-05T16:37:44.014 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:37:44.531 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 202 20 92406 92386 3.57234 2.11719 0.0326688 0.0437202 2024-09-05T16:37:44.610 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:37:44.610 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:37:44.644 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":159,"stamp":"2024-09-05T16:37:42.864884+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":748112928,"num_objects":91327,"num_object_clones":0,"num_object_copies":365308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33788,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91327,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":85658,"num_write_kb":685157,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":144,"num_bytes_recovered":974848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91383,"ondisk_log_size":91383,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2570184,"kb_used_data":1325000,"kb_used_omap":14,"kb_used_meta":1245105,"kb_avail":374917176,"statfs":{"total":386547056640,"available":383915188224,"internally_reserved":0,"allocated":1356800000,"data_stored":1356031836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1274988087},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[9,2,2,0,8,11,25,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":38,"apply_latency_ms":38,"commit_latency_ns":38000000,"apply_latency_ns":38000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":22113120,"num_objects":2699,"num_object_clones":0,"num_object_copies":10796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-188,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2699,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-2997,"num_write_kb":-23965,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-2104,"num_bytes_recovered":-17203200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.912730"},"pg_stats":[{"pgid":"1.f","version":"35'5767","reported_seq":5810,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:40.328921+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:37:40.328921+0000","last_peered":"2024-09-05T16:37:40.328921+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:37:40.328921+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5767,"log_dups_size":0,"ondisk_log_size":5767,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47243264,"num_objects":5767,"num_object_clones":0,"num_object_copies":23068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5767,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5767,"num_write_kb":46136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3505}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'5667","reported_seq":5735,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:41.043553+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:37:41.043553+0000","last_peered":"2024-09-05T16:37:41.043553+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:37:41.043553+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5667,"log_dups_size":0,"ondisk_log_size":5667,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46388384,"num_objects":5663,"num_object_clones":0,"num_object_copies":22652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5665,"num_write_kb":45316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3396}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'5677","reported_seq":5756,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:40.824909+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:37:40.824909+0000","last_peered":"2024-09-05T16:37:40.824909+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:37:40.824909+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5677,"log_dups_size":0,"ondisk_log_size":5677,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46479952,"num_objects":5674,"num_object_clones":0,"num_object_copies":22696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5674,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5676,"num_write_kb":45398,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3424}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'5770","reported_seq":10493,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-05T16:37:39.846148+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:37:39.846148+0000","last_peered":"2024-09-05T16:37:39.846148+0000","last_clean":"2024-09-05T16:37:39.846148+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:37:39.846148+0000","last_undegraded":"2024-09-05T16:37:39.846148+0000","last_fullsized":"2024-09-05T16:37:39.846148+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":5770,"log_dups_size":0,"ondisk_log_size":5770,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47218688,"num_objects":5764,"num_object_clones":0,"num_object_copies":23056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5764,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":70,"num_write_kb":560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'5626","reported_seq":5684,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:40.137708+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:37:40.137708+0000","last_peered":"2024-09-05T16:37:40.137708+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:37:40.137708+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5626,"log_dups_size":0,"ondisk_log_size":5626,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46060704,"num_objects":5623,"num_object_clones":0,"num_object_copies":22492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5623,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5624,"num_write_kb":44989,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3461}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'5681","reported_seq":5727,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:40.283125+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:37:40.283125+0000","last_peered":"2024-09-05T16:37:40.283125+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:37:40.283125+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5681,"log_dups_size":0,"ondisk_log_size":5681,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46538752,"num_objects":5681,"num_object_clones":0,"num_object_copies":22724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5681,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5681,"num_write_kb":45448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3431}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'5602","reported_seq":5671,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:40.984796+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:37:40.984796+0000","last_peered":"2024-09-05T16:37:40.984796+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:37:40.984796+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5602,"log_dups_size":0,"ondisk_log_size":5602,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45839520,"num_objects":5596,"num_object_clones":0,"num_object_copies":22384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5596,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5600,"num_write_kb":44780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3449}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'5685","reported_seq":5752,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:40.876033+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:37:40.876033+0000","last_peered":"2024-09-05T16:37:40.876033+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:37:40.876033+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5685,"log_dups_size":0,"ondisk_log_size":5685,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46527648,"num_objects":5680,"num_object_clones":0,"num_object_copies":22720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5680,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5683,"num_write_kb":45452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3441}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'5743","reported_seq":5803,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:40.170906+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:37:40.170906+0000","last_peered":"2024-09-05T16:37:40.170906+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:37:40.170906+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5743,"log_dups_size":0,"ondisk_log_size":5743,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46975296,"num_objects":5735,"num_object_clones":0,"num_object_copies":22940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5735,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5740,"num_write_kb":45897,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3474}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'5725","reported_seq":5806,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:40.927487+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:37:40.927487+0000","last_peered":"2024-09-05T16:37:40.927487+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:37:40.927487+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5725,"log_dups_size":0,"ondisk_log_size":5725,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46879904,"num_objects":5723,"num_object_clones":0,"num_object_copies":22892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5723,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5723,"num_write_kb":45782,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3461}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'5824","reported_seq":5884,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:40.333727+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:37:40.333727+0000","last_peered":"2024-09-05T16:37:40.333727+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:37:40.333727+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5824,"log_dups_size":0,"ondisk_log_size":5824,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47664880,"num_objects":5819,"num_object_clones":0,"num_object_copies":23276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5819,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5821,"num_write_kb":46556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3508}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'5654","reported_seq":5694,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:40.220060+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:37:40.220060+0000","last_peered":"2024-09-05T16:37:40.220060+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:37:40.220060+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5654,"log_dups_size":0,"ondisk_log_size":5654,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46307920,"num_objects":5653,"num_object_clones":0,"num_object_copies":22612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5653,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5653,"num_write_kb":45223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3389}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'5725","reported_seq":6041,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:40.989066+0000","last_change":"2024-09-05T16:37:30.634570+0000","last_active":"2024-09-05T16:37:40.989066+0000","last_peered":"2024-09-05T16:37:40.989066+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:37:40.989066+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5725,"log_dups_size":0,"ondisk_log_size":5725,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46889552,"num_objects":5724,"num_object_clones":0,"num_object_copies":22896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2175,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5724,"num_write_kb":45791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":120,"num_bytes_recovered":974848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3549}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'5830","reported_seq":5886,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:40.315773+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:37:40.315773+0000","last_peered":"2024-09-05T16:37:40.315773+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:37:40.315773+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5830,"log_dups_size":0,"ondisk_log_size":5830,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47723680,"num_objects":5826,"num_object_clones":0,"num_object_copies":23304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5826,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5828,"num_write_kb":46620,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3469}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'5719","reported_seq":5801,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:40.814166+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:37:40.814166+0000","last_peered":"2024-09-05T16:37:40.814166+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:37:40.814166+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5719,"log_dups_size":0,"ondisk_log_size":5719,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46822560,"num_objects":5716,"num_object_clones":0,"num_object_copies":22864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5716,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5717,"num_write_kb":45733,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3396}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'5688","reported_seq":5755,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:40.962403+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:37:40.962403+0000","last_peered":"2024-09-05T16:37:40.962403+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:37:40.962403+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5688,"log_dups_size":0,"ondisk_log_size":5688,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46552224,"num_objects":5683,"num_object_clones":0,"num_object_copies":22732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5683,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5686,"num_write_kb":45476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3446}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":748112928,"num_objects":91327,"num_object_clones":0,"num_object_copies":365308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33788,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91327,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":85658,"num_write_kb":685157,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":144,"num_bytes_recovered":974848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1357414400,"data_stored":1357414400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91383,"ondisk_log_size":91383,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084333,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":711340,"kb_used_data":364508,"kb_used_omap":4,"kb_used_meta":346811,"kb_avail":93660500,"statfs":{"total":96636764160,"available":95908352000,"internally_reserved":0,"allocated":373256192,"data_stored":373064151,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":355135187},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,0,7,6,8,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986210,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":459916,"kb_used_data":229692,"kb_used_omap":4,"kb_used_meta":230203,"kb_avail":93911924,"statfs":{"total":96636764160,"available":96165810176,"internally_reserved":0,"allocated":235204608,"data_stored":235012567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":235728594},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,1,5,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673016,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":698960,"kb_used_data":365120,"kb_used_omap":1,"kb_used_meta":333822,"kb_avail":93672880,"statfs":{"total":96636764160,"available":95921029120,"internally_reserved":0,"allocated":373882880,"data_stored":373690839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":341834181},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,1,0,0,0,8,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182447,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699968,"kb_used_data":365680,"kb_used_omap":4,"kb_used_meta":334267,"kb_avail":93671872,"statfs":{"total":96636764160,"available":95919996928,"internally_reserved":0,"allocated":374456320,"data_stored":374264279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":342290125},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,0,0,0,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":374415360,"data_stored":374415360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":235012096,"data_stored":235012096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":374308864,"data_stored":374308864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":373678080,"data_stored":373678080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:37:44.646 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:37:45.226 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:37:45.226 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:37:45.251 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":160,"stamp":"2024-09-05T16:37:44.865772+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":748112928,"num_objects":91327,"num_object_clones":0,"num_object_copies":365308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33788,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91327,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":85658,"num_write_kb":685157,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":144,"num_bytes_recovered":974848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91383,"ondisk_log_size":91383,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2570184,"kb_used_data":1325000,"kb_used_omap":14,"kb_used_meta":1245105,"kb_avail":374917176,"statfs":{"total":386547056640,"available":383915188224,"internally_reserved":0,"allocated":1356800000,"data_stored":1356031836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1274988087},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[9,2,2,0,8,11,25,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":38,"apply_latency_ms":38,"commit_latency_ns":38000000,"apply_latency_ns":38000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":22113120,"num_objects":2699,"num_object_clones":0,"num_object_copies":10796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-188,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2699,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-2997,"num_write_kb":-23965,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-2104,"num_bytes_recovered":-17203200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004611"},"pg_stats":[{"pgid":"1.f","version":"35'5767","reported_seq":5810,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:40.328921+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:37:40.328921+0000","last_peered":"2024-09-05T16:37:40.328921+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:37:40.328921+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5767,"log_dups_size":0,"ondisk_log_size":5767,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47243264,"num_objects":5767,"num_object_clones":0,"num_object_copies":23068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5767,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5767,"num_write_kb":46136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3505}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'5667","reported_seq":5735,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:41.043553+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:37:41.043553+0000","last_peered":"2024-09-05T16:37:41.043553+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:37:41.043553+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5667,"log_dups_size":0,"ondisk_log_size":5667,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46388384,"num_objects":5663,"num_object_clones":0,"num_object_copies":22652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5665,"num_write_kb":45316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3396}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'5677","reported_seq":5756,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:40.824909+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:37:40.824909+0000","last_peered":"2024-09-05T16:37:40.824909+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:37:40.824909+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5677,"log_dups_size":0,"ondisk_log_size":5677,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46479952,"num_objects":5674,"num_object_clones":0,"num_object_copies":22696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5674,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5676,"num_write_kb":45398,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3424}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'5770","reported_seq":10493,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-05T16:37:39.846148+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:37:39.846148+0000","last_peered":"2024-09-05T16:37:39.846148+0000","last_clean":"2024-09-05T16:37:39.846148+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:37:39.846148+0000","last_undegraded":"2024-09-05T16:37:39.846148+0000","last_fullsized":"2024-09-05T16:37:39.846148+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":5770,"log_dups_size":0,"ondisk_log_size":5770,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47218688,"num_objects":5764,"num_object_clones":0,"num_object_copies":23056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5764,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":70,"num_write_kb":560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'5626","reported_seq":5684,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:40.137708+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:37:40.137708+0000","last_peered":"2024-09-05T16:37:40.137708+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:37:40.137708+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5626,"log_dups_size":0,"ondisk_log_size":5626,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46060704,"num_objects":5623,"num_object_clones":0,"num_object_copies":22492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5623,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5624,"num_write_kb":44989,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3461}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'5681","reported_seq":5727,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:40.283125+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:37:40.283125+0000","last_peered":"2024-09-05T16:37:40.283125+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:37:40.283125+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5681,"log_dups_size":0,"ondisk_log_size":5681,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46538752,"num_objects":5681,"num_object_clones":0,"num_object_copies":22724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5681,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5681,"num_write_kb":45448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3431}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'5602","reported_seq":5671,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:40.984796+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:37:40.984796+0000","last_peered":"2024-09-05T16:37:40.984796+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:37:40.984796+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5602,"log_dups_size":0,"ondisk_log_size":5602,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45839520,"num_objects":5596,"num_object_clones":0,"num_object_copies":22384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5596,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5600,"num_write_kb":44780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3449}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'5685","reported_seq":5752,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:40.876033+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:37:40.876033+0000","last_peered":"2024-09-05T16:37:40.876033+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:37:40.876033+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5685,"log_dups_size":0,"ondisk_log_size":5685,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46527648,"num_objects":5680,"num_object_clones":0,"num_object_copies":22720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5680,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5683,"num_write_kb":45452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3441}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'5743","reported_seq":5803,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:40.170906+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:37:40.170906+0000","last_peered":"2024-09-05T16:37:40.170906+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:37:40.170906+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5743,"log_dups_size":0,"ondisk_log_size":5743,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46975296,"num_objects":5735,"num_object_clones":0,"num_object_copies":22940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5735,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5740,"num_write_kb":45897,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3474}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'5725","reported_seq":5806,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:40.927487+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:37:40.927487+0000","last_peered":"2024-09-05T16:37:40.927487+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:37:40.927487+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5725,"log_dups_size":0,"ondisk_log_size":5725,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46879904,"num_objects":5723,"num_object_clones":0,"num_object_copies":22892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5723,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5723,"num_write_kb":45782,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3461}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'5824","reported_seq":5884,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:40.333727+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:37:40.333727+0000","last_peered":"2024-09-05T16:37:40.333727+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:37:40.333727+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5824,"log_dups_size":0,"ondisk_log_size":5824,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47664880,"num_objects":5819,"num_object_clones":0,"num_object_copies":23276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5819,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5821,"num_write_kb":46556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3508}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'5654","reported_seq":5694,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:40.220060+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:37:40.220060+0000","last_peered":"2024-09-05T16:37:40.220060+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:37:40.220060+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5654,"log_dups_size":0,"ondisk_log_size":5654,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46307920,"num_objects":5653,"num_object_clones":0,"num_object_copies":22612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5653,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5653,"num_write_kb":45223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3389}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'5725","reported_seq":6041,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:40.989066+0000","last_change":"2024-09-05T16:37:30.634570+0000","last_active":"2024-09-05T16:37:40.989066+0000","last_peered":"2024-09-05T16:37:40.989066+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:37:40.989066+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5725,"log_dups_size":0,"ondisk_log_size":5725,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46889552,"num_objects":5724,"num_object_clones":0,"num_object_copies":22896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2175,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5724,"num_write_kb":45791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":120,"num_bytes_recovered":974848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3549}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'5830","reported_seq":5886,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:40.315773+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:37:40.315773+0000","last_peered":"2024-09-05T16:37:40.315773+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:37:40.315773+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5830,"log_dups_size":0,"ondisk_log_size":5830,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47723680,"num_objects":5826,"num_object_clones":0,"num_object_copies":23304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5826,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5828,"num_write_kb":46620,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3469}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'5719","reported_seq":5801,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:40.814166+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:37:40.814166+0000","last_peered":"2024-09-05T16:37:40.814166+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:37:40.814166+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5719,"log_dups_size":0,"ondisk_log_size":5719,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46822560,"num_objects":5716,"num_object_clones":0,"num_object_copies":22864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5716,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5717,"num_write_kb":45733,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3396}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'5688","reported_seq":5755,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:40.962403+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:37:40.962403+0000","last_peered":"2024-09-05T16:37:40.962403+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:37:40.962403+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5688,"log_dups_size":0,"ondisk_log_size":5688,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46552224,"num_objects":5683,"num_object_clones":0,"num_object_copies":22732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5683,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5686,"num_write_kb":45476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3446}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":748112928,"num_objects":91327,"num_object_clones":0,"num_object_copies":365308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33788,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91327,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":85658,"num_write_kb":685157,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":144,"num_bytes_recovered":974848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1357414400,"data_stored":1357414400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91383,"ondisk_log_size":91383,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084333,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":711340,"kb_used_data":364508,"kb_used_omap":4,"kb_used_meta":346811,"kb_avail":93660500,"statfs":{"total":96636764160,"available":95908352000,"internally_reserved":0,"allocated":373256192,"data_stored":373064151,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":355135187},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,0,7,6,8,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986210,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":459916,"kb_used_data":229692,"kb_used_omap":4,"kb_used_meta":230203,"kb_avail":93911924,"statfs":{"total":96636764160,"available":96165810176,"internally_reserved":0,"allocated":235204608,"data_stored":235012567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":235728594},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,1,5,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673016,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":698960,"kb_used_data":365120,"kb_used_omap":1,"kb_used_meta":333822,"kb_avail":93672880,"statfs":{"total":96636764160,"available":95921029120,"internally_reserved":0,"allocated":373882880,"data_stored":373690839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":341834181},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,1,0,0,0,8,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182447,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699968,"kb_used_data":365680,"kb_used_omap":4,"kb_used_meta":334267,"kb_avail":93671872,"statfs":{"total":96636764160,"available":95919996928,"internally_reserved":0,"allocated":374456320,"data_stored":374264279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":342290125},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,0,0,0,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":374415360,"data_stored":374415360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":235012096,"data_stored":235012096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":374308864,"data_stored":374308864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":373678080,"data_stored":373678080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:37:45.252 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-05T16:37:45.253 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'5767", 'reported_seq': 5810, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:40.328921+0000', 'last_change': '2024-09-05T16:34:56.833497+0000', 'last_active': '2024-09-05T16:37:40.328921+0000', 'last_peered': '2024-09-05T16:37:40.328921+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.828289+0000', 'last_became_peered': '2024-09-05T16:34:56.828289+0000', 'last_unstale': '2024-09-05T16:37:40.328921+0000', 'last_undegraded': '2024-09-05T16:34:56.612160+0000', 'last_fullsized': '2024-09-05T16:34:56.557632+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5767, 'log_dups_size': 0, 'ondisk_log_size': 5767, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47243264, 'num_objects': 5767, 'num_object_clones': 0, 'num_object_copies': 23068, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2262, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5767, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5767, 'num_write_kb': 46136, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3505}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:37:45.253 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-05T16:37:45.253 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "35'5667", 'reported_seq': 5735, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:41.043553+0000', 'last_change': '2024-09-05T16:34:56.825031+0000', 'last_active': '2024-09-05T16:37:41.043553+0000', 'last_peered': '2024-09-05T16:37:41.043553+0000', 'last_clean': '2024-09-05T16:33:53.200762+0000', 'last_became_active': '2024-09-05T16:34:56.820292+0000', 'last_became_peered': '2024-09-05T16:34:56.820292+0000', 'last_unstale': '2024-09-05T16:37:41.043553+0000', 'last_undegraded': '2024-09-05T16:34:56.605776+0000', 'last_fullsized': '2024-09-05T16:34:56.559383+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5667, 'log_dups_size': 0, 'ondisk_log_size': 5667, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46388384, 'num_objects': 5663, 'num_object_clones': 0, 'num_object_copies': 22652, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2269, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5663, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5665, 'num_write_kb': 45316, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3396}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:37:45.253 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:37:45.253 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'5677", 'reported_seq': 5756, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:40.824909+0000', 'last_change': '2024-09-05T16:34:56.867900+0000', 'last_active': '2024-09-05T16:37:40.824909+0000', 'last_peered': '2024-09-05T16:37:40.824909+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:37:40.824909+0000', 'last_undegraded': '2024-09-05T16:34:56.590972+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5677, 'log_dups_size': 0, 'ondisk_log_size': 5677, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46479952, 'num_objects': 5674, 'num_object_clones': 0, 'num_object_copies': 22696, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2251, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5674, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5676, 'num_write_kb': 45398, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3424}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:37:45.253 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:37:45.253 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'5626", 'reported_seq': 5684, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:40.137708+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:37:40.137708+0000', 'last_peered': '2024-09-05T16:37:40.137708+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:37:40.137708+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5626, 'log_dups_size': 0, 'ondisk_log_size': 5626, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46060704, 'num_objects': 5623, 'num_object_clones': 0, 'num_object_copies': 22492, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2164, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5623, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5624, 'num_write_kb': 44989, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3461}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:37:45.253 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:37:45.253 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'5681", 'reported_seq': 5727, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:40.283125+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:37:40.283125+0000', 'last_peered': '2024-09-05T16:37:40.283125+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:37:40.283125+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5681, 'log_dups_size': 0, 'ondisk_log_size': 5681, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46538752, 'num_objects': 5681, 'num_object_clones': 0, 'num_object_copies': 22724, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5681, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5681, 'num_write_kb': 45448, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3431}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:37:45.253 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:37:45.256 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'5602", 'reported_seq': 5671, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:40.984796+0000', 'last_change': '2024-09-05T16:34:56.903436+0000', 'last_active': '2024-09-05T16:37:40.984796+0000', 'last_peered': '2024-09-05T16:37:40.984796+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:37:40.984796+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5602, 'log_dups_size': 0, 'ondisk_log_size': 5602, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45839520, 'num_objects': 5596, 'num_object_clones': 0, 'num_object_copies': 22384, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2149, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5596, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5600, 'num_write_kb': 44780, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3449}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:37:45.256 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-05T16:37:45.256 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "35'5685", 'reported_seq': 5752, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:40.876033+0000', 'last_change': '2024-09-05T16:34:56.883319+0000', 'last_active': '2024-09-05T16:37:40.876033+0000', 'last_peered': '2024-09-05T16:37:40.876033+0000', 'last_clean': '2024-09-05T16:33:53.505305+0000', 'last_became_active': '2024-09-05T16:34:56.873389+0000', 'last_became_peered': '2024-09-05T16:34:56.873389+0000', 'last_unstale': '2024-09-05T16:37:40.876033+0000', 'last_undegraded': '2024-09-05T16:34:56.602567+0000', 'last_fullsized': '2024-09-05T16:34:56.559958+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5685, 'log_dups_size': 0, 'ondisk_log_size': 5685, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46527648, 'num_objects': 5680, 'num_object_clones': 0, 'num_object_copies': 22720, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2241, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5680, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5683, 'num_write_kb': 45452, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3441}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:37:45.256 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:37:45.256 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'5743", 'reported_seq': 5803, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:40.170906+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:37:40.170906+0000', 'last_peered': '2024-09-05T16:37:40.170906+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:37:40.170906+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5743, 'log_dups_size': 0, 'ondisk_log_size': 5743, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46975296, 'num_objects': 5735, 'num_object_clones': 0, 'num_object_copies': 22940, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2264, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5735, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5740, 'num_write_kb': 45897, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3474}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:37:45.256 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:37:45.256 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'5725", 'reported_seq': 5806, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:40.927487+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:37:40.927487+0000', 'last_peered': '2024-09-05T16:37:40.927487+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:37:40.927487+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5725, 'log_dups_size': 0, 'ondisk_log_size': 5725, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46879904, 'num_objects': 5723, 'num_object_clones': 0, 'num_object_copies': 22892, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2264, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5723, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5723, 'num_write_kb': 45782, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3461}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:37:45.256 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:37:45.256 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'5824", 'reported_seq': 5884, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:40.333727+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:37:40.333727+0000', 'last_peered': '2024-09-05T16:37:40.333727+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:37:40.333727+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5824, 'log_dups_size': 0, 'ondisk_log_size': 5824, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47664880, 'num_objects': 5819, 'num_object_clones': 0, 'num_object_copies': 23276, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2314, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5819, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5821, 'num_write_kb': 46556, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3508}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:37:45.257 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:37:45.257 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'5654", 'reported_seq': 5694, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:40.220060+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:37:40.220060+0000', 'last_peered': '2024-09-05T16:37:40.220060+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:37:40.220060+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5654, 'log_dups_size': 0, 'ondisk_log_size': 5654, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46307920, 'num_objects': 5653, 'num_object_clones': 0, 'num_object_copies': 22612, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2265, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5653, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5653, 'num_write_kb': 45223, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3389}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:37:45.257 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-05T16:37:45.257 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'5725", 'reported_seq': 6041, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:40.989066+0000', 'last_change': '2024-09-05T16:37:30.634570+0000', 'last_active': '2024-09-05T16:37:40.989066+0000', 'last_peered': '2024-09-05T16:37:40.989066+0000', 'last_clean': '2024-09-05T16:33:45.082039+0000', 'last_became_active': '2024-09-05T16:34:56.792386+0000', 'last_became_peered': '2024-09-05T16:34:56.792386+0000', 'last_unstale': '2024-09-05T16:37:40.989066+0000', 'last_undegraded': '2024-09-05T16:34:56.600595+0000', 'last_fullsized': '2024-09-05T16:34:56.558203+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5725, 'log_dups_size': 0, 'ondisk_log_size': 5725, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46889552, 'num_objects': 5724, 'num_object_clones': 0, 'num_object_copies': 22896, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2175, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5724, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5724, 'num_write_kb': 45791, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 120, 'num_bytes_recovered': 974848, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3549}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:37:45.257 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:37:45.257 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'5830", 'reported_seq': 5886, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:40.315773+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:37:40.315773+0000', 'last_peered': '2024-09-05T16:37:40.315773+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:37:40.315773+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5830, 'log_dups_size': 0, 'ondisk_log_size': 5830, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47723680, 'num_objects': 5826, 'num_object_clones': 0, 'num_object_copies': 23304, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2359, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5826, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5828, 'num_write_kb': 46620, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3469}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:37:45.257 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:37:45.257 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'5719", 'reported_seq': 5801, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:40.814166+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:37:40.814166+0000', 'last_peered': '2024-09-05T16:37:40.814166+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:37:40.814166+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5719, 'log_dups_size': 0, 'ondisk_log_size': 5719, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46822560, 'num_objects': 5716, 'num_object_clones': 0, 'num_object_copies': 22864, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2322, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5716, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5717, 'num_write_kb': 45733, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3396}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:37:45.257 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:37:45.257 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'5688", 'reported_seq': 5755, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:40.962403+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:37:40.962403+0000', 'last_peered': '2024-09-05T16:37:40.962403+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:37:40.962403+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5688, 'log_dups_size': 0, 'ondisk_log_size': 5688, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46552224, 'num_objects': 5683, 'num_object_clones': 0, 'num_object_copies': 22732, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5683, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5686, 'num_write_kb': 45476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3446}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:37:45.258 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:37:45.534 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 203 20 92668 92648 3.56482 2.04688 0.103852 0.0438184 2024-09-05T16:37:46.015 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:37:46.015 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 33788/365308 objects degraded (9.249%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":269,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":91327,"data_bytes":748112928,"bytes_used":2631868416,"bytes_avail":383915188224,"bytes_total":386547056640,"degraded_objects":33788,"degraded_total":365308,"degraded_ratio":0.092491815125866406},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:37:46.053 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-09-05T16:37:46.054 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:37:46.532 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 204 20 92924 92904 3.55715 2 0.110024 0.0439106 2024-09-05T16:37:46.634 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:37:46.634 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:37:46.658 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":160,"stamp":"2024-09-05T16:37:44.865772+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":748112928,"num_objects":91327,"num_object_clones":0,"num_object_copies":365308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33788,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91327,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":85658,"num_write_kb":685157,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":144,"num_bytes_recovered":974848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91383,"ondisk_log_size":91383,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2570184,"kb_used_data":1325000,"kb_used_omap":14,"kb_used_meta":1245105,"kb_avail":374917176,"statfs":{"total":386547056640,"available":383915188224,"internally_reserved":0,"allocated":1356800000,"data_stored":1356031836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1274988087},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[9,2,2,0,8,11,25,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":38,"apply_latency_ms":38,"commit_latency_ns":38000000,"apply_latency_ns":38000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":22113120,"num_objects":2699,"num_object_clones":0,"num_object_copies":10796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-188,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2699,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-2997,"num_write_kb":-23965,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-2104,"num_bytes_recovered":-17203200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004611"},"pg_stats":[{"pgid":"1.f","version":"35'5767","reported_seq":5810,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:40.328921+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:37:40.328921+0000","last_peered":"2024-09-05T16:37:40.328921+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:37:40.328921+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5767,"log_dups_size":0,"ondisk_log_size":5767,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47243264,"num_objects":5767,"num_object_clones":0,"num_object_copies":23068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5767,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5767,"num_write_kb":46136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3505}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'5667","reported_seq":5735,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:41.043553+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:37:41.043553+0000","last_peered":"2024-09-05T16:37:41.043553+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:37:41.043553+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5667,"log_dups_size":0,"ondisk_log_size":5667,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46388384,"num_objects":5663,"num_object_clones":0,"num_object_copies":22652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5665,"num_write_kb":45316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3396}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'5677","reported_seq":5756,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:40.824909+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:37:40.824909+0000","last_peered":"2024-09-05T16:37:40.824909+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:37:40.824909+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5677,"log_dups_size":0,"ondisk_log_size":5677,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46479952,"num_objects":5674,"num_object_clones":0,"num_object_copies":22696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5674,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5676,"num_write_kb":45398,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3424}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'5770","reported_seq":10493,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-05T16:37:39.846148+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:37:39.846148+0000","last_peered":"2024-09-05T16:37:39.846148+0000","last_clean":"2024-09-05T16:37:39.846148+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:37:39.846148+0000","last_undegraded":"2024-09-05T16:37:39.846148+0000","last_fullsized":"2024-09-05T16:37:39.846148+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":5770,"log_dups_size":0,"ondisk_log_size":5770,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47218688,"num_objects":5764,"num_object_clones":0,"num_object_copies":23056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5764,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":70,"num_write_kb":560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'5626","reported_seq":5684,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:40.137708+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:37:40.137708+0000","last_peered":"2024-09-05T16:37:40.137708+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:37:40.137708+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5626,"log_dups_size":0,"ondisk_log_size":5626,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46060704,"num_objects":5623,"num_object_clones":0,"num_object_copies":22492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5623,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5624,"num_write_kb":44989,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3461}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'5681","reported_seq":5727,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:40.283125+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:37:40.283125+0000","last_peered":"2024-09-05T16:37:40.283125+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:37:40.283125+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5681,"log_dups_size":0,"ondisk_log_size":5681,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46538752,"num_objects":5681,"num_object_clones":0,"num_object_copies":22724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5681,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5681,"num_write_kb":45448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3431}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'5602","reported_seq":5671,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:40.984796+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:37:40.984796+0000","last_peered":"2024-09-05T16:37:40.984796+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:37:40.984796+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5602,"log_dups_size":0,"ondisk_log_size":5602,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45839520,"num_objects":5596,"num_object_clones":0,"num_object_copies":22384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5596,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5600,"num_write_kb":44780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3449}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'5685","reported_seq":5752,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:40.876033+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:37:40.876033+0000","last_peered":"2024-09-05T16:37:40.876033+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:37:40.876033+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5685,"log_dups_size":0,"ondisk_log_size":5685,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46527648,"num_objects":5680,"num_object_clones":0,"num_object_copies":22720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5680,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5683,"num_write_kb":45452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3441}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'5743","reported_seq":5803,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:40.170906+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:37:40.170906+0000","last_peered":"2024-09-05T16:37:40.170906+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:37:40.170906+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5743,"log_dups_size":0,"ondisk_log_size":5743,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46975296,"num_objects":5735,"num_object_clones":0,"num_object_copies":22940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5735,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5740,"num_write_kb":45897,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3474}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'5725","reported_seq":5806,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:40.927487+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:37:40.927487+0000","last_peered":"2024-09-05T16:37:40.927487+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:37:40.927487+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5725,"log_dups_size":0,"ondisk_log_size":5725,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46879904,"num_objects":5723,"num_object_clones":0,"num_object_copies":22892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5723,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5723,"num_write_kb":45782,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3461}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'5824","reported_seq":5884,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:40.333727+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:37:40.333727+0000","last_peered":"2024-09-05T16:37:40.333727+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:37:40.333727+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5824,"log_dups_size":0,"ondisk_log_size":5824,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47664880,"num_objects":5819,"num_object_clones":0,"num_object_copies":23276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5819,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5821,"num_write_kb":46556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3508}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'5654","reported_seq":5694,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:40.220060+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:37:40.220060+0000","last_peered":"2024-09-05T16:37:40.220060+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:37:40.220060+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5654,"log_dups_size":0,"ondisk_log_size":5654,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46307920,"num_objects":5653,"num_object_clones":0,"num_object_copies":22612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5653,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5653,"num_write_kb":45223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3389}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'5725","reported_seq":6041,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:40.989066+0000","last_change":"2024-09-05T16:37:30.634570+0000","last_active":"2024-09-05T16:37:40.989066+0000","last_peered":"2024-09-05T16:37:40.989066+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:37:40.989066+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5725,"log_dups_size":0,"ondisk_log_size":5725,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46889552,"num_objects":5724,"num_object_clones":0,"num_object_copies":22896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2175,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5724,"num_write_kb":45791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":120,"num_bytes_recovered":974848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3549}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'5830","reported_seq":5886,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:40.315773+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:37:40.315773+0000","last_peered":"2024-09-05T16:37:40.315773+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:37:40.315773+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5830,"log_dups_size":0,"ondisk_log_size":5830,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47723680,"num_objects":5826,"num_object_clones":0,"num_object_copies":23304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5826,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5828,"num_write_kb":46620,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3469}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'5719","reported_seq":5801,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:40.814166+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:37:40.814166+0000","last_peered":"2024-09-05T16:37:40.814166+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:37:40.814166+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5719,"log_dups_size":0,"ondisk_log_size":5719,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46822560,"num_objects":5716,"num_object_clones":0,"num_object_copies":22864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5716,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5717,"num_write_kb":45733,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3396}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'5688","reported_seq":5755,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:40.962403+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:37:40.962403+0000","last_peered":"2024-09-05T16:37:40.962403+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:37:40.962403+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5688,"log_dups_size":0,"ondisk_log_size":5688,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46552224,"num_objects":5683,"num_object_clones":0,"num_object_copies":22732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5683,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5686,"num_write_kb":45476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3446}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":748112928,"num_objects":91327,"num_object_clones":0,"num_object_copies":365308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33788,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91327,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":85658,"num_write_kb":685157,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":144,"num_bytes_recovered":974848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1357414400,"data_stored":1357414400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91383,"ondisk_log_size":91383,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084333,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":711340,"kb_used_data":364508,"kb_used_omap":4,"kb_used_meta":346811,"kb_avail":93660500,"statfs":{"total":96636764160,"available":95908352000,"internally_reserved":0,"allocated":373256192,"data_stored":373064151,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":355135187},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,0,7,6,8,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986210,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":459916,"kb_used_data":229692,"kb_used_omap":4,"kb_used_meta":230203,"kb_avail":93911924,"statfs":{"total":96636764160,"available":96165810176,"internally_reserved":0,"allocated":235204608,"data_stored":235012567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":235728594},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,1,5,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673016,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":698960,"kb_used_data":365120,"kb_used_omap":1,"kb_used_meta":333822,"kb_avail":93672880,"statfs":{"total":96636764160,"available":95921029120,"internally_reserved":0,"allocated":373882880,"data_stored":373690839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":341834181},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,1,0,0,0,8,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182447,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699968,"kb_used_data":365680,"kb_used_omap":4,"kb_used_meta":334267,"kb_avail":93671872,"statfs":{"total":96636764160,"available":95919996928,"internally_reserved":0,"allocated":374456320,"data_stored":374264279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":342290125},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,0,0,0,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":374415360,"data_stored":374415360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":235012096,"data_stored":235012096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":374308864,"data_stored":374308864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":373678080,"data_stored":373678080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:37:47.532 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 205 20 93215 93195 3.55089 2.27344 0.0536937 0.0439907 2024-09-05T16:37:48.532 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 206 20 93509 93489 3.5448 2.29688 0.0838135 0.0440667 2024-09-05T16:37:49.532 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 207 20 93800 93780 3.53866 2.27344 0.0756025 0.044143 2024-09-05T16:37:49.660 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:37:50.266 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:37:50.266 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:37:50.290 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":162,"stamp":"2024-09-05T16:37:48.867545+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":759475232,"num_objects":92714,"num_object_clones":0,"num_object_copies":370856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33732,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":92714,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":87045,"num_write_kb":696253,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":200,"num_bytes_recovered":1433600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92770,"ondisk_log_size":92770,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2592824,"kb_used_data":1347636,"kb_used_omap":14,"kb_used_meta":1245105,"kb_avail":374894536,"statfs":{"total":386547056640,"available":383892004864,"internally_reserved":0,"allocated":1379979264,"data_stored":1379211100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1274988087},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,12,6,5,6,10,17,8,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":22555488,"num_objects":2753,"num_object_clones":0,"num_object_copies":11012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-95,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2753,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-2943,"num_write_kb":-23533,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-2197,"num_bytes_recovered":-17965056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004954"},"pg_stats":[{"pgid":"1.f","version":"35'5850","reported_seq":5896,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:45.230047+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:37:45.230047+0000","last_peered":"2024-09-05T16:37:45.230047+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:37:45.230047+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5850,"log_dups_size":0,"ondisk_log_size":5850,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47923200,"num_objects":5850,"num_object_clones":0,"num_object_copies":23400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5850,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5850,"num_write_kb":46800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3588}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'5744","reported_seq":5812,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:45.850223+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:37:45.850223+0000","last_peered":"2024-09-05T16:37:45.850223+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:37:45.850223+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5744,"log_dups_size":0,"ondisk_log_size":5744,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47019168,"num_objects":5740,"num_object_clones":0,"num_object_copies":22960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5740,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5742,"num_write_kb":45932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3473}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'5771","reported_seq":5848,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:46.048097+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:37:46.048097+0000","last_peered":"2024-09-05T16:37:46.048097+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:37:46.048097+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5771,"log_dups_size":0,"ondisk_log_size":5771,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47250000,"num_objects":5768,"num_object_clones":0,"num_object_copies":23072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5768,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5770,"num_write_kb":46150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3518}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'5865","reported_seq":10588,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-05T16:37:44.918844+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:37:44.918844+0000","last_peered":"2024-09-05T16:37:44.918844+0000","last_clean":"2024-09-05T16:37:44.918844+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:37:44.918844+0000","last_undegraded":"2024-09-05T16:37:44.918844+0000","last_fullsized":"2024-09-05T16:37:44.918844+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":5865,"log_dups_size":0,"ondisk_log_size":5865,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47996928,"num_objects":5859,"num_object_clones":0,"num_object_copies":23436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":165,"num_write_kb":1320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'5715","reported_seq":5772,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:45.203658+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:37:45.203658+0000","last_peered":"2024-09-05T16:37:45.203658+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:37:45.203658+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5715,"log_dups_size":0,"ondisk_log_size":5715,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46789792,"num_objects":5712,"num_object_clones":0,"num_object_copies":22848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5712,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5713,"num_write_kb":45701,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3550}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'5763","reported_seq":5809,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:45.404063+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:37:45.404063+0000","last_peered":"2024-09-05T16:37:45.404063+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:37:45.404063+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5763,"log_dups_size":0,"ondisk_log_size":5763,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47210496,"num_objects":5763,"num_object_clones":0,"num_object_copies":23052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5763,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5763,"num_write_kb":46104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3513}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'5679","reported_seq":5748,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:45.768285+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:37:45.768285+0000","last_peered":"2024-09-05T16:37:45.768285+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:37:45.768285+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5679,"log_dups_size":0,"ondisk_log_size":5679,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46470304,"num_objects":5673,"num_object_clones":0,"num_object_copies":22692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5673,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5677,"num_write_kb":45396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3526}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'5778","reported_seq":5846,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:45.964622+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:37:45.964622+0000","last_peered":"2024-09-05T16:37:45.964622+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:37:45.964622+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5778,"log_dups_size":0,"ondisk_log_size":5778,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47289504,"num_objects":5773,"num_object_clones":0,"num_object_copies":23092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5773,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5776,"num_write_kb":46196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'5830","reported_seq":5890,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:45.424019+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:37:45.424019+0000","last_peered":"2024-09-05T16:37:45.424019+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:37:45.424019+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5830,"log_dups_size":0,"ondisk_log_size":5830,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47688000,"num_objects":5822,"num_object_clones":0,"num_object_copies":23288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5822,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5827,"num_write_kb":46593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3561}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'5800","reported_seq":5880,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:45.932179+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:37:45.932179+0000","last_peered":"2024-09-05T16:37:45.932179+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:37:45.932179+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5800,"log_dups_size":0,"ondisk_log_size":5800,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47494304,"num_objects":5798,"num_object_clones":0,"num_object_copies":23192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5798,"num_write_kb":46382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3536}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'5917","reported_seq":5977,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:45.373132+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:37:45.373132+0000","last_peered":"2024-09-05T16:37:45.373132+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:37:45.373132+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5917,"log_dups_size":0,"ondisk_log_size":5917,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48426736,"num_objects":5912,"num_object_clones":0,"num_object_copies":23648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5912,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5914,"num_write_kb":47300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3601}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'5743","reported_seq":5783,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:45.424100+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:37:45.424100+0000","last_peered":"2024-09-05T16:37:45.424100+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:37:45.424100+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5743,"log_dups_size":0,"ondisk_log_size":5743,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47037008,"num_objects":5742,"num_object_clones":0,"num_object_copies":22968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5742,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5742,"num_write_kb":45935,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3478}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'5802","reported_seq":6231,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:46.024585+0000","last_change":"2024-09-05T16:37:30.634570+0000","last_active":"2024-09-05T16:37:46.024585+0000","last_peered":"2024-09-05T16:37:46.024585+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:37:46.024585+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5802,"log_dups_size":0,"ondisk_log_size":5802,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47520336,"num_objects":5801,"num_object_clones":0,"num_object_copies":23204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5801,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5801,"num_write_kb":46407,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":176,"num_bytes_recovered":1433600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3682}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'5916","reported_seq":5972,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:45.278851+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:37:45.278851+0000","last_peered":"2024-09-05T16:37:45.278851+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:37:45.278851+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5916,"log_dups_size":0,"ondisk_log_size":5916,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48428192,"num_objects":5912,"num_object_clones":0,"num_object_copies":23648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5912,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5914,"num_write_kb":47308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3555}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'5828","reported_seq":5910,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:45.923627+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:37:45.923627+0000","last_peered":"2024-09-05T16:37:45.923627+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:37:45.923627+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5828,"log_dups_size":0,"ondisk_log_size":5828,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47715488,"num_objects":5825,"num_object_clones":0,"num_object_copies":23300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5825,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5826,"num_write_kb":46605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3505}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'5769","reported_seq":5837,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:45.964811+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:37:45.964811+0000","last_peered":"2024-09-05T16:37:45.964811+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:37:45.964811+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5769,"log_dups_size":0,"ondisk_log_size":5769,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47215776,"num_objects":5764,"num_object_clones":0,"num_object_copies":23056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5764,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5767,"num_write_kb":46124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3527}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":759475232,"num_objects":92714,"num_object_clones":0,"num_object_copies":370856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33732,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":92714,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":87045,"num_write_kb":696253,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":200,"num_bytes_recovered":1433600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1380659200,"data_stored":1380659200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92770,"ondisk_log_size":92770,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084334,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":704420,"kb_used_data":370068,"kb_used_omap":4,"kb_used_meta":334331,"kb_avail":93667420,"statfs":{"total":96636764160,"available":95915438080,"internally_reserved":0,"allocated":378949632,"data_stored":378757591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":342355667},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,4,1,5,7,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986211,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":469952,"kb_used_data":235568,"kb_used_omap":4,"kb_used_meta":234363,"kb_avail":93901888,"statfs":{"total":96636764160,"available":96155533312,"internally_reserved":0,"allocated":241221632,"data_stored":241029591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":239988434},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,3,5,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673017,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708668,"kb_used_data":370664,"kb_used_omap":1,"kb_used_meta":337982,"kb_avail":93663172,"statfs":{"total":96636764160,"available":95911088128,"internally_reserved":0,"allocated":379559936,"data_stored":379367895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":346094021},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,1,0,0,0,8,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182448,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709784,"kb_used_data":371336,"kb_used_omap":4,"kb_used_meta":338427,"kb_avail":93662056,"statfs":{"total":96636764160,"available":95909945344,"internally_reserved":0,"allocated":380248064,"data_stored":380056023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":346549965},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,10,3,1,2,0,2,4,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":380121088,"data_stored":380121088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":240963584,"data_stored":240963584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":380084224,"data_stored":380084224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":379490304,"data_stored":379490304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:37:50.291 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:37:50.532 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 208 20 94057 94037 3.5313 2.00781 0.0637236 0.0442363 2024-09-05T16:37:50.904 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:37:50.905 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:37:50.942 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":163,"stamp":"2024-09-05T16:37:50.868987+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":765447200,"num_objects":93443,"num_object_clones":0,"num_object_copies":373772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33732,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":93443,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":87774,"num_write_kb":702085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":200,"num_bytes_recovered":1433600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":93499,"ondisk_log_size":93499,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2612300,"kb_used_data":1358792,"kb_used_omap":14,"kb_used_meta":1253425,"kb_avail":374875060,"statfs":{"total":386547056640,"available":383872061440,"internally_reserved":0,"allocated":1391403008,"data_stored":1390634844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1283507767},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,10,7,8,7,14,19,6,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":28527456,"num_objects":3482,"num_object_clones":0,"num_object_copies":13928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-95,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-2214,"num_write_kb":-17701,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-2197,"num_bytes_recovered":-17965056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005586"},"pg_stats":[{"pgid":"1.f","version":"35'5946","reported_seq":5992,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:50.292692+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:37:50.292692+0000","last_peered":"2024-09-05T16:37:50.292692+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:37:50.292692+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5946,"log_dups_size":0,"ondisk_log_size":5946,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48709632,"num_objects":5946,"num_object_clones":0,"num_object_copies":23784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5946,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5946,"num_write_kb":47568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3684}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'5744","reported_seq":5812,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:45.850223+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:37:45.850223+0000","last_peered":"2024-09-05T16:37:45.850223+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:37:45.850223+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5744,"log_dups_size":0,"ondisk_log_size":5744,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47019168,"num_objects":5740,"num_object_clones":0,"num_object_copies":22960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5740,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5742,"num_write_kb":45932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3473}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'5771","reported_seq":5848,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:46.048097+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:37:46.048097+0000","last_peered":"2024-09-05T16:37:46.048097+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:37:46.048097+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5771,"log_dups_size":0,"ondisk_log_size":5771,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47250000,"num_objects":5768,"num_object_clones":0,"num_object_copies":23072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5768,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5770,"num_write_kb":46150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3518}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'5967","reported_seq":10690,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-05T16:37:49.849352+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:37:49.849352+0000","last_peered":"2024-09-05T16:37:49.849352+0000","last_clean":"2024-09-05T16:37:49.849352+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:37:49.849352+0000","last_undegraded":"2024-09-05T16:37:49.849352+0000","last_fullsized":"2024-09-05T16:37:49.849352+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":5967,"log_dups_size":0,"ondisk_log_size":5967,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48832512,"num_objects":5961,"num_object_clones":0,"num_object_copies":23844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5961,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":267,"num_write_kb":2136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'5792","reported_seq":5849,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:50.490298+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:37:50.490298+0000","last_peered":"2024-09-05T16:37:50.490298+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:37:50.490298+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5792,"log_dups_size":0,"ondisk_log_size":5792,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47420576,"num_objects":5789,"num_object_clones":0,"num_object_copies":23156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5789,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5790,"num_write_kb":46317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3627}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'5835","reported_seq":5881,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:50.236375+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:37:50.236375+0000","last_peered":"2024-09-05T16:37:50.236375+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:37:50.236375+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5835,"log_dups_size":0,"ondisk_log_size":5835,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47800320,"num_objects":5835,"num_object_clones":0,"num_object_copies":23340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5835,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5835,"num_write_kb":46680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3585}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'5679","reported_seq":5748,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:45.768285+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:37:45.768285+0000","last_peered":"2024-09-05T16:37:45.768285+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:37:45.768285+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5679,"log_dups_size":0,"ondisk_log_size":5679,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46470304,"num_objects":5673,"num_object_clones":0,"num_object_copies":22692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5673,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5677,"num_write_kb":45396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3526}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'5778","reported_seq":5846,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:45.964622+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:37:45.964622+0000","last_peered":"2024-09-05T16:37:45.964622+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:37:45.964622+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5778,"log_dups_size":0,"ondisk_log_size":5778,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47289504,"num_objects":5773,"num_object_clones":0,"num_object_copies":23092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5773,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5776,"num_write_kb":46196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'5921","reported_seq":5981,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:50.236038+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:37:50.236038+0000","last_peered":"2024-09-05T16:37:50.236038+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:37:50.236038+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5921,"log_dups_size":0,"ondisk_log_size":5921,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48433472,"num_objects":5913,"num_object_clones":0,"num_object_copies":23652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5913,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5918,"num_write_kb":47321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3652}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'5800","reported_seq":5880,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:45.932179+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:37:45.932179+0000","last_peered":"2024-09-05T16:37:45.932179+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:37:45.932179+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5800,"log_dups_size":0,"ondisk_log_size":5800,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47494304,"num_objects":5798,"num_object_clones":0,"num_object_copies":23192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5798,"num_write_kb":46382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3536}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'6015","reported_seq":6073,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:50.522124+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:37:50.522124+0000","last_peered":"2024-09-05T16:37:50.522124+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:37:50.522124+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6015,"log_dups_size":0,"ondisk_log_size":6015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49229552,"num_objects":6010,"num_object_clones":0,"num_object_copies":24040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6010,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6012,"num_write_kb":48084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3699}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'5834","reported_seq":5874,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:50.475868+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:37:50.475868+0000","last_peered":"2024-09-05T16:37:50.475868+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:37:50.475868+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5834,"log_dups_size":0,"ondisk_log_size":5834,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47782480,"num_objects":5833,"num_object_clones":0,"num_object_copies":23332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5833,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5833,"num_write_kb":46663,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3569}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'5802","reported_seq":6231,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:46.024585+0000","last_change":"2024-09-05T16:37:30.634570+0000","last_active":"2024-09-05T16:37:46.024585+0000","last_peered":"2024-09-05T16:37:46.024585+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:37:46.024585+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5802,"log_dups_size":0,"ondisk_log_size":5802,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47520336,"num_objects":5801,"num_object_clones":0,"num_object_copies":23204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5801,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5801,"num_write_kb":46407,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":176,"num_bytes_recovered":1433600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3682}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'6018","reported_seq":6073,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:50.507880+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:37:50.507880+0000","last_peered":"2024-09-05T16:37:50.507880+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:37:50.507880+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6018,"log_dups_size":0,"ondisk_log_size":6018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49263776,"num_objects":6014,"num_object_clones":0,"num_object_copies":24056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6014,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6016,"num_write_kb":48124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3657}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'5828","reported_seq":5910,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:45.923627+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:37:45.923627+0000","last_peered":"2024-09-05T16:37:45.923627+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:37:45.923627+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5828,"log_dups_size":0,"ondisk_log_size":5828,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47715488,"num_objects":5825,"num_object_clones":0,"num_object_copies":23300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5825,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5826,"num_write_kb":46605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3505}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'5769","reported_seq":5837,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:45.964811+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:37:45.964811+0000","last_peered":"2024-09-05T16:37:45.964811+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:37:45.964811+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5769,"log_dups_size":0,"ondisk_log_size":5769,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47215776,"num_objects":5764,"num_object_clones":0,"num_object_copies":23056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5764,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5767,"num_write_kb":46124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3527}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":765447200,"num_objects":93443,"num_object_clones":0,"num_object_copies":373772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33732,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":93443,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":87774,"num_write_kb":702085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":200,"num_bytes_recovered":1433600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1392410624,"data_stored":1392410624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":93499,"ondisk_log_size":93499,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084335,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":714004,"kb_used_data":375492,"kb_used_omap":4,"kb_used_meta":338491,"kb_avail":93657836,"statfs":{"total":96636764160,"available":95905624064,"internally_reserved":0,"allocated":384503808,"data_stored":384311767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":346615507},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,6,3,9],"upper_bound":64},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986212,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":479844,"kb_used_data":241300,"kb_used_omap":4,"kb_used_meta":238523,"kb_avail":93891996,"statfs":{"total":96636764160,"available":96145403904,"internally_reserved":0,"allocated":247091200,"data_stored":246899159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":244248274},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,1,2,5,9],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673017,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708668,"kb_used_data":370664,"kb_used_omap":1,"kb_used_meta":337982,"kb_avail":93663172,"statfs":{"total":96636764160,"available":95911088128,"internally_reserved":0,"allocated":379559936,"data_stored":379367895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":346094021},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,1,0,0,0,8,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182448,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709784,"kb_used_data":371336,"kb_used_omap":4,"kb_used_meta":338427,"kb_avail":93662056,"statfs":{"total":96636764160,"available":95909945344,"internally_reserved":0,"allocated":380248064,"data_stored":380056023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":346549965},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,10,3,1,2,0,2,4,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":380121088,"data_stored":380121088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":246980608,"data_stored":246980608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":380084224,"data_stored":380084224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":385224704,"data_stored":385224704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:37:50.943 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-05T16:37:50.943 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'5946", 'reported_seq': 5992, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:50.292692+0000', 'last_change': '2024-09-05T16:34:56.833497+0000', 'last_active': '2024-09-05T16:37:50.292692+0000', 'last_peered': '2024-09-05T16:37:50.292692+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.828289+0000', 'last_became_peered': '2024-09-05T16:34:56.828289+0000', 'last_unstale': '2024-09-05T16:37:50.292692+0000', 'last_undegraded': '2024-09-05T16:34:56.612160+0000', 'last_fullsized': '2024-09-05T16:34:56.557632+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5946, 'log_dups_size': 0, 'ondisk_log_size': 5946, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48709632, 'num_objects': 5946, 'num_object_clones': 0, 'num_object_copies': 23784, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2262, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5946, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5946, 'num_write_kb': 47568, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3684}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:37:50.943 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-05T16:37:50.943 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "35'5744", 'reported_seq': 5812, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:45.850223+0000', 'last_change': '2024-09-05T16:34:56.825031+0000', 'last_active': '2024-09-05T16:37:45.850223+0000', 'last_peered': '2024-09-05T16:37:45.850223+0000', 'last_clean': '2024-09-05T16:33:53.200762+0000', 'last_became_active': '2024-09-05T16:34:56.820292+0000', 'last_became_peered': '2024-09-05T16:34:56.820292+0000', 'last_unstale': '2024-09-05T16:37:45.850223+0000', 'last_undegraded': '2024-09-05T16:34:56.605776+0000', 'last_fullsized': '2024-09-05T16:34:56.559383+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5744, 'log_dups_size': 0, 'ondisk_log_size': 5744, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47019168, 'num_objects': 5740, 'num_object_clones': 0, 'num_object_copies': 22960, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2269, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5740, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5742, 'num_write_kb': 45932, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3473}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:37:50.943 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:37:50.943 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'5771", 'reported_seq': 5848, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:46.048097+0000', 'last_change': '2024-09-05T16:34:56.867900+0000', 'last_active': '2024-09-05T16:37:46.048097+0000', 'last_peered': '2024-09-05T16:37:46.048097+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:37:46.048097+0000', 'last_undegraded': '2024-09-05T16:34:56.590972+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5771, 'log_dups_size': 0, 'ondisk_log_size': 5771, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47250000, 'num_objects': 5768, 'num_object_clones': 0, 'num_object_copies': 23072, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2251, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5768, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5770, 'num_write_kb': 46150, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3518}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:37:50.943 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:37:50.943 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'5792", 'reported_seq': 5849, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:50.490298+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:37:50.490298+0000', 'last_peered': '2024-09-05T16:37:50.490298+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:37:50.490298+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5792, 'log_dups_size': 0, 'ondisk_log_size': 5792, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47420576, 'num_objects': 5789, 'num_object_clones': 0, 'num_object_copies': 23156, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2164, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5789, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5790, 'num_write_kb': 46317, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3627}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:37:50.943 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:37:50.943 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'5835", 'reported_seq': 5881, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:50.236375+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:37:50.236375+0000', 'last_peered': '2024-09-05T16:37:50.236375+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:37:50.236375+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5835, 'log_dups_size': 0, 'ondisk_log_size': 5835, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47800320, 'num_objects': 5835, 'num_object_clones': 0, 'num_object_copies': 23340, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5835, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5835, 'num_write_kb': 46680, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3585}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:37:50.944 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:37:50.944 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'5679", 'reported_seq': 5748, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:45.768285+0000', 'last_change': '2024-09-05T16:34:56.903436+0000', 'last_active': '2024-09-05T16:37:45.768285+0000', 'last_peered': '2024-09-05T16:37:45.768285+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:37:45.768285+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5679, 'log_dups_size': 0, 'ondisk_log_size': 5679, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46470304, 'num_objects': 5673, 'num_object_clones': 0, 'num_object_copies': 22692, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2149, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5673, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5677, 'num_write_kb': 45396, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3526}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:37:50.944 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-05T16:37:50.944 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "35'5778", 'reported_seq': 5846, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:45.964622+0000', 'last_change': '2024-09-05T16:34:56.883319+0000', 'last_active': '2024-09-05T16:37:45.964622+0000', 'last_peered': '2024-09-05T16:37:45.964622+0000', 'last_clean': '2024-09-05T16:33:53.505305+0000', 'last_became_active': '2024-09-05T16:34:56.873389+0000', 'last_became_peered': '2024-09-05T16:34:56.873389+0000', 'last_unstale': '2024-09-05T16:37:45.964622+0000', 'last_undegraded': '2024-09-05T16:34:56.602567+0000', 'last_fullsized': '2024-09-05T16:34:56.559958+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5778, 'log_dups_size': 0, 'ondisk_log_size': 5778, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47289504, 'num_objects': 5773, 'num_object_clones': 0, 'num_object_copies': 23092, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2241, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5773, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5776, 'num_write_kb': 46196, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3534}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:37:50.944 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:37:50.944 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'5921", 'reported_seq': 5981, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:50.236038+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:37:50.236038+0000', 'last_peered': '2024-09-05T16:37:50.236038+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:37:50.236038+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5921, 'log_dups_size': 0, 'ondisk_log_size': 5921, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48433472, 'num_objects': 5913, 'num_object_clones': 0, 'num_object_copies': 23652, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2264, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5913, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5918, 'num_write_kb': 47321, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3652}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:37:50.944 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:37:50.944 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'5800", 'reported_seq': 5880, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:45.932179+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:37:45.932179+0000', 'last_peered': '2024-09-05T16:37:45.932179+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:37:45.932179+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5800, 'log_dups_size': 0, 'ondisk_log_size': 5800, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47494304, 'num_objects': 5798, 'num_object_clones': 0, 'num_object_copies': 23192, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2264, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5798, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5798, 'num_write_kb': 46382, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3536}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:37:50.944 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:37:50.944 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'6015", 'reported_seq': 6073, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:50.522124+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:37:50.522124+0000', 'last_peered': '2024-09-05T16:37:50.522124+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:37:50.522124+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6015, 'log_dups_size': 0, 'ondisk_log_size': 6015, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49229552, 'num_objects': 6010, 'num_object_clones': 0, 'num_object_copies': 24040, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2314, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6010, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6012, 'num_write_kb': 48084, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3699}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:37:50.945 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:37:50.945 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'5834", 'reported_seq': 5874, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:50.475868+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:37:50.475868+0000', 'last_peered': '2024-09-05T16:37:50.475868+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:37:50.475868+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5834, 'log_dups_size': 0, 'ondisk_log_size': 5834, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47782480, 'num_objects': 5833, 'num_object_clones': 0, 'num_object_copies': 23332, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2265, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5833, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5833, 'num_write_kb': 46663, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3569}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:37:50.945 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-05T16:37:50.945 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'5802", 'reported_seq': 6231, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:46.024585+0000', 'last_change': '2024-09-05T16:37:30.634570+0000', 'last_active': '2024-09-05T16:37:46.024585+0000', 'last_peered': '2024-09-05T16:37:46.024585+0000', 'last_clean': '2024-09-05T16:33:45.082039+0000', 'last_became_active': '2024-09-05T16:34:56.792386+0000', 'last_became_peered': '2024-09-05T16:34:56.792386+0000', 'last_unstale': '2024-09-05T16:37:46.024585+0000', 'last_undegraded': '2024-09-05T16:34:56.600595+0000', 'last_fullsized': '2024-09-05T16:34:56.558203+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5802, 'log_dups_size': 0, 'ondisk_log_size': 5802, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47520336, 'num_objects': 5801, 'num_object_clones': 0, 'num_object_copies': 23204, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2119, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5801, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5801, 'num_write_kb': 46407, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 176, 'num_bytes_recovered': 1433600, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3682}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:37:50.945 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:37:50.945 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'6018", 'reported_seq': 6073, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:50.507880+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:37:50.507880+0000', 'last_peered': '2024-09-05T16:37:50.507880+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:37:50.507880+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6018, 'log_dups_size': 0, 'ondisk_log_size': 6018, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49263776, 'num_objects': 6014, 'num_object_clones': 0, 'num_object_copies': 24056, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2359, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6014, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6016, 'num_write_kb': 48124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3657}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:37:50.945 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:37:50.945 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'5828", 'reported_seq': 5910, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:45.923627+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:37:45.923627+0000', 'last_peered': '2024-09-05T16:37:45.923627+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:37:45.923627+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5828, 'log_dups_size': 0, 'ondisk_log_size': 5828, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47715488, 'num_objects': 5825, 'num_object_clones': 0, 'num_object_copies': 23300, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2322, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5825, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5826, 'num_write_kb': 46605, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3505}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:37:50.945 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:37:50.945 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'5769", 'reported_seq': 5837, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:45.964811+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:37:45.964811+0000', 'last_peered': '2024-09-05T16:37:45.964811+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:37:45.964811+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5769, 'log_dups_size': 0, 'ondisk_log_size': 5769, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47215776, 'num_objects': 5764, 'num_object_clones': 0, 'num_object_copies': 23056, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5764, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5767, 'num_write_kb': 46124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3527}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:37:50.946 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:37:51.532 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 209 20 94316 94296 3.52409 2.02344 0.0611238 0.0443231 2024-09-05T16:37:51.683 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:37:51.683 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 33732/373772 objects degraded (9.025%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":274,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+clean","count":1}],"num_pgs":16,"num_pools":1,"num_objects":93443,"data_bytes":765447200,"bytes_used":2674995200,"bytes_avail":383872061440,"bytes_total":386547056640,"degraded_objects":33732,"degraded_total":373772,"degraded_ratio":0.090247530580139762},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:37:51.707 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-09-05T16:37:51.707 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:37:52.314 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:37:52.314 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:37:52.339 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":163,"stamp":"2024-09-05T16:37:50.868987+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":765447200,"num_objects":93443,"num_object_clones":0,"num_object_copies":373772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33732,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":93443,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":87774,"num_write_kb":702085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":200,"num_bytes_recovered":1433600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":93499,"ondisk_log_size":93499,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2612300,"kb_used_data":1358792,"kb_used_omap":14,"kb_used_meta":1253425,"kb_avail":374875060,"statfs":{"total":386547056640,"available":383872061440,"internally_reserved":0,"allocated":1391403008,"data_stored":1390634844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1283507767},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,10,7,8,7,14,19,6,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":28527456,"num_objects":3482,"num_object_clones":0,"num_object_copies":13928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-95,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-2214,"num_write_kb":-17701,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-2197,"num_bytes_recovered":-17965056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005586"},"pg_stats":[{"pgid":"1.f","version":"35'5946","reported_seq":5992,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:50.292692+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:37:50.292692+0000","last_peered":"2024-09-05T16:37:50.292692+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:37:50.292692+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5946,"log_dups_size":0,"ondisk_log_size":5946,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48709632,"num_objects":5946,"num_object_clones":0,"num_object_copies":23784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5946,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5946,"num_write_kb":47568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3684}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'5744","reported_seq":5812,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:45.850223+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:37:45.850223+0000","last_peered":"2024-09-05T16:37:45.850223+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:37:45.850223+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5744,"log_dups_size":0,"ondisk_log_size":5744,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47019168,"num_objects":5740,"num_object_clones":0,"num_object_copies":22960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5740,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5742,"num_write_kb":45932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3473}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'5771","reported_seq":5848,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:46.048097+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:37:46.048097+0000","last_peered":"2024-09-05T16:37:46.048097+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:37:46.048097+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5771,"log_dups_size":0,"ondisk_log_size":5771,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47250000,"num_objects":5768,"num_object_clones":0,"num_object_copies":23072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5768,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5770,"num_write_kb":46150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3518}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'5967","reported_seq":10690,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-05T16:37:49.849352+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:37:49.849352+0000","last_peered":"2024-09-05T16:37:49.849352+0000","last_clean":"2024-09-05T16:37:49.849352+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:37:49.849352+0000","last_undegraded":"2024-09-05T16:37:49.849352+0000","last_fullsized":"2024-09-05T16:37:49.849352+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":5967,"log_dups_size":0,"ondisk_log_size":5967,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48832512,"num_objects":5961,"num_object_clones":0,"num_object_copies":23844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5961,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":267,"num_write_kb":2136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'5792","reported_seq":5849,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:50.490298+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:37:50.490298+0000","last_peered":"2024-09-05T16:37:50.490298+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:37:50.490298+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5792,"log_dups_size":0,"ondisk_log_size":5792,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47420576,"num_objects":5789,"num_object_clones":0,"num_object_copies":23156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5789,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5790,"num_write_kb":46317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3627}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'5835","reported_seq":5881,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:50.236375+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:37:50.236375+0000","last_peered":"2024-09-05T16:37:50.236375+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:37:50.236375+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5835,"log_dups_size":0,"ondisk_log_size":5835,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47800320,"num_objects":5835,"num_object_clones":0,"num_object_copies":23340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5835,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5835,"num_write_kb":46680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3585}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'5679","reported_seq":5748,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:45.768285+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:37:45.768285+0000","last_peered":"2024-09-05T16:37:45.768285+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:37:45.768285+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5679,"log_dups_size":0,"ondisk_log_size":5679,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46470304,"num_objects":5673,"num_object_clones":0,"num_object_copies":22692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5673,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5677,"num_write_kb":45396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3526}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'5778","reported_seq":5846,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:45.964622+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:37:45.964622+0000","last_peered":"2024-09-05T16:37:45.964622+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:37:45.964622+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5778,"log_dups_size":0,"ondisk_log_size":5778,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47289504,"num_objects":5773,"num_object_clones":0,"num_object_copies":23092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5773,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5776,"num_write_kb":46196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'5921","reported_seq":5981,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:50.236038+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:37:50.236038+0000","last_peered":"2024-09-05T16:37:50.236038+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:37:50.236038+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5921,"log_dups_size":0,"ondisk_log_size":5921,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48433472,"num_objects":5913,"num_object_clones":0,"num_object_copies":23652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5913,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5918,"num_write_kb":47321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3652}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'5800","reported_seq":5880,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:45.932179+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:37:45.932179+0000","last_peered":"2024-09-05T16:37:45.932179+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:37:45.932179+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5800,"log_dups_size":0,"ondisk_log_size":5800,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47494304,"num_objects":5798,"num_object_clones":0,"num_object_copies":23192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5798,"num_write_kb":46382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3536}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'6015","reported_seq":6073,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:50.522124+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:37:50.522124+0000","last_peered":"2024-09-05T16:37:50.522124+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:37:50.522124+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6015,"log_dups_size":0,"ondisk_log_size":6015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49229552,"num_objects":6010,"num_object_clones":0,"num_object_copies":24040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6010,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6012,"num_write_kb":48084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3699}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'5834","reported_seq":5874,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:50.475868+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:37:50.475868+0000","last_peered":"2024-09-05T16:37:50.475868+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:37:50.475868+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5834,"log_dups_size":0,"ondisk_log_size":5834,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47782480,"num_objects":5833,"num_object_clones":0,"num_object_copies":23332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5833,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5833,"num_write_kb":46663,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3569}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'5802","reported_seq":6231,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:46.024585+0000","last_change":"2024-09-05T16:37:30.634570+0000","last_active":"2024-09-05T16:37:46.024585+0000","last_peered":"2024-09-05T16:37:46.024585+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:37:46.024585+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5802,"log_dups_size":0,"ondisk_log_size":5802,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47520336,"num_objects":5801,"num_object_clones":0,"num_object_copies":23204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5801,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5801,"num_write_kb":46407,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":176,"num_bytes_recovered":1433600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3682}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'6018","reported_seq":6073,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:50.507880+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:37:50.507880+0000","last_peered":"2024-09-05T16:37:50.507880+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:37:50.507880+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6018,"log_dups_size":0,"ondisk_log_size":6018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49263776,"num_objects":6014,"num_object_clones":0,"num_object_copies":24056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6014,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6016,"num_write_kb":48124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3657}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'5828","reported_seq":5910,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:45.923627+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:37:45.923627+0000","last_peered":"2024-09-05T16:37:45.923627+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:37:45.923627+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5828,"log_dups_size":0,"ondisk_log_size":5828,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47715488,"num_objects":5825,"num_object_clones":0,"num_object_copies":23300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5825,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5826,"num_write_kb":46605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3505}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'5769","reported_seq":5837,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:45.964811+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:37:45.964811+0000","last_peered":"2024-09-05T16:37:45.964811+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:37:45.964811+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5769,"log_dups_size":0,"ondisk_log_size":5769,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47215776,"num_objects":5764,"num_object_clones":0,"num_object_copies":23056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5764,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5767,"num_write_kb":46124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3527}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":765447200,"num_objects":93443,"num_object_clones":0,"num_object_copies":373772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33732,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":93443,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":87774,"num_write_kb":702085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":200,"num_bytes_recovered":1433600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1392410624,"data_stored":1392410624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":93499,"ondisk_log_size":93499,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084335,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":714004,"kb_used_data":375492,"kb_used_omap":4,"kb_used_meta":338491,"kb_avail":93657836,"statfs":{"total":96636764160,"available":95905624064,"internally_reserved":0,"allocated":384503808,"data_stored":384311767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":346615507},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,6,3,9],"upper_bound":64},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986212,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":479844,"kb_used_data":241300,"kb_used_omap":4,"kb_used_meta":238523,"kb_avail":93891996,"statfs":{"total":96636764160,"available":96145403904,"internally_reserved":0,"allocated":247091200,"data_stored":246899159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":244248274},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,1,2,5,9],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673017,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708668,"kb_used_data":370664,"kb_used_omap":1,"kb_used_meta":337982,"kb_avail":93663172,"statfs":{"total":96636764160,"available":95911088128,"internally_reserved":0,"allocated":379559936,"data_stored":379367895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":346094021},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,1,0,0,0,8,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182448,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709784,"kb_used_data":371336,"kb_used_omap":4,"kb_used_meta":338427,"kb_avail":93662056,"statfs":{"total":96636764160,"available":95909945344,"internally_reserved":0,"allocated":380248064,"data_stored":380056023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":346549965},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,10,3,1,2,0,2,4,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":380121088,"data_stored":380121088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":246980608,"data_stored":246980608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":380084224,"data_stored":380084224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":385224704,"data_stored":385224704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:37:52.532 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 210 20 94578 94558 3.51705 2.04688 0.065046 0.0444149 2024-09-05T16:37:53.533 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 211 20 94861 94841 3.51086 2.21094 0.0487003 0.0444897 2024-09-05T16:37:54.532 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 212 20 95149 95129 3.50491 2.25 0.0729021 0.0445681 2024-09-05T16:37:55.341 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:37:55.532 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 213 20 95434 95414 3.49891 2.22656 0.0607454 0.044644 2024-09-05T16:37:55.916 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:37:55.918 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:37:55.944 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":165,"stamp":"2024-09-05T16:37:54.870679+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":771107872,"num_objects":94134,"num_object_clones":0,"num_object_copies":376536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33685,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":94134,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":88465,"num_write_kb":707613,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":1818624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94190,"ondisk_log_size":94190,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2631724,"kb_used_data":1369900,"kb_used_omap":14,"kb_used_meta":1261745,"kb_avail":374855636,"statfs":{"total":386547056640,"available":383852171264,"internally_reserved":0,"allocated":1402777600,"data_stored":1402009436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1292027447},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,7,8,15,23,11,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":22994944,"num_objects":2807,"num_object_clones":0,"num_object_copies":11228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2807,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2807,"num_write_kb":22456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":843776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005794"},"pg_stats":[{"pgid":"1.f","version":"35'5946","reported_seq":5992,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:50.292692+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:37:50.292692+0000","last_peered":"2024-09-05T16:37:50.292692+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:37:50.292692+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5946,"log_dups_size":0,"ondisk_log_size":5946,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48709632,"num_objects":5946,"num_object_clones":0,"num_object_copies":23784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5946,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5946,"num_write_kb":47568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3684}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'5811","reported_seq":5879,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:51.042275+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:37:51.042275+0000","last_peered":"2024-09-05T16:37:51.042275+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:37:51.042275+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5811,"log_dups_size":0,"ondisk_log_size":5811,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47568032,"num_objects":5807,"num_object_clones":0,"num_object_copies":23228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5807,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5809,"num_write_kb":46468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3540}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'5862","reported_seq":5941,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:51.042216+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:37:51.042216+0000","last_peered":"2024-09-05T16:37:51.042216+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:37:51.042216+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5862,"log_dups_size":0,"ondisk_log_size":5862,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47995472,"num_objects":5859,"num_object_clones":0,"num_object_copies":23436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5861,"num_write_kb":46878,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3609}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'5967","reported_seq":10690,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-05T16:37:49.849352+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:37:49.849352+0000","last_peered":"2024-09-05T16:37:49.849352+0000","last_clean":"2024-09-05T16:37:49.849352+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:37:49.849352+0000","last_undegraded":"2024-09-05T16:37:49.849352+0000","last_fullsized":"2024-09-05T16:37:49.849352+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":5967,"log_dups_size":0,"ondisk_log_size":5967,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48832512,"num_objects":5961,"num_object_clones":0,"num_object_copies":23844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5961,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":267,"num_write_kb":2136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'5792","reported_seq":5849,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:50.490298+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:37:50.490298+0000","last_peered":"2024-09-05T16:37:50.490298+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:37:50.490298+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5792,"log_dups_size":0,"ondisk_log_size":5792,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47420576,"num_objects":5789,"num_object_clones":0,"num_object_copies":23156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5789,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5790,"num_write_kb":46317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3627}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'5835","reported_seq":5881,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:50.236375+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:37:50.236375+0000","last_peered":"2024-09-05T16:37:50.236375+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:37:50.236375+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5835,"log_dups_size":0,"ondisk_log_size":5835,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47800320,"num_objects":5835,"num_object_clones":0,"num_object_copies":23340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5835,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5835,"num_write_kb":46680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3585}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'5770","reported_seq":5839,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:51.037476+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:37:51.037476+0000","last_peered":"2024-09-05T16:37:51.037476+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:37:51.037476+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5770,"log_dups_size":0,"ondisk_log_size":5770,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47215776,"num_objects":5764,"num_object_clones":0,"num_object_copies":23056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5764,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5768,"num_write_kb":46124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3617}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'5858","reported_seq":5926,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:51.146016+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:37:51.146016+0000","last_peered":"2024-09-05T16:37:51.146016+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:37:51.146016+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5858,"log_dups_size":0,"ondisk_log_size":5858,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47944864,"num_objects":5853,"num_object_clones":0,"num_object_copies":23412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5853,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5856,"num_write_kb":46836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3614}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'5921","reported_seq":5981,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:50.236038+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:37:50.236038+0000","last_peered":"2024-09-05T16:37:50.236038+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:37:50.236038+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5921,"log_dups_size":0,"ondisk_log_size":5921,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48433472,"num_objects":5913,"num_object_clones":0,"num_object_copies":23652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5913,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5918,"num_write_kb":47321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3652}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'5899","reported_seq":5980,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:51.045128+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:37:51.045128+0000","last_peered":"2024-09-05T16:37:51.045128+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:37:51.045128+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5899,"log_dups_size":0,"ondisk_log_size":5899,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48305312,"num_objects":5897,"num_object_clones":0,"num_object_copies":23588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5897,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5897,"num_write_kb":47174,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3635}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'6015","reported_seq":6073,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:50.522124+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:37:50.522124+0000","last_peered":"2024-09-05T16:37:50.522124+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:37:50.522124+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6015,"log_dups_size":0,"ondisk_log_size":6015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49229552,"num_objects":6010,"num_object_clones":0,"num_object_copies":24040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6010,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6012,"num_write_kb":48084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3699}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'5834","reported_seq":5874,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:50.475868+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:37:50.475868+0000","last_peered":"2024-09-05T16:37:50.475868+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:37:50.475868+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5834,"log_dups_size":0,"ondisk_log_size":5834,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47782480,"num_objects":5833,"num_object_clones":0,"num_object_copies":23332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5833,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5833,"num_write_kb":46663,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3569}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'5885","reported_seq":6408,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:51.113330+0000","last_change":"2024-09-05T16:37:30.634570+0000","last_active":"2024-09-05T16:37:51.113330+0000","last_peered":"2024-09-05T16:37:51.113330+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:37:51.113330+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5885,"log_dups_size":0,"ondisk_log_size":5885,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48200272,"num_objects":5884,"num_object_clones":0,"num_object_copies":23536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2072,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5884,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5884,"num_write_kb":47071,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":223,"num_bytes_recovered":1818624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3812}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'6018","reported_seq":6073,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:50.507880+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:37:50.507880+0000","last_peered":"2024-09-05T16:37:50.507880+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:37:50.507880+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6018,"log_dups_size":0,"ondisk_log_size":6018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49263776,"num_objects":6014,"num_object_clones":0,"num_object_copies":24056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6014,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6016,"num_write_kb":48124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3657}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'5915","reported_seq":5997,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:51.178312+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:37:51.178312+0000","last_peered":"2024-09-05T16:37:51.178312+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:37:51.178312+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5915,"log_dups_size":0,"ondisk_log_size":5915,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48428192,"num_objects":5912,"num_object_clones":0,"num_object_copies":23648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5912,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5913,"num_write_kb":47301,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3592}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'5862","reported_seq":5930,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:51.093382+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:37:51.093382+0000","last_peered":"2024-09-05T16:37:51.093382+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:37:51.093382+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5862,"log_dups_size":0,"ondisk_log_size":5862,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47977632,"num_objects":5857,"num_object_clones":0,"num_object_copies":23428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5857,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5860,"num_write_kb":46868,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3620}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":771107872,"num_objects":94134,"num_object_clones":0,"num_object_copies":376536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33685,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":94134,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":88465,"num_write_kb":707613,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":1818624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1404030976,"data_stored":1404030976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94190,"ondisk_log_size":94190,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084335,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":714004,"kb_used_data":375492,"kb_used_omap":4,"kb_used_meta":338491,"kb_avail":93657836,"statfs":{"total":96636764160,"available":95905624064,"internally_reserved":0,"allocated":384503808,"data_stored":384311767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":346615507},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,6,3,9],"upper_bound":64},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986212,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":479844,"kb_used_data":241300,"kb_used_omap":4,"kb_used_meta":238523,"kb_avail":93891996,"statfs":{"total":96636764160,"available":96145403904,"internally_reserved":0,"allocated":247091200,"data_stored":246899159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":244248274},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,1,2,5,9],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673018,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":718364,"kb_used_data":376204,"kb_used_omap":1,"kb_used_meta":342142,"kb_avail":93653476,"statfs":{"total":96636764160,"available":95901159424,"internally_reserved":0,"allocated":385232896,"data_stored":385040855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":350353861},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,1,4,2,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182449,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":719512,"kb_used_data":376904,"kb_used_omap":4,"kb_used_meta":342587,"kb_avail":93652328,"statfs":{"total":96636764160,"available":95899983872,"internally_reserved":0,"allocated":385949696,"data_stored":385757655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":350809805},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,6,7,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":385888256,"data_stored":385888256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":246980608,"data_stored":246980608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":385937408,"data_stored":385937408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":385224704,"data_stored":385224704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:37:55.946 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:37:56.532 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 214 20 95684 95664 3.49169 1.95312 0.126156 0.0447344 2024-09-05T16:37:56.542 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:37:56.542 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:37:56.564 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":165,"stamp":"2024-09-05T16:37:54.870679+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":771107872,"num_objects":94134,"num_object_clones":0,"num_object_copies":376536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33685,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":94134,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":88465,"num_write_kb":707613,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":1818624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94190,"ondisk_log_size":94190,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2631724,"kb_used_data":1369900,"kb_used_omap":14,"kb_used_meta":1261745,"kb_avail":374855636,"statfs":{"total":386547056640,"available":383852171264,"internally_reserved":0,"allocated":1402777600,"data_stored":1402009436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1292027447},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,7,8,15,23,11,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":22994944,"num_objects":2807,"num_object_clones":0,"num_object_copies":11228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2807,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2807,"num_write_kb":22456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":843776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005794"},"pg_stats":[{"pgid":"1.f","version":"35'5946","reported_seq":5992,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:50.292692+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:37:50.292692+0000","last_peered":"2024-09-05T16:37:50.292692+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:37:50.292692+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5946,"log_dups_size":0,"ondisk_log_size":5946,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48709632,"num_objects":5946,"num_object_clones":0,"num_object_copies":23784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5946,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5946,"num_write_kb":47568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3684}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'5811","reported_seq":5879,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:51.042275+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:37:51.042275+0000","last_peered":"2024-09-05T16:37:51.042275+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:37:51.042275+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5811,"log_dups_size":0,"ondisk_log_size":5811,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47568032,"num_objects":5807,"num_object_clones":0,"num_object_copies":23228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5807,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5809,"num_write_kb":46468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3540}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'5862","reported_seq":5941,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:51.042216+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:37:51.042216+0000","last_peered":"2024-09-05T16:37:51.042216+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:37:51.042216+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5862,"log_dups_size":0,"ondisk_log_size":5862,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47995472,"num_objects":5859,"num_object_clones":0,"num_object_copies":23436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5861,"num_write_kb":46878,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3609}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'5967","reported_seq":10690,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-05T16:37:49.849352+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:37:49.849352+0000","last_peered":"2024-09-05T16:37:49.849352+0000","last_clean":"2024-09-05T16:37:49.849352+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:37:49.849352+0000","last_undegraded":"2024-09-05T16:37:49.849352+0000","last_fullsized":"2024-09-05T16:37:49.849352+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":5967,"log_dups_size":0,"ondisk_log_size":5967,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48832512,"num_objects":5961,"num_object_clones":0,"num_object_copies":23844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5961,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":267,"num_write_kb":2136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'5792","reported_seq":5849,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:50.490298+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:37:50.490298+0000","last_peered":"2024-09-05T16:37:50.490298+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:37:50.490298+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5792,"log_dups_size":0,"ondisk_log_size":5792,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47420576,"num_objects":5789,"num_object_clones":0,"num_object_copies":23156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5789,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5790,"num_write_kb":46317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3627}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'5835","reported_seq":5881,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:50.236375+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:37:50.236375+0000","last_peered":"2024-09-05T16:37:50.236375+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:37:50.236375+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5835,"log_dups_size":0,"ondisk_log_size":5835,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47800320,"num_objects":5835,"num_object_clones":0,"num_object_copies":23340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5835,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5835,"num_write_kb":46680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3585}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'5770","reported_seq":5839,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:51.037476+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:37:51.037476+0000","last_peered":"2024-09-05T16:37:51.037476+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:37:51.037476+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5770,"log_dups_size":0,"ondisk_log_size":5770,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47215776,"num_objects":5764,"num_object_clones":0,"num_object_copies":23056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5764,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5768,"num_write_kb":46124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3617}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'5858","reported_seq":5926,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:51.146016+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:37:51.146016+0000","last_peered":"2024-09-05T16:37:51.146016+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:37:51.146016+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5858,"log_dups_size":0,"ondisk_log_size":5858,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47944864,"num_objects":5853,"num_object_clones":0,"num_object_copies":23412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5853,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5856,"num_write_kb":46836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3614}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'5921","reported_seq":5981,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:50.236038+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:37:50.236038+0000","last_peered":"2024-09-05T16:37:50.236038+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:37:50.236038+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5921,"log_dups_size":0,"ondisk_log_size":5921,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48433472,"num_objects":5913,"num_object_clones":0,"num_object_copies":23652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5913,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5918,"num_write_kb":47321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3652}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'5899","reported_seq":5980,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:51.045128+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:37:51.045128+0000","last_peered":"2024-09-05T16:37:51.045128+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:37:51.045128+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5899,"log_dups_size":0,"ondisk_log_size":5899,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48305312,"num_objects":5897,"num_object_clones":0,"num_object_copies":23588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5897,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5897,"num_write_kb":47174,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3635}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'6015","reported_seq":6073,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:50.522124+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:37:50.522124+0000","last_peered":"2024-09-05T16:37:50.522124+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:37:50.522124+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6015,"log_dups_size":0,"ondisk_log_size":6015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49229552,"num_objects":6010,"num_object_clones":0,"num_object_copies":24040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6010,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6012,"num_write_kb":48084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3699}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'5834","reported_seq":5874,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:50.475868+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:37:50.475868+0000","last_peered":"2024-09-05T16:37:50.475868+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:37:50.475868+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5834,"log_dups_size":0,"ondisk_log_size":5834,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47782480,"num_objects":5833,"num_object_clones":0,"num_object_copies":23332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5833,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5833,"num_write_kb":46663,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3569}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'5885","reported_seq":6408,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:51.113330+0000","last_change":"2024-09-05T16:37:30.634570+0000","last_active":"2024-09-05T16:37:51.113330+0000","last_peered":"2024-09-05T16:37:51.113330+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:37:51.113330+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5885,"log_dups_size":0,"ondisk_log_size":5885,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48200272,"num_objects":5884,"num_object_clones":0,"num_object_copies":23536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2072,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5884,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5884,"num_write_kb":47071,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":223,"num_bytes_recovered":1818624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3812}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'6018","reported_seq":6073,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:50.507880+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:37:50.507880+0000","last_peered":"2024-09-05T16:37:50.507880+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:37:50.507880+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6018,"log_dups_size":0,"ondisk_log_size":6018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49263776,"num_objects":6014,"num_object_clones":0,"num_object_copies":24056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6014,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6016,"num_write_kb":48124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3657}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'5915","reported_seq":5997,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:51.178312+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:37:51.178312+0000","last_peered":"2024-09-05T16:37:51.178312+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:37:51.178312+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5915,"log_dups_size":0,"ondisk_log_size":5915,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48428192,"num_objects":5912,"num_object_clones":0,"num_object_copies":23648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5912,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5913,"num_write_kb":47301,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3592}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'5862","reported_seq":5930,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:51.093382+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:37:51.093382+0000","last_peered":"2024-09-05T16:37:51.093382+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:37:51.093382+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5862,"log_dups_size":0,"ondisk_log_size":5862,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47977632,"num_objects":5857,"num_object_clones":0,"num_object_copies":23428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5857,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5860,"num_write_kb":46868,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3620}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":771107872,"num_objects":94134,"num_object_clones":0,"num_object_copies":376536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33685,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":94134,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":88465,"num_write_kb":707613,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":1818624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1404030976,"data_stored":1404030976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94190,"ondisk_log_size":94190,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084335,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":714004,"kb_used_data":375492,"kb_used_omap":4,"kb_used_meta":338491,"kb_avail":93657836,"statfs":{"total":96636764160,"available":95905624064,"internally_reserved":0,"allocated":384503808,"data_stored":384311767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":346615507},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,6,3,9],"upper_bound":64},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986212,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":479844,"kb_used_data":241300,"kb_used_omap":4,"kb_used_meta":238523,"kb_avail":93891996,"statfs":{"total":96636764160,"available":96145403904,"internally_reserved":0,"allocated":247091200,"data_stored":246899159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":244248274},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,1,2,5,9],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673018,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":718364,"kb_used_data":376204,"kb_used_omap":1,"kb_used_meta":342142,"kb_avail":93653476,"statfs":{"total":96636764160,"available":95901159424,"internally_reserved":0,"allocated":385232896,"data_stored":385040855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":350353861},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,1,4,2,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182449,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":719512,"kb_used_data":376904,"kb_used_omap":4,"kb_used_meta":342587,"kb_avail":93652328,"statfs":{"total":96636764160,"available":95899983872,"internally_reserved":0,"allocated":385949696,"data_stored":385757655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":350809805},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,6,7,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":385888256,"data_stored":385888256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":246980608,"data_stored":246980608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":385937408,"data_stored":385937408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":385224704,"data_stored":385224704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:37:56.566 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-05T16:37:56.566 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'5946", 'reported_seq': 5992, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:50.292692+0000', 'last_change': '2024-09-05T16:34:56.833497+0000', 'last_active': '2024-09-05T16:37:50.292692+0000', 'last_peered': '2024-09-05T16:37:50.292692+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.828289+0000', 'last_became_peered': '2024-09-05T16:34:56.828289+0000', 'last_unstale': '2024-09-05T16:37:50.292692+0000', 'last_undegraded': '2024-09-05T16:34:56.612160+0000', 'last_fullsized': '2024-09-05T16:34:56.557632+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5946, 'log_dups_size': 0, 'ondisk_log_size': 5946, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48709632, 'num_objects': 5946, 'num_object_clones': 0, 'num_object_copies': 23784, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2262, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5946, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5946, 'num_write_kb': 47568, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3684}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:37:56.566 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-05T16:37:56.566 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "35'5811", 'reported_seq': 5879, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:51.042275+0000', 'last_change': '2024-09-05T16:34:56.825031+0000', 'last_active': '2024-09-05T16:37:51.042275+0000', 'last_peered': '2024-09-05T16:37:51.042275+0000', 'last_clean': '2024-09-05T16:33:53.200762+0000', 'last_became_active': '2024-09-05T16:34:56.820292+0000', 'last_became_peered': '2024-09-05T16:34:56.820292+0000', 'last_unstale': '2024-09-05T16:37:51.042275+0000', 'last_undegraded': '2024-09-05T16:34:56.605776+0000', 'last_fullsized': '2024-09-05T16:34:56.559383+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5811, 'log_dups_size': 0, 'ondisk_log_size': 5811, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47568032, 'num_objects': 5807, 'num_object_clones': 0, 'num_object_copies': 23228, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2269, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5807, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5809, 'num_write_kb': 46468, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3540}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:37:56.566 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:37:56.566 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'5862", 'reported_seq': 5941, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:51.042216+0000', 'last_change': '2024-09-05T16:34:56.867900+0000', 'last_active': '2024-09-05T16:37:51.042216+0000', 'last_peered': '2024-09-05T16:37:51.042216+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:37:51.042216+0000', 'last_undegraded': '2024-09-05T16:34:56.590972+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5862, 'log_dups_size': 0, 'ondisk_log_size': 5862, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47995472, 'num_objects': 5859, 'num_object_clones': 0, 'num_object_copies': 23436, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2251, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5859, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5861, 'num_write_kb': 46878, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3609}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:37:56.566 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:37:56.566 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'5792", 'reported_seq': 5849, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:50.490298+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:37:50.490298+0000', 'last_peered': '2024-09-05T16:37:50.490298+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:37:50.490298+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5792, 'log_dups_size': 0, 'ondisk_log_size': 5792, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47420576, 'num_objects': 5789, 'num_object_clones': 0, 'num_object_copies': 23156, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2164, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5789, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5790, 'num_write_kb': 46317, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3627}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:37:56.566 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:37:56.567 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'5835", 'reported_seq': 5881, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:50.236375+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:37:50.236375+0000', 'last_peered': '2024-09-05T16:37:50.236375+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:37:50.236375+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5835, 'log_dups_size': 0, 'ondisk_log_size': 5835, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47800320, 'num_objects': 5835, 'num_object_clones': 0, 'num_object_copies': 23340, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5835, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5835, 'num_write_kb': 46680, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3585}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:37:56.567 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:37:56.567 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'5770", 'reported_seq': 5839, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:51.037476+0000', 'last_change': '2024-09-05T16:34:56.903436+0000', 'last_active': '2024-09-05T16:37:51.037476+0000', 'last_peered': '2024-09-05T16:37:51.037476+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:37:51.037476+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5770, 'log_dups_size': 0, 'ondisk_log_size': 5770, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47215776, 'num_objects': 5764, 'num_object_clones': 0, 'num_object_copies': 23056, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2149, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5764, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5768, 'num_write_kb': 46124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3617}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:37:56.567 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-05T16:37:56.567 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "35'5858", 'reported_seq': 5926, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:51.146016+0000', 'last_change': '2024-09-05T16:34:56.883319+0000', 'last_active': '2024-09-05T16:37:51.146016+0000', 'last_peered': '2024-09-05T16:37:51.146016+0000', 'last_clean': '2024-09-05T16:33:53.505305+0000', 'last_became_active': '2024-09-05T16:34:56.873389+0000', 'last_became_peered': '2024-09-05T16:34:56.873389+0000', 'last_unstale': '2024-09-05T16:37:51.146016+0000', 'last_undegraded': '2024-09-05T16:34:56.602567+0000', 'last_fullsized': '2024-09-05T16:34:56.559958+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5858, 'log_dups_size': 0, 'ondisk_log_size': 5858, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47944864, 'num_objects': 5853, 'num_object_clones': 0, 'num_object_copies': 23412, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2241, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5853, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5856, 'num_write_kb': 46836, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3614}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:37:56.567 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:37:56.567 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'5921", 'reported_seq': 5981, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:50.236038+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:37:50.236038+0000', 'last_peered': '2024-09-05T16:37:50.236038+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:37:50.236038+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5921, 'log_dups_size': 0, 'ondisk_log_size': 5921, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48433472, 'num_objects': 5913, 'num_object_clones': 0, 'num_object_copies': 23652, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2264, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5913, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5918, 'num_write_kb': 47321, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3652}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:37:56.567 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:37:56.567 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'5899", 'reported_seq': 5980, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:51.045128+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:37:51.045128+0000', 'last_peered': '2024-09-05T16:37:51.045128+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:37:51.045128+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5899, 'log_dups_size': 0, 'ondisk_log_size': 5899, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48305312, 'num_objects': 5897, 'num_object_clones': 0, 'num_object_copies': 23588, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2264, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5897, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5897, 'num_write_kb': 47174, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3635}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:37:56.567 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:37:56.568 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'6015", 'reported_seq': 6073, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:50.522124+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:37:50.522124+0000', 'last_peered': '2024-09-05T16:37:50.522124+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:37:50.522124+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6015, 'log_dups_size': 0, 'ondisk_log_size': 6015, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49229552, 'num_objects': 6010, 'num_object_clones': 0, 'num_object_copies': 24040, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2314, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6010, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6012, 'num_write_kb': 48084, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3699}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:37:56.568 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:37:56.568 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'5834", 'reported_seq': 5874, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:50.475868+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:37:50.475868+0000', 'last_peered': '2024-09-05T16:37:50.475868+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:37:50.475868+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5834, 'log_dups_size': 0, 'ondisk_log_size': 5834, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47782480, 'num_objects': 5833, 'num_object_clones': 0, 'num_object_copies': 23332, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2265, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5833, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5833, 'num_write_kb': 46663, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3569}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:37:56.568 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-05T16:37:56.568 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'5885", 'reported_seq': 6408, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:51.113330+0000', 'last_change': '2024-09-05T16:37:30.634570+0000', 'last_active': '2024-09-05T16:37:51.113330+0000', 'last_peered': '2024-09-05T16:37:51.113330+0000', 'last_clean': '2024-09-05T16:33:45.082039+0000', 'last_became_active': '2024-09-05T16:34:56.792386+0000', 'last_became_peered': '2024-09-05T16:34:56.792386+0000', 'last_unstale': '2024-09-05T16:37:51.113330+0000', 'last_undegraded': '2024-09-05T16:34:56.600595+0000', 'last_fullsized': '2024-09-05T16:34:56.558203+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5885, 'log_dups_size': 0, 'ondisk_log_size': 5885, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48200272, 'num_objects': 5884, 'num_object_clones': 0, 'num_object_copies': 23536, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2072, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5884, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5884, 'num_write_kb': 47071, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 223, 'num_bytes_recovered': 1818624, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3812}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:37:56.568 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:37:56.568 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'6018", 'reported_seq': 6073, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:50.507880+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:37:50.507880+0000', 'last_peered': '2024-09-05T16:37:50.507880+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:37:50.507880+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6018, 'log_dups_size': 0, 'ondisk_log_size': 6018, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49263776, 'num_objects': 6014, 'num_object_clones': 0, 'num_object_copies': 24056, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2359, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6014, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6016, 'num_write_kb': 48124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3657}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:37:56.568 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:37:56.568 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'5915", 'reported_seq': 5997, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:51.178312+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:37:51.178312+0000', 'last_peered': '2024-09-05T16:37:51.178312+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:37:51.178312+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5915, 'log_dups_size': 0, 'ondisk_log_size': 5915, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48428192, 'num_objects': 5912, 'num_object_clones': 0, 'num_object_copies': 23648, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2322, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5912, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5913, 'num_write_kb': 47301, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3592}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:37:56.568 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:37:56.568 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'5862", 'reported_seq': 5930, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:51.093382+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:37:51.093382+0000', 'last_peered': '2024-09-05T16:37:51.093382+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:37:51.093382+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5862, 'log_dups_size': 0, 'ondisk_log_size': 5862, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47977632, 'num_objects': 5857, 'num_object_clones': 0, 'num_object_copies': 23428, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5857, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5860, 'num_write_kb': 46868, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3620}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:37:56.569 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:37:57.321 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:37:57.324 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 33633/382076 objects degraded (8.803%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":280,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":95519,"data_bytes":782453792,"bytes_used":2717409280,"bytes_avail":383829647360,"bytes_total":386547056640,"degraded_objects":33633,"degraded_total":382076,"degraded_ratio":0.088026989394780097,"recovering_objects_per_sec":12,"recovering_bytes_per_sec":105762,"recovering_keys_per_sec":0,"num_objects_recovered":155,"num_bytes_recovered":1269760,"num_keys_recovered":0,"write_bytes_sec":2860372,"read_op_per_sec":0,"write_op_per_sec":349},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:37:57.351 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:37:57.351 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:37:57.533 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 215 20 95938 95918 3.48468 1.98438 0.0518599 0.044828 2024-09-05T16:37:57.962 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:37:57.962 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:37:57.999 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":166,"stamp":"2024-09-05T16:37:56.871502+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":782453792,"num_objects":95519,"num_object_clones":0,"num_object_copies":382076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33633,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":95519,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":89850,"num_write_kb":718693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":299,"num_bytes_recovered":2244608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":95575,"ondisk_log_size":95575,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2653720,"kb_used_data":1391896,"kb_used_omap":14,"kb_used_meta":1261745,"kb_avail":374833640,"statfs":{"total":386547056640,"available":383829647360,"internally_reserved":0,"allocated":1425301504,"data_stored":1424533340,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1292027447},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[8,2,2,12,7,11,8,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":39,"apply_latency_ms":39,"commit_latency_ns":39000000,"apply_latency_ns":39000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":34340864,"num_objects":4192,"num_object_clones":0,"num_object_copies":16768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-155,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4192,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4192,"num_write_kb":33536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":155,"num_bytes_recovered":1269760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005730"},"pg_stats":[{"pgid":"1.f","version":"35'6043","reported_seq":6088,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:55.479932+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:37:55.479932+0000","last_peered":"2024-09-05T16:37:55.479932+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:37:55.479932+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6043,"log_dups_size":0,"ondisk_log_size":6043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49504256,"num_objects":6043,"num_object_clones":0,"num_object_copies":24172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6043,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6043,"num_write_kb":48344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3781}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'5887","reported_seq":5955,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:56.067834+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:37:56.067834+0000","last_peered":"2024-09-05T16:37:56.067834+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:37:56.067834+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5887,"log_dups_size":0,"ondisk_log_size":5887,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48190624,"num_objects":5883,"num_object_clones":0,"num_object_copies":23532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5883,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5885,"num_write_kb":47076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3616}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'5962","reported_seq":6041,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:56.146529+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:37:56.146529+0000","last_peered":"2024-09-05T16:37:56.146529+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:37:56.146529+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5962,"log_dups_size":0,"ondisk_log_size":5962,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48814672,"num_objects":5959,"num_object_clones":0,"num_object_copies":23836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5959,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5961,"num_write_kb":47678,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3709}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'6059","reported_seq":10782,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-05T16:37:55.062476+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:37:55.062476+0000","last_peered":"2024-09-05T16:37:55.062476+0000","last_clean":"2024-09-05T16:37:55.062476+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:37:55.062476+0000","last_undegraded":"2024-09-05T16:37:55.062476+0000","last_fullsized":"2024-09-05T16:37:55.062476+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":6059,"log_dups_size":0,"ondisk_log_size":6059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49586176,"num_objects":6053,"num_object_clones":0,"num_object_copies":24212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6053,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":359,"num_write_kb":2872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'5871","reported_seq":5929,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:55.491986+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:37:55.491986+0000","last_peered":"2024-09-05T16:37:55.491986+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:37:55.491986+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5871,"log_dups_size":0,"ondisk_log_size":5871,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48067744,"num_objects":5868,"num_object_clones":0,"num_object_copies":23472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5868,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5869,"num_write_kb":46949,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3706}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'5940","reported_seq":5984,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:55.400417+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:37:55.400417+0000","last_peered":"2024-09-05T16:37:55.400417+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:37:55.400417+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5940,"log_dups_size":0,"ondisk_log_size":5940,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48660480,"num_objects":5940,"num_object_clones":0,"num_object_copies":23760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5940,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5940,"num_write_kb":47520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3690}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'5854","reported_seq":5923,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:56.170051+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:37:56.170051+0000","last_peered":"2024-09-05T16:37:56.170051+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:37:56.170051+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5854,"log_dups_size":0,"ondisk_log_size":5854,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47903904,"num_objects":5848,"num_object_clones":0,"num_object_copies":23392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5848,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5852,"num_write_kb":46796,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3701}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'5927","reported_seq":5995,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:56.033825+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:37:56.033825+0000","last_peered":"2024-09-05T16:37:56.033825+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:37:56.033825+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5927,"log_dups_size":0,"ondisk_log_size":5927,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48510112,"num_objects":5922,"num_object_clones":0,"num_object_copies":23688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5922,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5925,"num_write_kb":47388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3683}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'6004","reported_seq":6064,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:55.504027+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:37:55.504027+0000","last_peered":"2024-09-05T16:37:55.504027+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:37:55.504027+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6004,"log_dups_size":0,"ondisk_log_size":6004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49113408,"num_objects":5996,"num_object_clones":0,"num_object_copies":23984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5996,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6001,"num_write_kb":47985,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3735}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'5974","reported_seq":6056,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:56.060306+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:37:56.060306+0000","last_peered":"2024-09-05T16:37:56.060306+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:37:56.060306+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5974,"log_dups_size":0,"ondisk_log_size":5974,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48919712,"num_objects":5972,"num_object_clones":0,"num_object_copies":23888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5972,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5972,"num_write_kb":47774,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3710}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'6095","reported_seq":6155,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:55.417212+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:37:55.417212+0000","last_peered":"2024-09-05T16:37:55.417212+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:37:55.417212+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6095,"log_dups_size":0,"ondisk_log_size":6095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49884912,"num_objects":6090,"num_object_clones":0,"num_object_copies":24360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6090,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6092,"num_write_kb":48724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3779}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'5941","reported_seq":5979,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:55.600171+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:37:55.600171+0000","last_peered":"2024-09-05T16:37:55.600171+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:37:55.600171+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5941,"log_dups_size":0,"ondisk_log_size":5941,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48659024,"num_objects":5940,"num_object_clones":0,"num_object_copies":23760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5940,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5940,"num_write_kb":47519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3676}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'5966","reported_seq":6593,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:56.138574+0000","last_change":"2024-09-05T16:37:30.634570+0000","last_active":"2024-09-05T16:37:56.138574+0000","last_peered":"2024-09-05T16:37:56.138574+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:37:56.138574+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5966,"log_dups_size":0,"ondisk_log_size":5966,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48863824,"num_objects":5965,"num_object_clones":0,"num_object_copies":23860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2020,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5965,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5965,"num_write_kb":47719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":275,"num_bytes_recovered":2244608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3945}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'6110","reported_seq":6166,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:55.456484+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:37:55.456484+0000","last_peered":"2024-09-05T16:37:55.456484+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:37:55.456484+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6110,"log_dups_size":0,"ondisk_log_size":6110,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50017440,"num_objects":6106,"num_object_clones":0,"num_object_copies":24424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6106,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6108,"num_write_kb":48860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3749}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'5999","reported_seq":6081,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:56.092237+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:37:56.092237+0000","last_peered":"2024-09-05T16:37:56.092237+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:37:56.092237+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5999,"log_dups_size":0,"ondisk_log_size":5999,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49116320,"num_objects":5996,"num_object_clones":0,"num_object_copies":23984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5996,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5997,"num_write_kb":47973,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3676}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'5943","reported_seq":6011,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:56.065302+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:37:56.065302+0000","last_peered":"2024-09-05T16:37:56.065302+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:37:56.065302+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5943,"log_dups_size":0,"ondisk_log_size":5943,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48641184,"num_objects":5938,"num_object_clones":0,"num_object_copies":23752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5938,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5941,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3701}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":782453792,"num_objects":95519,"num_object_clones":0,"num_object_copies":382076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33633,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":95519,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":89850,"num_write_kb":718693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":299,"num_bytes_recovered":2244608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1426989056,"data_stored":1426989056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":95575,"ondisk_log_size":95575,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084336,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":723608,"kb_used_data":380936,"kb_used_omap":4,"kb_used_meta":342651,"kb_avail":93648232,"statfs":{"total":96636764160,"available":95895789568,"internally_reserved":0,"allocated":390078464,"data_stored":389886423,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":350875347},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,7,0,2,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986213,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":472820,"kb_used_data":246756,"kb_used_omap":4,"kb_used_meta":226043,"kb_avail":93899020,"statfs":{"total":96636764160,"available":96152596480,"internally_reserved":0,"allocated":252678144,"data_stored":252486103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":231468754},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,2,2,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673019,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":728020,"kb_used_data":381700,"kb_used_omap":1,"kb_used_meta":346302,"kb_avail":93643820,"statfs":{"total":96636764160,"available":95891271680,"internally_reserved":0,"allocated":390860800,"data_stored":390668759,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":354613701},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,4,6,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182450,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":729272,"kb_used_data":382504,"kb_used_omap":4,"kb_used_meta":346747,"kb_avail":93642568,"statfs":{"total":96636764160,"available":95889989632,"internally_reserved":0,"allocated":391684096,"data_stored":391492055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":355069645},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[8,2,0,0,1,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":391536640,"data_stored":391536640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":252932096,"data_stored":252932096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":391577600,"data_stored":391577600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":390942720,"data_stored":390942720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:37:58.535 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 216 20 96213 96193 3.47845 2.14844 0.0912708 0.0449077 2024-09-05T16:37:59.535 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 217 20 96502 96482 3.47282 2.25781 0.0504439 0.0449802 2024-09-05T16:38:00.537 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 218 20 96784 96764 3.467 2.20312 0.0871506 0.0450565 2024-09-05T16:38:01.001 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:38:01.536 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 219 20 97027 97007 3.45984 1.89844 0.106684 0.0451506 2024-09-05T16:38:01.551 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:38:01.555 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:38:01.587 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":168,"stamp":"2024-09-05T16:38:00.873400+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":788057120,"num_objects":96203,"num_object_clones":0,"num_object_copies":384812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33633,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":96203,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":90534,"num_write_kb":724165,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":299,"num_bytes_recovered":2244608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96259,"ondisk_log_size":96259,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2674428,"kb_used_data":1404284,"kb_used_omap":14,"kb_used_meta":1270065,"kb_avail":374812932,"statfs":{"total":386547056640,"available":383808442368,"internally_reserved":0,"allocated":1437986816,"data_stored":1437218652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1300547127},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[11,5,3,4,9,12,12,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":28581888,"num_objects":3489,"num_object_clones":0,"num_object_copies":13956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-99,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3489,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3489,"num_write_kb":27912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":811008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005854"},"pg_stats":[{"pgid":"1.f","version":"35'6133","reported_seq":6177,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:00.565925+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:38:00.565925+0000","last_peered":"2024-09-05T16:38:00.565925+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:38:00.565925+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6133,"log_dups_size":0,"ondisk_log_size":6133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50241536,"num_objects":6133,"num_object_clones":0,"num_object_copies":24532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6133,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6133,"num_write_kb":49064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3871}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'5887","reported_seq":5955,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:56.067834+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:37:56.067834+0000","last_peered":"2024-09-05T16:37:56.067834+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:37:56.067834+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5887,"log_dups_size":0,"ondisk_log_size":5887,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48190624,"num_objects":5883,"num_object_clones":0,"num_object_copies":23532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5883,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5885,"num_write_kb":47076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3616}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'5962","reported_seq":6041,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:56.146529+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:37:56.146529+0000","last_peered":"2024-09-05T16:37:56.146529+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:37:56.146529+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5962,"log_dups_size":0,"ondisk_log_size":5962,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48814672,"num_objects":5959,"num_object_clones":0,"num_object_copies":23836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5959,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5961,"num_write_kb":47678,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3709}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'6147","reported_seq":10870,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-05T16:38:00.255903+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:38:00.255903+0000","last_peered":"2024-09-05T16:38:00.255903+0000","last_clean":"2024-09-05T16:38:00.255903+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:38:00.255903+0000","last_undegraded":"2024-09-05T16:38:00.255903+0000","last_fullsized":"2024-09-05T16:38:00.255903+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":6147,"log_dups_size":0,"ondisk_log_size":6147,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50307072,"num_objects":6141,"num_object_clones":0,"num_object_copies":24564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6141,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":447,"num_write_kb":3576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'5954","reported_seq":6012,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:00.481105+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:38:00.481105+0000","last_peered":"2024-09-05T16:38:00.481105+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:38:00.481105+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5954,"log_dups_size":0,"ondisk_log_size":5954,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48747680,"num_objects":5951,"num_object_clones":0,"num_object_copies":23804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5951,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5952,"num_write_kb":47613,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3789}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'6029","reported_seq":6075,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:00.583458+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:38:00.583458+0000","last_peered":"2024-09-05T16:38:00.583458+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:38:00.583458+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6029,"log_dups_size":0,"ondisk_log_size":6029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49389568,"num_objects":6029,"num_object_clones":0,"num_object_copies":24116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6029,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6029,"num_write_kb":48232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3779}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'5854","reported_seq":5923,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:56.170051+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:37:56.170051+0000","last_peered":"2024-09-05T16:37:56.170051+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:37:56.170051+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5854,"log_dups_size":0,"ondisk_log_size":5854,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47903904,"num_objects":5848,"num_object_clones":0,"num_object_copies":23392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5848,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5852,"num_write_kb":46796,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3701}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'5927","reported_seq":5995,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:56.033825+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:37:56.033825+0000","last_peered":"2024-09-05T16:37:56.033825+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:37:56.033825+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5927,"log_dups_size":0,"ondisk_log_size":5927,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48510112,"num_objects":5922,"num_object_clones":0,"num_object_copies":23688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5922,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5925,"num_write_kb":47388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3683}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'6086","reported_seq":6146,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:00.624023+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:38:00.624023+0000","last_peered":"2024-09-05T16:38:00.624023+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:38:00.624023+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6086,"log_dups_size":0,"ondisk_log_size":6086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49785152,"num_objects":6078,"num_object_clones":0,"num_object_copies":24312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6078,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6083,"num_write_kb":48641,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3817}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'5974","reported_seq":6056,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:56.060306+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:37:56.060306+0000","last_peered":"2024-09-05T16:37:56.060306+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:37:56.060306+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5974,"log_dups_size":0,"ondisk_log_size":5974,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48919712,"num_objects":5972,"num_object_clones":0,"num_object_copies":23888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5972,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5972,"num_write_kb":47774,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3710}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'6189","reported_seq":6249,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:00.619170+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:38:00.619170+0000","last_peered":"2024-09-05T16:38:00.619170+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:38:00.619170+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6189,"log_dups_size":0,"ondisk_log_size":6189,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50654960,"num_objects":6184,"num_object_clones":0,"num_object_copies":24736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6184,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6186,"num_write_kb":49476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3873}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'6009","reported_seq":6049,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:00.591951+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:38:00.591951+0000","last_peered":"2024-09-05T16:38:00.591951+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:38:00.591951+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6009,"log_dups_size":0,"ondisk_log_size":6009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49216080,"num_objects":6008,"num_object_clones":0,"num_object_copies":24032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6008,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6008,"num_write_kb":48063,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3744}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'5966","reported_seq":6593,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:56.138574+0000","last_change":"2024-09-05T16:37:30.634570+0000","last_active":"2024-09-05T16:37:56.138574+0000","last_peered":"2024-09-05T16:37:56.138574+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:37:56.138574+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5966,"log_dups_size":0,"ondisk_log_size":5966,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48863824,"num_objects":5965,"num_object_clones":0,"num_object_copies":23860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2020,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5965,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5965,"num_write_kb":47719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":275,"num_bytes_recovered":2244608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3945}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'6200","reported_seq":6256,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:00.601598+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:38:00.601598+0000","last_peered":"2024-09-05T16:38:00.601598+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:38:00.601598+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6200,"log_dups_size":0,"ondisk_log_size":6200,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50754720,"num_objects":6196,"num_object_clones":0,"num_object_copies":24784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6196,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6198,"num_write_kb":49580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3839}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'5999","reported_seq":6081,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:56.092237+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:37:56.092237+0000","last_peered":"2024-09-05T16:37:56.092237+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:37:56.092237+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5999,"log_dups_size":0,"ondisk_log_size":5999,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49116320,"num_objects":5996,"num_object_clones":0,"num_object_copies":23984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5996,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5997,"num_write_kb":47973,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3676}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'5943","reported_seq":6011,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:56.065302+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:37:56.065302+0000","last_peered":"2024-09-05T16:37:56.065302+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:37:56.065302+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5943,"log_dups_size":0,"ondisk_log_size":5943,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48641184,"num_objects":5938,"num_object_clones":0,"num_object_copies":23752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5938,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5941,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3701}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":788057120,"num_objects":96203,"num_object_clones":0,"num_object_copies":384812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33633,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":96203,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":90534,"num_write_kb":724165,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":299,"num_bytes_recovered":2244608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1438400512,"data_stored":1438400512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96259,"ondisk_log_size":96259,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084337,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":734420,"kb_used_data":387588,"kb_used_omap":4,"kb_used_meta":346811,"kb_avail":93637420,"statfs":{"total":96636764160,"available":95884718080,"internally_reserved":0,"allocated":396890112,"data_stored":396698071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":355135187},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,2,1,4,3,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986214,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":482716,"kb_used_data":252492,"kb_used_omap":4,"kb_used_meta":230203,"kb_avail":93889124,"statfs":{"total":96636764160,"available":96142462976,"internally_reserved":0,"allocated":258551808,"data_stored":258359767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":235728594},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,0,2,0,2,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673019,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":728020,"kb_used_data":381700,"kb_used_omap":1,"kb_used_meta":346302,"kb_avail":93643820,"statfs":{"total":96636764160,"available":95891271680,"internally_reserved":0,"allocated":390860800,"data_stored":390668759,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":354613701},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,4,6,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182450,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":729272,"kb_used_data":382504,"kb_used_omap":4,"kb_used_meta":346747,"kb_avail":93642568,"statfs":{"total":96636764160,"available":95889989632,"internally_reserved":0,"allocated":391684096,"data_stored":391492055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":355069645},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[8,2,0,0,1,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":391536640,"data_stored":391536640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":258691072,"data_stored":258691072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":391577600,"data_stored":391577600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":396595200,"data_stored":396595200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:38:01.588 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:38:02.234 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:38:02.235 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:38:02.265 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":168,"stamp":"2024-09-05T16:38:00.873400+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":788057120,"num_objects":96203,"num_object_clones":0,"num_object_copies":384812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33633,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":96203,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":90534,"num_write_kb":724165,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":299,"num_bytes_recovered":2244608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96259,"ondisk_log_size":96259,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2674428,"kb_used_data":1404284,"kb_used_omap":14,"kb_used_meta":1270065,"kb_avail":374812932,"statfs":{"total":386547056640,"available":383808442368,"internally_reserved":0,"allocated":1437986816,"data_stored":1437218652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1300547127},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[11,5,3,4,9,12,12,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":28581888,"num_objects":3489,"num_object_clones":0,"num_object_copies":13956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-99,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3489,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3489,"num_write_kb":27912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":811008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005854"},"pg_stats":[{"pgid":"1.f","version":"35'6133","reported_seq":6177,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:00.565925+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:38:00.565925+0000","last_peered":"2024-09-05T16:38:00.565925+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:38:00.565925+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6133,"log_dups_size":0,"ondisk_log_size":6133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50241536,"num_objects":6133,"num_object_clones":0,"num_object_copies":24532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6133,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6133,"num_write_kb":49064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3871}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'5887","reported_seq":5955,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:56.067834+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:37:56.067834+0000","last_peered":"2024-09-05T16:37:56.067834+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:37:56.067834+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5887,"log_dups_size":0,"ondisk_log_size":5887,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48190624,"num_objects":5883,"num_object_clones":0,"num_object_copies":23532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5883,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5885,"num_write_kb":47076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3616}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'5962","reported_seq":6041,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:56.146529+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:37:56.146529+0000","last_peered":"2024-09-05T16:37:56.146529+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:37:56.146529+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5962,"log_dups_size":0,"ondisk_log_size":5962,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48814672,"num_objects":5959,"num_object_clones":0,"num_object_copies":23836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5959,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5961,"num_write_kb":47678,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3709}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'6147","reported_seq":10870,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-05T16:38:00.255903+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:38:00.255903+0000","last_peered":"2024-09-05T16:38:00.255903+0000","last_clean":"2024-09-05T16:38:00.255903+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:38:00.255903+0000","last_undegraded":"2024-09-05T16:38:00.255903+0000","last_fullsized":"2024-09-05T16:38:00.255903+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":6147,"log_dups_size":0,"ondisk_log_size":6147,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50307072,"num_objects":6141,"num_object_clones":0,"num_object_copies":24564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6141,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":447,"num_write_kb":3576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'5954","reported_seq":6012,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:00.481105+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:38:00.481105+0000","last_peered":"2024-09-05T16:38:00.481105+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:38:00.481105+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5954,"log_dups_size":0,"ondisk_log_size":5954,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48747680,"num_objects":5951,"num_object_clones":0,"num_object_copies":23804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5951,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5952,"num_write_kb":47613,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3789}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'6029","reported_seq":6075,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:00.583458+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:38:00.583458+0000","last_peered":"2024-09-05T16:38:00.583458+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:38:00.583458+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6029,"log_dups_size":0,"ondisk_log_size":6029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49389568,"num_objects":6029,"num_object_clones":0,"num_object_copies":24116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6029,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6029,"num_write_kb":48232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3779}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'5854","reported_seq":5923,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:56.170051+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:37:56.170051+0000","last_peered":"2024-09-05T16:37:56.170051+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:37:56.170051+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5854,"log_dups_size":0,"ondisk_log_size":5854,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47903904,"num_objects":5848,"num_object_clones":0,"num_object_copies":23392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5848,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5852,"num_write_kb":46796,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3701}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'5927","reported_seq":5995,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:56.033825+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:37:56.033825+0000","last_peered":"2024-09-05T16:37:56.033825+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:37:56.033825+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5927,"log_dups_size":0,"ondisk_log_size":5927,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48510112,"num_objects":5922,"num_object_clones":0,"num_object_copies":23688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5922,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5925,"num_write_kb":47388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3683}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'6086","reported_seq":6146,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:00.624023+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:38:00.624023+0000","last_peered":"2024-09-05T16:38:00.624023+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:38:00.624023+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6086,"log_dups_size":0,"ondisk_log_size":6086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49785152,"num_objects":6078,"num_object_clones":0,"num_object_copies":24312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6078,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6083,"num_write_kb":48641,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3817}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'5974","reported_seq":6056,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:56.060306+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:37:56.060306+0000","last_peered":"2024-09-05T16:37:56.060306+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:37:56.060306+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5974,"log_dups_size":0,"ondisk_log_size":5974,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48919712,"num_objects":5972,"num_object_clones":0,"num_object_copies":23888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5972,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5972,"num_write_kb":47774,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3710}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'6189","reported_seq":6249,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:00.619170+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:38:00.619170+0000","last_peered":"2024-09-05T16:38:00.619170+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:38:00.619170+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6189,"log_dups_size":0,"ondisk_log_size":6189,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50654960,"num_objects":6184,"num_object_clones":0,"num_object_copies":24736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6184,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6186,"num_write_kb":49476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3873}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'6009","reported_seq":6049,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:00.591951+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:38:00.591951+0000","last_peered":"2024-09-05T16:38:00.591951+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:38:00.591951+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6009,"log_dups_size":0,"ondisk_log_size":6009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49216080,"num_objects":6008,"num_object_clones":0,"num_object_copies":24032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6008,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6008,"num_write_kb":48063,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3744}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'5966","reported_seq":6593,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:56.138574+0000","last_change":"2024-09-05T16:37:30.634570+0000","last_active":"2024-09-05T16:37:56.138574+0000","last_peered":"2024-09-05T16:37:56.138574+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:37:56.138574+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5966,"log_dups_size":0,"ondisk_log_size":5966,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48863824,"num_objects":5965,"num_object_clones":0,"num_object_copies":23860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2020,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5965,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5965,"num_write_kb":47719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":275,"num_bytes_recovered":2244608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3945}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'6200","reported_seq":6256,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:00.601598+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:38:00.601598+0000","last_peered":"2024-09-05T16:38:00.601598+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:38:00.601598+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6200,"log_dups_size":0,"ondisk_log_size":6200,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50754720,"num_objects":6196,"num_object_clones":0,"num_object_copies":24784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6196,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6198,"num_write_kb":49580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3839}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'5999","reported_seq":6081,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:56.092237+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:37:56.092237+0000","last_peered":"2024-09-05T16:37:56.092237+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:37:56.092237+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5999,"log_dups_size":0,"ondisk_log_size":5999,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49116320,"num_objects":5996,"num_object_clones":0,"num_object_copies":23984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5996,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5997,"num_write_kb":47973,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3676}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'5943","reported_seq":6011,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:37:56.065302+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:37:56.065302+0000","last_peered":"2024-09-05T16:37:56.065302+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:37:56.065302+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5943,"log_dups_size":0,"ondisk_log_size":5943,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48641184,"num_objects":5938,"num_object_clones":0,"num_object_copies":23752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5938,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5941,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3701}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":788057120,"num_objects":96203,"num_object_clones":0,"num_object_copies":384812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33633,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":96203,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":90534,"num_write_kb":724165,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":299,"num_bytes_recovered":2244608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1438400512,"data_stored":1438400512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96259,"ondisk_log_size":96259,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084337,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":734420,"kb_used_data":387588,"kb_used_omap":4,"kb_used_meta":346811,"kb_avail":93637420,"statfs":{"total":96636764160,"available":95884718080,"internally_reserved":0,"allocated":396890112,"data_stored":396698071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":355135187},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,2,1,4,3,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986214,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":482716,"kb_used_data":252492,"kb_used_omap":4,"kb_used_meta":230203,"kb_avail":93889124,"statfs":{"total":96636764160,"available":96142462976,"internally_reserved":0,"allocated":258551808,"data_stored":258359767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":235728594},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,0,2,0,2,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673019,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":728020,"kb_used_data":381700,"kb_used_omap":1,"kb_used_meta":346302,"kb_avail":93643820,"statfs":{"total":96636764160,"available":95891271680,"internally_reserved":0,"allocated":390860800,"data_stored":390668759,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":354613701},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,4,6,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182450,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":729272,"kb_used_data":382504,"kb_used_omap":4,"kb_used_meta":346747,"kb_avail":93642568,"statfs":{"total":96636764160,"available":95889989632,"internally_reserved":0,"allocated":391684096,"data_stored":391492055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":355069645},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[8,2,0,0,1,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":391536640,"data_stored":391536640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":258691072,"data_stored":258691072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":391577600,"data_stored":391577600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":396595200,"data_stored":396595200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:38:02.266 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-05T16:38:02.267 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'6133", 'reported_seq': 6177, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:00.565925+0000', 'last_change': '2024-09-05T16:34:56.833497+0000', 'last_active': '2024-09-05T16:38:00.565925+0000', 'last_peered': '2024-09-05T16:38:00.565925+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.828289+0000', 'last_became_peered': '2024-09-05T16:34:56.828289+0000', 'last_unstale': '2024-09-05T16:38:00.565925+0000', 'last_undegraded': '2024-09-05T16:34:56.612160+0000', 'last_fullsized': '2024-09-05T16:34:56.557632+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6133, 'log_dups_size': 0, 'ondisk_log_size': 6133, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50241536, 'num_objects': 6133, 'num_object_clones': 0, 'num_object_copies': 24532, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2262, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6133, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6133, 'num_write_kb': 49064, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3871}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:38:02.267 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-05T16:38:02.267 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "35'5887", 'reported_seq': 5955, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:56.067834+0000', 'last_change': '2024-09-05T16:34:56.825031+0000', 'last_active': '2024-09-05T16:37:56.067834+0000', 'last_peered': '2024-09-05T16:37:56.067834+0000', 'last_clean': '2024-09-05T16:33:53.200762+0000', 'last_became_active': '2024-09-05T16:34:56.820292+0000', 'last_became_peered': '2024-09-05T16:34:56.820292+0000', 'last_unstale': '2024-09-05T16:37:56.067834+0000', 'last_undegraded': '2024-09-05T16:34:56.605776+0000', 'last_fullsized': '2024-09-05T16:34:56.559383+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5887, 'log_dups_size': 0, 'ondisk_log_size': 5887, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48190624, 'num_objects': 5883, 'num_object_clones': 0, 'num_object_copies': 23532, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2269, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5883, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5885, 'num_write_kb': 47076, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3616}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:38:02.267 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:38:02.267 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'5962", 'reported_seq': 6041, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:56.146529+0000', 'last_change': '2024-09-05T16:34:56.867900+0000', 'last_active': '2024-09-05T16:37:56.146529+0000', 'last_peered': '2024-09-05T16:37:56.146529+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:37:56.146529+0000', 'last_undegraded': '2024-09-05T16:34:56.590972+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5962, 'log_dups_size': 0, 'ondisk_log_size': 5962, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48814672, 'num_objects': 5959, 'num_object_clones': 0, 'num_object_copies': 23836, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2251, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5959, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5961, 'num_write_kb': 47678, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3709}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:38:02.268 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:38:02.268 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'5954", 'reported_seq': 6012, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:00.481105+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:38:00.481105+0000', 'last_peered': '2024-09-05T16:38:00.481105+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:38:00.481105+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5954, 'log_dups_size': 0, 'ondisk_log_size': 5954, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48747680, 'num_objects': 5951, 'num_object_clones': 0, 'num_object_copies': 23804, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2164, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5951, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5952, 'num_write_kb': 47613, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3789}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:38:02.268 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:38:02.268 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'6029", 'reported_seq': 6075, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:00.583458+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:38:00.583458+0000', 'last_peered': '2024-09-05T16:38:00.583458+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:38:00.583458+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6029, 'log_dups_size': 0, 'ondisk_log_size': 6029, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49389568, 'num_objects': 6029, 'num_object_clones': 0, 'num_object_copies': 24116, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6029, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6029, 'num_write_kb': 48232, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3779}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:38:02.268 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:38:02.268 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'5854", 'reported_seq': 5923, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:56.170051+0000', 'last_change': '2024-09-05T16:34:56.903436+0000', 'last_active': '2024-09-05T16:37:56.170051+0000', 'last_peered': '2024-09-05T16:37:56.170051+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:37:56.170051+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5854, 'log_dups_size': 0, 'ondisk_log_size': 5854, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47903904, 'num_objects': 5848, 'num_object_clones': 0, 'num_object_copies': 23392, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2149, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5848, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5852, 'num_write_kb': 46796, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3701}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:38:02.268 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-05T16:38:02.268 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "35'5927", 'reported_seq': 5995, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:56.033825+0000', 'last_change': '2024-09-05T16:34:56.883319+0000', 'last_active': '2024-09-05T16:37:56.033825+0000', 'last_peered': '2024-09-05T16:37:56.033825+0000', 'last_clean': '2024-09-05T16:33:53.505305+0000', 'last_became_active': '2024-09-05T16:34:56.873389+0000', 'last_became_peered': '2024-09-05T16:34:56.873389+0000', 'last_unstale': '2024-09-05T16:37:56.033825+0000', 'last_undegraded': '2024-09-05T16:34:56.602567+0000', 'last_fullsized': '2024-09-05T16:34:56.559958+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5927, 'log_dups_size': 0, 'ondisk_log_size': 5927, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48510112, 'num_objects': 5922, 'num_object_clones': 0, 'num_object_copies': 23688, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2241, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5922, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5925, 'num_write_kb': 47388, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3683}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:38:02.268 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:38:02.268 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'6086", 'reported_seq': 6146, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:00.624023+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:38:00.624023+0000', 'last_peered': '2024-09-05T16:38:00.624023+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:38:00.624023+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6086, 'log_dups_size': 0, 'ondisk_log_size': 6086, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49785152, 'num_objects': 6078, 'num_object_clones': 0, 'num_object_copies': 24312, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2264, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6078, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6083, 'num_write_kb': 48641, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3817}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:38:02.268 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:38:02.269 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'5974", 'reported_seq': 6056, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:56.060306+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:37:56.060306+0000', 'last_peered': '2024-09-05T16:37:56.060306+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:37:56.060306+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5974, 'log_dups_size': 0, 'ondisk_log_size': 5974, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48919712, 'num_objects': 5972, 'num_object_clones': 0, 'num_object_copies': 23888, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2264, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5972, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5972, 'num_write_kb': 47774, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3710}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:38:02.269 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:38:02.269 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'6189", 'reported_seq': 6249, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:00.619170+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:38:00.619170+0000', 'last_peered': '2024-09-05T16:38:00.619170+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:38:00.619170+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6189, 'log_dups_size': 0, 'ondisk_log_size': 6189, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50654960, 'num_objects': 6184, 'num_object_clones': 0, 'num_object_copies': 24736, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2314, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6184, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6186, 'num_write_kb': 49476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3873}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:38:02.269 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:38:02.269 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'6009", 'reported_seq': 6049, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:00.591951+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:38:00.591951+0000', 'last_peered': '2024-09-05T16:38:00.591951+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:38:00.591951+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6009, 'log_dups_size': 0, 'ondisk_log_size': 6009, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49216080, 'num_objects': 6008, 'num_object_clones': 0, 'num_object_copies': 24032, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2265, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6008, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6008, 'num_write_kb': 48063, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3744}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:38:02.269 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-05T16:38:02.269 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'5966", 'reported_seq': 6593, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:56.138574+0000', 'last_change': '2024-09-05T16:37:30.634570+0000', 'last_active': '2024-09-05T16:37:56.138574+0000', 'last_peered': '2024-09-05T16:37:56.138574+0000', 'last_clean': '2024-09-05T16:33:45.082039+0000', 'last_became_active': '2024-09-05T16:34:56.792386+0000', 'last_became_peered': '2024-09-05T16:34:56.792386+0000', 'last_unstale': '2024-09-05T16:37:56.138574+0000', 'last_undegraded': '2024-09-05T16:34:56.600595+0000', 'last_fullsized': '2024-09-05T16:34:56.558203+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5966, 'log_dups_size': 0, 'ondisk_log_size': 5966, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48863824, 'num_objects': 5965, 'num_object_clones': 0, 'num_object_copies': 23860, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2020, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5965, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5965, 'num_write_kb': 47719, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 275, 'num_bytes_recovered': 2244608, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3945}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:38:02.269 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:38:02.269 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'6200", 'reported_seq': 6256, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:00.601598+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:38:00.601598+0000', 'last_peered': '2024-09-05T16:38:00.601598+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:38:00.601598+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6200, 'log_dups_size': 0, 'ondisk_log_size': 6200, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50754720, 'num_objects': 6196, 'num_object_clones': 0, 'num_object_copies': 24784, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2359, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6196, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6198, 'num_write_kb': 49580, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3839}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:38:02.269 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:38:02.269 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'5999", 'reported_seq': 6081, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:56.092237+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:37:56.092237+0000', 'last_peered': '2024-09-05T16:37:56.092237+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:37:56.092237+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5999, 'log_dups_size': 0, 'ondisk_log_size': 5999, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49116320, 'num_objects': 5996, 'num_object_clones': 0, 'num_object_copies': 23984, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2322, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5996, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5997, 'num_write_kb': 47973, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3676}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:38:02.270 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:38:02.270 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'5943", 'reported_seq': 6011, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:37:56.065302+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:37:56.065302+0000', 'last_peered': '2024-09-05T16:37:56.065302+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:37:56.065302+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 5943, 'log_dups_size': 0, 'ondisk_log_size': 5943, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48641184, 'num_objects': 5938, 'num_object_clones': 0, 'num_object_copies': 23752, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5938, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5941, 'num_write_kb': 47516, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3701}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:38:02.270 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:38:02.536 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout:2024-09-05T16:38:02.544379+0000 min lat: 0.00357551 max lat: 2.29705 avg lat: 0.0452385 2024-09-05T16:38:02.536 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-05T16:38:02.536 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 220 20 97276 97256 3.45295 1.94531 0.0673296 0.0452385 2024-09-05T16:38:03.033 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:38:03.034 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 33633/384812 objects degraded (8.740%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":286,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+clean","count":1}],"num_pgs":16,"num_pools":1,"num_objects":96203,"data_bytes":788057120,"bytes_used":2738614272,"bytes_avail":383808442368,"bytes_total":386547056640,"degraded_objects":33633,"degraded_total":384812,"degraded_ratio":0.087401120547176289,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":67551,"recovering_keys_per_sec":0,"num_objects_recovered":99,"num_bytes_recovered":811008,"num_keys_recovered":0,"write_bytes_sec":2380662,"read_op_per_sec":0,"write_op_per_sec":290},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:38:03.065 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:38:03.065 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:38:03.536 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 221 20 97528 97508 3.44624 1.96875 0.0540244 0.0453254 2024-09-05T16:38:03.687 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:38:03.687 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:38:03.718 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":169,"stamp":"2024-09-05T16:38:02.874098+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":793742368,"num_objects":96897,"num_object_clones":0,"num_object_copies":387588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33580,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":96897,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":91228,"num_write_kb":729717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":352,"num_bytes_recovered":2678784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96953,"ondisk_log_size":96953,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2696456,"kb_used_data":1415176,"kb_used_omap":14,"kb_used_meta":1281201,"kb_avail":374790904,"statfs":{"total":386547056640,"available":383785885696,"internally_reserved":0,"allocated":1449140224,"data_stored":1448372060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1311950391},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,3,3,4,17,16,8,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":28295168,"num_objects":3454,"num_object_clones":0,"num_object_copies":13816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-152,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3454,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3454,"num_write_kb":27632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":152,"num_bytes_recovered":1245184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005111"},"pg_stats":[{"pgid":"1.f","version":"35'6133","reported_seq":6177,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:00.565925+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:38:00.565925+0000","last_peered":"2024-09-05T16:38:00.565925+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:38:00.565925+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6133,"log_dups_size":0,"ondisk_log_size":6133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50241536,"num_objects":6133,"num_object_clones":0,"num_object_copies":24532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6133,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6133,"num_write_kb":49064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3871}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'5968","reported_seq":6036,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:01.194482+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:38:01.194482+0000","last_peered":"2024-09-05T16:38:01.194482+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:38:01.194482+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5968,"log_dups_size":0,"ondisk_log_size":5968,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48854176,"num_objects":5964,"num_object_clones":0,"num_object_copies":23856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5964,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5966,"num_write_kb":47724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3697}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'6041","reported_seq":6120,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:00.904050+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:38:00.904050+0000","last_peered":"2024-09-05T16:38:00.904050+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:38:00.904050+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6041,"log_dups_size":0,"ondisk_log_size":6041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49461840,"num_objects":6038,"num_object_clones":0,"num_object_copies":24152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6038,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6040,"num_write_kb":48310,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3788}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'6147","reported_seq":10870,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-05T16:38:00.255903+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:38:00.255903+0000","last_peered":"2024-09-05T16:38:00.255903+0000","last_clean":"2024-09-05T16:38:00.255903+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:38:00.255903+0000","last_undegraded":"2024-09-05T16:38:00.255903+0000","last_fullsized":"2024-09-05T16:38:00.255903+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":6147,"log_dups_size":0,"ondisk_log_size":6147,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50307072,"num_objects":6141,"num_object_clones":0,"num_object_copies":24564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6141,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":447,"num_write_kb":3576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'5954","reported_seq":6012,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:00.481105+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:38:00.481105+0000","last_peered":"2024-09-05T16:38:00.481105+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:38:00.481105+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5954,"log_dups_size":0,"ondisk_log_size":5954,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48747680,"num_objects":5951,"num_object_clones":0,"num_object_copies":23804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5951,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5952,"num_write_kb":47613,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3789}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'6029","reported_seq":6075,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:00.583458+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:38:00.583458+0000","last_peered":"2024-09-05T16:38:00.583458+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:38:00.583458+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6029,"log_dups_size":0,"ondisk_log_size":6029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49389568,"num_objects":6029,"num_object_clones":0,"num_object_copies":24116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6029,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6029,"num_write_kb":48232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3779}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'5953","reported_seq":6022,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:01.098237+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:38:01.098237+0000","last_peered":"2024-09-05T16:38:01.098237+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:38:01.098237+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":5953,"log_dups_size":0,"ondisk_log_size":5953,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48714912,"num_objects":5947,"num_object_clones":0,"num_object_copies":23788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5947,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5951,"num_write_kb":47588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3800}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'6024","reported_seq":6091,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:01.172767+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:38:01.172767+0000","last_peered":"2024-09-05T16:38:01.172767+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:38:01.172767+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6024,"log_dups_size":0,"ondisk_log_size":6024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49304736,"num_objects":6019,"num_object_clones":0,"num_object_copies":24076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6019,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6022,"num_write_kb":48164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3780}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'6086","reported_seq":6146,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:00.624023+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:38:00.624023+0000","last_peered":"2024-09-05T16:38:00.624023+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:38:00.624023+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6086,"log_dups_size":0,"ondisk_log_size":6086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49785152,"num_objects":6078,"num_object_clones":0,"num_object_copies":24312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6078,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6083,"num_write_kb":48641,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3817}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'6062","reported_seq":6144,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:01.180501+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:38:01.180501+0000","last_peered":"2024-09-05T16:38:01.180501+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:38:01.180501+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6062,"log_dups_size":0,"ondisk_log_size":6062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49640608,"num_objects":6060,"num_object_clones":0,"num_object_copies":24240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6060,"num_write_kb":48478,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3798}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'6189","reported_seq":6249,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:00.619170+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:38:00.619170+0000","last_peered":"2024-09-05T16:38:00.619170+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:38:00.619170+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6189,"log_dups_size":0,"ondisk_log_size":6189,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50654960,"num_objects":6184,"num_object_clones":0,"num_object_copies":24736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6184,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6186,"num_write_kb":49476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3873}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'6009","reported_seq":6049,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:00.591951+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:38:00.591951+0000","last_peered":"2024-09-05T16:38:00.591951+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:38:00.591951+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6009,"log_dups_size":0,"ondisk_log_size":6009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49216080,"num_objects":6008,"num_object_clones":0,"num_object_copies":24032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6008,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6008,"num_write_kb":48063,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3744}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'6046","reported_seq":6779,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:01.267067+0000","last_change":"2024-09-05T16:37:30.634570+0000","last_active":"2024-09-05T16:38:01.267067+0000","last_peered":"2024-09-05T16:38:01.267067+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:38:01.267067+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6046,"log_dups_size":0,"ondisk_log_size":6046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49519184,"num_objects":6045,"num_object_clones":0,"num_object_copies":24180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1967,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6045,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6045,"num_write_kb":48359,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":328,"num_bytes_recovered":2678784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4078}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'6200","reported_seq":6256,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:00.601598+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:38:00.601598+0000","last_peered":"2024-09-05T16:38:00.601598+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:38:00.601598+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6200,"log_dups_size":0,"ondisk_log_size":6200,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50754720,"num_objects":6196,"num_object_clones":0,"num_object_copies":24784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6196,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6198,"num_write_kb":49580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3839}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'6080","reported_seq":6162,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:01.273568+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:38:01.273568+0000","last_peered":"2024-09-05T16:38:01.273568+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:38:01.273568+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6080,"log_dups_size":0,"ondisk_log_size":6080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49779872,"num_objects":6077,"num_object_clones":0,"num_object_copies":24308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6077,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6078,"num_write_kb":48621,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3757}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'6032","reported_seq":6099,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:01.216955+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:38:01.216955+0000","last_peered":"2024-09-05T16:38:01.216955+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:38:01.216955+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6032,"log_dups_size":0,"ondisk_log_size":6032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49370272,"num_objects":6027,"num_object_clones":0,"num_object_copies":24108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6027,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6030,"num_write_kb":48228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3790}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":793742368,"num_objects":96897,"num_object_clones":0,"num_object_copies":387588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33580,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":96897,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":91228,"num_write_kb":729717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":352,"num_bytes_recovered":2678784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1449521152,"data_stored":1449521152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96953,"ondisk_log_size":96953,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084337,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":734420,"kb_used_data":387588,"kb_used_omap":4,"kb_used_meta":346811,"kb_avail":93637420,"statfs":{"total":96636764160,"available":95884718080,"internally_reserved":0,"allocated":396890112,"data_stored":396698071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":355135187},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,2,1,4,3,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986214,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":482716,"kb_used_data":252492,"kb_used_omap":4,"kb_used_meta":230203,"kb_avail":93889124,"statfs":{"total":96636764160,"available":96142462976,"internally_reserved":0,"allocated":258551808,"data_stored":258359767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":235728594},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,0,2,0,2,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673020,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":739028,"kb_used_data":387140,"kb_used_omap":1,"kb_used_meta":351870,"kb_avail":93632812,"statfs":{"total":96636764160,"available":95879999488,"internally_reserved":0,"allocated":396431360,"data_stored":396239319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":360315333},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1,0,5,4,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182451,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":740292,"kb_used_data":387956,"kb_used_omap":4,"kb_used_meta":352315,"kb_avail":93631548,"statfs":{"total":96636764160,"available":95878705152,"internally_reserved":0,"allocated":397266944,"data_stored":397074903,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":360771277},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,8,7],"upper_bound":64},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":397086720,"data_stored":397086720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":258691072,"data_stored":258691072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":397148160,"data_stored":397148160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":396595200,"data_stored":396595200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:38:04.536 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 222 20 97798 97778 3.44021 2.10938 0.0324118 0.0454064 2024-09-05T16:38:05.536 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 223 20 98080 98060 3.43467 2.20312 0.0705991 0.0454788 2024-09-05T16:38:06.536 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 224 20 98355 98335 3.42892 2.14844 0.0588743 0.0455539 2024-09-05T16:38:06.720 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:38:07.386 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:38:07.387 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:38:07.419 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":171,"stamp":"2024-09-05T16:38:06.875660+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":804686880,"num_objects":98233,"num_object_clones":0,"num_object_copies":392932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33528,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98233,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":92564,"num_write_kb":740405,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":404,"num_bytes_recovered":3104768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":98289,"ondisk_log_size":98289,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2736400,"kb_used_data":1437072,"kb_used_omap":14,"kb_used_meta":1299249,"kb_avail":374750960,"statfs":{"total":386547056640,"available":383744983040,"internally_reserved":0,"allocated":1471561728,"data_stored":1470793564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1330431543},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[11,3,3,3,21,20,23,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":33579008,"num_objects":4099,"num_object_clones":0,"num_object_copies":16396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-157,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4099,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4099,"num_write_kb":32792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":157,"num_bytes_recovered":1286144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004981"},"pg_stats":[{"pgid":"1.f","version":"35'6221","reported_seq":6267,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:05.646370+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:38:05.646370+0000","last_peered":"2024-09-05T16:38:05.646370+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:38:05.646370+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6221,"log_dups_size":0,"ondisk_log_size":6221,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50962432,"num_objects":6221,"num_object_clones":0,"num_object_copies":24884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6221,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6221,"num_write_kb":49768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3959}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'6035","reported_seq":6102,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:06.213250+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:38:06.213250+0000","last_peered":"2024-09-05T16:38:06.213250+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:38:06.213250+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6035,"log_dups_size":0,"ondisk_log_size":6035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49403040,"num_objects":6031,"num_object_clones":0,"num_object_copies":24124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6031,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6033,"num_write_kb":48260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3764}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'6124","reported_seq":6203,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:06.250751+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:38:06.250751+0000","last_peered":"2024-09-05T16:38:06.250751+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:38:06.250751+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6124,"log_dups_size":0,"ondisk_log_size":6124,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50141776,"num_objects":6121,"num_object_clones":0,"num_object_copies":24484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6121,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6123,"num_write_kb":48974,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3871}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'6222","reported_seq":10945,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-05T16:38:05.303042+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:38:05.303042+0000","last_peered":"2024-09-05T16:38:05.303042+0000","last_clean":"2024-09-05T16:38:05.303042+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:38:05.303042+0000","last_undegraded":"2024-09-05T16:38:05.303042+0000","last_fullsized":"2024-09-05T16:38:05.303042+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":6222,"log_dups_size":0,"ondisk_log_size":6222,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50921472,"num_objects":6216,"num_object_clones":0,"num_object_copies":24864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":522,"num_write_kb":4176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'6029","reported_seq":6087,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:05.673226+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:38:05.673226+0000","last_peered":"2024-09-05T16:38:05.673226+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:38:05.673226+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6029,"log_dups_size":0,"ondisk_log_size":6029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49362080,"num_objects":6026,"num_object_clones":0,"num_object_copies":24104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6026,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6027,"num_write_kb":48213,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3864}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'6102","reported_seq":6148,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:05.692487+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:38:05.692487+0000","last_peered":"2024-09-05T16:38:05.692487+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:38:05.692487+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6102,"log_dups_size":0,"ondisk_log_size":6102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49987584,"num_objects":6102,"num_object_clones":0,"num_object_copies":24408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6102,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6102,"num_write_kb":48816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3852}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'6048","reported_seq":6117,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:06.225509+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:38:06.225509+0000","last_peered":"2024-09-05T16:38:06.225509+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:38:06.225509+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6048,"log_dups_size":0,"ondisk_log_size":6048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49493152,"num_objects":6042,"num_object_clones":0,"num_object_copies":24168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6042,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6046,"num_write_kb":48348,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3895}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'6113","reported_seq":6180,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:06.170980+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:38:06.170980+0000","last_peered":"2024-09-05T16:38:06.170980+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:38:06.170980+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6113,"log_dups_size":0,"ondisk_log_size":6113,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50033824,"num_objects":6108,"num_object_clones":0,"num_object_copies":24432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6108,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6111,"num_write_kb":48876,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3869}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'6159","reported_seq":6219,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:05.623161+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:38:05.623161+0000","last_peered":"2024-09-05T16:38:05.623161+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:38:05.623161+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6159,"log_dups_size":0,"ondisk_log_size":6159,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50383168,"num_objects":6151,"num_object_clones":0,"num_object_copies":24604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6151,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6156,"num_write_kb":49225,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3890}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'6171","reported_seq":6253,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:06.114984+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:38:06.114984+0000","last_peered":"2024-09-05T16:38:06.114984+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:38:06.114984+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6171,"log_dups_size":0,"ondisk_log_size":6171,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50533536,"num_objects":6169,"num_object_clones":0,"num_object_copies":24676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6169,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6169,"num_write_kb":49350,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3907}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'6273","reported_seq":6333,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:05.673432+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:38:05.673432+0000","last_peered":"2024-09-05T16:38:05.673432+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:38:05.673432+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6273,"log_dups_size":0,"ondisk_log_size":6273,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51343088,"num_objects":6268,"num_object_clones":0,"num_object_copies":25072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6268,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6270,"num_write_kb":50148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3957}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'6097","reported_seq":6137,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:05.756908+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:38:05.756908+0000","last_peered":"2024-09-05T16:38:05.756908+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:38:05.756908+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6097,"log_dups_size":0,"ondisk_log_size":6097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49936976,"num_objects":6096,"num_object_clones":0,"num_object_copies":24384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6096,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6096,"num_write_kb":48767,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3832}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'6134","reported_seq":6971,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:06.313306+0000","last_change":"2024-09-05T16:37:30.634570+0000","last_active":"2024-09-05T16:38:06.313306+0000","last_peered":"2024-09-05T16:38:06.313306+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:38:06.313306+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6134,"log_dups_size":0,"ondisk_log_size":6134,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50240080,"num_objects":6133,"num_object_clones":0,"num_object_copies":24532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1915,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6133,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6133,"num_write_kb":49063,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":380,"num_bytes_recovered":3104768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4218}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'6275","reported_seq":6331,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:05.675699+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:38:05.675699+0000","last_peered":"2024-09-05T16:38:05.675699+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:38:05.675699+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6275,"log_dups_size":0,"ondisk_log_size":6275,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51369120,"num_objects":6271,"num_object_clones":0,"num_object_copies":25084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6271,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6273,"num_write_kb":50180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3914}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'6159","reported_seq":6241,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:06.234958+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:38:06.234958+0000","last_peered":"2024-09-05T16:38:06.234958+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:38:06.234958+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6159,"log_dups_size":0,"ondisk_log_size":6159,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50427040,"num_objects":6156,"num_object_clones":0,"num_object_copies":24624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6156,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6157,"num_write_kb":49253,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3836}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'6127","reported_seq":6195,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:06.191740+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:38:06.191740+0000","last_peered":"2024-09-05T16:38:06.191740+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:38:06.191740+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6127,"log_dups_size":0,"ondisk_log_size":6127,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50148512,"num_objects":6122,"num_object_clones":0,"num_object_copies":24488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6122,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6125,"num_write_kb":48988,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3885}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":804686880,"num_objects":98233,"num_object_clones":0,"num_object_copies":392932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33528,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98233,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":92564,"num_write_kb":740405,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":404,"num_bytes_recovered":3104768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1471447040,"data_stored":1471447040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":98289,"ondisk_log_size":98289,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084338,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":745088,"kb_used_data":392688,"kb_used_omap":4,"kb_used_meta":352379,"kb_avail":93626752,"statfs":{"total":96636764160,"available":95873794048,"internally_reserved":0,"allocated":402112512,"data_stored":401920471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":360836819},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[11,0,0,0,8,6,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986215,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":492272,"kb_used_data":257888,"kb_used_omap":4,"kb_used_meta":234363,"kb_avail":93879568,"statfs":{"total":96636764160,"available":96132677632,"internally_reserved":0,"allocated":264077312,"data_stored":263885271,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":239988434},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,1,3,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673021,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":749428,"kb_used_data":393380,"kb_used_omap":1,"kb_used_meta":356030,"kb_avail":93622412,"statfs":{"total":96636764160,"available":95869349888,"internally_reserved":0,"allocated":402821120,"data_stored":402629079,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":364575173},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1,7,8,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182452,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":749612,"kb_used_data":393116,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93622228,"statfs":{"total":96636764160,"available":95869161472,"internally_reserved":0,"allocated":402550784,"data_stored":402358743,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":365031117},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,5,3,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":402513920,"data_stored":402513920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":264327168,"data_stored":264327168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":402632704,"data_stored":402632704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":401973248,"data_stored":401973248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:38:07.420 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:38:07.536 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 225 20 98611 98591 3.42257 2 0.0911838 0.0456411 2024-09-05T16:38:08.013 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:38:08.013 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:38:08.046 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":171,"stamp":"2024-09-05T16:38:06.875660+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":804686880,"num_objects":98233,"num_object_clones":0,"num_object_copies":392932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33528,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98233,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":92564,"num_write_kb":740405,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":404,"num_bytes_recovered":3104768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":98289,"ondisk_log_size":98289,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2736400,"kb_used_data":1437072,"kb_used_omap":14,"kb_used_meta":1299249,"kb_avail":374750960,"statfs":{"total":386547056640,"available":383744983040,"internally_reserved":0,"allocated":1471561728,"data_stored":1470793564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1330431543},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[11,3,3,3,21,20,23,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":33579008,"num_objects":4099,"num_object_clones":0,"num_object_copies":16396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-157,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4099,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4099,"num_write_kb":32792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":157,"num_bytes_recovered":1286144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004981"},"pg_stats":[{"pgid":"1.f","version":"35'6221","reported_seq":6267,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:05.646370+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:38:05.646370+0000","last_peered":"2024-09-05T16:38:05.646370+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:38:05.646370+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6221,"log_dups_size":0,"ondisk_log_size":6221,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50962432,"num_objects":6221,"num_object_clones":0,"num_object_copies":24884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6221,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6221,"num_write_kb":49768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3959}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'6035","reported_seq":6102,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:06.213250+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:38:06.213250+0000","last_peered":"2024-09-05T16:38:06.213250+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:38:06.213250+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6035,"log_dups_size":0,"ondisk_log_size":6035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49403040,"num_objects":6031,"num_object_clones":0,"num_object_copies":24124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6031,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6033,"num_write_kb":48260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3764}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'6124","reported_seq":6203,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:06.250751+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:38:06.250751+0000","last_peered":"2024-09-05T16:38:06.250751+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:38:06.250751+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6124,"log_dups_size":0,"ondisk_log_size":6124,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50141776,"num_objects":6121,"num_object_clones":0,"num_object_copies":24484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6121,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6123,"num_write_kb":48974,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3871}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'6222","reported_seq":10945,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-05T16:38:05.303042+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:38:05.303042+0000","last_peered":"2024-09-05T16:38:05.303042+0000","last_clean":"2024-09-05T16:38:05.303042+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:38:05.303042+0000","last_undegraded":"2024-09-05T16:38:05.303042+0000","last_fullsized":"2024-09-05T16:38:05.303042+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":6222,"log_dups_size":0,"ondisk_log_size":6222,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50921472,"num_objects":6216,"num_object_clones":0,"num_object_copies":24864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":522,"num_write_kb":4176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'6029","reported_seq":6087,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:05.673226+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:38:05.673226+0000","last_peered":"2024-09-05T16:38:05.673226+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:38:05.673226+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6029,"log_dups_size":0,"ondisk_log_size":6029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49362080,"num_objects":6026,"num_object_clones":0,"num_object_copies":24104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6026,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6027,"num_write_kb":48213,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3864}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'6102","reported_seq":6148,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:05.692487+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:38:05.692487+0000","last_peered":"2024-09-05T16:38:05.692487+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:38:05.692487+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6102,"log_dups_size":0,"ondisk_log_size":6102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49987584,"num_objects":6102,"num_object_clones":0,"num_object_copies":24408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6102,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6102,"num_write_kb":48816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3852}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'6048","reported_seq":6117,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:06.225509+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:38:06.225509+0000","last_peered":"2024-09-05T16:38:06.225509+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:38:06.225509+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6048,"log_dups_size":0,"ondisk_log_size":6048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49493152,"num_objects":6042,"num_object_clones":0,"num_object_copies":24168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6042,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6046,"num_write_kb":48348,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3895}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'6113","reported_seq":6180,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:06.170980+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:38:06.170980+0000","last_peered":"2024-09-05T16:38:06.170980+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:38:06.170980+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6113,"log_dups_size":0,"ondisk_log_size":6113,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50033824,"num_objects":6108,"num_object_clones":0,"num_object_copies":24432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6108,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6111,"num_write_kb":48876,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3869}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'6159","reported_seq":6219,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:05.623161+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:38:05.623161+0000","last_peered":"2024-09-05T16:38:05.623161+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:38:05.623161+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6159,"log_dups_size":0,"ondisk_log_size":6159,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50383168,"num_objects":6151,"num_object_clones":0,"num_object_copies":24604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6151,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6156,"num_write_kb":49225,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3890}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'6171","reported_seq":6253,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:06.114984+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:38:06.114984+0000","last_peered":"2024-09-05T16:38:06.114984+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:38:06.114984+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6171,"log_dups_size":0,"ondisk_log_size":6171,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50533536,"num_objects":6169,"num_object_clones":0,"num_object_copies":24676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6169,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6169,"num_write_kb":49350,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3907}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'6273","reported_seq":6333,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:05.673432+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:38:05.673432+0000","last_peered":"2024-09-05T16:38:05.673432+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:38:05.673432+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6273,"log_dups_size":0,"ondisk_log_size":6273,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51343088,"num_objects":6268,"num_object_clones":0,"num_object_copies":25072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6268,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6270,"num_write_kb":50148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3957}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'6097","reported_seq":6137,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:05.756908+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:38:05.756908+0000","last_peered":"2024-09-05T16:38:05.756908+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:38:05.756908+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6097,"log_dups_size":0,"ondisk_log_size":6097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49936976,"num_objects":6096,"num_object_clones":0,"num_object_copies":24384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6096,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6096,"num_write_kb":48767,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3832}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'6134","reported_seq":6971,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:06.313306+0000","last_change":"2024-09-05T16:37:30.634570+0000","last_active":"2024-09-05T16:38:06.313306+0000","last_peered":"2024-09-05T16:38:06.313306+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:38:06.313306+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6134,"log_dups_size":0,"ondisk_log_size":6134,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50240080,"num_objects":6133,"num_object_clones":0,"num_object_copies":24532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1915,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6133,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6133,"num_write_kb":49063,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":380,"num_bytes_recovered":3104768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4218}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'6275","reported_seq":6331,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:05.675699+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:38:05.675699+0000","last_peered":"2024-09-05T16:38:05.675699+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:38:05.675699+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6275,"log_dups_size":0,"ondisk_log_size":6275,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51369120,"num_objects":6271,"num_object_clones":0,"num_object_copies":25084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6271,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6273,"num_write_kb":50180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3914}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'6159","reported_seq":6241,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:06.234958+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:38:06.234958+0000","last_peered":"2024-09-05T16:38:06.234958+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:38:06.234958+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6159,"log_dups_size":0,"ondisk_log_size":6159,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50427040,"num_objects":6156,"num_object_clones":0,"num_object_copies":24624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6156,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6157,"num_write_kb":49253,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3836}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'6127","reported_seq":6195,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:06.191740+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:38:06.191740+0000","last_peered":"2024-09-05T16:38:06.191740+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:38:06.191740+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6127,"log_dups_size":0,"ondisk_log_size":6127,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50148512,"num_objects":6122,"num_object_clones":0,"num_object_copies":24488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6122,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6125,"num_write_kb":48988,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3885}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":804686880,"num_objects":98233,"num_object_clones":0,"num_object_copies":392932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33528,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98233,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":92564,"num_write_kb":740405,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":404,"num_bytes_recovered":3104768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1471447040,"data_stored":1471447040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":98289,"ondisk_log_size":98289,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084338,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":745088,"kb_used_data":392688,"kb_used_omap":4,"kb_used_meta":352379,"kb_avail":93626752,"statfs":{"total":96636764160,"available":95873794048,"internally_reserved":0,"allocated":402112512,"data_stored":401920471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":360836819},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[11,0,0,0,8,6,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986215,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":492272,"kb_used_data":257888,"kb_used_omap":4,"kb_used_meta":234363,"kb_avail":93879568,"statfs":{"total":96636764160,"available":96132677632,"internally_reserved":0,"allocated":264077312,"data_stored":263885271,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":239988434},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,1,3,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673021,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":749428,"kb_used_data":393380,"kb_used_omap":1,"kb_used_meta":356030,"kb_avail":93622412,"statfs":{"total":96636764160,"available":95869349888,"internally_reserved":0,"allocated":402821120,"data_stored":402629079,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":364575173},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1,7,8,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182452,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":749612,"kb_used_data":393116,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93622228,"statfs":{"total":96636764160,"available":95869161472,"internally_reserved":0,"allocated":402550784,"data_stored":402358743,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":365031117},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,5,3,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":402513920,"data_stored":402513920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":264327168,"data_stored":264327168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":402632704,"data_stored":402632704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":401973248,"data_stored":401973248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:38:08.048 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-05T16:38:08.048 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'6221", 'reported_seq': 6267, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:05.646370+0000', 'last_change': '2024-09-05T16:34:56.833497+0000', 'last_active': '2024-09-05T16:38:05.646370+0000', 'last_peered': '2024-09-05T16:38:05.646370+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.828289+0000', 'last_became_peered': '2024-09-05T16:34:56.828289+0000', 'last_unstale': '2024-09-05T16:38:05.646370+0000', 'last_undegraded': '2024-09-05T16:34:56.612160+0000', 'last_fullsized': '2024-09-05T16:34:56.557632+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6221, 'log_dups_size': 0, 'ondisk_log_size': 6221, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50962432, 'num_objects': 6221, 'num_object_clones': 0, 'num_object_copies': 24884, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2262, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6221, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6221, 'num_write_kb': 49768, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3959}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:38:08.049 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-05T16:38:08.049 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "35'6035", 'reported_seq': 6102, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:06.213250+0000', 'last_change': '2024-09-05T16:34:56.825031+0000', 'last_active': '2024-09-05T16:38:06.213250+0000', 'last_peered': '2024-09-05T16:38:06.213250+0000', 'last_clean': '2024-09-05T16:33:53.200762+0000', 'last_became_active': '2024-09-05T16:34:56.820292+0000', 'last_became_peered': '2024-09-05T16:34:56.820292+0000', 'last_unstale': '2024-09-05T16:38:06.213250+0000', 'last_undegraded': '2024-09-05T16:34:56.605776+0000', 'last_fullsized': '2024-09-05T16:34:56.559383+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6035, 'log_dups_size': 0, 'ondisk_log_size': 6035, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49403040, 'num_objects': 6031, 'num_object_clones': 0, 'num_object_copies': 24124, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2269, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6031, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6033, 'num_write_kb': 48260, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3764}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:38:08.049 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:38:08.049 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'6124", 'reported_seq': 6203, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:06.250751+0000', 'last_change': '2024-09-05T16:34:56.867900+0000', 'last_active': '2024-09-05T16:38:06.250751+0000', 'last_peered': '2024-09-05T16:38:06.250751+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:38:06.250751+0000', 'last_undegraded': '2024-09-05T16:34:56.590972+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6124, 'log_dups_size': 0, 'ondisk_log_size': 6124, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50141776, 'num_objects': 6121, 'num_object_clones': 0, 'num_object_copies': 24484, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2251, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6121, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6123, 'num_write_kb': 48974, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3871}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:38:08.050 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:38:08.050 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'6029", 'reported_seq': 6087, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:05.673226+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:38:05.673226+0000', 'last_peered': '2024-09-05T16:38:05.673226+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:38:05.673226+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6029, 'log_dups_size': 0, 'ondisk_log_size': 6029, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49362080, 'num_objects': 6026, 'num_object_clones': 0, 'num_object_copies': 24104, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2164, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6026, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6027, 'num_write_kb': 48213, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3864}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:38:08.050 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:38:08.050 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'6102", 'reported_seq': 6148, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:05.692487+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:38:05.692487+0000', 'last_peered': '2024-09-05T16:38:05.692487+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:38:05.692487+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6102, 'log_dups_size': 0, 'ondisk_log_size': 6102, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49987584, 'num_objects': 6102, 'num_object_clones': 0, 'num_object_copies': 24408, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6102, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6102, 'num_write_kb': 48816, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3852}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:38:08.050 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:38:08.051 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'6048", 'reported_seq': 6117, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:06.225509+0000', 'last_change': '2024-09-05T16:34:56.903436+0000', 'last_active': '2024-09-05T16:38:06.225509+0000', 'last_peered': '2024-09-05T16:38:06.225509+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:38:06.225509+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6048, 'log_dups_size': 0, 'ondisk_log_size': 6048, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49493152, 'num_objects': 6042, 'num_object_clones': 0, 'num_object_copies': 24168, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2149, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6042, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6046, 'num_write_kb': 48348, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3895}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:38:08.051 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-05T16:38:08.051 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "35'6113", 'reported_seq': 6180, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:06.170980+0000', 'last_change': '2024-09-05T16:34:56.883319+0000', 'last_active': '2024-09-05T16:38:06.170980+0000', 'last_peered': '2024-09-05T16:38:06.170980+0000', 'last_clean': '2024-09-05T16:33:53.505305+0000', 'last_became_active': '2024-09-05T16:34:56.873389+0000', 'last_became_peered': '2024-09-05T16:34:56.873389+0000', 'last_unstale': '2024-09-05T16:38:06.170980+0000', 'last_undegraded': '2024-09-05T16:34:56.602567+0000', 'last_fullsized': '2024-09-05T16:34:56.559958+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6113, 'log_dups_size': 0, 'ondisk_log_size': 6113, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50033824, 'num_objects': 6108, 'num_object_clones': 0, 'num_object_copies': 24432, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2241, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6108, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6111, 'num_write_kb': 48876, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3869}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:38:08.051 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:38:08.051 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'6159", 'reported_seq': 6219, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:05.623161+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:38:05.623161+0000', 'last_peered': '2024-09-05T16:38:05.623161+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:38:05.623161+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6159, 'log_dups_size': 0, 'ondisk_log_size': 6159, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50383168, 'num_objects': 6151, 'num_object_clones': 0, 'num_object_copies': 24604, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2264, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6151, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6156, 'num_write_kb': 49225, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3890}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:38:08.052 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:38:08.052 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'6171", 'reported_seq': 6253, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:06.114984+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:38:06.114984+0000', 'last_peered': '2024-09-05T16:38:06.114984+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:38:06.114984+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6171, 'log_dups_size': 0, 'ondisk_log_size': 6171, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50533536, 'num_objects': 6169, 'num_object_clones': 0, 'num_object_copies': 24676, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2264, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6169, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6169, 'num_write_kb': 49350, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3907}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:38:08.052 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:38:08.052 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'6273", 'reported_seq': 6333, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:05.673432+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:38:05.673432+0000', 'last_peered': '2024-09-05T16:38:05.673432+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:38:05.673432+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6273, 'log_dups_size': 0, 'ondisk_log_size': 6273, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51343088, 'num_objects': 6268, 'num_object_clones': 0, 'num_object_copies': 25072, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2314, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6268, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6270, 'num_write_kb': 50148, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3957}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:38:08.053 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:38:08.053 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'6097", 'reported_seq': 6137, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:05.756908+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:38:05.756908+0000', 'last_peered': '2024-09-05T16:38:05.756908+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:38:05.756908+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6097, 'log_dups_size': 0, 'ondisk_log_size': 6097, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49936976, 'num_objects': 6096, 'num_object_clones': 0, 'num_object_copies': 24384, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2265, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6096, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6096, 'num_write_kb': 48767, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3832}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:38:08.053 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-05T16:38:08.053 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'6134", 'reported_seq': 6971, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:06.313306+0000', 'last_change': '2024-09-05T16:37:30.634570+0000', 'last_active': '2024-09-05T16:38:06.313306+0000', 'last_peered': '2024-09-05T16:38:06.313306+0000', 'last_clean': '2024-09-05T16:33:45.082039+0000', 'last_became_active': '2024-09-05T16:34:56.792386+0000', 'last_became_peered': '2024-09-05T16:34:56.792386+0000', 'last_unstale': '2024-09-05T16:38:06.313306+0000', 'last_undegraded': '2024-09-05T16:34:56.600595+0000', 'last_fullsized': '2024-09-05T16:34:56.558203+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6134, 'log_dups_size': 0, 'ondisk_log_size': 6134, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50240080, 'num_objects': 6133, 'num_object_clones': 0, 'num_object_copies': 24532, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1915, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6133, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6133, 'num_write_kb': 49063, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 380, 'num_bytes_recovered': 3104768, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 4218}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:38:08.053 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:38:08.054 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'6275", 'reported_seq': 6331, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:05.675699+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:38:05.675699+0000', 'last_peered': '2024-09-05T16:38:05.675699+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:38:05.675699+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6275, 'log_dups_size': 0, 'ondisk_log_size': 6275, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51369120, 'num_objects': 6271, 'num_object_clones': 0, 'num_object_copies': 25084, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2359, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6271, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6273, 'num_write_kb': 50180, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3914}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:38:08.054 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:38:08.054 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'6159", 'reported_seq': 6241, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:06.234958+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:38:06.234958+0000', 'last_peered': '2024-09-05T16:38:06.234958+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:38:06.234958+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6159, 'log_dups_size': 0, 'ondisk_log_size': 6159, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50427040, 'num_objects': 6156, 'num_object_clones': 0, 'num_object_copies': 24624, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2322, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6156, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6157, 'num_write_kb': 49253, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3836}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:38:08.054 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:38:08.054 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'6127", 'reported_seq': 6195, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:06.191740+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:38:06.191740+0000', 'last_peered': '2024-09-05T16:38:06.191740+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:38:06.191740+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6127, 'log_dups_size': 0, 'ondisk_log_size': 6127, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50148512, 'num_objects': 6122, 'num_object_clones': 0, 'num_object_copies': 24488, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6122, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6125, 'num_write_kb': 48988, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3885}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:38:08.055 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:38:08.537 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 226 20 98845 98825 3.41551 1.82812 0.0812391 0.0457328 2024-09-05T16:38:08.740 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:38:08.740 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 33528/392932 objects degraded (8.533%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":291,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":98233,"data_bytes":804686880,"bytes_used":2802073600,"bytes_avail":383744983040,"bytes_total":386547056640,"degraded_objects":33528,"degraded_total":392932,"degraded_ratio":0.085327741186770237,"recovering_objects_per_sec":13,"recovering_bytes_per_sec":107134,"recovering_keys_per_sec":0,"num_objects_recovered":157,"num_bytes_recovered":1286144,"num_keys_recovered":0,"write_bytes_sec":2797089,"read_op_per_sec":0,"write_op_per_sec":341},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:38:08.762 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:38:08.762 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:38:09.451 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:38:09.451 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:38:09.477 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":172,"stamp":"2024-09-05T16:38:08.876630+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":804686880,"num_objects":98233,"num_object_clones":0,"num_object_copies":392932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33528,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98233,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":92564,"num_write_kb":740405,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":404,"num_bytes_recovered":3104768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":98289,"ondisk_log_size":98289,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2736400,"kb_used_data":1437072,"kb_used_omap":14,"kb_used_meta":1299249,"kb_avail":374750960,"statfs":{"total":386547056640,"available":383744983040,"internally_reserved":0,"allocated":1471561728,"data_stored":1470793564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1330431543},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[11,3,3,3,21,20,23,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":22233088,"num_objects":2714,"num_object_clones":0,"num_object_copies":10856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-105,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2714,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2714,"num_write_kb":21712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":105,"num_bytes_recovered":860160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005127"},"pg_stats":[{"pgid":"1.f","version":"35'6221","reported_seq":6267,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:05.646370+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:38:05.646370+0000","last_peered":"2024-09-05T16:38:05.646370+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:38:05.646370+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6221,"log_dups_size":0,"ondisk_log_size":6221,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50962432,"num_objects":6221,"num_object_clones":0,"num_object_copies":24884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6221,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6221,"num_write_kb":49768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3959}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'6035","reported_seq":6102,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:06.213250+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:38:06.213250+0000","last_peered":"2024-09-05T16:38:06.213250+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:38:06.213250+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6035,"log_dups_size":0,"ondisk_log_size":6035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49403040,"num_objects":6031,"num_object_clones":0,"num_object_copies":24124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6031,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6033,"num_write_kb":48260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3764}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'6124","reported_seq":6203,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:06.250751+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:38:06.250751+0000","last_peered":"2024-09-05T16:38:06.250751+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:38:06.250751+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6124,"log_dups_size":0,"ondisk_log_size":6124,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50141776,"num_objects":6121,"num_object_clones":0,"num_object_copies":24484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6121,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6123,"num_write_kb":48974,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3871}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'6222","reported_seq":10945,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-05T16:38:05.303042+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:38:05.303042+0000","last_peered":"2024-09-05T16:38:05.303042+0000","last_clean":"2024-09-05T16:38:05.303042+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:38:05.303042+0000","last_undegraded":"2024-09-05T16:38:05.303042+0000","last_fullsized":"2024-09-05T16:38:05.303042+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":6222,"log_dups_size":0,"ondisk_log_size":6222,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50921472,"num_objects":6216,"num_object_clones":0,"num_object_copies":24864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":522,"num_write_kb":4176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'6029","reported_seq":6087,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:05.673226+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:38:05.673226+0000","last_peered":"2024-09-05T16:38:05.673226+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:38:05.673226+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6029,"log_dups_size":0,"ondisk_log_size":6029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49362080,"num_objects":6026,"num_object_clones":0,"num_object_copies":24104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6026,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6027,"num_write_kb":48213,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3864}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'6102","reported_seq":6148,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:05.692487+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:38:05.692487+0000","last_peered":"2024-09-05T16:38:05.692487+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:38:05.692487+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6102,"log_dups_size":0,"ondisk_log_size":6102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49987584,"num_objects":6102,"num_object_clones":0,"num_object_copies":24408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6102,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6102,"num_write_kb":48816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3852}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'6048","reported_seq":6117,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:06.225509+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:38:06.225509+0000","last_peered":"2024-09-05T16:38:06.225509+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:38:06.225509+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6048,"log_dups_size":0,"ondisk_log_size":6048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49493152,"num_objects":6042,"num_object_clones":0,"num_object_copies":24168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6042,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6046,"num_write_kb":48348,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3895}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'6113","reported_seq":6180,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:06.170980+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:38:06.170980+0000","last_peered":"2024-09-05T16:38:06.170980+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:38:06.170980+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6113,"log_dups_size":0,"ondisk_log_size":6113,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50033824,"num_objects":6108,"num_object_clones":0,"num_object_copies":24432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6108,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6111,"num_write_kb":48876,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3869}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'6159","reported_seq":6219,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:05.623161+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:38:05.623161+0000","last_peered":"2024-09-05T16:38:05.623161+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:38:05.623161+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6159,"log_dups_size":0,"ondisk_log_size":6159,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50383168,"num_objects":6151,"num_object_clones":0,"num_object_copies":24604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6151,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6156,"num_write_kb":49225,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3890}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'6171","reported_seq":6253,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:06.114984+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:38:06.114984+0000","last_peered":"2024-09-05T16:38:06.114984+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:38:06.114984+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6171,"log_dups_size":0,"ondisk_log_size":6171,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50533536,"num_objects":6169,"num_object_clones":0,"num_object_copies":24676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6169,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6169,"num_write_kb":49350,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3907}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'6273","reported_seq":6333,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:05.673432+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:38:05.673432+0000","last_peered":"2024-09-05T16:38:05.673432+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:38:05.673432+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6273,"log_dups_size":0,"ondisk_log_size":6273,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51343088,"num_objects":6268,"num_object_clones":0,"num_object_copies":25072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6268,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6270,"num_write_kb":50148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3957}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'6097","reported_seq":6137,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:05.756908+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:38:05.756908+0000","last_peered":"2024-09-05T16:38:05.756908+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:38:05.756908+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6097,"log_dups_size":0,"ondisk_log_size":6097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49936976,"num_objects":6096,"num_object_clones":0,"num_object_copies":24384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6096,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6096,"num_write_kb":48767,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3832}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'6134","reported_seq":6971,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:06.313306+0000","last_change":"2024-09-05T16:37:30.634570+0000","last_active":"2024-09-05T16:38:06.313306+0000","last_peered":"2024-09-05T16:38:06.313306+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:38:06.313306+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6134,"log_dups_size":0,"ondisk_log_size":6134,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50240080,"num_objects":6133,"num_object_clones":0,"num_object_copies":24532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1915,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6133,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6133,"num_write_kb":49063,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":380,"num_bytes_recovered":3104768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4218}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'6275","reported_seq":6331,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:05.675699+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:38:05.675699+0000","last_peered":"2024-09-05T16:38:05.675699+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:38:05.675699+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6275,"log_dups_size":0,"ondisk_log_size":6275,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51369120,"num_objects":6271,"num_object_clones":0,"num_object_copies":25084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6271,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6273,"num_write_kb":50180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3914}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'6159","reported_seq":6241,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:06.234958+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:38:06.234958+0000","last_peered":"2024-09-05T16:38:06.234958+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:38:06.234958+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6159,"log_dups_size":0,"ondisk_log_size":6159,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50427040,"num_objects":6156,"num_object_clones":0,"num_object_copies":24624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6156,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6157,"num_write_kb":49253,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3836}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'6127","reported_seq":6195,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:06.191740+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:38:06.191740+0000","last_peered":"2024-09-05T16:38:06.191740+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:38:06.191740+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6127,"log_dups_size":0,"ondisk_log_size":6127,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50148512,"num_objects":6122,"num_object_clones":0,"num_object_copies":24488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6122,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6125,"num_write_kb":48988,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3885}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":804686880,"num_objects":98233,"num_object_clones":0,"num_object_copies":392932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33528,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98233,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":92564,"num_write_kb":740405,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":404,"num_bytes_recovered":3104768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1471447040,"data_stored":1471447040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":98289,"ondisk_log_size":98289,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084338,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":745088,"kb_used_data":392688,"kb_used_omap":4,"kb_used_meta":352379,"kb_avail":93626752,"statfs":{"total":96636764160,"available":95873794048,"internally_reserved":0,"allocated":402112512,"data_stored":401920471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":360836819},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[11,0,0,0,8,6,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986215,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":492272,"kb_used_data":257888,"kb_used_omap":4,"kb_used_meta":234363,"kb_avail":93879568,"statfs":{"total":96636764160,"available":96132677632,"internally_reserved":0,"allocated":264077312,"data_stored":263885271,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":239988434},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,1,3,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673021,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":749428,"kb_used_data":393380,"kb_used_omap":1,"kb_used_meta":356030,"kb_avail":93622412,"statfs":{"total":96636764160,"available":95869349888,"internally_reserved":0,"allocated":402821120,"data_stored":402629079,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":364575173},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1,7,8,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182452,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":749612,"kb_used_data":393116,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93622228,"statfs":{"total":96636764160,"available":95869161472,"internally_reserved":0,"allocated":402550784,"data_stored":402358743,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":365031117},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,5,3,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":402513920,"data_stored":402513920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":264327168,"data_stored":264327168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":402632704,"data_stored":402632704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":401973248,"data_stored":401973248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:38:09.538 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 227 20 99058 99038 3.40778 1.66406 0.0592548 0.0458313 2024-09-05T16:38:10.538 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 228 20 99301 99281 3.40116 1.89844 0.0744814 0.045927 2024-09-05T16:38:11.538 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 229 20 99573 99553 3.39559 2.125 0.0618941 0.0460029 2024-09-05T16:38:12.479 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:38:12.538 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 230 20 99853 99833 3.39034 2.1875 0.049391 0.0460737 2024-09-05T16:38:13.056 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:38:13.056 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:38:13.084 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":174,"stamp":"2024-09-05T16:38:12.877890+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":814795808,"num_objects":99467,"num_object_clones":0,"num_object_copies":397868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33486,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99467,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":93798,"num_write_kb":750277,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":446,"num_bytes_recovered":3448832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99523,"ondisk_log_size":99523,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2772888,"kb_used_data":1456920,"kb_used_omap":14,"kb_used_meta":1315889,"kb_avail":374714472,"statfs":{"total":386547056640,"available":383707619328,"internally_reserved":0,"allocated":1491886080,"data_stored":1491117916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1347470903},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,7,8,6,11,10,6,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":38,"apply_latency_ms":38,"commit_latency_ns":38000000,"apply_latency_ns":38000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":26738688,"num_objects":3264,"num_object_clones":0,"num_object_copies":13056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3264,"num_write_kb":26112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":147,"num_bytes_recovered":1204224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004490"},"pg_stats":[{"pgid":"1.f","version":"35'6288","reported_seq":6333,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:10.693284+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:38:10.693284+0000","last_peered":"2024-09-05T16:38:10.693284+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:38:10.693284+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6288,"log_dups_size":0,"ondisk_log_size":6288,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51511296,"num_objects":6288,"num_object_clones":0,"num_object_copies":25152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6288,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6288,"num_write_kb":50304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4026}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'6099","reported_seq":6167,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:11.225709+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:38:11.225709+0000","last_peered":"2024-09-05T16:38:11.225709+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:38:11.225709+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6099,"log_dups_size":0,"ondisk_log_size":6099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49927328,"num_objects":6095,"num_object_clones":0,"num_object_copies":24380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6095,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6097,"num_write_kb":48772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3828}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'6209","reported_seq":6288,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:11.161657+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:38:11.161657+0000","last_peered":"2024-09-05T16:38:11.161657+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:38:11.161657+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6209,"log_dups_size":0,"ondisk_log_size":6209,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50838096,"num_objects":6206,"num_object_clones":0,"num_object_copies":24824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6206,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6208,"num_write_kb":49654,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3956}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'6291","reported_seq":11014,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-05T16:38:10.508711+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:38:10.508711+0000","last_peered":"2024-09-05T16:38:10.508711+0000","last_clean":"2024-09-05T16:38:10.508711+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:38:10.508711+0000","last_undegraded":"2024-09-05T16:38:10.508711+0000","last_fullsized":"2024-09-05T16:38:10.508711+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":6291,"log_dups_size":0,"ondisk_log_size":6291,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51486720,"num_objects":6285,"num_object_clones":0,"num_object_copies":25140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6285,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":591,"num_write_kb":4728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'6113","reported_seq":6171,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:10.624812+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:38:10.624812+0000","last_peered":"2024-09-05T16:38:10.624812+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:38:10.624812+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6113,"log_dups_size":0,"ondisk_log_size":6113,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50050208,"num_objects":6110,"num_object_clones":0,"num_object_copies":24440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6110,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6111,"num_write_kb":48885,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3948}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'6182","reported_seq":6227,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:10.671813+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:38:10.671813+0000","last_peered":"2024-09-05T16:38:10.671813+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:38:10.671813+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6182,"log_dups_size":0,"ondisk_log_size":6182,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50642944,"num_objects":6182,"num_object_clones":0,"num_object_copies":24728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6182,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6182,"num_write_kb":49456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3932}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'6161","reported_seq":6223,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:11.254629+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:38:11.254629+0000","last_peered":"2024-09-05T16:38:11.254629+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:38:11.254629+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6161,"log_dups_size":0,"ondisk_log_size":6161,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50418848,"num_objects":6155,"num_object_clones":0,"num_object_copies":24620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6155,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6159,"num_write_kb":49252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":4008}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'6203","reported_seq":6271,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:11.278413+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:38:11.278413+0000","last_peered":"2024-09-05T16:38:11.278413+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:38:11.278413+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6203,"log_dups_size":0,"ondisk_log_size":6203,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50771104,"num_objects":6198,"num_object_clones":0,"num_object_copies":24792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6198,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6201,"num_write_kb":49596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3959}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'6222","reported_seq":6281,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:10.735151+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:38:10.735151+0000","last_peered":"2024-09-05T16:38:10.735151+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:38:10.735151+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6222,"log_dups_size":0,"ondisk_log_size":6222,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50899264,"num_objects":6214,"num_object_clones":0,"num_object_copies":24856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6214,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6219,"num_write_kb":49729,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3953}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'6237","reported_seq":6319,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:11.104078+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:38:11.104078+0000","last_peered":"2024-09-05T16:38:11.104078+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:38:11.104078+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6237,"log_dups_size":0,"ondisk_log_size":6237,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51074208,"num_objects":6235,"num_object_clones":0,"num_object_copies":24940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6235,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6235,"num_write_kb":49878,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3973}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'6352","reported_seq":6411,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:10.778666+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:38:10.778666+0000","last_peered":"2024-09-05T16:38:10.778666+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:38:10.778666+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6352,"log_dups_size":0,"ondisk_log_size":6352,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51990256,"num_objects":6347,"num_object_clones":0,"num_object_copies":25388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6347,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6349,"num_write_kb":50780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4036}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'6154","reported_seq":6194,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:10.740242+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:38:10.740242+0000","last_peered":"2024-09-05T16:38:10.740242+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:38:10.740242+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6154,"log_dups_size":0,"ondisk_log_size":6154,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50403920,"num_objects":6153,"num_object_clones":0,"num_object_copies":24612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6153,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6153,"num_write_kb":49223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3889}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'6217","reported_seq":7138,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:11.326864+0000","last_change":"2024-09-05T16:37:30.634570+0000","last_active":"2024-09-05T16:38:11.326864+0000","last_peered":"2024-09-05T16:38:11.326864+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:38:11.326864+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6217,"log_dups_size":0,"ondisk_log_size":6217,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50920016,"num_objects":6216,"num_object_clones":0,"num_object_copies":24864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1873,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6216,"num_write_kb":49727,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":422,"num_bytes_recovered":3448832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4343}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'6355","reported_seq":6411,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:10.723354+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:38:10.723354+0000","last_peered":"2024-09-05T16:38:10.723354+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:38:10.723354+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6355,"log_dups_size":0,"ondisk_log_size":6355,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52024480,"num_objects":6351,"num_object_clones":0,"num_object_copies":25404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6351,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6353,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3994}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'6235","reported_seq":6317,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:11.188629+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:38:11.188629+0000","last_peered":"2024-09-05T16:38:11.188629+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:38:11.188629+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6235,"log_dups_size":0,"ondisk_log_size":6235,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51049632,"num_objects":6232,"num_object_clones":0,"num_object_copies":24928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6232,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6233,"num_write_kb":49861,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3912}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'6205","reported_seq":6273,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:11.246732+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:38:11.246732+0000","last_peered":"2024-09-05T16:38:11.246732+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:38:11.246732+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6205,"log_dups_size":0,"ondisk_log_size":6205,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50787488,"num_objects":6200,"num_object_clones":0,"num_object_copies":24800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6200,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6203,"num_write_kb":49612,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3963}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":814795808,"num_objects":99467,"num_object_clones":0,"num_object_copies":397868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33486,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99467,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":93798,"num_write_kb":750277,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":446,"num_bytes_recovered":3448832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1491988480,"data_stored":1491988480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99523,"ondisk_log_size":99523,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084339,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":754032,"kb_used_data":397472,"kb_used_omap":4,"kb_used_meta":356539,"kb_avail":93617808,"statfs":{"total":96636764160,"available":95864635392,"internally_reserved":0,"allocated":407011328,"data_stored":406819287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":365096659},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,1,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986216,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":501644,"kb_used_data":263100,"kb_used_omap":4,"kb_used_meta":238523,"kb_avail":93870196,"statfs":{"total":96636764160,"available":96123080704,"internally_reserved":0,"allocated":269414400,"data_stored":269222359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":244248274},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,1,5,7,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673022,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":758556,"kb_used_data":398348,"kb_used_omap":1,"kb_used_meta":360190,"kb_avail":93613284,"statfs":{"total":96636764160,"available":95860002816,"internally_reserved":0,"allocated":407908352,"data_stored":407716311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":368835013},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,4,3,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182453,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":758656,"kb_used_data":398000,"kb_used_omap":4,"kb_used_meta":360635,"kb_avail":93613184,"statfs":{"total":96636764160,"available":95859900416,"internally_reserved":0,"allocated":407552000,"data_stored":407359959,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":369290957},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,7,0,0,2,1,2,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":407576576,"data_stored":407576576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":269737984,"data_stored":269737984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":407633920,"data_stored":407633920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":407040000,"data_stored":407040000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:38:13.085 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:38:13.538 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 231 20 100090 100070 3.38367 1.85156 0.0545302 0.046165 2024-09-05T16:38:13.654 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:38:13.654 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:38:13.679 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":174,"stamp":"2024-09-05T16:38:12.877890+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":814795808,"num_objects":99467,"num_object_clones":0,"num_object_copies":397868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33486,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99467,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":93798,"num_write_kb":750277,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":446,"num_bytes_recovered":3448832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99523,"ondisk_log_size":99523,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2772888,"kb_used_data":1456920,"kb_used_omap":14,"kb_used_meta":1315889,"kb_avail":374714472,"statfs":{"total":386547056640,"available":383707619328,"internally_reserved":0,"allocated":1491886080,"data_stored":1491117916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1347470903},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,7,8,6,11,10,6,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":38,"apply_latency_ms":38,"commit_latency_ns":38000000,"apply_latency_ns":38000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":26738688,"num_objects":3264,"num_object_clones":0,"num_object_copies":13056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3264,"num_write_kb":26112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":147,"num_bytes_recovered":1204224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004490"},"pg_stats":[{"pgid":"1.f","version":"35'6288","reported_seq":6333,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:10.693284+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:38:10.693284+0000","last_peered":"2024-09-05T16:38:10.693284+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:38:10.693284+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6288,"log_dups_size":0,"ondisk_log_size":6288,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51511296,"num_objects":6288,"num_object_clones":0,"num_object_copies":25152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6288,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6288,"num_write_kb":50304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4026}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'6099","reported_seq":6167,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:11.225709+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:38:11.225709+0000","last_peered":"2024-09-05T16:38:11.225709+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:38:11.225709+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6099,"log_dups_size":0,"ondisk_log_size":6099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49927328,"num_objects":6095,"num_object_clones":0,"num_object_copies":24380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6095,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6097,"num_write_kb":48772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3828}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'6209","reported_seq":6288,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:11.161657+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:38:11.161657+0000","last_peered":"2024-09-05T16:38:11.161657+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:38:11.161657+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6209,"log_dups_size":0,"ondisk_log_size":6209,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50838096,"num_objects":6206,"num_object_clones":0,"num_object_copies":24824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6206,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6208,"num_write_kb":49654,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3956}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'6291","reported_seq":11014,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-05T16:38:10.508711+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:38:10.508711+0000","last_peered":"2024-09-05T16:38:10.508711+0000","last_clean":"2024-09-05T16:38:10.508711+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:38:10.508711+0000","last_undegraded":"2024-09-05T16:38:10.508711+0000","last_fullsized":"2024-09-05T16:38:10.508711+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":6291,"log_dups_size":0,"ondisk_log_size":6291,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51486720,"num_objects":6285,"num_object_clones":0,"num_object_copies":25140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6285,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":591,"num_write_kb":4728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'6113","reported_seq":6171,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:10.624812+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:38:10.624812+0000","last_peered":"2024-09-05T16:38:10.624812+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:38:10.624812+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6113,"log_dups_size":0,"ondisk_log_size":6113,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50050208,"num_objects":6110,"num_object_clones":0,"num_object_copies":24440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6110,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6111,"num_write_kb":48885,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3948}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'6182","reported_seq":6227,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:10.671813+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:38:10.671813+0000","last_peered":"2024-09-05T16:38:10.671813+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:38:10.671813+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6182,"log_dups_size":0,"ondisk_log_size":6182,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50642944,"num_objects":6182,"num_object_clones":0,"num_object_copies":24728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6182,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6182,"num_write_kb":49456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3932}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'6161","reported_seq":6223,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:11.254629+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:38:11.254629+0000","last_peered":"2024-09-05T16:38:11.254629+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:38:11.254629+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6161,"log_dups_size":0,"ondisk_log_size":6161,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50418848,"num_objects":6155,"num_object_clones":0,"num_object_copies":24620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6155,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6159,"num_write_kb":49252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":4008}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'6203","reported_seq":6271,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:11.278413+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:38:11.278413+0000","last_peered":"2024-09-05T16:38:11.278413+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:38:11.278413+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6203,"log_dups_size":0,"ondisk_log_size":6203,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50771104,"num_objects":6198,"num_object_clones":0,"num_object_copies":24792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6198,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6201,"num_write_kb":49596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3959}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'6222","reported_seq":6281,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:10.735151+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:38:10.735151+0000","last_peered":"2024-09-05T16:38:10.735151+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:38:10.735151+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6222,"log_dups_size":0,"ondisk_log_size":6222,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50899264,"num_objects":6214,"num_object_clones":0,"num_object_copies":24856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6214,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6219,"num_write_kb":49729,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3953}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'6237","reported_seq":6319,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:11.104078+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:38:11.104078+0000","last_peered":"2024-09-05T16:38:11.104078+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:38:11.104078+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6237,"log_dups_size":0,"ondisk_log_size":6237,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51074208,"num_objects":6235,"num_object_clones":0,"num_object_copies":24940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6235,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6235,"num_write_kb":49878,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3973}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'6352","reported_seq":6411,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:10.778666+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:38:10.778666+0000","last_peered":"2024-09-05T16:38:10.778666+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:38:10.778666+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6352,"log_dups_size":0,"ondisk_log_size":6352,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51990256,"num_objects":6347,"num_object_clones":0,"num_object_copies":25388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6347,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6349,"num_write_kb":50780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4036}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'6154","reported_seq":6194,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:10.740242+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:38:10.740242+0000","last_peered":"2024-09-05T16:38:10.740242+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:38:10.740242+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6154,"log_dups_size":0,"ondisk_log_size":6154,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50403920,"num_objects":6153,"num_object_clones":0,"num_object_copies":24612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6153,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6153,"num_write_kb":49223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3889}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'6217","reported_seq":7138,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:11.326864+0000","last_change":"2024-09-05T16:37:30.634570+0000","last_active":"2024-09-05T16:38:11.326864+0000","last_peered":"2024-09-05T16:38:11.326864+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:38:11.326864+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6217,"log_dups_size":0,"ondisk_log_size":6217,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50920016,"num_objects":6216,"num_object_clones":0,"num_object_copies":24864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1873,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6216,"num_write_kb":49727,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":422,"num_bytes_recovered":3448832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4343}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'6355","reported_seq":6411,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:10.723354+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:38:10.723354+0000","last_peered":"2024-09-05T16:38:10.723354+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:38:10.723354+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6355,"log_dups_size":0,"ondisk_log_size":6355,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52024480,"num_objects":6351,"num_object_clones":0,"num_object_copies":25404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6351,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6353,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3994}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'6235","reported_seq":6317,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:11.188629+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:38:11.188629+0000","last_peered":"2024-09-05T16:38:11.188629+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:38:11.188629+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6235,"log_dups_size":0,"ondisk_log_size":6235,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51049632,"num_objects":6232,"num_object_clones":0,"num_object_copies":24928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6232,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6233,"num_write_kb":49861,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3912}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'6205","reported_seq":6273,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:11.246732+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:38:11.246732+0000","last_peered":"2024-09-05T16:38:11.246732+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:38:11.246732+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6205,"log_dups_size":0,"ondisk_log_size":6205,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50787488,"num_objects":6200,"num_object_clones":0,"num_object_copies":24800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6200,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6203,"num_write_kb":49612,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3963}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":814795808,"num_objects":99467,"num_object_clones":0,"num_object_copies":397868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33486,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99467,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":93798,"num_write_kb":750277,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":446,"num_bytes_recovered":3448832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1491988480,"data_stored":1491988480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99523,"ondisk_log_size":99523,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084339,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":754032,"kb_used_data":397472,"kb_used_omap":4,"kb_used_meta":356539,"kb_avail":93617808,"statfs":{"total":96636764160,"available":95864635392,"internally_reserved":0,"allocated":407011328,"data_stored":406819287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":365096659},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,1,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986216,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":501644,"kb_used_data":263100,"kb_used_omap":4,"kb_used_meta":238523,"kb_avail":93870196,"statfs":{"total":96636764160,"available":96123080704,"internally_reserved":0,"allocated":269414400,"data_stored":269222359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":244248274},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,1,5,7,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673022,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":758556,"kb_used_data":398348,"kb_used_omap":1,"kb_used_meta":360190,"kb_avail":93613284,"statfs":{"total":96636764160,"available":95860002816,"internally_reserved":0,"allocated":407908352,"data_stored":407716311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":368835013},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,4,3,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182453,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":758656,"kb_used_data":398000,"kb_used_omap":4,"kb_used_meta":360635,"kb_avail":93613184,"statfs":{"total":96636764160,"available":95859900416,"internally_reserved":0,"allocated":407552000,"data_stored":407359959,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":369290957},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,7,0,0,2,1,2,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":407576576,"data_stored":407576576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":269737984,"data_stored":269737984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":407633920,"data_stored":407633920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":407040000,"data_stored":407040000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:38:13.680 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-05T16:38:13.680 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'6288", 'reported_seq': 6333, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:10.693284+0000', 'last_change': '2024-09-05T16:34:56.833497+0000', 'last_active': '2024-09-05T16:38:10.693284+0000', 'last_peered': '2024-09-05T16:38:10.693284+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.828289+0000', 'last_became_peered': '2024-09-05T16:34:56.828289+0000', 'last_unstale': '2024-09-05T16:38:10.693284+0000', 'last_undegraded': '2024-09-05T16:34:56.612160+0000', 'last_fullsized': '2024-09-05T16:34:56.557632+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6288, 'log_dups_size': 0, 'ondisk_log_size': 6288, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51511296, 'num_objects': 6288, 'num_object_clones': 0, 'num_object_copies': 25152, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2262, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6288, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6288, 'num_write_kb': 50304, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 4026}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:38:13.681 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-05T16:38:13.681 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "35'6099", 'reported_seq': 6167, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:11.225709+0000', 'last_change': '2024-09-05T16:34:56.825031+0000', 'last_active': '2024-09-05T16:38:11.225709+0000', 'last_peered': '2024-09-05T16:38:11.225709+0000', 'last_clean': '2024-09-05T16:33:53.200762+0000', 'last_became_active': '2024-09-05T16:34:56.820292+0000', 'last_became_peered': '2024-09-05T16:34:56.820292+0000', 'last_unstale': '2024-09-05T16:38:11.225709+0000', 'last_undegraded': '2024-09-05T16:34:56.605776+0000', 'last_fullsized': '2024-09-05T16:34:56.559383+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6099, 'log_dups_size': 0, 'ondisk_log_size': 6099, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49927328, 'num_objects': 6095, 'num_object_clones': 0, 'num_object_copies': 24380, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2269, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6095, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6097, 'num_write_kb': 48772, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3828}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:38:13.681 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:38:13.681 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'6209", 'reported_seq': 6288, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:11.161657+0000', 'last_change': '2024-09-05T16:34:56.867900+0000', 'last_active': '2024-09-05T16:38:11.161657+0000', 'last_peered': '2024-09-05T16:38:11.161657+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:38:11.161657+0000', 'last_undegraded': '2024-09-05T16:34:56.590972+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6209, 'log_dups_size': 0, 'ondisk_log_size': 6209, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50838096, 'num_objects': 6206, 'num_object_clones': 0, 'num_object_copies': 24824, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2251, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6206, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6208, 'num_write_kb': 49654, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3956}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:38:13.681 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:38:13.681 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'6113", 'reported_seq': 6171, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:10.624812+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:38:10.624812+0000', 'last_peered': '2024-09-05T16:38:10.624812+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:38:10.624812+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6113, 'log_dups_size': 0, 'ondisk_log_size': 6113, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50050208, 'num_objects': 6110, 'num_object_clones': 0, 'num_object_copies': 24440, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2164, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6110, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6111, 'num_write_kb': 48885, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3948}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:38:13.681 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:38:13.681 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'6182", 'reported_seq': 6227, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:10.671813+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:38:10.671813+0000', 'last_peered': '2024-09-05T16:38:10.671813+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:38:10.671813+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6182, 'log_dups_size': 0, 'ondisk_log_size': 6182, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50642944, 'num_objects': 6182, 'num_object_clones': 0, 'num_object_copies': 24728, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6182, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6182, 'num_write_kb': 49456, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3932}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:38:13.681 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:38:13.681 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'6161", 'reported_seq': 6223, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:11.254629+0000', 'last_change': '2024-09-05T16:34:56.903436+0000', 'last_active': '2024-09-05T16:38:11.254629+0000', 'last_peered': '2024-09-05T16:38:11.254629+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:38:11.254629+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6161, 'log_dups_size': 0, 'ondisk_log_size': 6161, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50418848, 'num_objects': 6155, 'num_object_clones': 0, 'num_object_copies': 24620, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2149, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6155, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6159, 'num_write_kb': 49252, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 4008}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:38:13.682 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-05T16:38:13.682 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "35'6203", 'reported_seq': 6271, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:11.278413+0000', 'last_change': '2024-09-05T16:34:56.883319+0000', 'last_active': '2024-09-05T16:38:11.278413+0000', 'last_peered': '2024-09-05T16:38:11.278413+0000', 'last_clean': '2024-09-05T16:33:53.505305+0000', 'last_became_active': '2024-09-05T16:34:56.873389+0000', 'last_became_peered': '2024-09-05T16:34:56.873389+0000', 'last_unstale': '2024-09-05T16:38:11.278413+0000', 'last_undegraded': '2024-09-05T16:34:56.602567+0000', 'last_fullsized': '2024-09-05T16:34:56.559958+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6203, 'log_dups_size': 0, 'ondisk_log_size': 6203, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50771104, 'num_objects': 6198, 'num_object_clones': 0, 'num_object_copies': 24792, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2241, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6198, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6201, 'num_write_kb': 49596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3959}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:38:13.682 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:38:13.682 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'6222", 'reported_seq': 6281, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:10.735151+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:38:10.735151+0000', 'last_peered': '2024-09-05T16:38:10.735151+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:38:10.735151+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6222, 'log_dups_size': 0, 'ondisk_log_size': 6222, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50899264, 'num_objects': 6214, 'num_object_clones': 0, 'num_object_copies': 24856, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2264, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6214, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6219, 'num_write_kb': 49729, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3953}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:38:13.682 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:38:13.682 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'6237", 'reported_seq': 6319, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:11.104078+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:38:11.104078+0000', 'last_peered': '2024-09-05T16:38:11.104078+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:38:11.104078+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6237, 'log_dups_size': 0, 'ondisk_log_size': 6237, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51074208, 'num_objects': 6235, 'num_object_clones': 0, 'num_object_copies': 24940, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2264, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6235, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6235, 'num_write_kb': 49878, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3973}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:38:13.682 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:38:13.682 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'6352", 'reported_seq': 6411, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:10.778666+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:38:10.778666+0000', 'last_peered': '2024-09-05T16:38:10.778666+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:38:10.778666+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6352, 'log_dups_size': 0, 'ondisk_log_size': 6352, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51990256, 'num_objects': 6347, 'num_object_clones': 0, 'num_object_copies': 25388, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2314, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6347, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6349, 'num_write_kb': 50780, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 4036}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:38:13.682 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:38:13.682 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'6154", 'reported_seq': 6194, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:10.740242+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:38:10.740242+0000', 'last_peered': '2024-09-05T16:38:10.740242+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:38:10.740242+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6154, 'log_dups_size': 0, 'ondisk_log_size': 6154, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50403920, 'num_objects': 6153, 'num_object_clones': 0, 'num_object_copies': 24612, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2265, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6153, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6153, 'num_write_kb': 49223, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3889}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:38:13.683 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-05T16:38:13.683 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'6217", 'reported_seq': 7138, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:11.326864+0000', 'last_change': '2024-09-05T16:37:30.634570+0000', 'last_active': '2024-09-05T16:38:11.326864+0000', 'last_peered': '2024-09-05T16:38:11.326864+0000', 'last_clean': '2024-09-05T16:33:45.082039+0000', 'last_became_active': '2024-09-05T16:34:56.792386+0000', 'last_became_peered': '2024-09-05T16:34:56.792386+0000', 'last_unstale': '2024-09-05T16:38:11.326864+0000', 'last_undegraded': '2024-09-05T16:34:56.600595+0000', 'last_fullsized': '2024-09-05T16:34:56.558203+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6217, 'log_dups_size': 0, 'ondisk_log_size': 6217, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50920016, 'num_objects': 6216, 'num_object_clones': 0, 'num_object_copies': 24864, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1873, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6216, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6216, 'num_write_kb': 49727, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 422, 'num_bytes_recovered': 3448832, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 4343}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:38:13.683 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:38:13.683 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'6355", 'reported_seq': 6411, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:10.723354+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:38:10.723354+0000', 'last_peered': '2024-09-05T16:38:10.723354+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:38:10.723354+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6355, 'log_dups_size': 0, 'ondisk_log_size': 6355, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 52024480, 'num_objects': 6351, 'num_object_clones': 0, 'num_object_copies': 25404, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2359, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6351, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6353, 'num_write_kb': 50820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3994}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:38:13.683 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:38:13.683 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'6235", 'reported_seq': 6317, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:11.188629+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:38:11.188629+0000', 'last_peered': '2024-09-05T16:38:11.188629+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:38:11.188629+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6235, 'log_dups_size': 0, 'ondisk_log_size': 6235, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51049632, 'num_objects': 6232, 'num_object_clones': 0, 'num_object_copies': 24928, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2322, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6232, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6233, 'num_write_kb': 49861, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3912}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:38:13.683 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:38:13.683 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'6205", 'reported_seq': 6273, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:11.246732+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:38:11.246732+0000', 'last_peered': '2024-09-05T16:38:11.246732+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:38:11.246732+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6205, 'log_dups_size': 0, 'ondisk_log_size': 6205, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50787488, 'num_objects': 6200, 'num_object_clones': 0, 'num_object_copies': 24800, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6200, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6203, 'num_write_kb': 49612, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3963}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:38:13.684 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:38:14.397 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:38:14.397 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 33486/397868 objects degraded (8.416%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":297,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":99467,"data_bytes":814795808,"bytes_used":2839437312,"bytes_avail":383707619328,"bytes_total":386547056640,"degraded_objects":33486,"degraded_total":397868,"degraded_ratio":0.084163591945067215,"recovering_objects_per_sec":12,"recovering_bytes_per_sec":100314,"recovering_keys_per_sec":0,"num_objects_recovered":147,"num_bytes_recovered":1204224,"num_keys_recovered":0,"write_bytes_sec":2227390,"read_op_per_sec":0,"write_op_per_sec":271},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:38:14.426 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:38:14.426 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:38:14.538 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 232 20 100327 100307 3.37707 1.85156 0.0738016 0.0462533 2024-09-05T16:38:15.057 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:38:15.057 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:38:15.089 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":175,"stamp":"2024-09-05T16:38:14.878576+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":814795808,"num_objects":99467,"num_object_clones":0,"num_object_copies":397868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33486,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99467,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":93798,"num_write_kb":750277,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":446,"num_bytes_recovered":3448832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99523,"ondisk_log_size":99523,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2772888,"kb_used_data":1456920,"kb_used_omap":14,"kb_used_meta":1315889,"kb_avail":374714472,"statfs":{"total":386547056640,"available":383707619328,"internally_reserved":0,"allocated":1491886080,"data_stored":1491117916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1347470903},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,7,8,6,11,10,6,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":38,"apply_latency_ms":38,"commit_latency_ns":38000000,"apply_latency_ns":38000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":21053440,"num_objects":2570,"num_object_clones":0,"num_object_copies":10280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-94,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2570,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":770048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004478"},"pg_stats":[{"pgid":"1.f","version":"35'6288","reported_seq":6333,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:10.693284+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:38:10.693284+0000","last_peered":"2024-09-05T16:38:10.693284+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:38:10.693284+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6288,"log_dups_size":0,"ondisk_log_size":6288,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51511296,"num_objects":6288,"num_object_clones":0,"num_object_copies":25152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6288,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6288,"num_write_kb":50304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4026}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'6099","reported_seq":6167,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:11.225709+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:38:11.225709+0000","last_peered":"2024-09-05T16:38:11.225709+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:38:11.225709+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6099,"log_dups_size":0,"ondisk_log_size":6099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49927328,"num_objects":6095,"num_object_clones":0,"num_object_copies":24380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6095,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6097,"num_write_kb":48772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3828}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'6209","reported_seq":6288,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:11.161657+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:38:11.161657+0000","last_peered":"2024-09-05T16:38:11.161657+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:38:11.161657+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6209,"log_dups_size":0,"ondisk_log_size":6209,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50838096,"num_objects":6206,"num_object_clones":0,"num_object_copies":24824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6206,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6208,"num_write_kb":49654,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3956}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'6291","reported_seq":11014,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-05T16:38:10.508711+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:38:10.508711+0000","last_peered":"2024-09-05T16:38:10.508711+0000","last_clean":"2024-09-05T16:38:10.508711+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:38:10.508711+0000","last_undegraded":"2024-09-05T16:38:10.508711+0000","last_fullsized":"2024-09-05T16:38:10.508711+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":6291,"log_dups_size":0,"ondisk_log_size":6291,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51486720,"num_objects":6285,"num_object_clones":0,"num_object_copies":25140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6285,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":591,"num_write_kb":4728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'6113","reported_seq":6171,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:10.624812+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:38:10.624812+0000","last_peered":"2024-09-05T16:38:10.624812+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:38:10.624812+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6113,"log_dups_size":0,"ondisk_log_size":6113,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50050208,"num_objects":6110,"num_object_clones":0,"num_object_copies":24440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6110,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6111,"num_write_kb":48885,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3948}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'6182","reported_seq":6227,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:10.671813+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:38:10.671813+0000","last_peered":"2024-09-05T16:38:10.671813+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:38:10.671813+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6182,"log_dups_size":0,"ondisk_log_size":6182,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50642944,"num_objects":6182,"num_object_clones":0,"num_object_copies":24728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6182,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6182,"num_write_kb":49456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3932}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'6161","reported_seq":6223,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:11.254629+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:38:11.254629+0000","last_peered":"2024-09-05T16:38:11.254629+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:38:11.254629+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6161,"log_dups_size":0,"ondisk_log_size":6161,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50418848,"num_objects":6155,"num_object_clones":0,"num_object_copies":24620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6155,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6159,"num_write_kb":49252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":4008}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'6203","reported_seq":6271,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:11.278413+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:38:11.278413+0000","last_peered":"2024-09-05T16:38:11.278413+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:38:11.278413+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6203,"log_dups_size":0,"ondisk_log_size":6203,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50771104,"num_objects":6198,"num_object_clones":0,"num_object_copies":24792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6198,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6201,"num_write_kb":49596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3959}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'6222","reported_seq":6281,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:10.735151+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:38:10.735151+0000","last_peered":"2024-09-05T16:38:10.735151+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:38:10.735151+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6222,"log_dups_size":0,"ondisk_log_size":6222,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50899264,"num_objects":6214,"num_object_clones":0,"num_object_copies":24856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6214,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6219,"num_write_kb":49729,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3953}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'6237","reported_seq":6319,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:11.104078+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:38:11.104078+0000","last_peered":"2024-09-05T16:38:11.104078+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:38:11.104078+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6237,"log_dups_size":0,"ondisk_log_size":6237,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51074208,"num_objects":6235,"num_object_clones":0,"num_object_copies":24940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6235,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6235,"num_write_kb":49878,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3973}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'6352","reported_seq":6411,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:10.778666+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:38:10.778666+0000","last_peered":"2024-09-05T16:38:10.778666+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:38:10.778666+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6352,"log_dups_size":0,"ondisk_log_size":6352,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51990256,"num_objects":6347,"num_object_clones":0,"num_object_copies":25388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6347,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6349,"num_write_kb":50780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4036}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'6154","reported_seq":6194,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:10.740242+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:38:10.740242+0000","last_peered":"2024-09-05T16:38:10.740242+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:38:10.740242+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6154,"log_dups_size":0,"ondisk_log_size":6154,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50403920,"num_objects":6153,"num_object_clones":0,"num_object_copies":24612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6153,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6153,"num_write_kb":49223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3889}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'6217","reported_seq":7138,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:11.326864+0000","last_change":"2024-09-05T16:37:30.634570+0000","last_active":"2024-09-05T16:38:11.326864+0000","last_peered":"2024-09-05T16:38:11.326864+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:38:11.326864+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6217,"log_dups_size":0,"ondisk_log_size":6217,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50920016,"num_objects":6216,"num_object_clones":0,"num_object_copies":24864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1873,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6216,"num_write_kb":49727,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":422,"num_bytes_recovered":3448832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4343}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'6355","reported_seq":6411,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:10.723354+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:38:10.723354+0000","last_peered":"2024-09-05T16:38:10.723354+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:38:10.723354+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6355,"log_dups_size":0,"ondisk_log_size":6355,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52024480,"num_objects":6351,"num_object_clones":0,"num_object_copies":25404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6351,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6353,"num_write_kb":50820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3994}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'6235","reported_seq":6317,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:11.188629+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:38:11.188629+0000","last_peered":"2024-09-05T16:38:11.188629+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:38:11.188629+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6235,"log_dups_size":0,"ondisk_log_size":6235,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51049632,"num_objects":6232,"num_object_clones":0,"num_object_copies":24928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6232,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6233,"num_write_kb":49861,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3912}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'6205","reported_seq":6273,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:11.246732+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:38:11.246732+0000","last_peered":"2024-09-05T16:38:11.246732+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:38:11.246732+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6205,"log_dups_size":0,"ondisk_log_size":6205,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50787488,"num_objects":6200,"num_object_clones":0,"num_object_copies":24800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6200,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6203,"num_write_kb":49612,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3963}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":814795808,"num_objects":99467,"num_object_clones":0,"num_object_copies":397868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33486,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99467,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":93798,"num_write_kb":750277,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":446,"num_bytes_recovered":3448832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1491988480,"data_stored":1491988480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99523,"ondisk_log_size":99523,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084339,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":754032,"kb_used_data":397472,"kb_used_omap":4,"kb_used_meta":356539,"kb_avail":93617808,"statfs":{"total":96636764160,"available":95864635392,"internally_reserved":0,"allocated":407011328,"data_stored":406819287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":365096659},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,1,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986216,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":501644,"kb_used_data":263100,"kb_used_omap":4,"kb_used_meta":238523,"kb_avail":93870196,"statfs":{"total":96636764160,"available":96123080704,"internally_reserved":0,"allocated":269414400,"data_stored":269222359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":244248274},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,1,5,7,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673022,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":758556,"kb_used_data":398348,"kb_used_omap":1,"kb_used_meta":360190,"kb_avail":93613284,"statfs":{"total":96636764160,"available":95860002816,"internally_reserved":0,"allocated":407908352,"data_stored":407716311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":368835013},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,4,3,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182453,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":758656,"kb_used_data":398000,"kb_used_omap":4,"kb_used_meta":360635,"kb_avail":93613184,"statfs":{"total":96636764160,"available":95859900416,"internally_reserved":0,"allocated":407552000,"data_stored":407359959,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":369290957},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,7,0,0,2,1,2,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":407576576,"data_stored":407576576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":269737984,"data_stored":269737984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":407633920,"data_stored":407633920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":407040000,"data_stored":407040000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:38:15.541 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 233 20 100595 100575 3.37156 2.09375 0.0537841 0.0463318 2024-09-05T16:38:16.538 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 234 20 100872 100852 3.3664 2.16406 0.0320229 0.046403 2024-09-05T16:38:17.538 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 235 20 101149 101129 3.36129 2.16406 0.0812271 0.0464734 2024-09-05T16:38:18.092 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:38:18.542 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 236 20 101416 101396 3.35583 2.08594 0.051286 0.0465451 2024-09-05T16:38:18.719 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:38:18.720 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:38:18.744 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":176,"stamp":"2024-09-05T16:38:16.879420+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":825388064,"num_objects":100760,"num_object_clones":0,"num_object_copies":403040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33434,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100760,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":95091,"num_write_kb":760621,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":498,"num_bytes_recovered":3874816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100816,"ondisk_log_size":100816,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2809356,"kb_used_data":1476748,"kb_used_omap":14,"kb_used_meta":1332529,"kb_avail":374678004,"statfs":{"total":386547056640,"available":383670276096,"internally_reserved":0,"allocated":1512189952,"data_stored":1511421788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1364510263},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,7,6,8,17,8,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":31645696,"num_objects":3863,"num_object_clones":0,"num_object_copies":15452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-146,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3863,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3863,"num_write_kb":30904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":146,"num_bytes_recovered":1196032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004605"},"pg_stats":[{"pgid":"1.f","version":"35'6346","reported_seq":6392,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:15.755541+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:38:15.755541+0000","last_peered":"2024-09-05T16:38:15.755541+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:38:15.755541+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6346,"log_dups_size":0,"ondisk_log_size":6346,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51986432,"num_objects":6346,"num_object_clones":0,"num_object_copies":25384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6346,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6346,"num_write_kb":50768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4084}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'6171","reported_seq":6239,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:16.236181+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:38:16.236181+0000","last_peered":"2024-09-05T16:38:16.236181+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:38:16.236181+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6171,"log_dups_size":0,"ondisk_log_size":6171,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50517152,"num_objects":6167,"num_object_clones":0,"num_object_copies":24668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6167,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6169,"num_write_kb":49348,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3900}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'6292","reported_seq":6372,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:16.341690+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:38:16.341690+0000","last_peered":"2024-09-05T16:38:16.341690+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:38:16.341690+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6292,"log_dups_size":0,"ondisk_log_size":6292,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51518032,"num_objects":6289,"num_object_clones":0,"num_object_copies":25156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6289,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6291,"num_write_kb":50318,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":4039}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'6369","reported_seq":11092,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-05T16:38:15.456615+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:38:15.456615+0000","last_peered":"2024-09-05T16:38:15.456615+0000","last_clean":"2024-09-05T16:38:15.456615+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:38:15.456615+0000","last_undegraded":"2024-09-05T16:38:15.456615+0000","last_fullsized":"2024-09-05T16:38:15.456615+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":6369,"log_dups_size":0,"ondisk_log_size":6369,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52125696,"num_objects":6363,"num_object_clones":0,"num_object_copies":25452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6363,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":669,"num_write_kb":5352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'6198","reported_seq":6256,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:15.538502+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:38:15.538502+0000","last_peered":"2024-09-05T16:38:15.538502+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:38:15.538502+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6198,"log_dups_size":0,"ondisk_log_size":6198,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50746528,"num_objects":6195,"num_object_clones":0,"num_object_copies":24780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6195,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6196,"num_write_kb":49565,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":4033}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'6269","reported_seq":6315,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:15.803874+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:38:15.803874+0000","last_peered":"2024-09-05T16:38:15.803874+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:38:15.803874+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6269,"log_dups_size":0,"ondisk_log_size":6269,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51355648,"num_objects":6269,"num_object_clones":0,"num_object_copies":25076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6269,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6269,"num_write_kb":50152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":4019}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'6242","reported_seq":6312,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:16.314126+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:38:16.314126+0000","last_peered":"2024-09-05T16:38:16.314126+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:38:16.314126+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6242,"log_dups_size":0,"ondisk_log_size":6242,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51082400,"num_objects":6236,"num_object_clones":0,"num_object_copies":24944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6240,"num_write_kb":49900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":4089}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'6286","reported_seq":6353,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:16.158997+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:38:16.158997+0000","last_peered":"2024-09-05T16:38:16.158997+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:38:16.158997+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6286,"log_dups_size":0,"ondisk_log_size":6286,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51451040,"num_objects":6281,"num_object_clones":0,"num_object_copies":25124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6281,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6284,"num_write_kb":50260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4042}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'6298","reported_seq":6358,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:15.806643+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:38:15.806643+0000","last_peered":"2024-09-05T16:38:15.806643+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:38:15.806643+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6298,"log_dups_size":0,"ondisk_log_size":6298,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51521856,"num_objects":6290,"num_object_clones":0,"num_object_copies":25160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6290,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6295,"num_write_kb":50337,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":4029}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'6316","reported_seq":6398,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:16.340090+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:38:16.340090+0000","last_peered":"2024-09-05T16:38:16.340090+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:38:16.340090+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6316,"log_dups_size":0,"ondisk_log_size":6316,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51721376,"num_objects":6314,"num_object_clones":0,"num_object_copies":25256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6314,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6314,"num_write_kb":50510,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":4052}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'6439","reported_seq":6497,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:15.726019+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:38:15.726019+0000","last_peered":"2024-09-05T16:38:15.726019+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:38:15.726019+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6439,"log_dups_size":0,"ondisk_log_size":6439,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52702960,"num_objects":6434,"num_object_clones":0,"num_object_copies":25736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6434,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6436,"num_write_kb":51476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4123}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'6261","reported_seq":6299,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:15.847804+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:38:15.847804+0000","last_peered":"2024-09-05T16:38:15.847804+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:38:15.847804+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6261,"log_dups_size":0,"ondisk_log_size":6261,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51280464,"num_objects":6260,"num_object_clones":0,"num_object_copies":25040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6260,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6260,"num_write_kb":50079,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3996}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'6287","reported_seq":7309,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:16.332167+0000","last_change":"2024-09-05T16:37:30.634570+0000","last_active":"2024-09-05T16:38:16.332167+0000","last_peered":"2024-09-05T16:38:16.332167+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:38:16.332167+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6287,"log_dups_size":0,"ondisk_log_size":6287,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51493456,"num_objects":6286,"num_object_clones":0,"num_object_copies":25144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1821,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6286,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6286,"num_write_kb":50287,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":474,"num_bytes_recovered":3874816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4465}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'6444","reported_seq":6500,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:15.838152+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:38:15.838152+0000","last_peered":"2024-09-05T16:38:15.838152+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:38:15.838152+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6444,"log_dups_size":0,"ondisk_log_size":6444,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52753568,"num_objects":6440,"num_object_clones":0,"num_object_copies":25760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6440,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6442,"num_write_kb":51532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":4083}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'6312","reported_seq":6394,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:16.133393+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:38:16.133393+0000","last_peered":"2024-09-05T16:38:16.133393+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:38:16.133393+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6312,"log_dups_size":0,"ondisk_log_size":6312,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51680416,"num_objects":6309,"num_object_clones":0,"num_object_copies":25236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6309,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6310,"num_write_kb":50477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3989}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'6286","reported_seq":6353,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:16.104487+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:38:16.104487+0000","last_peered":"2024-09-05T16:38:16.104487+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:38:16.104487+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6286,"log_dups_size":0,"ondisk_log_size":6286,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51451040,"num_objects":6281,"num_object_clones":0,"num_object_copies":25124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6281,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6284,"num_write_kb":50260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4044}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":825388064,"num_objects":100760,"num_object_clones":0,"num_object_copies":403040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33434,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100760,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":95091,"num_write_kb":760621,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":498,"num_bytes_recovered":3874816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1513521152,"data_stored":1513521152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100816,"ondisk_log_size":100816,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084340,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":763344,"kb_used_data":402624,"kb_used_omap":4,"kb_used_meta":360699,"kb_avail":93608496,"statfs":{"total":96636764160,"available":95855099904,"internally_reserved":0,"allocated":412286976,"data_stored":412094935,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":369356499},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,3,6],"upper_bound":64},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986217,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511172,"kb_used_data":268468,"kb_used_omap":4,"kb_used_meta":242683,"kb_avail":93860668,"statfs":{"total":96636764160,"available":96113324032,"internally_reserved":0,"allocated":274911232,"data_stored":274719191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":248508114},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,3,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673023,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":766820,"kb_used_data":402452,"kb_used_omap":1,"kb_used_meta":364350,"kb_avail":93605020,"statfs":{"total":96636764160,"available":95851540480,"internally_reserved":0,"allocated":412110848,"data_stored":411918807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":373094853},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,2,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182454,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":768020,"kb_used_data":403204,"kb_used_omap":4,"kb_used_meta":364795,"kb_avail":93603820,"statfs":{"total":96636764160,"available":95850311680,"internally_reserved":0,"allocated":412880896,"data_stored":412688855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":373550797},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,2,1,5,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":412901376,"data_stored":412901376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":275288064,"data_stored":275288064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":412942336,"data_stored":412942336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":412389376,"data_stored":412389376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:38:18.745 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:38:19.325 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:38:19.326 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:38:19.362 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":177,"stamp":"2024-09-05T16:38:18.880184+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":825388064,"num_objects":100760,"num_object_clones":0,"num_object_copies":403040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33434,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100760,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":95091,"num_write_kb":760621,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":498,"num_bytes_recovered":3874816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100816,"ondisk_log_size":100816,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2809356,"kb_used_data":1476748,"kb_used_omap":14,"kb_used_meta":1332529,"kb_avail":374678004,"statfs":{"total":386547056640,"available":383670276096,"internally_reserved":0,"allocated":1512189952,"data_stored":1511421788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1364510263},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,7,6,8,17,8,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":20701184,"num_objects":2527,"num_object_clones":0,"num_object_copies":10108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-94,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2527,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2527,"num_write_kb":20216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":770048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004523"},"pg_stats":[{"pgid":"1.f","version":"35'6346","reported_seq":6392,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:15.755541+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:38:15.755541+0000","last_peered":"2024-09-05T16:38:15.755541+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:38:15.755541+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6346,"log_dups_size":0,"ondisk_log_size":6346,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51986432,"num_objects":6346,"num_object_clones":0,"num_object_copies":25384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6346,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6346,"num_write_kb":50768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4084}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'6171","reported_seq":6239,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:16.236181+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:38:16.236181+0000","last_peered":"2024-09-05T16:38:16.236181+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:38:16.236181+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6171,"log_dups_size":0,"ondisk_log_size":6171,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50517152,"num_objects":6167,"num_object_clones":0,"num_object_copies":24668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6167,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6169,"num_write_kb":49348,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3900}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'6292","reported_seq":6372,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:16.341690+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:38:16.341690+0000","last_peered":"2024-09-05T16:38:16.341690+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:38:16.341690+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6292,"log_dups_size":0,"ondisk_log_size":6292,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51518032,"num_objects":6289,"num_object_clones":0,"num_object_copies":25156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6289,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6291,"num_write_kb":50318,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":4039}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'6369","reported_seq":11092,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-05T16:38:15.456615+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:38:15.456615+0000","last_peered":"2024-09-05T16:38:15.456615+0000","last_clean":"2024-09-05T16:38:15.456615+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:38:15.456615+0000","last_undegraded":"2024-09-05T16:38:15.456615+0000","last_fullsized":"2024-09-05T16:38:15.456615+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":6369,"log_dups_size":0,"ondisk_log_size":6369,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52125696,"num_objects":6363,"num_object_clones":0,"num_object_copies":25452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6363,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":669,"num_write_kb":5352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'6198","reported_seq":6256,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:15.538502+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:38:15.538502+0000","last_peered":"2024-09-05T16:38:15.538502+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:38:15.538502+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6198,"log_dups_size":0,"ondisk_log_size":6198,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50746528,"num_objects":6195,"num_object_clones":0,"num_object_copies":24780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6195,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6196,"num_write_kb":49565,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":4033}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'6269","reported_seq":6315,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:15.803874+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:38:15.803874+0000","last_peered":"2024-09-05T16:38:15.803874+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:38:15.803874+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6269,"log_dups_size":0,"ondisk_log_size":6269,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51355648,"num_objects":6269,"num_object_clones":0,"num_object_copies":25076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6269,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6269,"num_write_kb":50152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":4019}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'6242","reported_seq":6312,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:16.314126+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:38:16.314126+0000","last_peered":"2024-09-05T16:38:16.314126+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:38:16.314126+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6242,"log_dups_size":0,"ondisk_log_size":6242,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51082400,"num_objects":6236,"num_object_clones":0,"num_object_copies":24944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6240,"num_write_kb":49900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":4089}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'6286","reported_seq":6353,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:16.158997+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:38:16.158997+0000","last_peered":"2024-09-05T16:38:16.158997+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:38:16.158997+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6286,"log_dups_size":0,"ondisk_log_size":6286,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51451040,"num_objects":6281,"num_object_clones":0,"num_object_copies":25124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6281,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6284,"num_write_kb":50260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4042}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'6298","reported_seq":6358,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:15.806643+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:38:15.806643+0000","last_peered":"2024-09-05T16:38:15.806643+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:38:15.806643+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6298,"log_dups_size":0,"ondisk_log_size":6298,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51521856,"num_objects":6290,"num_object_clones":0,"num_object_copies":25160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6290,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6295,"num_write_kb":50337,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":4029}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'6316","reported_seq":6398,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:16.340090+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:38:16.340090+0000","last_peered":"2024-09-05T16:38:16.340090+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:38:16.340090+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6316,"log_dups_size":0,"ondisk_log_size":6316,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51721376,"num_objects":6314,"num_object_clones":0,"num_object_copies":25256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6314,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6314,"num_write_kb":50510,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":4052}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'6439","reported_seq":6497,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:15.726019+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:38:15.726019+0000","last_peered":"2024-09-05T16:38:15.726019+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:38:15.726019+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6439,"log_dups_size":0,"ondisk_log_size":6439,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52702960,"num_objects":6434,"num_object_clones":0,"num_object_copies":25736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6434,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6436,"num_write_kb":51476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4123}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'6261","reported_seq":6299,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:15.847804+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:38:15.847804+0000","last_peered":"2024-09-05T16:38:15.847804+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:38:15.847804+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6261,"log_dups_size":0,"ondisk_log_size":6261,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51280464,"num_objects":6260,"num_object_clones":0,"num_object_copies":25040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6260,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6260,"num_write_kb":50079,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3996}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'6287","reported_seq":7309,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:16.332167+0000","last_change":"2024-09-05T16:37:30.634570+0000","last_active":"2024-09-05T16:38:16.332167+0000","last_peered":"2024-09-05T16:38:16.332167+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:38:16.332167+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6287,"log_dups_size":0,"ondisk_log_size":6287,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51493456,"num_objects":6286,"num_object_clones":0,"num_object_copies":25144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1821,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6286,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6286,"num_write_kb":50287,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":474,"num_bytes_recovered":3874816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4465}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'6444","reported_seq":6500,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:15.838152+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:38:15.838152+0000","last_peered":"2024-09-05T16:38:15.838152+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:38:15.838152+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6444,"log_dups_size":0,"ondisk_log_size":6444,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52753568,"num_objects":6440,"num_object_clones":0,"num_object_copies":25760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6440,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6442,"num_write_kb":51532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":4083}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'6312","reported_seq":6394,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:16.133393+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:38:16.133393+0000","last_peered":"2024-09-05T16:38:16.133393+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:38:16.133393+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6312,"log_dups_size":0,"ondisk_log_size":6312,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51680416,"num_objects":6309,"num_object_clones":0,"num_object_copies":25236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6309,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6310,"num_write_kb":50477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3989}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'6286","reported_seq":6353,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:16.104487+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:38:16.104487+0000","last_peered":"2024-09-05T16:38:16.104487+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:38:16.104487+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6286,"log_dups_size":0,"ondisk_log_size":6286,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51451040,"num_objects":6281,"num_object_clones":0,"num_object_copies":25124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6281,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6284,"num_write_kb":50260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4044}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":825388064,"num_objects":100760,"num_object_clones":0,"num_object_copies":403040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33434,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100760,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":95091,"num_write_kb":760621,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":498,"num_bytes_recovered":3874816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1513521152,"data_stored":1513521152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100816,"ondisk_log_size":100816,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084340,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":763344,"kb_used_data":402624,"kb_used_omap":4,"kb_used_meta":360699,"kb_avail":93608496,"statfs":{"total":96636764160,"available":95855099904,"internally_reserved":0,"allocated":412286976,"data_stored":412094935,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":369356499},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,3,6],"upper_bound":64},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986217,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511172,"kb_used_data":268468,"kb_used_omap":4,"kb_used_meta":242683,"kb_avail":93860668,"statfs":{"total":96636764160,"available":96113324032,"internally_reserved":0,"allocated":274911232,"data_stored":274719191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":248508114},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,3,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673023,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":766820,"kb_used_data":402452,"kb_used_omap":1,"kb_used_meta":364350,"kb_avail":93605020,"statfs":{"total":96636764160,"available":95851540480,"internally_reserved":0,"allocated":412110848,"data_stored":411918807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":373094853},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,2,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182454,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":768020,"kb_used_data":403204,"kb_used_omap":4,"kb_used_meta":364795,"kb_avail":93603820,"statfs":{"total":96636764160,"available":95850311680,"internally_reserved":0,"allocated":412880896,"data_stored":412688855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":373550797},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,2,1,5,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":412901376,"data_stored":412901376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":275288064,"data_stored":275288064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":412942336,"data_stored":412942336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":412389376,"data_stored":412389376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:38:19.363 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-05T16:38:19.363 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'6346", 'reported_seq': 6392, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:15.755541+0000', 'last_change': '2024-09-05T16:34:56.833497+0000', 'last_active': '2024-09-05T16:38:15.755541+0000', 'last_peered': '2024-09-05T16:38:15.755541+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.828289+0000', 'last_became_peered': '2024-09-05T16:34:56.828289+0000', 'last_unstale': '2024-09-05T16:38:15.755541+0000', 'last_undegraded': '2024-09-05T16:34:56.612160+0000', 'last_fullsized': '2024-09-05T16:34:56.557632+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6346, 'log_dups_size': 0, 'ondisk_log_size': 6346, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51986432, 'num_objects': 6346, 'num_object_clones': 0, 'num_object_copies': 25384, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2262, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6346, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6346, 'num_write_kb': 50768, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 4084}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:38:19.363 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-05T16:38:19.363 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "35'6171", 'reported_seq': 6239, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:16.236181+0000', 'last_change': '2024-09-05T16:34:56.825031+0000', 'last_active': '2024-09-05T16:38:16.236181+0000', 'last_peered': '2024-09-05T16:38:16.236181+0000', 'last_clean': '2024-09-05T16:33:53.200762+0000', 'last_became_active': '2024-09-05T16:34:56.820292+0000', 'last_became_peered': '2024-09-05T16:34:56.820292+0000', 'last_unstale': '2024-09-05T16:38:16.236181+0000', 'last_undegraded': '2024-09-05T16:34:56.605776+0000', 'last_fullsized': '2024-09-05T16:34:56.559383+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6171, 'log_dups_size': 0, 'ondisk_log_size': 6171, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50517152, 'num_objects': 6167, 'num_object_clones': 0, 'num_object_copies': 24668, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2269, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6167, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6169, 'num_write_kb': 49348, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3900}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:38:19.364 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:38:19.364 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'6292", 'reported_seq': 6372, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:16.341690+0000', 'last_change': '2024-09-05T16:34:56.867900+0000', 'last_active': '2024-09-05T16:38:16.341690+0000', 'last_peered': '2024-09-05T16:38:16.341690+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:38:16.341690+0000', 'last_undegraded': '2024-09-05T16:34:56.590972+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6292, 'log_dups_size': 0, 'ondisk_log_size': 6292, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51518032, 'num_objects': 6289, 'num_object_clones': 0, 'num_object_copies': 25156, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2251, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6289, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6291, 'num_write_kb': 50318, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 4039}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:38:19.364 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:38:19.364 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'6198", 'reported_seq': 6256, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:15.538502+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:38:15.538502+0000', 'last_peered': '2024-09-05T16:38:15.538502+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:38:15.538502+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6198, 'log_dups_size': 0, 'ondisk_log_size': 6198, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50746528, 'num_objects': 6195, 'num_object_clones': 0, 'num_object_copies': 24780, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2164, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6195, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6196, 'num_write_kb': 49565, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 4033}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:38:19.364 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:38:19.365 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'6269", 'reported_seq': 6315, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:15.803874+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:38:15.803874+0000', 'last_peered': '2024-09-05T16:38:15.803874+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:38:15.803874+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6269, 'log_dups_size': 0, 'ondisk_log_size': 6269, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51355648, 'num_objects': 6269, 'num_object_clones': 0, 'num_object_copies': 25076, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6269, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6269, 'num_write_kb': 50152, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 4019}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:38:19.365 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:38:19.365 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'6242", 'reported_seq': 6312, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:16.314126+0000', 'last_change': '2024-09-05T16:34:56.903436+0000', 'last_active': '2024-09-05T16:38:16.314126+0000', 'last_peered': '2024-09-05T16:38:16.314126+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:38:16.314126+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6242, 'log_dups_size': 0, 'ondisk_log_size': 6242, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51082400, 'num_objects': 6236, 'num_object_clones': 0, 'num_object_copies': 24944, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2149, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6236, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6240, 'num_write_kb': 49900, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 4089}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:38:19.365 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-05T16:38:19.365 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "35'6286", 'reported_seq': 6353, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:16.158997+0000', 'last_change': '2024-09-05T16:34:56.883319+0000', 'last_active': '2024-09-05T16:38:16.158997+0000', 'last_peered': '2024-09-05T16:38:16.158997+0000', 'last_clean': '2024-09-05T16:33:53.505305+0000', 'last_became_active': '2024-09-05T16:34:56.873389+0000', 'last_became_peered': '2024-09-05T16:34:56.873389+0000', 'last_unstale': '2024-09-05T16:38:16.158997+0000', 'last_undegraded': '2024-09-05T16:34:56.602567+0000', 'last_fullsized': '2024-09-05T16:34:56.559958+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6286, 'log_dups_size': 0, 'ondisk_log_size': 6286, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51451040, 'num_objects': 6281, 'num_object_clones': 0, 'num_object_copies': 25124, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2241, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6281, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6284, 'num_write_kb': 50260, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 4042}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:38:19.366 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:38:19.366 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'6298", 'reported_seq': 6358, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:15.806643+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:38:15.806643+0000', 'last_peered': '2024-09-05T16:38:15.806643+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:38:15.806643+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6298, 'log_dups_size': 0, 'ondisk_log_size': 6298, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51521856, 'num_objects': 6290, 'num_object_clones': 0, 'num_object_copies': 25160, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2264, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6290, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6295, 'num_write_kb': 50337, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 4029}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:38:19.366 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:38:19.366 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'6316", 'reported_seq': 6398, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:16.340090+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:38:16.340090+0000', 'last_peered': '2024-09-05T16:38:16.340090+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:38:16.340090+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6316, 'log_dups_size': 0, 'ondisk_log_size': 6316, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51721376, 'num_objects': 6314, 'num_object_clones': 0, 'num_object_copies': 25256, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2264, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6314, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6314, 'num_write_kb': 50510, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 4052}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:38:19.366 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:38:19.366 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'6439", 'reported_seq': 6497, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:15.726019+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:38:15.726019+0000', 'last_peered': '2024-09-05T16:38:15.726019+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:38:15.726019+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6439, 'log_dups_size': 0, 'ondisk_log_size': 6439, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 52702960, 'num_objects': 6434, 'num_object_clones': 0, 'num_object_copies': 25736, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2314, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6434, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6436, 'num_write_kb': 51476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 4123}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:38:19.367 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:38:19.367 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'6261", 'reported_seq': 6299, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:15.847804+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:38:15.847804+0000', 'last_peered': '2024-09-05T16:38:15.847804+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:38:15.847804+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6261, 'log_dups_size': 0, 'ondisk_log_size': 6261, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51280464, 'num_objects': 6260, 'num_object_clones': 0, 'num_object_copies': 25040, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2265, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6260, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6260, 'num_write_kb': 50079, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3996}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:38:19.367 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-05T16:38:19.367 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'6287", 'reported_seq': 7309, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:16.332167+0000', 'last_change': '2024-09-05T16:37:30.634570+0000', 'last_active': '2024-09-05T16:38:16.332167+0000', 'last_peered': '2024-09-05T16:38:16.332167+0000', 'last_clean': '2024-09-05T16:33:45.082039+0000', 'last_became_active': '2024-09-05T16:34:56.792386+0000', 'last_became_peered': '2024-09-05T16:34:56.792386+0000', 'last_unstale': '2024-09-05T16:38:16.332167+0000', 'last_undegraded': '2024-09-05T16:34:56.600595+0000', 'last_fullsized': '2024-09-05T16:34:56.558203+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6287, 'log_dups_size': 0, 'ondisk_log_size': 6287, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51493456, 'num_objects': 6286, 'num_object_clones': 0, 'num_object_copies': 25144, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1821, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6286, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6286, 'num_write_kb': 50287, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 474, 'num_bytes_recovered': 3874816, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 4465}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:38:19.367 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:38:19.367 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'6444", 'reported_seq': 6500, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:15.838152+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:38:15.838152+0000', 'last_peered': '2024-09-05T16:38:15.838152+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:38:15.838152+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6444, 'log_dups_size': 0, 'ondisk_log_size': 6444, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 52753568, 'num_objects': 6440, 'num_object_clones': 0, 'num_object_copies': 25760, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2359, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6440, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6442, 'num_write_kb': 51532, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 4083}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:38:19.367 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:38:19.367 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'6312", 'reported_seq': 6394, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:16.133393+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:38:16.133393+0000', 'last_peered': '2024-09-05T16:38:16.133393+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:38:16.133393+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6312, 'log_dups_size': 0, 'ondisk_log_size': 6312, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51680416, 'num_objects': 6309, 'num_object_clones': 0, 'num_object_copies': 25236, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2322, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6309, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6310, 'num_write_kb': 50477, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3989}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:38:19.367 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:38:19.367 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'6286", 'reported_seq': 6353, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:16.104487+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:38:16.104487+0000', 'last_peered': '2024-09-05T16:38:16.104487+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:38:16.104487+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6286, 'log_dups_size': 0, 'ondisk_log_size': 6286, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51451040, 'num_objects': 6281, 'num_object_clones': 0, 'num_object_copies': 25124, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6281, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6284, 'num_write_kb': 50260, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 4044}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:38:19.368 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:38:19.542 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 237 20 101663 101643 3.34981 1.92969 0.0670249 0.0466321 2024-09-05T16:38:20.135 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:38:20.135 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 33434/403040 objects degraded (8.295%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":303,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":100760,"data_bytes":825388064,"bytes_used":2876780544,"bytes_avail":383670276096,"bytes_total":386547056640,"degraded_objects":33434,"degraded_total":403040,"degraded_ratio":0.082954545454545461,"recovering_objects_per_sec":7,"recovering_bytes_per_sec":64146,"recovering_keys_per_sec":0,"num_objects_recovered":94,"num_bytes_recovered":770048,"num_keys_recovered":0,"write_bytes_sec":1724448,"read_op_per_sec":0,"write_op_per_sec":210},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:38:20.155 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:38:20.156 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:38:20.542 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 238 20 101901 101881 3.34355 1.85938 0.0888149 0.0467187 2024-09-05T16:38:20.822 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:38:20.823 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:38:20.860 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":177,"stamp":"2024-09-05T16:38:18.880184+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":825388064,"num_objects":100760,"num_object_clones":0,"num_object_copies":403040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33434,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100760,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":95091,"num_write_kb":760621,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":498,"num_bytes_recovered":3874816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100816,"ondisk_log_size":100816,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2809356,"kb_used_data":1476748,"kb_used_omap":14,"kb_used_meta":1332529,"kb_avail":374678004,"statfs":{"total":386547056640,"available":383670276096,"internally_reserved":0,"allocated":1512189952,"data_stored":1511421788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1364510263},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,7,6,8,17,8,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":20701184,"num_objects":2527,"num_object_clones":0,"num_object_copies":10108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-94,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2527,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2527,"num_write_kb":20216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":770048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004523"},"pg_stats":[{"pgid":"1.f","version":"35'6346","reported_seq":6392,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:15.755541+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:38:15.755541+0000","last_peered":"2024-09-05T16:38:15.755541+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:38:15.755541+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6346,"log_dups_size":0,"ondisk_log_size":6346,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51986432,"num_objects":6346,"num_object_clones":0,"num_object_copies":25384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6346,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6346,"num_write_kb":50768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4084}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'6171","reported_seq":6239,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:16.236181+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:38:16.236181+0000","last_peered":"2024-09-05T16:38:16.236181+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:38:16.236181+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6171,"log_dups_size":0,"ondisk_log_size":6171,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50517152,"num_objects":6167,"num_object_clones":0,"num_object_copies":24668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6167,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6169,"num_write_kb":49348,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3900}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'6292","reported_seq":6372,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:16.341690+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:38:16.341690+0000","last_peered":"2024-09-05T16:38:16.341690+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:38:16.341690+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6292,"log_dups_size":0,"ondisk_log_size":6292,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51518032,"num_objects":6289,"num_object_clones":0,"num_object_copies":25156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6289,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6291,"num_write_kb":50318,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":4039}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'6369","reported_seq":11092,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-05T16:38:15.456615+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:38:15.456615+0000","last_peered":"2024-09-05T16:38:15.456615+0000","last_clean":"2024-09-05T16:38:15.456615+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:38:15.456615+0000","last_undegraded":"2024-09-05T16:38:15.456615+0000","last_fullsized":"2024-09-05T16:38:15.456615+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":6369,"log_dups_size":0,"ondisk_log_size":6369,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52125696,"num_objects":6363,"num_object_clones":0,"num_object_copies":25452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6363,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":669,"num_write_kb":5352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'6198","reported_seq":6256,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:15.538502+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:38:15.538502+0000","last_peered":"2024-09-05T16:38:15.538502+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:38:15.538502+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6198,"log_dups_size":0,"ondisk_log_size":6198,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50746528,"num_objects":6195,"num_object_clones":0,"num_object_copies":24780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6195,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6196,"num_write_kb":49565,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":4033}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'6269","reported_seq":6315,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:15.803874+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:38:15.803874+0000","last_peered":"2024-09-05T16:38:15.803874+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:38:15.803874+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6269,"log_dups_size":0,"ondisk_log_size":6269,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51355648,"num_objects":6269,"num_object_clones":0,"num_object_copies":25076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6269,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6269,"num_write_kb":50152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":4019}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'6242","reported_seq":6312,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:16.314126+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:38:16.314126+0000","last_peered":"2024-09-05T16:38:16.314126+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:38:16.314126+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6242,"log_dups_size":0,"ondisk_log_size":6242,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51082400,"num_objects":6236,"num_object_clones":0,"num_object_copies":24944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6240,"num_write_kb":49900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":4089}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'6286","reported_seq":6353,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:16.158997+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:38:16.158997+0000","last_peered":"2024-09-05T16:38:16.158997+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:38:16.158997+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6286,"log_dups_size":0,"ondisk_log_size":6286,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51451040,"num_objects":6281,"num_object_clones":0,"num_object_copies":25124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6281,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6284,"num_write_kb":50260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4042}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'6298","reported_seq":6358,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:15.806643+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:38:15.806643+0000","last_peered":"2024-09-05T16:38:15.806643+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:38:15.806643+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6298,"log_dups_size":0,"ondisk_log_size":6298,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51521856,"num_objects":6290,"num_object_clones":0,"num_object_copies":25160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6290,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6295,"num_write_kb":50337,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":4029}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'6316","reported_seq":6398,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:16.340090+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:38:16.340090+0000","last_peered":"2024-09-05T16:38:16.340090+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:38:16.340090+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6316,"log_dups_size":0,"ondisk_log_size":6316,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51721376,"num_objects":6314,"num_object_clones":0,"num_object_copies":25256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6314,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6314,"num_write_kb":50510,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":4052}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'6439","reported_seq":6497,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:15.726019+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:38:15.726019+0000","last_peered":"2024-09-05T16:38:15.726019+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:38:15.726019+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6439,"log_dups_size":0,"ondisk_log_size":6439,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52702960,"num_objects":6434,"num_object_clones":0,"num_object_copies":25736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6434,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6436,"num_write_kb":51476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4123}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'6261","reported_seq":6299,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:15.847804+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:38:15.847804+0000","last_peered":"2024-09-05T16:38:15.847804+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:38:15.847804+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6261,"log_dups_size":0,"ondisk_log_size":6261,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51280464,"num_objects":6260,"num_object_clones":0,"num_object_copies":25040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6260,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6260,"num_write_kb":50079,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3996}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'6287","reported_seq":7309,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:16.332167+0000","last_change":"2024-09-05T16:37:30.634570+0000","last_active":"2024-09-05T16:38:16.332167+0000","last_peered":"2024-09-05T16:38:16.332167+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:38:16.332167+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6287,"log_dups_size":0,"ondisk_log_size":6287,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51493456,"num_objects":6286,"num_object_clones":0,"num_object_copies":25144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1821,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6286,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6286,"num_write_kb":50287,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":474,"num_bytes_recovered":3874816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4465}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'6444","reported_seq":6500,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:15.838152+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:38:15.838152+0000","last_peered":"2024-09-05T16:38:15.838152+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:38:15.838152+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6444,"log_dups_size":0,"ondisk_log_size":6444,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52753568,"num_objects":6440,"num_object_clones":0,"num_object_copies":25760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6440,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6442,"num_write_kb":51532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":4083}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'6312","reported_seq":6394,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:16.133393+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:38:16.133393+0000","last_peered":"2024-09-05T16:38:16.133393+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:38:16.133393+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6312,"log_dups_size":0,"ondisk_log_size":6312,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51680416,"num_objects":6309,"num_object_clones":0,"num_object_copies":25236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6309,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6310,"num_write_kb":50477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3989}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'6286","reported_seq":6353,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:16.104487+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:38:16.104487+0000","last_peered":"2024-09-05T16:38:16.104487+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:38:16.104487+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6286,"log_dups_size":0,"ondisk_log_size":6286,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51451040,"num_objects":6281,"num_object_clones":0,"num_object_copies":25124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6281,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6284,"num_write_kb":50260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4044}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":825388064,"num_objects":100760,"num_object_clones":0,"num_object_copies":403040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33434,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100760,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":95091,"num_write_kb":760621,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":498,"num_bytes_recovered":3874816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1513521152,"data_stored":1513521152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100816,"ondisk_log_size":100816,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084340,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":763344,"kb_used_data":402624,"kb_used_omap":4,"kb_used_meta":360699,"kb_avail":93608496,"statfs":{"total":96636764160,"available":95855099904,"internally_reserved":0,"allocated":412286976,"data_stored":412094935,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":369356499},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,3,6],"upper_bound":64},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986217,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511172,"kb_used_data":268468,"kb_used_omap":4,"kb_used_meta":242683,"kb_avail":93860668,"statfs":{"total":96636764160,"available":96113324032,"internally_reserved":0,"allocated":274911232,"data_stored":274719191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":248508114},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,3,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673023,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":766820,"kb_used_data":402452,"kb_used_omap":1,"kb_used_meta":364350,"kb_avail":93605020,"statfs":{"total":96636764160,"available":95851540480,"internally_reserved":0,"allocated":412110848,"data_stored":411918807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":373094853},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,2,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182454,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":768020,"kb_used_data":403204,"kb_used_omap":4,"kb_used_meta":364795,"kb_avail":93603820,"statfs":{"total":96636764160,"available":95850311680,"internally_reserved":0,"allocated":412880896,"data_stored":412688855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":373550797},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,2,1,5,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":412901376,"data_stored":412901376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":275288064,"data_stored":275288064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":412942336,"data_stored":412942336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":412389376,"data_stored":412389376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:38:21.542 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout: 239 20 102170 102150 3.33836 2.10156 0.0606375 0.0467948 2024-09-05T16:38:22.542 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout:Total time run: 240.044 2024-09-05T16:38:22.543 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout:Total writes made: 102426 2024-09-05T16:38:22.543 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout:Write size: 8192 2024-09-05T16:38:22.543 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout:Object size: 8192 2024-09-05T16:38:22.543 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout:Bandwidth (MB/sec): 3.33357 2024-09-05T16:38:22.543 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout:Stddev Bandwidth: 2.31447 2024-09-05T16:38:22.543 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout:Max bandwidth (MB/sec): 13.0781 2024-09-05T16:38:22.543 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout:Min bandwidth (MB/sec): 0 2024-09-05T16:38:22.543 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout:Average IOPS: 426 2024-09-05T16:38:22.543 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout:Stddev IOPS: 296.253 2024-09-05T16:38:22.543 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout:Max IOPS: 1674 2024-09-05T16:38:22.543 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout:Min IOPS: 0 2024-09-05T16:38:22.544 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout:Average Latency(s): 0.0468636 2024-09-05T16:38:22.544 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout:Stddev Latency(s): 0.0443049 2024-09-05T16:38:22.544 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout:Max latency(s): 2.29705 2024-09-05T16:38:22.544 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout:Min latency(s): 0.00357551 2024-09-05T16:38:22.557 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout:Cleaning up (deleting benchmark objects) 2024-09-05T16:38:23.862 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:38:24.454 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:38:24.455 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:38:24.466 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":179,"stamp":"2024-09-05T16:38:22.881412+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":836201504,"num_objects":102080,"num_object_clones":0,"num_object_copies":408320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33393,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102080,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":96411,"num_write_kb":771181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":539,"num_bytes_recovered":4210688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":102136,"ondisk_log_size":102136,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2815244,"kb_used_data":1497868,"kb_used_omap":14,"kb_used_meta":1317297,"kb_avail":374672116,"statfs":{"total":386547056640,"available":383664246784,"internally_reserved":0,"allocated":1533816832,"data_stored":1533048668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1348912695},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,8,4,4,11,8,13,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":26771456,"num_objects":3268,"num_object_clones":0,"num_object_copies":13072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-135,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3268,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3268,"num_write_kb":26144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":135,"num_bytes_recovered":1105920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004033"},"pg_stats":[{"pgid":"1.f","version":"35'6427","reported_seq":6473,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:20.874536+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:38:20.874536+0000","last_peered":"2024-09-05T16:38:20.874536+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:38:20.874536+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6427,"log_dups_size":0,"ondisk_log_size":6427,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52649984,"num_objects":6427,"num_object_clones":0,"num_object_copies":25708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6427,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6427,"num_write_kb":51416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4165}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'6262","reported_seq":6329,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:21.339928+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:38:21.339928+0000","last_peered":"2024-09-05T16:38:21.339928+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:38:21.339928+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6262,"log_dups_size":0,"ondisk_log_size":6262,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51262624,"num_objects":6258,"num_object_clones":0,"num_object_copies":25032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6258,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6260,"num_write_kb":50076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3991}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'6359","reported_seq":6439,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:21.326411+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:38:21.326411+0000","last_peered":"2024-09-05T16:38:21.326411+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:38:21.326411+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6359,"log_dups_size":0,"ondisk_log_size":6359,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52066896,"num_objects":6356,"num_object_clones":0,"num_object_copies":25424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6356,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6358,"num_write_kb":50854,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":4106}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'6446","reported_seq":11169,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-05T16:38:20.616103+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:38:20.616103+0000","last_peered":"2024-09-05T16:38:20.616103+0000","last_clean":"2024-09-05T16:38:20.616103+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:38:20.616103+0000","last_undegraded":"2024-09-05T16:38:20.616103+0000","last_fullsized":"2024-09-05T16:38:20.616103+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":6446,"log_dups_size":0,"ondisk_log_size":6446,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52756480,"num_objects":6440,"num_object_clones":0,"num_object_copies":25760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6440,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":746,"num_write_kb":5968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'6274","reported_seq":6329,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:20.769552+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:38:20.769552+0000","last_peered":"2024-09-05T16:38:20.769552+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:38:20.769552+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6274,"log_dups_size":0,"ondisk_log_size":6274,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51369120,"num_objects":6271,"num_object_clones":0,"num_object_copies":25084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6271,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6272,"num_write_kb":50173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":4109}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'6349","reported_seq":6395,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:20.671522+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:38:20.671522+0000","last_peered":"2024-09-05T16:38:20.671522+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:38:20.671522+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6349,"log_dups_size":0,"ondisk_log_size":6349,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52011008,"num_objects":6349,"num_object_clones":0,"num_object_copies":25396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6349,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6349,"num_write_kb":50792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":4099}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'6314","reported_seq":6384,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:20.854023+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:38:20.854023+0000","last_peered":"2024-09-05T16:38:20.854023+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:38:20.854023+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6314,"log_dups_size":0,"ondisk_log_size":6314,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51672224,"num_objects":6308,"num_object_clones":0,"num_object_copies":25232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6308,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6312,"num_write_kb":50476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":4161}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'6375","reported_seq":6441,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:21.337303+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:38:21.337303+0000","last_peered":"2024-09-05T16:38:21.337303+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:38:21.337303+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6375,"log_dups_size":0,"ondisk_log_size":6375,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52180128,"num_objects":6370,"num_object_clones":0,"num_object_copies":25480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6370,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6373,"num_write_kb":50972,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4131}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'6371","reported_seq":6431,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:20.780083+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:38:20.780083+0000","last_peered":"2024-09-05T16:38:20.780083+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:38:20.780083+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6371,"log_dups_size":0,"ondisk_log_size":6371,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52119872,"num_objects":6363,"num_object_clones":0,"num_object_copies":25452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6363,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6368,"num_write_kb":50921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":4102}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'6387","reported_seq":6468,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:21.412638+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:38:21.412638+0000","last_peered":"2024-09-05T16:38:21.412638+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:38:21.412638+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6387,"log_dups_size":0,"ondisk_log_size":6387,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52303008,"num_objects":6385,"num_object_clones":0,"num_object_copies":25540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6385,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6385,"num_write_kb":51078,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":4123}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'6542","reported_seq":6602,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:20.877105+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:38:20.877105+0000","last_peered":"2024-09-05T16:38:20.877105+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:38:20.877105+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6542,"log_dups_size":0,"ondisk_log_size":6542,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53546736,"num_objects":6537,"num_object_clones":0,"num_object_copies":26148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6537,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6539,"num_write_kb":52300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4226}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'6348","reported_seq":6387,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:20.887738+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:38:20.887738+0000","last_peered":"2024-09-05T16:38:20.887738+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:38:20.887738+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6348,"log_dups_size":0,"ondisk_log_size":6348,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51993168,"num_objects":6347,"num_object_clones":0,"num_object_copies":25388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6347,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6347,"num_write_kb":50775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":4083}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'6394","reported_seq":7502,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:21.395122+0000","last_change":"2024-09-05T16:37:30.634570+0000","last_active":"2024-09-05T16:38:21.395122+0000","last_peered":"2024-09-05T16:38:21.395122+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:38:21.395122+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6394,"log_dups_size":0,"ondisk_log_size":6394,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52370000,"num_objects":6393,"num_object_clones":0,"num_object_copies":25572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1780,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6393,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6393,"num_write_kb":51143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":515,"num_bytes_recovered":4210688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4613}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'6530","reported_seq":6584,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:20.771696+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:38:20.771696+0000","last_peered":"2024-09-05T16:38:20.771696+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:38:20.771696+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6530,"log_dups_size":0,"ondisk_log_size":6530,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53458080,"num_objects":6526,"num_object_clones":0,"num_object_copies":26104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6526,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6528,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":4169}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'6391","reported_seq":6474,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:21.410274+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:38:21.410274+0000","last_peered":"2024-09-05T16:38:21.410274+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:38:21.410274+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6391,"log_dups_size":0,"ondisk_log_size":6391,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52327584,"num_objects":6388,"num_object_clones":0,"num_object_copies":25552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6388,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6389,"num_write_kb":51109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4068}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'6367","reported_seq":6435,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:21.359908+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:38:21.359908+0000","last_peered":"2024-09-05T16:38:21.359908+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:38:21.359908+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6367,"log_dups_size":0,"ondisk_log_size":6367,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52114592,"num_objects":6362,"num_object_clones":0,"num_object_copies":25448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6362,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6365,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4125}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":836201504,"num_objects":102080,"num_object_clones":0,"num_object_copies":408320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33393,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102080,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":96411,"num_write_kb":771181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":539,"num_bytes_recovered":4210688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1535275008,"data_stored":1535275008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":102136,"ondisk_log_size":102136,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084341,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":772692,"kb_used_data":407812,"kb_used_omap":4,"kb_used_meta":364859,"kb_avail":93599148,"statfs":{"total":96636764160,"available":95845527552,"internally_reserved":0,"allocated":417599488,"data_stored":417407447,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":373616339},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,1,2,5,1,7,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986218,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":522204,"kb_used_data":273932,"kb_used_omap":4,"kb_used_meta":248251,"kb_avail":93849636,"statfs":{"total":96636764160,"available":96102027264,"internally_reserved":0,"allocated":280506368,"data_stored":280314327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":254209746},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,3,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673024,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":759680,"kb_used_data":407792,"kb_used_omap":1,"kb_used_meta":351870,"kb_avail":93612160,"statfs":{"total":96636764160,"available":95858851840,"internally_reserved":0,"allocated":417579008,"data_stored":417386967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":360315333},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,5,0,0,0,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182455,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":760668,"kb_used_data":408332,"kb_used_omap":4,"kb_used_meta":352315,"kb_avail":93611172,"statfs":{"total":96636764160,"available":95857840128,"internally_reserved":0,"allocated":418131968,"data_stored":417939927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":360771277},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,0,3,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":418308096,"data_stored":418308096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":280895488,"data_stored":280895488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":418316288,"data_stored":418316288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":417755136,"data_stored":417755136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:38:24.468 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:38:25.032 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:38:25.035 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:38:25.059 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":180,"stamp":"2024-09-05T16:38:24.882386+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":836201504,"num_objects":102080,"num_object_clones":0,"num_object_copies":408320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33393,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102080,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":96411,"num_write_kb":771181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":539,"num_bytes_recovered":4210688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":102136,"ondisk_log_size":102136,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2815244,"kb_used_data":1497868,"kb_used_omap":14,"kb_used_meta":1317297,"kb_avail":374672116,"statfs":{"total":386547056640,"available":383664246784,"internally_reserved":0,"allocated":1533816832,"data_stored":1533048668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1348912695},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,8,4,4,11,8,13,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":21405696,"num_objects":2613,"num_object_clones":0,"num_object_copies":10452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-93,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2613,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2613,"num_write_kb":20904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":93,"num_bytes_recovered":761856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004496"},"pg_stats":[{"pgid":"1.f","version":"35'6427","reported_seq":6473,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:20.874536+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:38:20.874536+0000","last_peered":"2024-09-05T16:38:20.874536+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:38:20.874536+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6427,"log_dups_size":0,"ondisk_log_size":6427,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52649984,"num_objects":6427,"num_object_clones":0,"num_object_copies":25708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6427,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6427,"num_write_kb":51416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4165}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'6262","reported_seq":6329,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:21.339928+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:38:21.339928+0000","last_peered":"2024-09-05T16:38:21.339928+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:38:21.339928+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6262,"log_dups_size":0,"ondisk_log_size":6262,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51262624,"num_objects":6258,"num_object_clones":0,"num_object_copies":25032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6258,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6260,"num_write_kb":50076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3991}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'6359","reported_seq":6439,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:21.326411+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:38:21.326411+0000","last_peered":"2024-09-05T16:38:21.326411+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:38:21.326411+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6359,"log_dups_size":0,"ondisk_log_size":6359,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52066896,"num_objects":6356,"num_object_clones":0,"num_object_copies":25424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6356,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6358,"num_write_kb":50854,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":4106}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'6446","reported_seq":11169,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-05T16:38:20.616103+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:38:20.616103+0000","last_peered":"2024-09-05T16:38:20.616103+0000","last_clean":"2024-09-05T16:38:20.616103+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:38:20.616103+0000","last_undegraded":"2024-09-05T16:38:20.616103+0000","last_fullsized":"2024-09-05T16:38:20.616103+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":6446,"log_dups_size":0,"ondisk_log_size":6446,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52756480,"num_objects":6440,"num_object_clones":0,"num_object_copies":25760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6440,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":746,"num_write_kb":5968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'6274","reported_seq":6329,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:20.769552+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:38:20.769552+0000","last_peered":"2024-09-05T16:38:20.769552+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:38:20.769552+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6274,"log_dups_size":0,"ondisk_log_size":6274,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51369120,"num_objects":6271,"num_object_clones":0,"num_object_copies":25084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6271,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6272,"num_write_kb":50173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":4109}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'6349","reported_seq":6395,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:20.671522+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:38:20.671522+0000","last_peered":"2024-09-05T16:38:20.671522+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:38:20.671522+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6349,"log_dups_size":0,"ondisk_log_size":6349,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52011008,"num_objects":6349,"num_object_clones":0,"num_object_copies":25396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6349,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6349,"num_write_kb":50792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":4099}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'6314","reported_seq":6384,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:20.854023+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:38:20.854023+0000","last_peered":"2024-09-05T16:38:20.854023+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:38:20.854023+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6314,"log_dups_size":0,"ondisk_log_size":6314,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51672224,"num_objects":6308,"num_object_clones":0,"num_object_copies":25232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6308,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6312,"num_write_kb":50476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":4161}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'6375","reported_seq":6441,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:21.337303+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:38:21.337303+0000","last_peered":"2024-09-05T16:38:21.337303+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:38:21.337303+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6375,"log_dups_size":0,"ondisk_log_size":6375,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52180128,"num_objects":6370,"num_object_clones":0,"num_object_copies":25480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6370,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6373,"num_write_kb":50972,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4131}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'6371","reported_seq":6431,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:20.780083+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:38:20.780083+0000","last_peered":"2024-09-05T16:38:20.780083+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:38:20.780083+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6371,"log_dups_size":0,"ondisk_log_size":6371,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52119872,"num_objects":6363,"num_object_clones":0,"num_object_copies":25452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6363,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6368,"num_write_kb":50921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":4102}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'6387","reported_seq":6468,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:21.412638+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:38:21.412638+0000","last_peered":"2024-09-05T16:38:21.412638+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:38:21.412638+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6387,"log_dups_size":0,"ondisk_log_size":6387,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52303008,"num_objects":6385,"num_object_clones":0,"num_object_copies":25540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6385,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6385,"num_write_kb":51078,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":4123}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'6542","reported_seq":6602,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:20.877105+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:38:20.877105+0000","last_peered":"2024-09-05T16:38:20.877105+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:38:20.877105+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6542,"log_dups_size":0,"ondisk_log_size":6542,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53546736,"num_objects":6537,"num_object_clones":0,"num_object_copies":26148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6537,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6539,"num_write_kb":52300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4226}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'6348","reported_seq":6387,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:20.887738+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:38:20.887738+0000","last_peered":"2024-09-05T16:38:20.887738+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:38:20.887738+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6348,"log_dups_size":0,"ondisk_log_size":6348,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51993168,"num_objects":6347,"num_object_clones":0,"num_object_copies":25388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6347,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6347,"num_write_kb":50775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":4083}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'6394","reported_seq":7502,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:21.395122+0000","last_change":"2024-09-05T16:37:30.634570+0000","last_active":"2024-09-05T16:38:21.395122+0000","last_peered":"2024-09-05T16:38:21.395122+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:38:21.395122+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6394,"log_dups_size":0,"ondisk_log_size":6394,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52370000,"num_objects":6393,"num_object_clones":0,"num_object_copies":25572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1780,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6393,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6393,"num_write_kb":51143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":515,"num_bytes_recovered":4210688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4613}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'6530","reported_seq":6584,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:20.771696+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:38:20.771696+0000","last_peered":"2024-09-05T16:38:20.771696+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:38:20.771696+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6530,"log_dups_size":0,"ondisk_log_size":6530,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53458080,"num_objects":6526,"num_object_clones":0,"num_object_copies":26104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6526,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6528,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":4169}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'6391","reported_seq":6474,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:21.410274+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:38:21.410274+0000","last_peered":"2024-09-05T16:38:21.410274+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:38:21.410274+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6391,"log_dups_size":0,"ondisk_log_size":6391,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52327584,"num_objects":6388,"num_object_clones":0,"num_object_copies":25552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6388,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6389,"num_write_kb":51109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4068}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'6367","reported_seq":6435,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:21.359908+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:38:21.359908+0000","last_peered":"2024-09-05T16:38:21.359908+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:38:21.359908+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6367,"log_dups_size":0,"ondisk_log_size":6367,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52114592,"num_objects":6362,"num_object_clones":0,"num_object_copies":25448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6362,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6365,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4125}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":836201504,"num_objects":102080,"num_object_clones":0,"num_object_copies":408320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33393,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102080,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":96411,"num_write_kb":771181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":539,"num_bytes_recovered":4210688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1535275008,"data_stored":1535275008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":102136,"ondisk_log_size":102136,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084341,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":772692,"kb_used_data":407812,"kb_used_omap":4,"kb_used_meta":364859,"kb_avail":93599148,"statfs":{"total":96636764160,"available":95845527552,"internally_reserved":0,"allocated":417599488,"data_stored":417407447,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":373616339},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,1,2,5,1,7,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986218,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":522204,"kb_used_data":273932,"kb_used_omap":4,"kb_used_meta":248251,"kb_avail":93849636,"statfs":{"total":96636764160,"available":96102027264,"internally_reserved":0,"allocated":280506368,"data_stored":280314327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":254209746},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,3,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673024,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":759680,"kb_used_data":407792,"kb_used_omap":1,"kb_used_meta":351870,"kb_avail":93612160,"statfs":{"total":96636764160,"available":95858851840,"internally_reserved":0,"allocated":417579008,"data_stored":417386967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":360315333},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,5,0,0,0,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182455,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":760668,"kb_used_data":408332,"kb_used_omap":4,"kb_used_meta":352315,"kb_avail":93611172,"statfs":{"total":96636764160,"available":95857840128,"internally_reserved":0,"allocated":418131968,"data_stored":417939927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":360771277},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,0,3,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":418308096,"data_stored":418308096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":280895488,"data_stored":280895488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":418316288,"data_stored":418316288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":417755136,"data_stored":417755136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:38:25.060 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-05T16:38:25.060 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'6427", 'reported_seq': 6473, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:20.874536+0000', 'last_change': '2024-09-05T16:34:56.833497+0000', 'last_active': '2024-09-05T16:38:20.874536+0000', 'last_peered': '2024-09-05T16:38:20.874536+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.828289+0000', 'last_became_peered': '2024-09-05T16:34:56.828289+0000', 'last_unstale': '2024-09-05T16:38:20.874536+0000', 'last_undegraded': '2024-09-05T16:34:56.612160+0000', 'last_fullsized': '2024-09-05T16:34:56.557632+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6427, 'log_dups_size': 0, 'ondisk_log_size': 6427, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 52649984, 'num_objects': 6427, 'num_object_clones': 0, 'num_object_copies': 25708, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2262, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6427, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6427, 'num_write_kb': 51416, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 4165}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:38:25.061 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-05T16:38:25.061 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "35'6262", 'reported_seq': 6329, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:21.339928+0000', 'last_change': '2024-09-05T16:34:56.825031+0000', 'last_active': '2024-09-05T16:38:21.339928+0000', 'last_peered': '2024-09-05T16:38:21.339928+0000', 'last_clean': '2024-09-05T16:33:53.200762+0000', 'last_became_active': '2024-09-05T16:34:56.820292+0000', 'last_became_peered': '2024-09-05T16:34:56.820292+0000', 'last_unstale': '2024-09-05T16:38:21.339928+0000', 'last_undegraded': '2024-09-05T16:34:56.605776+0000', 'last_fullsized': '2024-09-05T16:34:56.559383+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6262, 'log_dups_size': 0, 'ondisk_log_size': 6262, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51262624, 'num_objects': 6258, 'num_object_clones': 0, 'num_object_copies': 25032, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2269, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6258, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6260, 'num_write_kb': 50076, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3991}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:38:25.061 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:38:25.061 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'6359", 'reported_seq': 6439, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:21.326411+0000', 'last_change': '2024-09-05T16:34:56.867900+0000', 'last_active': '2024-09-05T16:38:21.326411+0000', 'last_peered': '2024-09-05T16:38:21.326411+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:38:21.326411+0000', 'last_undegraded': '2024-09-05T16:34:56.590972+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6359, 'log_dups_size': 0, 'ondisk_log_size': 6359, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 52066896, 'num_objects': 6356, 'num_object_clones': 0, 'num_object_copies': 25424, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2251, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6356, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6358, 'num_write_kb': 50854, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 4106}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:38:25.061 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:38:25.061 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'6274", 'reported_seq': 6329, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:20.769552+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:38:20.769552+0000', 'last_peered': '2024-09-05T16:38:20.769552+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:38:20.769552+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6274, 'log_dups_size': 0, 'ondisk_log_size': 6274, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51369120, 'num_objects': 6271, 'num_object_clones': 0, 'num_object_copies': 25084, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2164, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6271, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6272, 'num_write_kb': 50173, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 4109}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:38:25.061 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:38:25.061 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'6349", 'reported_seq': 6395, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:20.671522+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:38:20.671522+0000', 'last_peered': '2024-09-05T16:38:20.671522+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:38:20.671522+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6349, 'log_dups_size': 0, 'ondisk_log_size': 6349, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 52011008, 'num_objects': 6349, 'num_object_clones': 0, 'num_object_copies': 25396, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6349, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6349, 'num_write_kb': 50792, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 4099}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:38:25.061 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:38:25.061 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'6314", 'reported_seq': 6384, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:20.854023+0000', 'last_change': '2024-09-05T16:34:56.903436+0000', 'last_active': '2024-09-05T16:38:20.854023+0000', 'last_peered': '2024-09-05T16:38:20.854023+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:38:20.854023+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6314, 'log_dups_size': 0, 'ondisk_log_size': 6314, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51672224, 'num_objects': 6308, 'num_object_clones': 0, 'num_object_copies': 25232, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2149, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6308, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6312, 'num_write_kb': 50476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 4161}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:38:25.062 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-05T16:38:25.062 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "35'6375", 'reported_seq': 6441, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:21.337303+0000', 'last_change': '2024-09-05T16:34:56.883319+0000', 'last_active': '2024-09-05T16:38:21.337303+0000', 'last_peered': '2024-09-05T16:38:21.337303+0000', 'last_clean': '2024-09-05T16:33:53.505305+0000', 'last_became_active': '2024-09-05T16:34:56.873389+0000', 'last_became_peered': '2024-09-05T16:34:56.873389+0000', 'last_unstale': '2024-09-05T16:38:21.337303+0000', 'last_undegraded': '2024-09-05T16:34:56.602567+0000', 'last_fullsized': '2024-09-05T16:34:56.559958+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6375, 'log_dups_size': 0, 'ondisk_log_size': 6375, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 52180128, 'num_objects': 6370, 'num_object_clones': 0, 'num_object_copies': 25480, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2241, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6370, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6373, 'num_write_kb': 50972, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 4131}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:38:25.062 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:38:25.062 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'6371", 'reported_seq': 6431, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:20.780083+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:38:20.780083+0000', 'last_peered': '2024-09-05T16:38:20.780083+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:38:20.780083+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6371, 'log_dups_size': 0, 'ondisk_log_size': 6371, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 52119872, 'num_objects': 6363, 'num_object_clones': 0, 'num_object_copies': 25452, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2264, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6363, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6368, 'num_write_kb': 50921, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 4102}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:38:25.062 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:38:25.062 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'6387", 'reported_seq': 6468, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:21.412638+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:38:21.412638+0000', 'last_peered': '2024-09-05T16:38:21.412638+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:38:21.412638+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6387, 'log_dups_size': 0, 'ondisk_log_size': 6387, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 52303008, 'num_objects': 6385, 'num_object_clones': 0, 'num_object_copies': 25540, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2264, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6385, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6385, 'num_write_kb': 51078, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 4123}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:38:25.062 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:38:25.062 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'6542", 'reported_seq': 6602, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:20.877105+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:38:20.877105+0000', 'last_peered': '2024-09-05T16:38:20.877105+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:38:20.877105+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6542, 'log_dups_size': 0, 'ondisk_log_size': 6542, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 53546736, 'num_objects': 6537, 'num_object_clones': 0, 'num_object_copies': 26148, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2314, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6537, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6539, 'num_write_kb': 52300, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 4226}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:38:25.063 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:38:25.063 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'6348", 'reported_seq': 6387, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:20.887738+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:38:20.887738+0000', 'last_peered': '2024-09-05T16:38:20.887738+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:38:20.887738+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6348, 'log_dups_size': 0, 'ondisk_log_size': 6348, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51993168, 'num_objects': 6347, 'num_object_clones': 0, 'num_object_copies': 25388, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2265, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6347, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6347, 'num_write_kb': 50775, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 4083}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:38:25.063 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-05T16:38:25.063 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'6394", 'reported_seq': 7502, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:21.395122+0000', 'last_change': '2024-09-05T16:37:30.634570+0000', 'last_active': '2024-09-05T16:38:21.395122+0000', 'last_peered': '2024-09-05T16:38:21.395122+0000', 'last_clean': '2024-09-05T16:33:45.082039+0000', 'last_became_active': '2024-09-05T16:34:56.792386+0000', 'last_became_peered': '2024-09-05T16:34:56.792386+0000', 'last_unstale': '2024-09-05T16:38:21.395122+0000', 'last_undegraded': '2024-09-05T16:34:56.600595+0000', 'last_fullsized': '2024-09-05T16:34:56.558203+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6394, 'log_dups_size': 0, 'ondisk_log_size': 6394, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 52370000, 'num_objects': 6393, 'num_object_clones': 0, 'num_object_copies': 25572, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1780, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6393, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6393, 'num_write_kb': 51143, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 515, 'num_bytes_recovered': 4210688, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 4613}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:38:25.063 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:38:25.063 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'6530", 'reported_seq': 6584, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:20.771696+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:38:20.771696+0000', 'last_peered': '2024-09-05T16:38:20.771696+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:38:20.771696+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6530, 'log_dups_size': 0, 'ondisk_log_size': 6530, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 53458080, 'num_objects': 6526, 'num_object_clones': 0, 'num_object_copies': 26104, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2359, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6526, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6528, 'num_write_kb': 52220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 4169}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:38:25.063 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:38:25.063 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'6391", 'reported_seq': 6474, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:21.410274+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:38:21.410274+0000', 'last_peered': '2024-09-05T16:38:21.410274+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:38:21.410274+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6391, 'log_dups_size': 0, 'ondisk_log_size': 6391, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 52327584, 'num_objects': 6388, 'num_object_clones': 0, 'num_object_copies': 25552, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2322, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6388, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6389, 'num_write_kb': 51109, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 4068}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:38:25.063 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:38:25.063 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'6367", 'reported_seq': 6435, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:21.359908+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:38:21.359908+0000', 'last_peered': '2024-09-05T16:38:21.359908+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:38:21.359908+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6367, 'log_dups_size': 0, 'ondisk_log_size': 6367, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 52114592, 'num_objects': 6362, 'num_object_clones': 0, 'num_object_copies': 25448, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6362, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6365, 'num_write_kb': 50908, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 4125}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:38:25.064 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:38:25.774 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:38:25.774 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 33393/408320 objects degraded (8.178%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":309,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":102080,"data_bytes":836201504,"bytes_used":2882809856,"bytes_avail":383664246784,"bytes_total":386547056640,"degraded_objects":33393,"degraded_total":408320,"degraded_ratio":0.081781445924764884,"recovering_objects_per_sec":7,"recovering_bytes_per_sec":63464,"recovering_keys_per_sec":0,"num_objects_recovered":93,"num_bytes_recovered":761856,"num_keys_recovered":0,"write_bytes_sec":1783139,"read_op_per_sec":0,"write_op_per_sec":217},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:38:25.801 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:38:25.802 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:38:26.402 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:38:26.402 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:38:26.427 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":180,"stamp":"2024-09-05T16:38:24.882386+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":836201504,"num_objects":102080,"num_object_clones":0,"num_object_copies":408320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33393,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102080,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":96411,"num_write_kb":771181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":539,"num_bytes_recovered":4210688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":102136,"ondisk_log_size":102136,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2815244,"kb_used_data":1497868,"kb_used_omap":14,"kb_used_meta":1317297,"kb_avail":374672116,"statfs":{"total":386547056640,"available":383664246784,"internally_reserved":0,"allocated":1533816832,"data_stored":1533048668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1348912695},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,8,4,4,11,8,13,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":21405696,"num_objects":2613,"num_object_clones":0,"num_object_copies":10452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-93,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2613,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2613,"num_write_kb":20904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":93,"num_bytes_recovered":761856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004496"},"pg_stats":[{"pgid":"1.f","version":"35'6427","reported_seq":6473,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:20.874536+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:38:20.874536+0000","last_peered":"2024-09-05T16:38:20.874536+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:38:20.874536+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6427,"log_dups_size":0,"ondisk_log_size":6427,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52649984,"num_objects":6427,"num_object_clones":0,"num_object_copies":25708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6427,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6427,"num_write_kb":51416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4165}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'6262","reported_seq":6329,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:21.339928+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:38:21.339928+0000","last_peered":"2024-09-05T16:38:21.339928+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:38:21.339928+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6262,"log_dups_size":0,"ondisk_log_size":6262,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51262624,"num_objects":6258,"num_object_clones":0,"num_object_copies":25032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6258,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6260,"num_write_kb":50076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3991}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'6359","reported_seq":6439,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:21.326411+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:38:21.326411+0000","last_peered":"2024-09-05T16:38:21.326411+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:38:21.326411+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6359,"log_dups_size":0,"ondisk_log_size":6359,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52066896,"num_objects":6356,"num_object_clones":0,"num_object_copies":25424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6356,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6358,"num_write_kb":50854,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":4106}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'6446","reported_seq":11169,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-05T16:38:20.616103+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:38:20.616103+0000","last_peered":"2024-09-05T16:38:20.616103+0000","last_clean":"2024-09-05T16:38:20.616103+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:38:20.616103+0000","last_undegraded":"2024-09-05T16:38:20.616103+0000","last_fullsized":"2024-09-05T16:38:20.616103+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":6446,"log_dups_size":0,"ondisk_log_size":6446,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52756480,"num_objects":6440,"num_object_clones":0,"num_object_copies":25760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6440,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":746,"num_write_kb":5968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'6274","reported_seq":6329,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:20.769552+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:38:20.769552+0000","last_peered":"2024-09-05T16:38:20.769552+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:38:20.769552+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6274,"log_dups_size":0,"ondisk_log_size":6274,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51369120,"num_objects":6271,"num_object_clones":0,"num_object_copies":25084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6271,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6272,"num_write_kb":50173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":4109}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'6349","reported_seq":6395,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:20.671522+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:38:20.671522+0000","last_peered":"2024-09-05T16:38:20.671522+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:38:20.671522+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6349,"log_dups_size":0,"ondisk_log_size":6349,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52011008,"num_objects":6349,"num_object_clones":0,"num_object_copies":25396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6349,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6349,"num_write_kb":50792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":4099}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'6314","reported_seq":6384,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:20.854023+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:38:20.854023+0000","last_peered":"2024-09-05T16:38:20.854023+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:38:20.854023+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6314,"log_dups_size":0,"ondisk_log_size":6314,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51672224,"num_objects":6308,"num_object_clones":0,"num_object_copies":25232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6308,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6312,"num_write_kb":50476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":4161}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'6375","reported_seq":6441,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:21.337303+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:38:21.337303+0000","last_peered":"2024-09-05T16:38:21.337303+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:38:21.337303+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6375,"log_dups_size":0,"ondisk_log_size":6375,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52180128,"num_objects":6370,"num_object_clones":0,"num_object_copies":25480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6370,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6373,"num_write_kb":50972,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4131}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'6371","reported_seq":6431,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:20.780083+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:38:20.780083+0000","last_peered":"2024-09-05T16:38:20.780083+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:38:20.780083+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6371,"log_dups_size":0,"ondisk_log_size":6371,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52119872,"num_objects":6363,"num_object_clones":0,"num_object_copies":25452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6363,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6368,"num_write_kb":50921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":4102}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'6387","reported_seq":6468,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:21.412638+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:38:21.412638+0000","last_peered":"2024-09-05T16:38:21.412638+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:38:21.412638+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6387,"log_dups_size":0,"ondisk_log_size":6387,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52303008,"num_objects":6385,"num_object_clones":0,"num_object_copies":25540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6385,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6385,"num_write_kb":51078,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":4123}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'6542","reported_seq":6602,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:20.877105+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:38:20.877105+0000","last_peered":"2024-09-05T16:38:20.877105+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:38:20.877105+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6542,"log_dups_size":0,"ondisk_log_size":6542,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53546736,"num_objects":6537,"num_object_clones":0,"num_object_copies":26148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6537,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6539,"num_write_kb":52300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4226}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'6348","reported_seq":6387,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:20.887738+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:38:20.887738+0000","last_peered":"2024-09-05T16:38:20.887738+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:38:20.887738+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6348,"log_dups_size":0,"ondisk_log_size":6348,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51993168,"num_objects":6347,"num_object_clones":0,"num_object_copies":25388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6347,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6347,"num_write_kb":50775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":4083}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'6394","reported_seq":7502,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:21.395122+0000","last_change":"2024-09-05T16:37:30.634570+0000","last_active":"2024-09-05T16:38:21.395122+0000","last_peered":"2024-09-05T16:38:21.395122+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:38:21.395122+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6394,"log_dups_size":0,"ondisk_log_size":6394,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52370000,"num_objects":6393,"num_object_clones":0,"num_object_copies":25572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1780,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6393,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6393,"num_write_kb":51143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":515,"num_bytes_recovered":4210688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4613}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'6530","reported_seq":6584,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:20.771696+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:38:20.771696+0000","last_peered":"2024-09-05T16:38:20.771696+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:38:20.771696+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6530,"log_dups_size":0,"ondisk_log_size":6530,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53458080,"num_objects":6526,"num_object_clones":0,"num_object_copies":26104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6526,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6528,"num_write_kb":52220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":4169}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'6391","reported_seq":6474,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:21.410274+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:38:21.410274+0000","last_peered":"2024-09-05T16:38:21.410274+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:38:21.410274+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6391,"log_dups_size":0,"ondisk_log_size":6391,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52327584,"num_objects":6388,"num_object_clones":0,"num_object_copies":25552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6388,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6389,"num_write_kb":51109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4068}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'6367","reported_seq":6435,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:21.359908+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:38:21.359908+0000","last_peered":"2024-09-05T16:38:21.359908+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:38:21.359908+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6367,"log_dups_size":0,"ondisk_log_size":6367,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52114592,"num_objects":6362,"num_object_clones":0,"num_object_copies":25448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6362,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6365,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4125}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":836201504,"num_objects":102080,"num_object_clones":0,"num_object_copies":408320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33393,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102080,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":96411,"num_write_kb":771181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":539,"num_bytes_recovered":4210688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1535275008,"data_stored":1535275008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":102136,"ondisk_log_size":102136,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084341,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":772692,"kb_used_data":407812,"kb_used_omap":4,"kb_used_meta":364859,"kb_avail":93599148,"statfs":{"total":96636764160,"available":95845527552,"internally_reserved":0,"allocated":417599488,"data_stored":417407447,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":373616339},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,1,2,5,1,7,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986218,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":522204,"kb_used_data":273932,"kb_used_omap":4,"kb_used_meta":248251,"kb_avail":93849636,"statfs":{"total":96636764160,"available":96102027264,"internally_reserved":0,"allocated":280506368,"data_stored":280314327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":254209746},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,3,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673024,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":759680,"kb_used_data":407792,"kb_used_omap":1,"kb_used_meta":351870,"kb_avail":93612160,"statfs":{"total":96636764160,"available":95858851840,"internally_reserved":0,"allocated":417579008,"data_stored":417386967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":360315333},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,5,0,0,0,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182455,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":760668,"kb_used_data":408332,"kb_used_omap":4,"kb_used_meta":352315,"kb_avail":93611172,"statfs":{"total":96636764160,"available":95857840128,"internally_reserved":0,"allocated":418131968,"data_stored":417939927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":360771277},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,0,3,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":418308096,"data_stored":418308096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":280895488,"data_stored":280895488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":418316288,"data_stored":418316288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":417755136,"data_stored":417755136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:38:29.429 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:38:30.026 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:38:30.026 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:38:30.047 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":182,"stamp":"2024-09-05T16:38:28.883779+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":831327288,"num_objects":101486,"num_object_clones":0,"num_object_copies":405944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101486,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":97751,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":596,"num_bytes_recovered":4677632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":103476,"ondisk_log_size":103476,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2831844,"kb_used_data":1496392,"kb_used_omap":14,"kb_used_meta":1335345,"kb_avail":374655516,"statfs":{"total":386547056640,"available":383647248384,"internally_reserved":0,"allocated":1532305408,"data_stored":1531537244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1367393847},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,3,5,18,18,12,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5939224,"num_objects":726,"num_object_clones":0,"num_object_copies":2904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-98,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":726,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2660,"num_write_kb":13537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":802816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004359"},"pg_stats":[{"pgid":"1.f","version":"35'6490","reported_seq":6536,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:25.792850+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:38:25.792850+0000","last_peered":"2024-09-05T16:38:25.792850+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:38:25.792850+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6490,"log_dups_size":0,"ondisk_log_size":6490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52543488,"num_objects":6414,"num_object_clones":0,"num_object_copies":25656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6414,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6490,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4152}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'6355","reported_seq":6423,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:26.436535+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:38:26.436535+0000","last_peered":"2024-09-05T16:38:26.436535+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:38:26.436535+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6355,"log_dups_size":0,"ondisk_log_size":6355,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50779296,"num_objects":6199,"num_object_clones":0,"num_object_copies":24796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6199,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6353,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3932}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'6444","reported_seq":6525,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:26.185785+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:38:26.185785+0000","last_peered":"2024-09-05T16:38:26.185785+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:38:26.185785+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6444,"log_dups_size":0,"ondisk_log_size":6444,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51673704,"num_objects":6309,"num_object_clones":0,"num_object_copies":25236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6309,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6443,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":4059}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'6536","reported_seq":11259,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-05T16:38:25.524568+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:38:25.524568+0000","last_peered":"2024-09-05T16:38:25.524568+0000","last_clean":"2024-09-05T16:38:25.524568+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:38:25.524568+0000","last_undegraded":"2024-09-05T16:38:25.524568+0000","last_fullsized":"2024-09-05T16:38:25.524568+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":6536,"log_dups_size":0,"ondisk_log_size":6536,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52412416,"num_objects":6398,"num_object_clones":0,"num_object_copies":25592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6398,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":836,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'6352","reported_seq":6411,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:25.726220+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:38:25.726220+0000","last_peered":"2024-09-05T16:38:25.726220+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:38:25.726220+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6352,"log_dups_size":0,"ondisk_log_size":6352,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51139744,"num_objects":6243,"num_object_clones":0,"num_object_copies":24972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6243,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6350,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":4081}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'6434","reported_seq":6480,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:25.734050+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:38:25.734050+0000","last_peered":"2024-09-05T16:38:25.734050+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:38:25.734050+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6434,"log_dups_size":0,"ondisk_log_size":6434,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51920896,"num_objects":6338,"num_object_clones":0,"num_object_copies":25352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6338,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6434,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":4088}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'6389","reported_seq":6459,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:26.272371+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:38:26.272371+0000","last_peered":"2024-09-05T16:38:26.272371+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:38:26.272371+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6389,"log_dups_size":0,"ondisk_log_size":6389,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51385504,"num_objects":6273,"num_object_clones":0,"num_object_copies":25092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6273,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6387,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":4126}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'6462","reported_seq":6531,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:26.231750+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:38:26.231750+0000","last_peered":"2024-09-05T16:38:26.231750+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:38:26.231750+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6462,"log_dups_size":0,"ondisk_log_size":6462,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51745952,"num_objects":6317,"num_object_clones":0,"num_object_copies":25268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6317,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6460,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4078}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'6449","reported_seq":6508,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:25.897075+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:38:25.897075+0000","last_peered":"2024-09-05T16:38:25.897075+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:38:25.897075+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6449,"log_dups_size":0,"ondisk_log_size":6449,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51808576,"num_objects":6325,"num_object_clones":0,"num_object_copies":25300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6325,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6446,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":4064}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'6486","reported_seq":6568,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:26.448972+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:38:26.448972+0000","last_peered":"2024-09-05T16:38:26.448972+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:38:26.448972+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6486,"log_dups_size":0,"ondisk_log_size":6486,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51803296,"num_objects":6324,"num_object_clones":0,"num_object_copies":25296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6324,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6484,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":4062}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'6625","reported_seq":6685,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:25.956068+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:38:25.956068+0000","last_peered":"2024-09-05T16:38:25.956068+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:38:25.956068+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6625,"log_dups_size":0,"ondisk_log_size":6625,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53227248,"num_objects":6498,"num_object_clones":0,"num_object_copies":25992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6498,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6622,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4187}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'6446","reported_seq":6486,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:25.950381+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:38:25.950381+0000","last_peered":"2024-09-05T16:38:25.950381+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:38:25.950381+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6446,"log_dups_size":0,"ondisk_log_size":6446,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51714640,"num_objects":6313,"num_object_clones":0,"num_object_copies":25252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6313,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6445,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":4049}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'6468","reported_seq":7690,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:26.438203+0000","last_change":"2024-09-05T16:37:30.634570+0000","last_active":"2024-09-05T16:38:26.438203+0000","last_peered":"2024-09-05T16:38:26.438203+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:38:26.438203+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6468,"log_dups_size":0,"ondisk_log_size":6468,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51944016,"num_objects":6341,"num_object_clones":0,"num_object_copies":25364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1723,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6341,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6467,"num_write_kb":51231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":572,"num_bytes_recovered":4677632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4618}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'6611","reported_seq":6667,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:25.645936+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:38:25.645936+0000","last_peered":"2024-09-05T16:38:25.645936+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:38:25.645936+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6611,"log_dups_size":0,"ondisk_log_size":6611,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53367968,"num_objects":6515,"num_object_clones":0,"num_object_copies":26060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6515,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6609,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":4158}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'6484","reported_seq":6566,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:26.445640+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:38:26.445640+0000","last_peered":"2024-09-05T16:38:26.445640+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:38:26.445640+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6484,"log_dups_size":0,"ondisk_log_size":6484,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52008096,"num_objects":6349,"num_object_clones":0,"num_object_copies":25396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6349,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6482,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4029}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'6445","reported_seq":6513,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:26.231604+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:38:26.231604+0000","last_peered":"2024-09-05T16:38:26.231604+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:38:26.231604+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6445,"log_dups_size":0,"ondisk_log_size":6445,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51852448,"num_objects":6330,"num_object_clones":0,"num_object_copies":25320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6330,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6443,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4093}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":831327288,"num_objects":101486,"num_object_clones":0,"num_object_copies":405944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101486,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":97751,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":596,"num_bytes_recovered":4677632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1529139200,"data_stored":1529139200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":103476,"ondisk_log_size":103476,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084342,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":759268,"kb_used_data":406848,"kb_used_omap":4,"kb_used_meta":352379,"kb_avail":93612572,"statfs":{"total":96636764160,"available":95859273728,"internally_reserved":0,"allocated":416612352,"data_stored":416420311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":360836819},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,2,4,8,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986219,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":546864,"kb_used_data":276384,"kb_used_omap":4,"kb_used_meta":270459,"kb_avail":93824976,"statfs":{"total":96636764160,"available":96076775424,"internally_reserved":0,"allocated":283017216,"data_stored":282825175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":276950738},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,8,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673025,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":762968,"kb_used_data":406912,"kb_used_omap":1,"kb_used_meta":356030,"kb_avail":93608872,"statfs":{"total":96636764160,"available":95855484928,"internally_reserved":0,"allocated":416677888,"data_stored":416485847,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":364575173},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,6,2,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182456,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":762744,"kb_used_data":406248,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93609096,"statfs":{"total":96636764160,"available":95855714304,"internally_reserved":0,"allocated":415997952,"data_stored":415805911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":365031117},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,8,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":415350784,"data_stored":415350784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":282660864,"data_stored":282660864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":415313920,"data_stored":415313920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":415813632,"data_stored":415813632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:38:30.049 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:38:30.604 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:38:30.605 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:38:30.627 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":182,"stamp":"2024-09-05T16:38:28.883779+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":831327288,"num_objects":101486,"num_object_clones":0,"num_object_copies":405944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101486,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":97751,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":596,"num_bytes_recovered":4677632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":103476,"ondisk_log_size":103476,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2831844,"kb_used_data":1496392,"kb_used_omap":14,"kb_used_meta":1335345,"kb_avail":374655516,"statfs":{"total":386547056640,"available":383647248384,"internally_reserved":0,"allocated":1532305408,"data_stored":1531537244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1367393847},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,3,5,18,18,12,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5939224,"num_objects":726,"num_object_clones":0,"num_object_copies":2904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-98,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":726,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2660,"num_write_kb":13537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":802816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004359"},"pg_stats":[{"pgid":"1.f","version":"35'6490","reported_seq":6536,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:25.792850+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:38:25.792850+0000","last_peered":"2024-09-05T16:38:25.792850+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:38:25.792850+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6490,"log_dups_size":0,"ondisk_log_size":6490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52543488,"num_objects":6414,"num_object_clones":0,"num_object_copies":25656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6414,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6490,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4152}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'6355","reported_seq":6423,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:26.436535+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:38:26.436535+0000","last_peered":"2024-09-05T16:38:26.436535+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:38:26.436535+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6355,"log_dups_size":0,"ondisk_log_size":6355,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50779296,"num_objects":6199,"num_object_clones":0,"num_object_copies":24796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6199,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6353,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3932}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'6444","reported_seq":6525,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:26.185785+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:38:26.185785+0000","last_peered":"2024-09-05T16:38:26.185785+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:38:26.185785+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6444,"log_dups_size":0,"ondisk_log_size":6444,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51673704,"num_objects":6309,"num_object_clones":0,"num_object_copies":25236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6309,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6443,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":4059}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'6536","reported_seq":11259,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-05T16:38:25.524568+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:38:25.524568+0000","last_peered":"2024-09-05T16:38:25.524568+0000","last_clean":"2024-09-05T16:38:25.524568+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:38:25.524568+0000","last_undegraded":"2024-09-05T16:38:25.524568+0000","last_fullsized":"2024-09-05T16:38:25.524568+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":6536,"log_dups_size":0,"ondisk_log_size":6536,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52412416,"num_objects":6398,"num_object_clones":0,"num_object_copies":25592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6398,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":836,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'6352","reported_seq":6411,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:25.726220+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:38:25.726220+0000","last_peered":"2024-09-05T16:38:25.726220+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:38:25.726220+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6352,"log_dups_size":0,"ondisk_log_size":6352,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51139744,"num_objects":6243,"num_object_clones":0,"num_object_copies":24972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6243,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6350,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":4081}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'6434","reported_seq":6480,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:25.734050+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:38:25.734050+0000","last_peered":"2024-09-05T16:38:25.734050+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:38:25.734050+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6434,"log_dups_size":0,"ondisk_log_size":6434,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51920896,"num_objects":6338,"num_object_clones":0,"num_object_copies":25352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6338,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6434,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":4088}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'6389","reported_seq":6459,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:26.272371+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:38:26.272371+0000","last_peered":"2024-09-05T16:38:26.272371+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:38:26.272371+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6389,"log_dups_size":0,"ondisk_log_size":6389,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51385504,"num_objects":6273,"num_object_clones":0,"num_object_copies":25092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6273,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6387,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":4126}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'6462","reported_seq":6531,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:26.231750+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:38:26.231750+0000","last_peered":"2024-09-05T16:38:26.231750+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:38:26.231750+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6462,"log_dups_size":0,"ondisk_log_size":6462,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51745952,"num_objects":6317,"num_object_clones":0,"num_object_copies":25268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6317,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6460,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4078}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'6449","reported_seq":6508,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:25.897075+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:38:25.897075+0000","last_peered":"2024-09-05T16:38:25.897075+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:38:25.897075+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6449,"log_dups_size":0,"ondisk_log_size":6449,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51808576,"num_objects":6325,"num_object_clones":0,"num_object_copies":25300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6325,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6446,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":4064}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'6486","reported_seq":6568,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:26.448972+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:38:26.448972+0000","last_peered":"2024-09-05T16:38:26.448972+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:38:26.448972+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6486,"log_dups_size":0,"ondisk_log_size":6486,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51803296,"num_objects":6324,"num_object_clones":0,"num_object_copies":25296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6324,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6484,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":4062}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'6625","reported_seq":6685,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:25.956068+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:38:25.956068+0000","last_peered":"2024-09-05T16:38:25.956068+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:38:25.956068+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6625,"log_dups_size":0,"ondisk_log_size":6625,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53227248,"num_objects":6498,"num_object_clones":0,"num_object_copies":25992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6498,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6622,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4187}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'6446","reported_seq":6486,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:25.950381+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:38:25.950381+0000","last_peered":"2024-09-05T16:38:25.950381+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:38:25.950381+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6446,"log_dups_size":0,"ondisk_log_size":6446,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51714640,"num_objects":6313,"num_object_clones":0,"num_object_copies":25252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6313,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6445,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":4049}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'6468","reported_seq":7690,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:26.438203+0000","last_change":"2024-09-05T16:37:30.634570+0000","last_active":"2024-09-05T16:38:26.438203+0000","last_peered":"2024-09-05T16:38:26.438203+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:38:26.438203+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6468,"log_dups_size":0,"ondisk_log_size":6468,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51944016,"num_objects":6341,"num_object_clones":0,"num_object_copies":25364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1723,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6341,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6467,"num_write_kb":51231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":572,"num_bytes_recovered":4677632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4618}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'6611","reported_seq":6667,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:25.645936+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:38:25.645936+0000","last_peered":"2024-09-05T16:38:25.645936+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:38:25.645936+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6611,"log_dups_size":0,"ondisk_log_size":6611,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53367968,"num_objects":6515,"num_object_clones":0,"num_object_copies":26060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6515,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6609,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":4158}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'6484","reported_seq":6566,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:26.445640+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:38:26.445640+0000","last_peered":"2024-09-05T16:38:26.445640+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:38:26.445640+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6484,"log_dups_size":0,"ondisk_log_size":6484,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52008096,"num_objects":6349,"num_object_clones":0,"num_object_copies":25396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6349,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6482,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4029}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'6445","reported_seq":6513,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:26.231604+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:38:26.231604+0000","last_peered":"2024-09-05T16:38:26.231604+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:38:26.231604+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6445,"log_dups_size":0,"ondisk_log_size":6445,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51852448,"num_objects":6330,"num_object_clones":0,"num_object_copies":25320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6330,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6443,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4093}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":831327288,"num_objects":101486,"num_object_clones":0,"num_object_copies":405944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101486,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":97751,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":596,"num_bytes_recovered":4677632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1529139200,"data_stored":1529139200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":103476,"ondisk_log_size":103476,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084342,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":759268,"kb_used_data":406848,"kb_used_omap":4,"kb_used_meta":352379,"kb_avail":93612572,"statfs":{"total":96636764160,"available":95859273728,"internally_reserved":0,"allocated":416612352,"data_stored":416420311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":360836819},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,2,4,8,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986219,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":546864,"kb_used_data":276384,"kb_used_omap":4,"kb_used_meta":270459,"kb_avail":93824976,"statfs":{"total":96636764160,"available":96076775424,"internally_reserved":0,"allocated":283017216,"data_stored":282825175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":276950738},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,8,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673025,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":762968,"kb_used_data":406912,"kb_used_omap":1,"kb_used_meta":356030,"kb_avail":93608872,"statfs":{"total":96636764160,"available":95855484928,"internally_reserved":0,"allocated":416677888,"data_stored":416485847,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":364575173},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,6,2,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182456,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":762744,"kb_used_data":406248,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93609096,"statfs":{"total":96636764160,"available":95855714304,"internally_reserved":0,"allocated":415997952,"data_stored":415805911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":365031117},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,8,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":415350784,"data_stored":415350784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":282660864,"data_stored":282660864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":415313920,"data_stored":415313920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":415813632,"data_stored":415813632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:38:30.628 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-05T16:38:30.628 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'6490", 'reported_seq': 6536, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:25.792850+0000', 'last_change': '2024-09-05T16:34:56.833497+0000', 'last_active': '2024-09-05T16:38:25.792850+0000', 'last_peered': '2024-09-05T16:38:25.792850+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.828289+0000', 'last_became_peered': '2024-09-05T16:34:56.828289+0000', 'last_unstale': '2024-09-05T16:38:25.792850+0000', 'last_undegraded': '2024-09-05T16:34:56.612160+0000', 'last_fullsized': '2024-09-05T16:34:56.557632+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6490, 'log_dups_size': 0, 'ondisk_log_size': 6490, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 52543488, 'num_objects': 6414, 'num_object_clones': 0, 'num_object_copies': 25656, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2262, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6414, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6490, 'num_write_kb': 51616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 4152}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:38:30.629 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-05T16:38:30.629 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "35'6355", 'reported_seq': 6423, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:26.436535+0000', 'last_change': '2024-09-05T16:34:56.825031+0000', 'last_active': '2024-09-05T16:38:26.436535+0000', 'last_peered': '2024-09-05T16:38:26.436535+0000', 'last_clean': '2024-09-05T16:33:53.200762+0000', 'last_became_active': '2024-09-05T16:34:56.820292+0000', 'last_became_peered': '2024-09-05T16:34:56.820292+0000', 'last_unstale': '2024-09-05T16:38:26.436535+0000', 'last_undegraded': '2024-09-05T16:34:56.605776+0000', 'last_fullsized': '2024-09-05T16:34:56.559383+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6355, 'log_dups_size': 0, 'ondisk_log_size': 6355, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50779296, 'num_objects': 6199, 'num_object_clones': 0, 'num_object_copies': 24796, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2269, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6199, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6353, 'num_write_kb': 50212, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3932}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:38:30.629 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:38:30.629 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'6444", 'reported_seq': 6525, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:26.185785+0000', 'last_change': '2024-09-05T16:34:56.867900+0000', 'last_active': '2024-09-05T16:38:26.185785+0000', 'last_peered': '2024-09-05T16:38:26.185785+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:38:26.185785+0000', 'last_undegraded': '2024-09-05T16:34:56.590972+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6444, 'log_dups_size': 0, 'ondisk_log_size': 6444, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51673704, 'num_objects': 6309, 'num_object_clones': 0, 'num_object_copies': 25236, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2251, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6309, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 6443, 'num_write_kb': 50999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 4059}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:38:30.629 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:38:30.629 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'6352", 'reported_seq': 6411, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:25.726220+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:38:25.726220+0000', 'last_peered': '2024-09-05T16:38:25.726220+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:38:25.726220+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6352, 'log_dups_size': 0, 'ondisk_log_size': 6352, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51139744, 'num_objects': 6243, 'num_object_clones': 0, 'num_object_copies': 24972, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2164, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6243, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6350, 'num_write_kb': 50373, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 4081}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:38:30.629 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:38:30.629 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'6434", 'reported_seq': 6480, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:25.734050+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:38:25.734050+0000', 'last_peered': '2024-09-05T16:38:25.734050+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:38:25.734050+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6434, 'log_dups_size': 0, 'ondisk_log_size': 6434, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51920896, 'num_objects': 6338, 'num_object_clones': 0, 'num_object_copies': 25352, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6338, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6434, 'num_write_kb': 51088, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 4088}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:38:30.629 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:38:30.630 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'6389", 'reported_seq': 6459, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:26.272371+0000', 'last_change': '2024-09-05T16:34:56.903436+0000', 'last_active': '2024-09-05T16:38:26.272371+0000', 'last_peered': '2024-09-05T16:38:26.272371+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:38:26.272371+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6389, 'log_dups_size': 0, 'ondisk_log_size': 6389, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51385504, 'num_objects': 6273, 'num_object_clones': 0, 'num_object_copies': 25092, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2149, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6273, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6387, 'num_write_kb': 50636, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 4126}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:38:30.630 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-05T16:38:30.630 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "35'6462", 'reported_seq': 6531, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:26.231750+0000', 'last_change': '2024-09-05T16:34:56.883319+0000', 'last_active': '2024-09-05T16:38:26.231750+0000', 'last_peered': '2024-09-05T16:38:26.231750+0000', 'last_clean': '2024-09-05T16:33:53.505305+0000', 'last_became_active': '2024-09-05T16:34:56.873389+0000', 'last_became_peered': '2024-09-05T16:34:56.873389+0000', 'last_unstale': '2024-09-05T16:38:26.231750+0000', 'last_undegraded': '2024-09-05T16:34:56.602567+0000', 'last_fullsized': '2024-09-05T16:34:56.559958+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6462, 'log_dups_size': 0, 'ondisk_log_size': 6462, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51745952, 'num_objects': 6317, 'num_object_clones': 0, 'num_object_copies': 25268, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2241, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6317, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6460, 'num_write_kb': 51108, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 4078}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:38:30.630 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:38:30.630 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'6449", 'reported_seq': 6508, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:25.897075+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:38:25.897075+0000', 'last_peered': '2024-09-05T16:38:25.897075+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:38:25.897075+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6449, 'log_dups_size': 0, 'ondisk_log_size': 6449, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51808576, 'num_objects': 6325, 'num_object_clones': 0, 'num_object_copies': 25300, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2264, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6325, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6446, 'num_write_kb': 51081, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 4064}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:38:30.630 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:38:30.630 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'6486", 'reported_seq': 6568, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:26.448972+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:38:26.448972+0000', 'last_peered': '2024-09-05T16:38:26.448972+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:38:26.448972+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6486, 'log_dups_size': 0, 'ondisk_log_size': 6486, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51803296, 'num_objects': 6324, 'num_object_clones': 0, 'num_object_copies': 25296, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2264, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6324, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6484, 'num_write_kb': 51230, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 4062}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:38:30.630 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:38:30.630 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'6625", 'reported_seq': 6685, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:25.956068+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:38:25.956068+0000', 'last_peered': '2024-09-05T16:38:25.956068+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:38:25.956068+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6625, 'log_dups_size': 0, 'ondisk_log_size': 6625, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 53227248, 'num_objects': 6498, 'num_object_clones': 0, 'num_object_copies': 25992, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2314, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6498, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6622, 'num_write_kb': 52476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 4187}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:38:30.630 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:38:30.631 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'6446", 'reported_seq': 6486, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:25.950381+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:38:25.950381+0000', 'last_peered': '2024-09-05T16:38:25.950381+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:38:25.950381+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6446, 'log_dups_size': 0, 'ondisk_log_size': 6446, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51714640, 'num_objects': 6313, 'num_object_clones': 0, 'num_object_copies': 25252, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2265, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6313, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6445, 'num_write_kb': 51031, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 4049}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:38:30.631 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-05T16:38:30.631 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'6468", 'reported_seq': 7690, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:26.438203+0000', 'last_change': '2024-09-05T16:37:30.634570+0000', 'last_active': '2024-09-05T16:38:26.438203+0000', 'last_peered': '2024-09-05T16:38:26.438203+0000', 'last_clean': '2024-09-05T16:33:45.082039+0000', 'last_became_active': '2024-09-05T16:34:56.792386+0000', 'last_became_peered': '2024-09-05T16:34:56.792386+0000', 'last_unstale': '2024-09-05T16:38:26.438203+0000', 'last_undegraded': '2024-09-05T16:34:56.600595+0000', 'last_fullsized': '2024-09-05T16:34:56.558203+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6468, 'log_dups_size': 0, 'ondisk_log_size': 6468, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51944016, 'num_objects': 6341, 'num_object_clones': 0, 'num_object_copies': 25364, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1723, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6341, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6467, 'num_write_kb': 51231, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 572, 'num_bytes_recovered': 4677632, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 4618}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:38:30.631 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:38:30.631 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'6611", 'reported_seq': 6667, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:25.645936+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:38:25.645936+0000', 'last_peered': '2024-09-05T16:38:25.645936+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:38:25.645936+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6611, 'log_dups_size': 0, 'ondisk_log_size': 6611, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 53367968, 'num_objects': 6515, 'num_object_clones': 0, 'num_object_copies': 26060, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2359, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6515, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6609, 'num_write_kb': 52500, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 4158}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:38:30.631 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:38:30.631 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'6484", 'reported_seq': 6566, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:26.445640+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:38:26.445640+0000', 'last_peered': '2024-09-05T16:38:26.445640+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:38:26.445640+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6484, 'log_dups_size': 0, 'ondisk_log_size': 6484, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 52008096, 'num_objects': 6349, 'num_object_clones': 0, 'num_object_copies': 25396, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2322, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6349, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6482, 'num_write_kb': 51325, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 4029}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:38:30.631 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:38:30.631 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'6445", 'reported_seq': 6513, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:26.231604+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:38:26.231604+0000', 'last_peered': '2024-09-05T16:38:26.231604+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:38:26.231604+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6445, 'log_dups_size': 0, 'ondisk_log_size': 6445, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51852448, 'num_objects': 6330, 'num_object_clones': 0, 'num_object_copies': 25320, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6330, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6443, 'num_write_kb': 51092, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 4093}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:38:30.632 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:38:31.376 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:38:31.377 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 33336/405628 objects degraded (8.218%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":314,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+clean","count":1}],"num_pgs":16,"num_pools":1,"num_objects":101407,"data_bytes":830680120,"bytes_used":2903687168,"bytes_avail":383643369472,"bytes_total":386547056640,"degraded_objects":33336,"degraded_total":405628,"degraded_ratio":0.082183675683138238,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":66876,"recovering_keys_per_sec":0,"num_objects_recovered":98,"num_bytes_recovered":802816,"num_keys_recovered":0,"read_bytes_sec":85,"write_bytes_sec":1154731,"read_op_per_sec":0,"write_op_per_sec":228},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:38:31.404 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:38:31.405 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:38:32.025 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:38:32.025 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:38:32.051 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":183,"stamp":"2024-09-05T16:38:30.884613+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":830680120,"num_objects":101407,"num_object_clones":0,"num_object_copies":405628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101407,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":97830,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":596,"num_bytes_recovered":4677632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":103555,"ondisk_log_size":103555,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2835632,"kb_used_data":1496020,"kb_used_omap":14,"kb_used_meta":1339505,"kb_avail":374651728,"statfs":{"total":386547056640,"available":383643369472,"internally_reserved":0,"allocated":1531924480,"data_stored":1531156316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1371653687},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,3,4,18,13,9,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5292056,"num_objects":647,"num_object_clones":0,"num_object_copies":2588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-98,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":647,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2739,"num_write_kb":13537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":802816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004429"},"pg_stats":[{"pgid":"1.f","version":"35'6490","reported_seq":6536,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:25.792850+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:38:25.792850+0000","last_peered":"2024-09-05T16:38:25.792850+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:38:25.792850+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6490,"log_dups_size":0,"ondisk_log_size":6490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52543488,"num_objects":6414,"num_object_clones":0,"num_object_copies":25656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6414,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6490,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4152}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'6355","reported_seq":6423,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:26.436535+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:38:26.436535+0000","last_peered":"2024-09-05T16:38:26.436535+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:38:26.436535+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6355,"log_dups_size":0,"ondisk_log_size":6355,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50779296,"num_objects":6199,"num_object_clones":0,"num_object_copies":24796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6199,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6353,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3932}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'6444","reported_seq":6525,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:26.185785+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:38:26.185785+0000","last_peered":"2024-09-05T16:38:26.185785+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:38:26.185785+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6444,"log_dups_size":0,"ondisk_log_size":6444,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51673704,"num_objects":6309,"num_object_clones":0,"num_object_copies":25236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6309,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6443,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":4059}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'6615","reported_seq":11338,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-05T16:38:30.563703+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:38:30.563703+0000","last_peered":"2024-09-05T16:38:30.563703+0000","last_clean":"2024-09-05T16:38:30.563703+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:38:30.563703+0000","last_undegraded":"2024-09-05T16:38:30.563703+0000","last_fullsized":"2024-09-05T16:38:30.563703+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":6615,"log_dups_size":0,"ondisk_log_size":6615,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51765248,"num_objects":6319,"num_object_clones":0,"num_object_copies":25276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6319,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":915,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'6352","reported_seq":6411,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:25.726220+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:38:25.726220+0000","last_peered":"2024-09-05T16:38:25.726220+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:38:25.726220+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6352,"log_dups_size":0,"ondisk_log_size":6352,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51139744,"num_objects":6243,"num_object_clones":0,"num_object_copies":24972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6243,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6350,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":4081}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'6434","reported_seq":6480,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:25.734050+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:38:25.734050+0000","last_peered":"2024-09-05T16:38:25.734050+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:38:25.734050+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6434,"log_dups_size":0,"ondisk_log_size":6434,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51920896,"num_objects":6338,"num_object_clones":0,"num_object_copies":25352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6338,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6434,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":4088}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'6389","reported_seq":6459,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:26.272371+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:38:26.272371+0000","last_peered":"2024-09-05T16:38:26.272371+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:38:26.272371+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6389,"log_dups_size":0,"ondisk_log_size":6389,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51385504,"num_objects":6273,"num_object_clones":0,"num_object_copies":25092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6273,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6387,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":4126}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'6462","reported_seq":6531,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:26.231750+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:38:26.231750+0000","last_peered":"2024-09-05T16:38:26.231750+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:38:26.231750+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6462,"log_dups_size":0,"ondisk_log_size":6462,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51745952,"num_objects":6317,"num_object_clones":0,"num_object_copies":25268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6317,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6460,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4078}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'6449","reported_seq":6508,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:25.897075+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:38:25.897075+0000","last_peered":"2024-09-05T16:38:25.897075+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:38:25.897075+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6449,"log_dups_size":0,"ondisk_log_size":6449,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51808576,"num_objects":6325,"num_object_clones":0,"num_object_copies":25300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6325,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6446,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":4064}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'6486","reported_seq":6568,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:26.448972+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:38:26.448972+0000","last_peered":"2024-09-05T16:38:26.448972+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:38:26.448972+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6486,"log_dups_size":0,"ondisk_log_size":6486,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51803296,"num_objects":6324,"num_object_clones":0,"num_object_copies":25296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6324,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6484,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":4062}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'6625","reported_seq":6685,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:25.956068+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:38:25.956068+0000","last_peered":"2024-09-05T16:38:25.956068+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:38:25.956068+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6625,"log_dups_size":0,"ondisk_log_size":6625,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53227248,"num_objects":6498,"num_object_clones":0,"num_object_copies":25992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6498,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6622,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4187}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'6446","reported_seq":6486,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:25.950381+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:38:25.950381+0000","last_peered":"2024-09-05T16:38:25.950381+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:38:25.950381+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6446,"log_dups_size":0,"ondisk_log_size":6446,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51714640,"num_objects":6313,"num_object_clones":0,"num_object_copies":25252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6313,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6445,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":4049}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'6468","reported_seq":7690,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:26.438203+0000","last_change":"2024-09-05T16:37:30.634570+0000","last_active":"2024-09-05T16:38:26.438203+0000","last_peered":"2024-09-05T16:38:26.438203+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:38:26.438203+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6468,"log_dups_size":0,"ondisk_log_size":6468,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51944016,"num_objects":6341,"num_object_clones":0,"num_object_copies":25364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1723,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6341,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6467,"num_write_kb":51231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":572,"num_bytes_recovered":4677632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4618}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'6611","reported_seq":6667,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:25.645936+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:38:25.645936+0000","last_peered":"2024-09-05T16:38:25.645936+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:38:25.645936+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6611,"log_dups_size":0,"ondisk_log_size":6611,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53367968,"num_objects":6515,"num_object_clones":0,"num_object_copies":26060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6515,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6609,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":4158}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'6484","reported_seq":6566,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:26.445640+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:38:26.445640+0000","last_peered":"2024-09-05T16:38:26.445640+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:38:26.445640+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6484,"log_dups_size":0,"ondisk_log_size":6484,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52008096,"num_objects":6349,"num_object_clones":0,"num_object_copies":25396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6349,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6482,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4029}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'6445","reported_seq":6513,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:26.231604+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:38:26.231604+0000","last_peered":"2024-09-05T16:38:26.231604+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:38:26.231604+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6445,"log_dups_size":0,"ondisk_log_size":6445,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51852448,"num_objects":6330,"num_object_clones":0,"num_object_copies":25320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6330,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6443,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4093}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":830680120,"num_objects":101407,"num_object_clones":0,"num_object_copies":405628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101407,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":97830,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":596,"num_bytes_recovered":4677632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1528758272,"data_stored":1528758272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":103555,"ondisk_log_size":103555,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084342,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":759268,"kb_used_data":406848,"kb_used_omap":4,"kb_used_meta":352379,"kb_avail":93612572,"statfs":{"total":96636764160,"available":95859273728,"internally_reserved":0,"allocated":416612352,"data_stored":416420311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":360836819},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,2,4,8,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986220,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":550652,"kb_used_data":276012,"kb_used_omap":4,"kb_used_meta":274619,"kb_avail":93821188,"statfs":{"total":96636764160,"available":96072896512,"internally_reserved":0,"allocated":282636288,"data_stored":282444247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":281210578},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673025,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":762968,"kb_used_data":406912,"kb_used_omap":1,"kb_used_meta":356030,"kb_avail":93608872,"statfs":{"total":96636764160,"available":95855484928,"internally_reserved":0,"allocated":416677888,"data_stored":416485847,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":364575173},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,6,2,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182456,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":762744,"kb_used_data":406248,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93609096,"statfs":{"total":96636764160,"available":95855714304,"internally_reserved":0,"allocated":415997952,"data_stored":415805911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":365031117},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,8,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":415350784,"data_stored":415350784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":282279936,"data_stored":282279936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":415313920,"data_stored":415313920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":415813632,"data_stored":415813632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:38:35.053 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:38:35.660 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:38:35.661 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:38:35.680 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":185,"stamp":"2024-09-05T16:38:34.886474+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":820579384,"num_objects":100174,"num_object_clones":0,"num_object_copies":400696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33272,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100174,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":99063,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":660,"num_bytes_recovered":5201920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":104788,"ondisk_log_size":104788,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2831424,"kb_used_data":1479348,"kb_used_omap":14,"kb_used_meta":1351985,"kb_avail":374655936,"statfs":{"total":386547056640,"available":383647678464,"internally_reserved":0,"allocated":1514852352,"data_stored":1514084188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1384433207},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[7,0,2,9,12,19,6,4,0,0,7],"upper_bound":2048},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-15622120,"num_objects":-1906,"num_object_clones":0,"num_object_copies":-7624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-121,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1906,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2652,"num_write_kb":2977,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":121,"num_bytes_recovered":991232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005062"},"pg_stats":[{"pgid":"1.f","version":"35'6588","reported_seq":6634,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:31.006418+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:38:31.006418+0000","last_peered":"2024-09-05T16:38:31.006418+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:38:31.006418+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6588,"log_dups_size":0,"ondisk_log_size":6588,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51740672,"num_objects":6316,"num_object_clones":0,"num_object_copies":25264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6316,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6588,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4054}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'6436","reported_seq":6504,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:31.485647+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:38:31.485647+0000","last_peered":"2024-09-05T16:38:31.485647+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:38:31.485647+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6436,"log_dups_size":0,"ondisk_log_size":6436,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50115744,"num_objects":6118,"num_object_clones":0,"num_object_copies":24472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6118,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6434,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3851}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'6509","reported_seq":6590,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:31.174591+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:38:31.174591+0000","last_peered":"2024-09-05T16:38:31.174591+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:38:31.174591+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6509,"log_dups_size":0,"ondisk_log_size":6509,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51141224,"num_objects":6244,"num_object_clones":0,"num_object_copies":24976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6244,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6508,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3994}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'6615","reported_seq":11338,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-05T16:38:30.563703+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:38:30.563703+0000","last_peered":"2024-09-05T16:38:30.563703+0000","last_clean":"2024-09-05T16:38:30.563703+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:38:30.563703+0000","last_undegraded":"2024-09-05T16:38:30.563703+0000","last_fullsized":"2024-09-05T16:38:30.563703+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":6615,"log_dups_size":0,"ondisk_log_size":6615,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51765248,"num_objects":6319,"num_object_clones":0,"num_object_copies":25276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6319,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":915,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'6435","reported_seq":6494,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:30.919236+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:38:30.919236+0000","last_peered":"2024-09-05T16:38:30.919236+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:38:30.919236+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6435,"log_dups_size":0,"ondisk_log_size":6435,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50459808,"num_objects":6160,"num_object_clones":0,"num_object_copies":24640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6160,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6433,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3998}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'6512","reported_seq":6558,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:30.866573+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:38:30.866573+0000","last_peered":"2024-09-05T16:38:30.866573+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:38:30.866573+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6512,"log_dups_size":0,"ondisk_log_size":6512,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51281920,"num_objects":6260,"num_object_clones":0,"num_object_copies":25040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6260,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6512,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":4010}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'6457","reported_seq":6527,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:31.458721+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:38:31.458721+0000","last_peered":"2024-09-05T16:38:31.458721+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:38:31.458721+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6457,"log_dups_size":0,"ondisk_log_size":6457,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50828448,"num_objects":6205,"num_object_clones":0,"num_object_copies":24820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6205,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6455,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":4058}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'6550","reported_seq":6619,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:31.419992+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:38:31.419992+0000","last_peered":"2024-09-05T16:38:31.419992+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:38:31.419992+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6550,"log_dups_size":0,"ondisk_log_size":6550,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51025056,"num_objects":6229,"num_object_clones":0,"num_object_copies":24916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6229,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6548,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3990}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'6524","reported_seq":6585,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:30.796087+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:38:30.796087+0000","last_peered":"2024-09-05T16:38:30.796087+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:38:30.796087+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6524,"log_dups_size":0,"ondisk_log_size":6524,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51194176,"num_objects":6250,"num_object_clones":0,"num_object_copies":25000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6250,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6521,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3989}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'6579","reported_seq":6661,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:31.482226+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:38:31.482226+0000","last_peered":"2024-09-05T16:38:31.482226+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:38:31.482226+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6579,"log_dups_size":0,"ondisk_log_size":6579,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51041440,"num_objects":6231,"num_object_clones":0,"num_object_copies":24924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6231,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6577,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3969}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'6699","reported_seq":6759,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:31.005015+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:38:31.005015+0000","last_peered":"2024-09-05T16:38:31.005015+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:38:31.005015+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6699,"log_dups_size":0,"ondisk_log_size":6699,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52621040,"num_objects":6424,"num_object_clones":0,"num_object_copies":25696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6424,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6696,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4113}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'6528","reported_seq":6566,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:31.006199+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:38:31.006199+0000","last_peered":"2024-09-05T16:38:31.006199+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:38:31.006199+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6528,"log_dups_size":0,"ondisk_log_size":6528,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51042896,"num_objects":6231,"num_object_clones":0,"num_object_copies":24924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6231,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6527,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3967}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'6546","reported_seq":7896,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:31.465915+0000","last_change":"2024-09-05T16:37:30.634570+0000","last_active":"2024-09-05T16:38:31.465915+0000","last_peered":"2024-09-05T16:38:31.465915+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:38:31.465915+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6546,"log_dups_size":0,"ondisk_log_size":6546,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51305040,"num_objects":6263,"num_object_clones":0,"num_object_copies":25052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1659,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6263,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6545,"num_write_kb":51231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":636,"num_bytes_recovered":5201920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4604}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'6710","reported_seq":6766,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:30.916725+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:38:30.916725+0000","last_peered":"2024-09-05T16:38:30.916725+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:38:30.916725+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6710,"log_dups_size":0,"ondisk_log_size":6710,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52556960,"num_objects":6416,"num_object_clones":0,"num_object_copies":25664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6416,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6708,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":4059}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'6560","reported_seq":6643,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:31.393987+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:38:31.393987+0000","last_peered":"2024-09-05T16:38:31.393987+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:38:31.393987+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6560,"log_dups_size":0,"ondisk_log_size":6560,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51385504,"num_objects":6273,"num_object_clones":0,"num_object_copies":25092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6273,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6558,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3953}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'6540","reported_seq":6608,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:31.363462+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:38:31.363462+0000","last_peered":"2024-09-05T16:38:31.363462+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:38:31.363462+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6540,"log_dups_size":0,"ondisk_log_size":6540,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51074208,"num_objects":6235,"num_object_clones":0,"num_object_copies":24940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6235,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6538,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3998}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":820579384,"num_objects":100174,"num_object_clones":0,"num_object_copies":400696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33272,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100174,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":99063,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":660,"num_bytes_recovered":5201920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1512747008,"data_stored":1512747008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":104788,"ondisk_log_size":104788,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084343,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":758132,"kb_used_data":401572,"kb_used_omap":4,"kb_used_meta":356539,"kb_avail":93613708,"statfs":{"total":96636764160,"available":95860436992,"internally_reserved":0,"allocated":411209728,"data_stored":411017687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":365096659},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,6,7,2,4,0,0,4],"upper_bound":2048},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986220,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":550652,"kb_used_data":276012,"kb_used_omap":4,"kb_used_meta":274619,"kb_avail":93821188,"statfs":{"total":96636764160,"available":96072896512,"internally_reserved":0,"allocated":282636288,"data_stored":282444247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":281210578},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673026,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":760960,"kb_used_data":400752,"kb_used_omap":1,"kb_used_meta":360190,"kb_avail":93610880,"statfs":{"total":96636764160,"available":95857541120,"internally_reserved":0,"allocated":410370048,"data_stored":410178007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":368835013},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,7,5,3,2,0,0,0,3],"upper_bound":2048},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182457,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":761680,"kb_used_data":401012,"kb_used_omap":4,"kb_used_meta":360635,"kb_avail":93610160,"statfs":{"total":96636764160,"available":95856803840,"internally_reserved":0,"allocated":410636288,"data_stored":410444247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":369290957},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,0,0,0,1,6,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":410025984,"data_stored":410025984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":282279936,"data_stored":282279936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":409989120,"data_stored":409989120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":410451968,"data_stored":410451968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:38:35.681 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:38:36.271 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:38:36.271 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:38:36.290 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":185,"stamp":"2024-09-05T16:38:34.886474+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":820579384,"num_objects":100174,"num_object_clones":0,"num_object_copies":400696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33272,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100174,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":99063,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":660,"num_bytes_recovered":5201920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":104788,"ondisk_log_size":104788,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2831424,"kb_used_data":1479348,"kb_used_omap":14,"kb_used_meta":1351985,"kb_avail":374655936,"statfs":{"total":386547056640,"available":383647678464,"internally_reserved":0,"allocated":1514852352,"data_stored":1514084188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1384433207},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[7,0,2,9,12,19,6,4,0,0,7],"upper_bound":2048},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-15622120,"num_objects":-1906,"num_object_clones":0,"num_object_copies":-7624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-121,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1906,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2652,"num_write_kb":2977,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":121,"num_bytes_recovered":991232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005062"},"pg_stats":[{"pgid":"1.f","version":"35'6588","reported_seq":6634,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:31.006418+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:38:31.006418+0000","last_peered":"2024-09-05T16:38:31.006418+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:38:31.006418+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6588,"log_dups_size":0,"ondisk_log_size":6588,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51740672,"num_objects":6316,"num_object_clones":0,"num_object_copies":25264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6316,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6588,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4054}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'6436","reported_seq":6504,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:31.485647+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:38:31.485647+0000","last_peered":"2024-09-05T16:38:31.485647+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:38:31.485647+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6436,"log_dups_size":0,"ondisk_log_size":6436,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50115744,"num_objects":6118,"num_object_clones":0,"num_object_copies":24472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6118,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6434,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3851}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'6509","reported_seq":6590,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:31.174591+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:38:31.174591+0000","last_peered":"2024-09-05T16:38:31.174591+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:38:31.174591+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6509,"log_dups_size":0,"ondisk_log_size":6509,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51141224,"num_objects":6244,"num_object_clones":0,"num_object_copies":24976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6244,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6508,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3994}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'6615","reported_seq":11338,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-05T16:38:30.563703+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:38:30.563703+0000","last_peered":"2024-09-05T16:38:30.563703+0000","last_clean":"2024-09-05T16:38:30.563703+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:38:30.563703+0000","last_undegraded":"2024-09-05T16:38:30.563703+0000","last_fullsized":"2024-09-05T16:38:30.563703+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":6615,"log_dups_size":0,"ondisk_log_size":6615,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51765248,"num_objects":6319,"num_object_clones":0,"num_object_copies":25276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6319,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":915,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'6435","reported_seq":6494,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:30.919236+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:38:30.919236+0000","last_peered":"2024-09-05T16:38:30.919236+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:38:30.919236+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6435,"log_dups_size":0,"ondisk_log_size":6435,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50459808,"num_objects":6160,"num_object_clones":0,"num_object_copies":24640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6160,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6433,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3998}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'6512","reported_seq":6558,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:30.866573+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:38:30.866573+0000","last_peered":"2024-09-05T16:38:30.866573+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:38:30.866573+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6512,"log_dups_size":0,"ondisk_log_size":6512,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51281920,"num_objects":6260,"num_object_clones":0,"num_object_copies":25040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6260,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6512,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":4010}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'6457","reported_seq":6527,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:31.458721+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:38:31.458721+0000","last_peered":"2024-09-05T16:38:31.458721+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:38:31.458721+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6457,"log_dups_size":0,"ondisk_log_size":6457,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50828448,"num_objects":6205,"num_object_clones":0,"num_object_copies":24820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6205,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6455,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":4058}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'6550","reported_seq":6619,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:31.419992+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:38:31.419992+0000","last_peered":"2024-09-05T16:38:31.419992+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:38:31.419992+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6550,"log_dups_size":0,"ondisk_log_size":6550,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51025056,"num_objects":6229,"num_object_clones":0,"num_object_copies":24916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6229,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6548,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3990}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'6524","reported_seq":6585,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:30.796087+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:38:30.796087+0000","last_peered":"2024-09-05T16:38:30.796087+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:38:30.796087+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6524,"log_dups_size":0,"ondisk_log_size":6524,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51194176,"num_objects":6250,"num_object_clones":0,"num_object_copies":25000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6250,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6521,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3989}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'6579","reported_seq":6661,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:31.482226+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:38:31.482226+0000","last_peered":"2024-09-05T16:38:31.482226+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:38:31.482226+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6579,"log_dups_size":0,"ondisk_log_size":6579,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51041440,"num_objects":6231,"num_object_clones":0,"num_object_copies":24924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6231,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6577,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3969}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'6699","reported_seq":6759,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:31.005015+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:38:31.005015+0000","last_peered":"2024-09-05T16:38:31.005015+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:38:31.005015+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6699,"log_dups_size":0,"ondisk_log_size":6699,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52621040,"num_objects":6424,"num_object_clones":0,"num_object_copies":25696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6424,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6696,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4113}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'6528","reported_seq":6566,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:31.006199+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:38:31.006199+0000","last_peered":"2024-09-05T16:38:31.006199+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:38:31.006199+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6528,"log_dups_size":0,"ondisk_log_size":6528,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51042896,"num_objects":6231,"num_object_clones":0,"num_object_copies":24924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6231,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6527,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3967}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'6546","reported_seq":7896,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:31.465915+0000","last_change":"2024-09-05T16:37:30.634570+0000","last_active":"2024-09-05T16:38:31.465915+0000","last_peered":"2024-09-05T16:38:31.465915+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:38:31.465915+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6546,"log_dups_size":0,"ondisk_log_size":6546,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51305040,"num_objects":6263,"num_object_clones":0,"num_object_copies":25052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1659,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6263,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6545,"num_write_kb":51231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":636,"num_bytes_recovered":5201920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4604}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'6710","reported_seq":6766,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:30.916725+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:38:30.916725+0000","last_peered":"2024-09-05T16:38:30.916725+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:38:30.916725+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6710,"log_dups_size":0,"ondisk_log_size":6710,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52556960,"num_objects":6416,"num_object_clones":0,"num_object_copies":25664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6416,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6708,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":4059}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'6560","reported_seq":6643,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:31.393987+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:38:31.393987+0000","last_peered":"2024-09-05T16:38:31.393987+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:38:31.393987+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6560,"log_dups_size":0,"ondisk_log_size":6560,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51385504,"num_objects":6273,"num_object_clones":0,"num_object_copies":25092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6273,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6558,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3953}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'6540","reported_seq":6608,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:31.363462+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:38:31.363462+0000","last_peered":"2024-09-05T16:38:31.363462+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:38:31.363462+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6540,"log_dups_size":0,"ondisk_log_size":6540,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51074208,"num_objects":6235,"num_object_clones":0,"num_object_copies":24940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6235,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6538,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3998}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":820579384,"num_objects":100174,"num_object_clones":0,"num_object_copies":400696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33272,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100174,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":99063,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":660,"num_bytes_recovered":5201920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1512747008,"data_stored":1512747008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":104788,"ondisk_log_size":104788,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084343,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":758132,"kb_used_data":401572,"kb_used_omap":4,"kb_used_meta":356539,"kb_avail":93613708,"statfs":{"total":96636764160,"available":95860436992,"internally_reserved":0,"allocated":411209728,"data_stored":411017687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":365096659},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,6,7,2,4,0,0,4],"upper_bound":2048},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986220,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":550652,"kb_used_data":276012,"kb_used_omap":4,"kb_used_meta":274619,"kb_avail":93821188,"statfs":{"total":96636764160,"available":96072896512,"internally_reserved":0,"allocated":282636288,"data_stored":282444247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":281210578},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673026,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":760960,"kb_used_data":400752,"kb_used_omap":1,"kb_used_meta":360190,"kb_avail":93610880,"statfs":{"total":96636764160,"available":95857541120,"internally_reserved":0,"allocated":410370048,"data_stored":410178007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":368835013},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,7,5,3,2,0,0,0,3],"upper_bound":2048},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182457,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":761680,"kb_used_data":401012,"kb_used_omap":4,"kb_used_meta":360635,"kb_avail":93610160,"statfs":{"total":96636764160,"available":95856803840,"internally_reserved":0,"allocated":410636288,"data_stored":410444247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":369290957},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,0,0,0,1,6,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":410025984,"data_stored":410025984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":282279936,"data_stored":282279936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":409989120,"data_stored":409989120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":410451968,"data_stored":410451968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:38:36.291 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-05T16:38:36.291 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'6588", 'reported_seq': 6634, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:31.006418+0000', 'last_change': '2024-09-05T16:34:56.833497+0000', 'last_active': '2024-09-05T16:38:31.006418+0000', 'last_peered': '2024-09-05T16:38:31.006418+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.828289+0000', 'last_became_peered': '2024-09-05T16:34:56.828289+0000', 'last_unstale': '2024-09-05T16:38:31.006418+0000', 'last_undegraded': '2024-09-05T16:34:56.612160+0000', 'last_fullsized': '2024-09-05T16:34:56.557632+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6588, 'log_dups_size': 0, 'ondisk_log_size': 6588, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51740672, 'num_objects': 6316, 'num_object_clones': 0, 'num_object_copies': 25264, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2262, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6316, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6588, 'num_write_kb': 51616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 4054}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:38:36.292 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-05T16:38:36.294 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "35'6436", 'reported_seq': 6504, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:31.485647+0000', 'last_change': '2024-09-05T16:34:56.825031+0000', 'last_active': '2024-09-05T16:38:31.485647+0000', 'last_peered': '2024-09-05T16:38:31.485647+0000', 'last_clean': '2024-09-05T16:33:53.200762+0000', 'last_became_active': '2024-09-05T16:34:56.820292+0000', 'last_became_peered': '2024-09-05T16:34:56.820292+0000', 'last_unstale': '2024-09-05T16:38:31.485647+0000', 'last_undegraded': '2024-09-05T16:34:56.605776+0000', 'last_fullsized': '2024-09-05T16:34:56.559383+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6436, 'log_dups_size': 0, 'ondisk_log_size': 6436, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50115744, 'num_objects': 6118, 'num_object_clones': 0, 'num_object_copies': 24472, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2269, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6118, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6434, 'num_write_kb': 50212, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3851}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:38:36.294 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:38:36.294 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'6509", 'reported_seq': 6590, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:31.174591+0000', 'last_change': '2024-09-05T16:34:56.867900+0000', 'last_active': '2024-09-05T16:38:31.174591+0000', 'last_peered': '2024-09-05T16:38:31.174591+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:38:31.174591+0000', 'last_undegraded': '2024-09-05T16:34:56.590972+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6509, 'log_dups_size': 0, 'ondisk_log_size': 6509, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51141224, 'num_objects': 6244, 'num_object_clones': 0, 'num_object_copies': 24976, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2251, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6244, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 6508, 'num_write_kb': 50999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3994}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:38:36.294 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:38:36.294 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'6435", 'reported_seq': 6494, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:30.919236+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:38:30.919236+0000', 'last_peered': '2024-09-05T16:38:30.919236+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:38:30.919236+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6435, 'log_dups_size': 0, 'ondisk_log_size': 6435, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50459808, 'num_objects': 6160, 'num_object_clones': 0, 'num_object_copies': 24640, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2164, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6160, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6433, 'num_write_kb': 50373, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3998}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:38:36.294 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:38:36.294 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'6512", 'reported_seq': 6558, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:30.866573+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:38:30.866573+0000', 'last_peered': '2024-09-05T16:38:30.866573+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:38:30.866573+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6512, 'log_dups_size': 0, 'ondisk_log_size': 6512, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51281920, 'num_objects': 6260, 'num_object_clones': 0, 'num_object_copies': 25040, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6260, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6512, 'num_write_kb': 51088, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 4010}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:38:36.294 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:38:36.295 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'6457", 'reported_seq': 6527, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:31.458721+0000', 'last_change': '2024-09-05T16:34:56.903436+0000', 'last_active': '2024-09-05T16:38:31.458721+0000', 'last_peered': '2024-09-05T16:38:31.458721+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:38:31.458721+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6457, 'log_dups_size': 0, 'ondisk_log_size': 6457, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50828448, 'num_objects': 6205, 'num_object_clones': 0, 'num_object_copies': 24820, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2149, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6205, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6455, 'num_write_kb': 50636, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 4058}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:38:36.295 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-05T16:38:36.295 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "35'6550", 'reported_seq': 6619, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:31.419992+0000', 'last_change': '2024-09-05T16:34:56.883319+0000', 'last_active': '2024-09-05T16:38:31.419992+0000', 'last_peered': '2024-09-05T16:38:31.419992+0000', 'last_clean': '2024-09-05T16:33:53.505305+0000', 'last_became_active': '2024-09-05T16:34:56.873389+0000', 'last_became_peered': '2024-09-05T16:34:56.873389+0000', 'last_unstale': '2024-09-05T16:38:31.419992+0000', 'last_undegraded': '2024-09-05T16:34:56.602567+0000', 'last_fullsized': '2024-09-05T16:34:56.559958+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6550, 'log_dups_size': 0, 'ondisk_log_size': 6550, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51025056, 'num_objects': 6229, 'num_object_clones': 0, 'num_object_copies': 24916, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2241, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6229, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6548, 'num_write_kb': 51108, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3990}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:38:36.295 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:38:36.295 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'6524", 'reported_seq': 6585, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:30.796087+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:38:30.796087+0000', 'last_peered': '2024-09-05T16:38:30.796087+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:38:30.796087+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6524, 'log_dups_size': 0, 'ondisk_log_size': 6524, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51194176, 'num_objects': 6250, 'num_object_clones': 0, 'num_object_copies': 25000, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2264, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6250, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6521, 'num_write_kb': 51081, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3989}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:38:36.295 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:38:36.295 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'6579", 'reported_seq': 6661, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:31.482226+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:38:31.482226+0000', 'last_peered': '2024-09-05T16:38:31.482226+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:38:31.482226+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6579, 'log_dups_size': 0, 'ondisk_log_size': 6579, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51041440, 'num_objects': 6231, 'num_object_clones': 0, 'num_object_copies': 24924, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2264, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6231, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6577, 'num_write_kb': 51230, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3969}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:38:36.295 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:38:36.295 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'6699", 'reported_seq': 6759, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:31.005015+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:38:31.005015+0000', 'last_peered': '2024-09-05T16:38:31.005015+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:38:31.005015+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6699, 'log_dups_size': 0, 'ondisk_log_size': 6699, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 52621040, 'num_objects': 6424, 'num_object_clones': 0, 'num_object_copies': 25696, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2314, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6424, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6696, 'num_write_kb': 52476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 4113}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:38:36.295 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:38:36.296 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'6528", 'reported_seq': 6566, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:31.006199+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:38:31.006199+0000', 'last_peered': '2024-09-05T16:38:31.006199+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:38:31.006199+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6528, 'log_dups_size': 0, 'ondisk_log_size': 6528, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51042896, 'num_objects': 6231, 'num_object_clones': 0, 'num_object_copies': 24924, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2265, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6231, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6527, 'num_write_kb': 51031, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3967}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:38:36.296 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-05T16:38:36.296 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'6546", 'reported_seq': 7896, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:31.465915+0000', 'last_change': '2024-09-05T16:37:30.634570+0000', 'last_active': '2024-09-05T16:38:31.465915+0000', 'last_peered': '2024-09-05T16:38:31.465915+0000', 'last_clean': '2024-09-05T16:33:45.082039+0000', 'last_became_active': '2024-09-05T16:34:56.792386+0000', 'last_became_peered': '2024-09-05T16:34:56.792386+0000', 'last_unstale': '2024-09-05T16:38:31.465915+0000', 'last_undegraded': '2024-09-05T16:34:56.600595+0000', 'last_fullsized': '2024-09-05T16:34:56.558203+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6546, 'log_dups_size': 0, 'ondisk_log_size': 6546, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51305040, 'num_objects': 6263, 'num_object_clones': 0, 'num_object_copies': 25052, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1659, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6263, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6545, 'num_write_kb': 51231, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 636, 'num_bytes_recovered': 5201920, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 4604}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:38:36.296 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:38:36.296 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'6710", 'reported_seq': 6766, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:30.916725+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:38:30.916725+0000', 'last_peered': '2024-09-05T16:38:30.916725+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:38:30.916725+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6710, 'log_dups_size': 0, 'ondisk_log_size': 6710, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 52556960, 'num_objects': 6416, 'num_object_clones': 0, 'num_object_copies': 25664, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2359, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6416, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6708, 'num_write_kb': 52500, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 4059}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:38:36.296 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:38:36.296 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'6560", 'reported_seq': 6643, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:31.393987+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:38:31.393987+0000', 'last_peered': '2024-09-05T16:38:31.393987+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:38:31.393987+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6560, 'log_dups_size': 0, 'ondisk_log_size': 6560, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51385504, 'num_objects': 6273, 'num_object_clones': 0, 'num_object_copies': 25092, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2322, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6273, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6558, 'num_write_kb': 51325, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3953}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:38:36.296 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:38:36.296 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'6540", 'reported_seq': 6608, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:31.363462+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:38:31.363462+0000', 'last_peered': '2024-09-05T16:38:31.363462+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:38:31.363462+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6540, 'log_dups_size': 0, 'ondisk_log_size': 6540, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51074208, 'num_objects': 6235, 'num_object_clones': 0, 'num_object_copies': 24940, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6235, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6538, 'num_write_kb': 51092, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3998}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:38:36.297 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:38:37.032 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:38:37.033 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 33204/395408 objects degraded (8.397%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":320,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":98852,"data_bytes":809749560,"bytes_used":2882994176,"bytes_avail":383664062464,"bytes_total":386547056640,"degraded_objects":33204,"degraded_total":395408,"degraded_ratio":0.083974021769918666,"recovering_objects_per_sec":15,"recovering_bytes_per_sec":120782,"recovering_keys_per_sec":0,"num_objects_recovered":189,"num_bytes_recovered":1449984,"num_keys_recovered":0,"read_bytes_sec":85,"write_bytes_sec":253932,"read_op_per_sec":0,"write_op_per_sec":331},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:38:37.060 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:38:37.060 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:38:37.661 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:38:37.661 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:38:37.683 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":186,"stamp":"2024-09-05T16:38:36.887325+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":809749560,"num_objects":98852,"num_object_clones":0,"num_object_copies":395408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33204,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98852,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":100385,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":728,"num_bytes_recovered":5660672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":106110,"ondisk_log_size":106110,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2815424,"kb_used_data":1463336,"kb_used_omap":14,"kb_used_meta":1351985,"kb_avail":374671936,"statfs":{"total":386547056640,"available":383664062464,"internally_reserved":0,"allocated":1498456064,"data_stored":1497687900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1384433207},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,8,13,15,2,2,0,1,0,0,0,3],"upper_bound":16384},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-26451944,"num_objects":-3228,"num_object_clones":0,"num_object_copies":-12912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-189,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3228,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3974,"num_write_kb":2977,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":189,"num_bytes_recovered":1449984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004938"},"pg_stats":[{"pgid":"1.f","version":"35'6677","reported_seq":6725,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:35.827800+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:38:35.827800+0000","last_peered":"2024-09-05T16:38:35.827800+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:38:35.827800+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6677,"log_dups_size":0,"ondisk_log_size":6677,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51011584,"num_objects":6227,"num_object_clones":0,"num_object_copies":24908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2261,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6227,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6677,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3966}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'6524","reported_seq":6592,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:36.444708+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:38:36.444708+0000","last_peered":"2024-09-05T16:38:36.444708+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:38:36.444708+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6524,"log_dups_size":0,"ondisk_log_size":6524,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49394848,"num_objects":6030,"num_object_clones":0,"num_object_copies":24120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6030,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6522,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3763}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'6588","reported_seq":6671,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:36.536583+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:38:36.536583+0000","last_peered":"2024-09-05T16:38:36.536583+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:38:36.536583+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6588,"log_dups_size":0,"ondisk_log_size":6588,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50494056,"num_objects":6165,"num_object_clones":0,"num_object_copies":24660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6165,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6587,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3916}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'6698","reported_seq":11419,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-05T16:38:35.725897+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:38:35.725897+0000","last_peered":"2024-09-05T16:38:35.725897+0000","last_clean":"2024-09-05T16:38:35.725897+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:38:35.725897+0000","last_undegraded":"2024-09-05T16:38:35.725897+0000","last_fullsized":"2024-09-05T16:38:35.725897+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":6698,"log_dups_size":0,"ondisk_log_size":6698,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51085312,"num_objects":6236,"num_object_clones":0,"num_object_copies":24944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":998,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'6533","reported_seq":6598,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:35.968276+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:38:35.968276+0000","last_peered":"2024-09-05T16:38:35.968276+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:38:35.968276+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6533,"log_dups_size":0,"ondisk_log_size":6533,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49656992,"num_objects":6062,"num_object_clones":0,"num_object_copies":24248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2161,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6062,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6531,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3903}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'6593","reported_seq":6641,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:36.064879+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:38:36.064879+0000","last_peered":"2024-09-05T16:38:36.064879+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:38:36.064879+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6593,"log_dups_size":0,"ondisk_log_size":6593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50618368,"num_objects":6179,"num_object_clones":0,"num_object_copies":24716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2249,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6179,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6593,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3930}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'6544","reported_seq":6614,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:36.436852+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:38:36.436852+0000","last_peered":"2024-09-05T16:38:36.436852+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:38:36.436852+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6544,"log_dups_size":0,"ondisk_log_size":6544,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50115744,"num_objects":6118,"num_object_clones":0,"num_object_copies":24472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6118,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6542,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3971}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'6621","reported_seq":6690,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:36.391771+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:38:36.391771+0000","last_peered":"2024-09-05T16:38:36.391771+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:38:36.391771+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6621,"log_dups_size":0,"ondisk_log_size":6621,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50443424,"num_objects":6158,"num_object_clones":0,"num_object_copies":24632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6158,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6619,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3919}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'6605","reported_seq":6666,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:35.882230+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:38:35.882230+0000","last_peered":"2024-09-05T16:38:35.882230+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:38:35.882230+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6605,"log_dups_size":0,"ondisk_log_size":6605,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50530624,"num_objects":6169,"num_object_clones":0,"num_object_copies":24676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6169,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6602,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3908}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'6651","reported_seq":6739,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:36.414086+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:38:36.414086+0000","last_peered":"2024-09-05T16:38:36.414086+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:38:36.414086+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6651,"log_dups_size":0,"ondisk_log_size":6651,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50451616,"num_objects":6159,"num_object_clones":0,"num_object_copies":24636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2261,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6159,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6649,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3900}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'6774","reported_seq":6837,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:35.913259+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:38:35.913259+0000","last_peered":"2024-09-05T16:38:35.913259+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:38:35.913259+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6774,"log_dups_size":0,"ondisk_log_size":6774,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52006640,"num_objects":6349,"num_object_clones":0,"num_object_copies":25396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6349,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6771,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4039}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'6607","reported_seq":6647,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:35.952803+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:38:35.952803+0000","last_peered":"2024-09-05T16:38:35.952803+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:38:35.952803+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6607,"log_dups_size":0,"ondisk_log_size":6607,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50395728,"num_objects":6152,"num_object_clones":0,"num_object_copies":24608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6152,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6606,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3888}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'6625","reported_seq":8087,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:36.584411+0000","last_change":"2024-09-05T16:37:30.634570+0000","last_active":"2024-09-05T16:38:36.584411+0000","last_peered":"2024-09-05T16:38:36.584411+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:38:36.584411+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6625,"log_dups_size":0,"ondisk_log_size":6625,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50657872,"num_objects":6184,"num_object_clones":0,"num_object_copies":24736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1603,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6184,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6624,"num_write_kb":51231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":692,"num_bytes_recovered":5660672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4581}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'6799","reported_seq":6855,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:36.075843+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:38:36.075843+0000","last_peered":"2024-09-05T16:38:36.075843+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:38:36.075843+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6799,"log_dups_size":0,"ondisk_log_size":6799,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51827872,"num_objects":6327,"num_object_clones":0,"num_object_copies":25308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2358,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6327,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6797,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3971}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'6649","reported_seq":6734,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:36.523001+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:38:36.523001+0000","last_peered":"2024-09-05T16:38:36.523001+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:38:36.523001+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6649,"log_dups_size":0,"ondisk_log_size":6649,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50656416,"num_objects":6184,"num_object_clones":0,"num_object_copies":24736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6184,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6647,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3865}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'6622","reported_seq":6690,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:36.454537+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:38:36.454537+0000","last_peered":"2024-09-05T16:38:36.454537+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:38:36.454537+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6622,"log_dups_size":0,"ondisk_log_size":6622,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50402464,"num_objects":6153,"num_object_clones":0,"num_object_copies":24612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6153,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6620,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3916}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":809749560,"num_objects":98852,"num_object_clones":0,"num_object_copies":395408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33204,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98852,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":100385,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":728,"num_bytes_recovered":5660672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1496092672,"data_stored":1496092672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":106110,"ondisk_log_size":106110,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084344,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":757096,"kb_used_data":396368,"kb_used_omap":4,"kb_used_meta":360699,"kb_avail":93614744,"statfs":{"total":96636764160,"available":95861497856,"internally_reserved":0,"allocated":405880832,"data_stored":405688791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":369356499},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,5,11,1,2,0,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986221,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":537812,"kb_used_data":275652,"kb_used_omap":4,"kb_used_meta":262139,"kb_avail":93834028,"statfs":{"total":96636764160,"available":96086044672,"internally_reserved":0,"allocated":282267648,"data_stored":282075607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":268431058},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,4,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673027,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":759832,"kb_used_data":395464,"kb_used_omap":1,"kb_used_meta":364350,"kb_avail":93612008,"statfs":{"total":96636764160,"available":95858696192,"internally_reserved":0,"allocated":404955136,"data_stored":404763095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":373094853},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,0,1,1,0,0,0,0,0,0,3],"upper_bound":16384},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182458,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":760684,"kb_used_data":395852,"kb_used_omap":4,"kb_used_meta":364795,"kb_avail":93611156,"statfs":{"total":96636764160,"available":95857823744,"internally_reserved":0,"allocated":405352448,"data_stored":405160407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":373550797},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,4,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":404619264,"data_stored":404619264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":281882624,"data_stored":281882624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":404529152,"data_stored":404529152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":405061632,"data_stored":405061632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:38:40.685 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:38:41.251 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:38:41.252 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:38:41.283 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":188,"stamp":"2024-09-05T16:38:40.889121+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":808995896,"num_objects":98760,"num_object_clones":0,"num_object_copies":395040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33204,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98760,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":100477,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":728,"num_bytes_recovered":5660672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":106202,"ondisk_log_size":106202,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2819100,"kb_used_data":1462852,"kb_used_omap":14,"kb_used_meta":1356145,"kb_avail":374668260,"statfs":{"total":386547056640,"available":383660298240,"internally_reserved":0,"allocated":1497960448,"data_stored":1497192284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1388693047},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,3,4,14,23,2,2,0,1,0,0,0,3],"upper_bound":16384},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22331392,"num_objects":-2726,"num_object_clones":0,"num_object_copies":-10904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-132,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2726,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2726,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":132,"num_bytes_recovered":983040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005341"},"pg_stats":[{"pgid":"1.f","version":"35'6677","reported_seq":6725,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:35.827800+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:38:35.827800+0000","last_peered":"2024-09-05T16:38:35.827800+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:38:35.827800+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6677,"log_dups_size":0,"ondisk_log_size":6677,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51011584,"num_objects":6227,"num_object_clones":0,"num_object_copies":24908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2261,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6227,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6677,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3966}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'6524","reported_seq":6592,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:36.444708+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:38:36.444708+0000","last_peered":"2024-09-05T16:38:36.444708+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:38:36.444708+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6524,"log_dups_size":0,"ondisk_log_size":6524,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49394848,"num_objects":6030,"num_object_clones":0,"num_object_copies":24120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6030,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6522,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3763}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'6588","reported_seq":6671,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:36.536583+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:38:36.536583+0000","last_peered":"2024-09-05T16:38:36.536583+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:38:36.536583+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6588,"log_dups_size":0,"ondisk_log_size":6588,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50494056,"num_objects":6165,"num_object_clones":0,"num_object_copies":24660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6165,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6587,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3916}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'6790","reported_seq":11513,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-05T16:38:40.658059+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:38:40.658059+0000","last_peered":"2024-09-05T16:38:40.658059+0000","last_clean":"2024-09-05T16:38:40.658059+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:38:40.658059+0000","last_undegraded":"2024-09-05T16:38:40.658059+0000","last_fullsized":"2024-09-05T16:38:40.658059+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":6790,"log_dups_size":0,"ondisk_log_size":6790,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50331648,"num_objects":6144,"num_object_clones":0,"num_object_copies":24576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6144,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1090,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'6533","reported_seq":6598,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:35.968276+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:38:35.968276+0000","last_peered":"2024-09-05T16:38:35.968276+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:38:35.968276+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6533,"log_dups_size":0,"ondisk_log_size":6533,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49656992,"num_objects":6062,"num_object_clones":0,"num_object_copies":24248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2161,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6062,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6531,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3903}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'6593","reported_seq":6641,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:36.064879+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:38:36.064879+0000","last_peered":"2024-09-05T16:38:36.064879+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:38:36.064879+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6593,"log_dups_size":0,"ondisk_log_size":6593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50618368,"num_objects":6179,"num_object_clones":0,"num_object_copies":24716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2249,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6179,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6593,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3930}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'6544","reported_seq":6614,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:36.436852+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:38:36.436852+0000","last_peered":"2024-09-05T16:38:36.436852+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:38:36.436852+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6544,"log_dups_size":0,"ondisk_log_size":6544,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50115744,"num_objects":6118,"num_object_clones":0,"num_object_copies":24472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6118,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6542,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3971}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'6621","reported_seq":6690,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:36.391771+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:38:36.391771+0000","last_peered":"2024-09-05T16:38:36.391771+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:38:36.391771+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6621,"log_dups_size":0,"ondisk_log_size":6621,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50443424,"num_objects":6158,"num_object_clones":0,"num_object_copies":24632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6158,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6619,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3919}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'6605","reported_seq":6666,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:35.882230+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:38:35.882230+0000","last_peered":"2024-09-05T16:38:35.882230+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:38:35.882230+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6605,"log_dups_size":0,"ondisk_log_size":6605,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50530624,"num_objects":6169,"num_object_clones":0,"num_object_copies":24676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6169,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6602,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3908}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'6651","reported_seq":6739,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:36.414086+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:38:36.414086+0000","last_peered":"2024-09-05T16:38:36.414086+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:38:36.414086+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6651,"log_dups_size":0,"ondisk_log_size":6651,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50451616,"num_objects":6159,"num_object_clones":0,"num_object_copies":24636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2261,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6159,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6649,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3900}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'6774","reported_seq":6837,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:35.913259+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:38:35.913259+0000","last_peered":"2024-09-05T16:38:35.913259+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:38:35.913259+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6774,"log_dups_size":0,"ondisk_log_size":6774,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52006640,"num_objects":6349,"num_object_clones":0,"num_object_copies":25396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6349,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6771,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4039}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'6607","reported_seq":6647,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:35.952803+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:38:35.952803+0000","last_peered":"2024-09-05T16:38:35.952803+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:38:35.952803+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6607,"log_dups_size":0,"ondisk_log_size":6607,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50395728,"num_objects":6152,"num_object_clones":0,"num_object_copies":24608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6152,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6606,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3888}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'6625","reported_seq":8087,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:36.584411+0000","last_change":"2024-09-05T16:37:30.634570+0000","last_active":"2024-09-05T16:38:36.584411+0000","last_peered":"2024-09-05T16:38:36.584411+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:38:36.584411+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6625,"log_dups_size":0,"ondisk_log_size":6625,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50657872,"num_objects":6184,"num_object_clones":0,"num_object_copies":24736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1603,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6184,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6624,"num_write_kb":51231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":692,"num_bytes_recovered":5660672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4581}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'6799","reported_seq":6855,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:36.075843+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:38:36.075843+0000","last_peered":"2024-09-05T16:38:36.075843+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:38:36.075843+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6799,"log_dups_size":0,"ondisk_log_size":6799,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51827872,"num_objects":6327,"num_object_clones":0,"num_object_copies":25308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2358,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6327,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6797,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3971}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'6649","reported_seq":6734,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:36.523001+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:38:36.523001+0000","last_peered":"2024-09-05T16:38:36.523001+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:38:36.523001+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6649,"log_dups_size":0,"ondisk_log_size":6649,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50656416,"num_objects":6184,"num_object_clones":0,"num_object_copies":24736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6184,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6647,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3865}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'6622","reported_seq":6690,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:36.454537+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:38:36.454537+0000","last_peered":"2024-09-05T16:38:36.454537+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:38:36.454537+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6622,"log_dups_size":0,"ondisk_log_size":6622,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50402464,"num_objects":6153,"num_object_clones":0,"num_object_copies":24612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6153,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6620,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3916}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":808995896,"num_objects":98760,"num_object_clones":0,"num_object_copies":395040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33204,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98760,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":100477,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":728,"num_bytes_recovered":5660672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1495560192,"data_stored":1495560192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":106202,"ondisk_log_size":106202,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084344,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":757096,"kb_used_data":396368,"kb_used_omap":4,"kb_used_meta":360699,"kb_avail":93614744,"statfs":{"total":96636764160,"available":95861497856,"internally_reserved":0,"allocated":405880832,"data_stored":405688791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":369356499},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,5,11,1,2,0,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986222,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":541488,"kb_used_data":275168,"kb_used_omap":4,"kb_used_meta":266299,"kb_avail":93830352,"statfs":{"total":96636764160,"available":96082280448,"internally_reserved":0,"allocated":281772032,"data_stored":281579991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":272690898},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,5,10],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673027,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":759832,"kb_used_data":395464,"kb_used_omap":1,"kb_used_meta":364350,"kb_avail":93612008,"statfs":{"total":96636764160,"available":95858696192,"internally_reserved":0,"allocated":404955136,"data_stored":404763095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":373094853},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,0,1,1,0,0,0,0,0,0,3],"upper_bound":16384},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182458,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":760684,"kb_used_data":395852,"kb_used_omap":4,"kb_used_meta":364795,"kb_avail":93611156,"statfs":{"total":96636764160,"available":95857823744,"internally_reserved":0,"allocated":405352448,"data_stored":405160407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":373550797},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,4,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":404619264,"data_stored":404619264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":281350144,"data_stored":281350144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":404529152,"data_stored":404529152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":405061632,"data_stored":405061632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:38:41.284 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:38:41.898 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:38:41.899 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:38:41.928 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":188,"stamp":"2024-09-05T16:38:40.889121+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":808995896,"num_objects":98760,"num_object_clones":0,"num_object_copies":395040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33204,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98760,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":100477,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":728,"num_bytes_recovered":5660672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":106202,"ondisk_log_size":106202,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2819100,"kb_used_data":1462852,"kb_used_omap":14,"kb_used_meta":1356145,"kb_avail":374668260,"statfs":{"total":386547056640,"available":383660298240,"internally_reserved":0,"allocated":1497960448,"data_stored":1497192284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1388693047},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,3,4,14,23,2,2,0,1,0,0,0,3],"upper_bound":16384},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22331392,"num_objects":-2726,"num_object_clones":0,"num_object_copies":-10904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-132,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2726,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2726,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":132,"num_bytes_recovered":983040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005341"},"pg_stats":[{"pgid":"1.f","version":"35'6677","reported_seq":6725,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:35.827800+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:38:35.827800+0000","last_peered":"2024-09-05T16:38:35.827800+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:38:35.827800+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6677,"log_dups_size":0,"ondisk_log_size":6677,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51011584,"num_objects":6227,"num_object_clones":0,"num_object_copies":24908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2261,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6227,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6677,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3966}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'6524","reported_seq":6592,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:36.444708+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:38:36.444708+0000","last_peered":"2024-09-05T16:38:36.444708+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:38:36.444708+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6524,"log_dups_size":0,"ondisk_log_size":6524,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49394848,"num_objects":6030,"num_object_clones":0,"num_object_copies":24120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6030,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6522,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3763}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'6588","reported_seq":6671,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:36.536583+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:38:36.536583+0000","last_peered":"2024-09-05T16:38:36.536583+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:38:36.536583+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6588,"log_dups_size":0,"ondisk_log_size":6588,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50494056,"num_objects":6165,"num_object_clones":0,"num_object_copies":24660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6165,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6587,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3916}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'6790","reported_seq":11513,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-05T16:38:40.658059+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:38:40.658059+0000","last_peered":"2024-09-05T16:38:40.658059+0000","last_clean":"2024-09-05T16:38:40.658059+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:38:40.658059+0000","last_undegraded":"2024-09-05T16:38:40.658059+0000","last_fullsized":"2024-09-05T16:38:40.658059+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":6790,"log_dups_size":0,"ondisk_log_size":6790,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50331648,"num_objects":6144,"num_object_clones":0,"num_object_copies":24576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6144,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1090,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'6533","reported_seq":6598,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:35.968276+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:38:35.968276+0000","last_peered":"2024-09-05T16:38:35.968276+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:38:35.968276+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6533,"log_dups_size":0,"ondisk_log_size":6533,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49656992,"num_objects":6062,"num_object_clones":0,"num_object_copies":24248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2161,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6062,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6531,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3903}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'6593","reported_seq":6641,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:36.064879+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:38:36.064879+0000","last_peered":"2024-09-05T16:38:36.064879+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:38:36.064879+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6593,"log_dups_size":0,"ondisk_log_size":6593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50618368,"num_objects":6179,"num_object_clones":0,"num_object_copies":24716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2249,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6179,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6593,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3930}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'6544","reported_seq":6614,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:36.436852+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:38:36.436852+0000","last_peered":"2024-09-05T16:38:36.436852+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:38:36.436852+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6544,"log_dups_size":0,"ondisk_log_size":6544,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50115744,"num_objects":6118,"num_object_clones":0,"num_object_copies":24472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6118,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6542,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3971}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'6621","reported_seq":6690,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:36.391771+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:38:36.391771+0000","last_peered":"2024-09-05T16:38:36.391771+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:38:36.391771+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6621,"log_dups_size":0,"ondisk_log_size":6621,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50443424,"num_objects":6158,"num_object_clones":0,"num_object_copies":24632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6158,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6619,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3919}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'6605","reported_seq":6666,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:35.882230+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:38:35.882230+0000","last_peered":"2024-09-05T16:38:35.882230+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:38:35.882230+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6605,"log_dups_size":0,"ondisk_log_size":6605,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50530624,"num_objects":6169,"num_object_clones":0,"num_object_copies":24676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6169,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6602,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3908}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'6651","reported_seq":6739,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:36.414086+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:38:36.414086+0000","last_peered":"2024-09-05T16:38:36.414086+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:38:36.414086+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6651,"log_dups_size":0,"ondisk_log_size":6651,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50451616,"num_objects":6159,"num_object_clones":0,"num_object_copies":24636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2261,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6159,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6649,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3900}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'6774","reported_seq":6837,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:35.913259+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:38:35.913259+0000","last_peered":"2024-09-05T16:38:35.913259+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:38:35.913259+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6774,"log_dups_size":0,"ondisk_log_size":6774,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52006640,"num_objects":6349,"num_object_clones":0,"num_object_copies":25396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6349,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6771,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4039}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'6607","reported_seq":6647,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:35.952803+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:38:35.952803+0000","last_peered":"2024-09-05T16:38:35.952803+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:38:35.952803+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6607,"log_dups_size":0,"ondisk_log_size":6607,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50395728,"num_objects":6152,"num_object_clones":0,"num_object_copies":24608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6152,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6606,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3888}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'6625","reported_seq":8087,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:36.584411+0000","last_change":"2024-09-05T16:37:30.634570+0000","last_active":"2024-09-05T16:38:36.584411+0000","last_peered":"2024-09-05T16:38:36.584411+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:38:36.584411+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6625,"log_dups_size":0,"ondisk_log_size":6625,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50657872,"num_objects":6184,"num_object_clones":0,"num_object_copies":24736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1603,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6184,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6624,"num_write_kb":51231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":692,"num_bytes_recovered":5660672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4581}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'6799","reported_seq":6855,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:36.075843+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:38:36.075843+0000","last_peered":"2024-09-05T16:38:36.075843+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:38:36.075843+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6799,"log_dups_size":0,"ondisk_log_size":6799,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51827872,"num_objects":6327,"num_object_clones":0,"num_object_copies":25308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2358,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6327,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6797,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3971}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'6649","reported_seq":6734,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:36.523001+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:38:36.523001+0000","last_peered":"2024-09-05T16:38:36.523001+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:38:36.523001+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6649,"log_dups_size":0,"ondisk_log_size":6649,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50656416,"num_objects":6184,"num_object_clones":0,"num_object_copies":24736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6184,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6647,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3865}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'6622","reported_seq":6690,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:36.454537+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:38:36.454537+0000","last_peered":"2024-09-05T16:38:36.454537+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:38:36.454537+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6622,"log_dups_size":0,"ondisk_log_size":6622,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50402464,"num_objects":6153,"num_object_clones":0,"num_object_copies":24612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6153,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6620,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3916}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":808995896,"num_objects":98760,"num_object_clones":0,"num_object_copies":395040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33204,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98760,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":100477,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":728,"num_bytes_recovered":5660672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1495560192,"data_stored":1495560192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":106202,"ondisk_log_size":106202,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084344,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":757096,"kb_used_data":396368,"kb_used_omap":4,"kb_used_meta":360699,"kb_avail":93614744,"statfs":{"total":96636764160,"available":95861497856,"internally_reserved":0,"allocated":405880832,"data_stored":405688791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":369356499},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,5,11,1,2,0,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986222,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":541488,"kb_used_data":275168,"kb_used_omap":4,"kb_used_meta":266299,"kb_avail":93830352,"statfs":{"total":96636764160,"available":96082280448,"internally_reserved":0,"allocated":281772032,"data_stored":281579991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":272690898},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,5,10],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673027,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":759832,"kb_used_data":395464,"kb_used_omap":1,"kb_used_meta":364350,"kb_avail":93612008,"statfs":{"total":96636764160,"available":95858696192,"internally_reserved":0,"allocated":404955136,"data_stored":404763095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":373094853},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,0,1,1,0,0,0,0,0,0,3],"upper_bound":16384},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182458,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":760684,"kb_used_data":395852,"kb_used_omap":4,"kb_used_meta":364795,"kb_avail":93611156,"statfs":{"total":96636764160,"available":95857823744,"internally_reserved":0,"allocated":405352448,"data_stored":405160407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":373550797},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,4,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":404619264,"data_stored":404619264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":281350144,"data_stored":281350144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":404529152,"data_stored":404529152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":405061632,"data_stored":405061632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:38:41.929 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-05T16:38:41.929 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'6677", 'reported_seq': 6725, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:35.827800+0000', 'last_change': '2024-09-05T16:34:56.833497+0000', 'last_active': '2024-09-05T16:38:35.827800+0000', 'last_peered': '2024-09-05T16:38:35.827800+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.828289+0000', 'last_became_peered': '2024-09-05T16:34:56.828289+0000', 'last_unstale': '2024-09-05T16:38:35.827800+0000', 'last_undegraded': '2024-09-05T16:34:56.612160+0000', 'last_fullsized': '2024-09-05T16:34:56.557632+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6677, 'log_dups_size': 0, 'ondisk_log_size': 6677, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51011584, 'num_objects': 6227, 'num_object_clones': 0, 'num_object_copies': 24908, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2261, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6227, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6677, 'num_write_kb': 51616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3966}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:38:41.930 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-05T16:38:41.930 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "35'6524", 'reported_seq': 6592, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:36.444708+0000', 'last_change': '2024-09-05T16:34:56.825031+0000', 'last_active': '2024-09-05T16:38:36.444708+0000', 'last_peered': '2024-09-05T16:38:36.444708+0000', 'last_clean': '2024-09-05T16:33:53.200762+0000', 'last_became_active': '2024-09-05T16:34:56.820292+0000', 'last_became_peered': '2024-09-05T16:34:56.820292+0000', 'last_unstale': '2024-09-05T16:38:36.444708+0000', 'last_undegraded': '2024-09-05T16:34:56.605776+0000', 'last_fullsized': '2024-09-05T16:34:56.559383+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6524, 'log_dups_size': 0, 'ondisk_log_size': 6524, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49394848, 'num_objects': 6030, 'num_object_clones': 0, 'num_object_copies': 24120, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2269, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6030, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6522, 'num_write_kb': 50212, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3763}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:38:41.930 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:38:41.930 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'6588", 'reported_seq': 6671, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:36.536583+0000', 'last_change': '2024-09-05T16:34:56.867900+0000', 'last_active': '2024-09-05T16:38:36.536583+0000', 'last_peered': '2024-09-05T16:38:36.536583+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:38:36.536583+0000', 'last_undegraded': '2024-09-05T16:34:56.590972+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6588, 'log_dups_size': 0, 'ondisk_log_size': 6588, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50494056, 'num_objects': 6165, 'num_object_clones': 0, 'num_object_copies': 24660, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6165, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 6587, 'num_write_kb': 50999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3916}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:38:41.931 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:38:41.931 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'6533", 'reported_seq': 6598, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:35.968276+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:38:35.968276+0000', 'last_peered': '2024-09-05T16:38:35.968276+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:38:35.968276+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6533, 'log_dups_size': 0, 'ondisk_log_size': 6533, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49656992, 'num_objects': 6062, 'num_object_clones': 0, 'num_object_copies': 24248, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2161, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6062, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6531, 'num_write_kb': 50373, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3903}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:38:41.931 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:38:41.931 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'6593", 'reported_seq': 6641, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:36.064879+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:38:36.064879+0000', 'last_peered': '2024-09-05T16:38:36.064879+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:38:36.064879+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6593, 'log_dups_size': 0, 'ondisk_log_size': 6593, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50618368, 'num_objects': 6179, 'num_object_clones': 0, 'num_object_copies': 24716, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2249, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6179, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6593, 'num_write_kb': 51088, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3930}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:38:41.931 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:38:41.931 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'6544", 'reported_seq': 6614, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:36.436852+0000', 'last_change': '2024-09-05T16:34:56.903436+0000', 'last_active': '2024-09-05T16:38:36.436852+0000', 'last_peered': '2024-09-05T16:38:36.436852+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:38:36.436852+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6544, 'log_dups_size': 0, 'ondisk_log_size': 6544, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50115744, 'num_objects': 6118, 'num_object_clones': 0, 'num_object_copies': 24472, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2149, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6118, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6542, 'num_write_kb': 50636, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3971}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:38:41.931 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-05T16:38:41.931 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "35'6621", 'reported_seq': 6690, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:36.391771+0000', 'last_change': '2024-09-05T16:34:56.883319+0000', 'last_active': '2024-09-05T16:38:36.391771+0000', 'last_peered': '2024-09-05T16:38:36.391771+0000', 'last_clean': '2024-09-05T16:33:53.505305+0000', 'last_became_active': '2024-09-05T16:34:56.873389+0000', 'last_became_peered': '2024-09-05T16:34:56.873389+0000', 'last_unstale': '2024-09-05T16:38:36.391771+0000', 'last_undegraded': '2024-09-05T16:34:56.602567+0000', 'last_fullsized': '2024-09-05T16:34:56.559958+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6621, 'log_dups_size': 0, 'ondisk_log_size': 6621, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50443424, 'num_objects': 6158, 'num_object_clones': 0, 'num_object_copies': 24632, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2241, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6158, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6619, 'num_write_kb': 51108, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3919}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:38:41.931 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:38:41.931 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'6605", 'reported_seq': 6666, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:35.882230+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:38:35.882230+0000', 'last_peered': '2024-09-05T16:38:35.882230+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:38:35.882230+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6605, 'log_dups_size': 0, 'ondisk_log_size': 6605, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50530624, 'num_objects': 6169, 'num_object_clones': 0, 'num_object_copies': 24676, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2264, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6169, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6602, 'num_write_kb': 51081, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3908}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:38:41.932 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:38:41.932 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'6651", 'reported_seq': 6739, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:36.414086+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:38:36.414086+0000', 'last_peered': '2024-09-05T16:38:36.414086+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:38:36.414086+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6651, 'log_dups_size': 0, 'ondisk_log_size': 6651, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50451616, 'num_objects': 6159, 'num_object_clones': 0, 'num_object_copies': 24636, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2261, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6159, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6649, 'num_write_kb': 51230, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3900}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:38:41.932 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:38:41.932 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'6774", 'reported_seq': 6837, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:35.913259+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:38:35.913259+0000', 'last_peered': '2024-09-05T16:38:35.913259+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:38:35.913259+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6774, 'log_dups_size': 0, 'ondisk_log_size': 6774, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 52006640, 'num_objects': 6349, 'num_object_clones': 0, 'num_object_copies': 25396, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6349, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6771, 'num_write_kb': 52476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 4039}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:38:41.932 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:38:41.932 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'6607", 'reported_seq': 6647, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:35.952803+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:38:35.952803+0000', 'last_peered': '2024-09-05T16:38:35.952803+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:38:35.952803+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6607, 'log_dups_size': 0, 'ondisk_log_size': 6607, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50395728, 'num_objects': 6152, 'num_object_clones': 0, 'num_object_copies': 24608, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2265, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6152, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6606, 'num_write_kb': 51031, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3888}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:38:41.932 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-05T16:38:41.932 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'6625", 'reported_seq': 8087, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:36.584411+0000', 'last_change': '2024-09-05T16:37:30.634570+0000', 'last_active': '2024-09-05T16:38:36.584411+0000', 'last_peered': '2024-09-05T16:38:36.584411+0000', 'last_clean': '2024-09-05T16:33:45.082039+0000', 'last_became_active': '2024-09-05T16:34:56.792386+0000', 'last_became_peered': '2024-09-05T16:34:56.792386+0000', 'last_unstale': '2024-09-05T16:38:36.584411+0000', 'last_undegraded': '2024-09-05T16:34:56.600595+0000', 'last_fullsized': '2024-09-05T16:34:56.558203+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6625, 'log_dups_size': 0, 'ondisk_log_size': 6625, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50657872, 'num_objects': 6184, 'num_object_clones': 0, 'num_object_copies': 24736, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1603, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6184, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6624, 'num_write_kb': 51231, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 692, 'num_bytes_recovered': 5660672, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 4581}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:38:41.932 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:38:41.932 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'6799", 'reported_seq': 6855, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:36.075843+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:38:36.075843+0000', 'last_peered': '2024-09-05T16:38:36.075843+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:38:36.075843+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6799, 'log_dups_size': 0, 'ondisk_log_size': 6799, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51827872, 'num_objects': 6327, 'num_object_clones': 0, 'num_object_copies': 25308, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2358, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6327, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6797, 'num_write_kb': 52500, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3971}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:38:41.933 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:38:41.933 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'6649", 'reported_seq': 6734, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:36.523001+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:38:36.523001+0000', 'last_peered': '2024-09-05T16:38:36.523001+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:38:36.523001+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6649, 'log_dups_size': 0, 'ondisk_log_size': 6649, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50656416, 'num_objects': 6184, 'num_object_clones': 0, 'num_object_copies': 24736, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2321, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6184, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6647, 'num_write_kb': 51325, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3865}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:38:41.933 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:38:41.933 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'6622", 'reported_seq': 6690, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:36.454537+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:38:36.454537+0000', 'last_peered': '2024-09-05T16:38:36.454537+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:38:36.454537+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6622, 'log_dups_size': 0, 'ondisk_log_size': 6622, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50402464, 'num_objects': 6153, 'num_object_clones': 0, 'num_object_copies': 24612, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6153, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6620, 'num_write_kb': 51092, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3916}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:38:41.933 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:38:42.687 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:38:42.687 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 33204/395040 objects degraded (8.405%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":325,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+clean","count":1}],"num_pgs":16,"num_pools":1,"num_objects":98760,"data_bytes":808995896,"bytes_used":2886758400,"bytes_avail":383660298240,"bytes_total":386547056640,"degraded_objects":33204,"degraded_total":395040,"degraded_ratio":0.084052247873633051,"recovering_objects_per_sec":10,"recovering_bytes_per_sec":81883,"recovering_keys_per_sec":0,"num_objects_recovered":132,"num_bytes_recovered":983040,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":227},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:38:42.711 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:38:42.711 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:38:43.276 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:38:43.276 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:38:43.300 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":189,"stamp":"2024-09-05T16:38:42.889966+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":798919736,"num_objects":97530,"num_object_clones":0,"num_object_copies":390120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97530,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":101707,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":789,"num_bytes_recovered":6078464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":107432,"ondisk_log_size":107432,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2836684,"kb_used_data":1447044,"kb_used_omap":14,"kb_used_meta":1389489,"kb_avail":374650676,"statfs":{"total":386547056640,"available":383642292224,"internally_reserved":0,"allocated":1481773056,"data_stored":1481004892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1422837303},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,2,1,6,24,19,6,5,0,0,0,0,2,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-31760384,"num_objects":-3877,"num_object_clones":0,"num_object_copies":-15508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-193,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3877,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3877,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":193,"num_bytes_recovered":1400832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005352"},"pg_stats":[{"pgid":"1.f","version":"35'6762","reported_seq":6810,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:40.850497+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:38:40.850497+0000","last_peered":"2024-09-05T16:38:40.850497+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:38:40.850497+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6762,"log_dups_size":0,"ondisk_log_size":6762,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50315264,"num_objects":6142,"num_object_clones":0,"num_object_copies":24568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2261,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6142,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6762,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3881}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'6597","reported_seq":6668,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:41.466535+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:38:41.466535+0000","last_peered":"2024-09-05T16:38:41.466535+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:38:41.466535+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6597,"log_dups_size":0,"ondisk_log_size":6597,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48796832,"num_objects":5957,"num_object_clones":0,"num_object_copies":23828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2268,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5957,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6595,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3691}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'6669","reported_seq":6752,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:41.703003+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:38:41.703003+0000","last_peered":"2024-09-05T16:38:41.703003+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:38:41.703003+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6669,"log_dups_size":0,"ondisk_log_size":6669,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49830504,"num_objects":6084,"num_object_clones":0,"num_object_copies":24336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6084,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6668,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3835}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'6790","reported_seq":11513,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-05T16:38:40.658059+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:38:40.658059+0000","last_peered":"2024-09-05T16:38:40.658059+0000","last_clean":"2024-09-05T16:38:40.658059+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:38:40.658059+0000","last_undegraded":"2024-09-05T16:38:40.658059+0000","last_fullsized":"2024-09-05T16:38:40.658059+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":6790,"log_dups_size":0,"ondisk_log_size":6790,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50331648,"num_objects":6144,"num_object_clones":0,"num_object_copies":24576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6144,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1090,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'6625","reported_seq":6690,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:40.999515+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:38:40.999515+0000","last_peered":"2024-09-05T16:38:40.999515+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:38:40.999515+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6625,"log_dups_size":0,"ondisk_log_size":6625,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48903328,"num_objects":5970,"num_object_clones":0,"num_object_copies":23880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2161,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5970,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6623,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3811}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'6683","reported_seq":6731,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:40.920483+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:38:40.920483+0000","last_peered":"2024-09-05T16:38:40.920483+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:38:40.920483+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6683,"log_dups_size":0,"ondisk_log_size":6683,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49881088,"num_objects":6089,"num_object_clones":0,"num_object_copies":24356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2249,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6089,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6683,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3840}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'6617","reported_seq":6691,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:41.401703+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:38:41.401703+0000","last_peered":"2024-09-05T16:38:41.401703+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:38:41.401703+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6617,"log_dups_size":0,"ondisk_log_size":6617,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49517728,"num_objects":6045,"num_object_clones":0,"num_object_copies":24180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6045,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6615,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3900}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'6694","reported_seq":6765,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:41.525106+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:38:41.525106+0000","last_peered":"2024-09-05T16:38:41.525106+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:38:41.525106+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6694,"log_dups_size":0,"ondisk_log_size":6694,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49845408,"num_objects":6085,"num_object_clones":0,"num_object_copies":24340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6085,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6692,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3847}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'6690","reported_seq":6751,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:41.003483+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:38:41.003483+0000","last_peered":"2024-09-05T16:38:41.003483+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:38:41.003483+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6690,"log_dups_size":0,"ondisk_log_size":6690,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49834304,"num_objects":6084,"num_object_clones":0,"num_object_copies":24336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6084,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6687,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3823}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'6738","reported_seq":6830,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:41.621665+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:38:41.621665+0000","last_peered":"2024-09-05T16:38:41.621665+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:38:41.621665+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6738,"log_dups_size":0,"ondisk_log_size":6738,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49738912,"num_objects":6072,"num_object_clones":0,"num_object_copies":24288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6072,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6736,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3815}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'6847","reported_seq":6910,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:41.062605+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:38:41.062605+0000","last_peered":"2024-09-05T16:38:41.062605+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:38:41.062605+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6847,"log_dups_size":0,"ondisk_log_size":6847,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51408624,"num_objects":6276,"num_object_clones":0,"num_object_copies":25104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6276,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6844,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3966}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'6688","reported_seq":6728,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:41.031392+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:38:41.031392+0000","last_peered":"2024-09-05T16:38:41.031392+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:38:41.031392+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6688,"log_dups_size":0,"ondisk_log_size":6688,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49732176,"num_objects":6071,"num_object_clones":0,"num_object_copies":24284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6071,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6687,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3807}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'6716","reported_seq":8276,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:41.685949+0000","last_change":"2024-09-05T16:37:30.634570+0000","last_active":"2024-09-05T16:38:41.685949+0000","last_peered":"2024-09-05T16:38:41.685949+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:38:41.685949+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6716,"log_dups_size":0,"ondisk_log_size":6716,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49912400,"num_objects":6093,"num_object_clones":0,"num_object_copies":24372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6093,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6715,"num_write_kb":51231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":743,"num_bytes_recovered":6078464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4541}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'6874","reported_seq":6932,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:41.052740+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:38:41.052740+0000","last_peered":"2024-09-05T16:38:41.052740+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:38:41.052740+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6874,"log_dups_size":0,"ondisk_log_size":6874,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51213472,"num_objects":6252,"num_object_clones":0,"num_object_copies":25008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2358,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6252,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6872,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3896}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'6736","reported_seq":6825,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:41.546041+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:38:41.546041+0000","last_peered":"2024-09-05T16:38:41.546041+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:38:41.546041+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6736,"log_dups_size":0,"ondisk_log_size":6736,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49943712,"num_objects":6097,"num_object_clones":0,"num_object_copies":24388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6097,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6734,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3780}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'6706","reported_seq":6779,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:41.536779+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:38:41.536779+0000","last_peered":"2024-09-05T16:38:41.536779+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:38:41.536779+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6706,"log_dups_size":0,"ondisk_log_size":6706,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49714336,"num_objects":6069,"num_object_clones":0,"num_object_copies":24276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2237,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6069,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6704,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3834}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":798919736,"num_objects":97530,"num_object_clones":0,"num_object_copies":390120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97530,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":101707,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":789,"num_bytes_recovered":6078464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1479225344,"data_stored":1479225344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":107432,"ondisk_log_size":107432,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084345,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":774028,"kb_used_data":391064,"kb_used_omap":4,"kb_used_meta":382907,"kb_avail":93597812,"statfs":{"total":96636764160,"available":95844159488,"internally_reserved":0,"allocated":400449536,"data_stored":400257495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":392097491},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,3,10,8,6,4,0,0,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986222,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":541488,"kb_used_data":275168,"kb_used_omap":4,"kb_used_meta":266299,"kb_avail":93830352,"statfs":{"total":96636764160,"available":96082280448,"internally_reserved":0,"allocated":281772032,"data_stored":281579991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":272690898},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,5,10],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673028,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":760124,"kb_used_data":390188,"kb_used_omap":1,"kb_used_meta":369918,"kb_avail":93611716,"statfs":{"total":96636764160,"available":95858397184,"internally_reserved":0,"allocated":399552512,"data_stored":399360471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":378796485},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,6,1,0,1,0,0,0,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182459,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":761044,"kb_used_data":390624,"kb_used_omap":4,"kb_used_meta":370363,"kb_avail":93610796,"statfs":{"total":96636764160,"available":95857455104,"internally_reserved":0,"allocated":399998976,"data_stored":399806935,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":379252429},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":399204352,"data_stored":399204352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":281350144,"data_stored":281350144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":399015936,"data_stored":399015936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":399654912,"data_stored":399654912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:38:46.302 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:38:46.923 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:38:46.923 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:38:46.945 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":191,"stamp":"2024-09-05T16:38:46.891297+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":788261944,"num_objects":96229,"num_object_clones":0,"num_object_copies":384916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33081,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":96229,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":103008,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":851,"num_bytes_recovered":6561792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":108733,"ondisk_log_size":108733,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2839064,"kb_used_data":1431432,"kb_used_omap":14,"kb_used_meta":1407537,"kb_avail":374648296,"statfs":{"total":386547056640,"available":383639855104,"internally_reserved":0,"allocated":1465786368,"data_stored":1465018204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1441318455},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,4,7,14,11,11,1,0,0,0,0,3,2],"upper_bound":16384},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-32317440,"num_objects":-3945,"num_object_clones":0,"num_object_copies":-15780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3945,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3945,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":191,"num_bytes_recovered":1359872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004822"},"pg_stats":[{"pgid":"1.f","version":"35'6849","reported_seq":6898,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:46.235798+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:38:46.235798+0000","last_peered":"2024-09-05T16:38:46.235798+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:38:46.235798+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6849,"log_dups_size":0,"ondisk_log_size":6849,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49602560,"num_objects":6055,"num_object_clones":0,"num_object_copies":24220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2261,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6055,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6849,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3794}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'6672","reported_seq":6743,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:46.555650+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:38:46.555650+0000","last_peered":"2024-09-05T16:38:46.555650+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:38:46.555650+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6672,"log_dups_size":0,"ondisk_log_size":6672,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48182432,"num_objects":5882,"num_object_clones":0,"num_object_copies":23528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2268,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5882,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6670,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3616}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'6745","reported_seq":6828,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:46.588306+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:38:46.588306+0000","last_peered":"2024-09-05T16:38:46.588306+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:38:46.588306+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6745,"log_dups_size":0,"ondisk_log_size":6745,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49207912,"num_objects":6008,"num_object_clones":0,"num_object_copies":24032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6008,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6744,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3759}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'6865","reported_seq":11588,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-05T16:38:45.730564+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:38:45.730564+0000","last_peered":"2024-09-05T16:38:45.730564+0000","last_clean":"2024-09-05T16:38:45.730564+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:38:45.730564+0000","last_undegraded":"2024-09-05T16:38:45.730564+0000","last_fullsized":"2024-09-05T16:38:45.730564+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":6865,"log_dups_size":0,"ondisk_log_size":6865,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49717248,"num_objects":6069,"num_object_clones":0,"num_object_copies":24276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6069,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1165,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'6701","reported_seq":6766,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:46.072775+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:38:46.072775+0000","last_peered":"2024-09-05T16:38:46.072775+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:38:46.072775+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6701,"log_dups_size":0,"ondisk_log_size":6701,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48280736,"num_objects":5894,"num_object_clones":0,"num_object_copies":23576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2161,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5894,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6699,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3735}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'6771","reported_seq":6822,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:46.161865+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:38:46.161865+0000","last_peered":"2024-09-05T16:38:46.161865+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:38:46.161865+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6771,"log_dups_size":0,"ondisk_log_size":6771,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49160192,"num_objects":6001,"num_object_clones":0,"num_object_copies":24004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2248,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6001,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6771,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3753}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'6684","reported_seq":6758,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:46.283734+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:38:46.283734+0000","last_peered":"2024-09-05T16:38:46.283734+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:38:46.283734+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6684,"log_dups_size":0,"ondisk_log_size":6684,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48968864,"num_objects":5978,"num_object_clones":0,"num_object_copies":23912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5978,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6682,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3833}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'6774","reported_seq":6845,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:46.464329+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:38:46.464329+0000","last_peered":"2024-09-05T16:38:46.464329+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:38:46.464329+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6774,"log_dups_size":0,"ondisk_log_size":6774,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49190048,"num_objects":6005,"num_object_clones":0,"num_object_copies":24020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6005,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6772,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3767}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'6781","reported_seq":6840,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:46.167975+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:38:46.167975+0000","last_peered":"2024-09-05T16:38:46.167975+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:38:46.167975+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6781,"log_dups_size":0,"ondisk_log_size":6781,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49088832,"num_objects":5993,"num_object_clones":0,"num_object_copies":23972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5993,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6778,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3732}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'6816","reported_seq":6908,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:46.634548+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:38:46.634548+0000","last_peered":"2024-09-05T16:38:46.634548+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:38:46.634548+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6816,"log_dups_size":0,"ondisk_log_size":6816,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49099936,"num_objects":5994,"num_object_clones":0,"num_object_copies":23976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5994,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6814,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3737}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'6953","reported_seq":7017,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:46.147498+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:38:46.147498+0000","last_peered":"2024-09-05T16:38:46.147498+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:38:46.147498+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6953,"log_dups_size":0,"ondisk_log_size":6953,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50540272,"num_objects":6170,"num_object_clones":0,"num_object_copies":24680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6170,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6950,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3861}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'6757","reported_seq":6798,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:46.065207+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:38:46.065207+0000","last_peered":"2024-09-05T16:38:46.065207+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:38:46.065207+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6757,"log_dups_size":0,"ondisk_log_size":6757,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49166928,"num_objects":6002,"num_object_clones":0,"num_object_copies":24008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6002,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6756,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3738}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'6798","reported_seq":8480,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:46.701702+0000","last_change":"2024-09-05T16:37:30.634570+0000","last_active":"2024-09-05T16:38:46.701702+0000","last_peered":"2024-09-05T16:38:46.701702+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:38:46.701702+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6798,"log_dups_size":0,"ondisk_log_size":6798,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49240656,"num_objects":6011,"num_object_clones":0,"num_object_copies":24044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1493,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6011,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6797,"num_write_kb":51231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":802,"num_bytes_recovered":6561792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4518}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'6964","reported_seq":7025,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:46.076076+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:38:46.076076+0000","last_peered":"2024-09-05T16:38:46.076076+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:38:46.076076+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6964,"log_dups_size":0,"ondisk_log_size":6964,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50476192,"num_objects":6162,"num_object_clones":0,"num_object_copies":24648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2357,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6162,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6962,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3807}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'6808","reported_seq":6896,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:46.717422+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:38:46.717422+0000","last_peered":"2024-09-05T16:38:46.717422+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:38:46.717422+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6808,"log_dups_size":0,"ondisk_log_size":6808,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49353888,"num_objects":6025,"num_object_clones":0,"num_object_copies":24100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6025,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6806,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3708}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'6795","reported_seq":6868,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:46.533847+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:38:46.533847+0000","last_peered":"2024-09-05T16:38:46.533847+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:38:46.533847+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6795,"log_dups_size":0,"ondisk_log_size":6795,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48985248,"num_objects":5980,"num_object_clones":0,"num_object_copies":23920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2237,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5980,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6793,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3745}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":788261944,"num_objects":96229,"num_object_clones":0,"num_object_copies":384916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33081,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":96229,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":103008,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":851,"num_bytes_recovered":6561792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1462857728,"data_stored":1462857728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":108733,"ondisk_log_size":108733,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084346,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":756480,"kb_used_data":386020,"kb_used_omap":4,"kb_used_meta":370427,"kb_avail":93615360,"statfs":{"total":96636764160,"available":95862128640,"internally_reserved":0,"allocated":395284480,"data_stored":395092439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":379317971},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,1,2,1,6,0,0,0,0,0,3],"upper_bound":8192},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986223,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":545188,"kb_used_data":274704,"kb_used_omap":4,"kb_used_meta":270459,"kb_avail":93826652,"statfs":{"total":96636764160,"available":96078491648,"internally_reserved":0,"allocated":281296896,"data_stored":281104855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":276950738},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,7,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673029,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":777244,"kb_used_data":385100,"kb_used_omap":1,"kb_used_meta":392126,"kb_avail":93594596,"statfs":{"total":96636764160,"available":95840866304,"internally_reserved":0,"allocated":394342400,"data_stored":394150359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":401537477},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2,5,3,3,1,0,0,0,0,0,2],"upper_bound":16384},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182460,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":760152,"kb_used_data":385608,"kb_used_omap":4,"kb_used_meta":374523,"kb_avail":93611688,"statfs":{"total":96636764160,"available":95858368512,"internally_reserved":0,"allocated":394862592,"data_stored":394670551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":383512269},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,0,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":393891840,"data_stored":393891840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":280928256,"data_stored":280928256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":393781248,"data_stored":393781248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":394256384,"data_stored":394256384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:38:46.946 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:38:47.576 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:38:47.577 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:38:47.612 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":191,"stamp":"2024-09-05T16:38:46.891297+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":788261944,"num_objects":96229,"num_object_clones":0,"num_object_copies":384916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33081,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":96229,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":103008,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":851,"num_bytes_recovered":6561792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":108733,"ondisk_log_size":108733,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2839064,"kb_used_data":1431432,"kb_used_omap":14,"kb_used_meta":1407537,"kb_avail":374648296,"statfs":{"total":386547056640,"available":383639855104,"internally_reserved":0,"allocated":1465786368,"data_stored":1465018204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1441318455},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,4,7,14,11,11,1,0,0,0,0,3,2],"upper_bound":16384},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-32317440,"num_objects":-3945,"num_object_clones":0,"num_object_copies":-15780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3945,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3945,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":191,"num_bytes_recovered":1359872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004822"},"pg_stats":[{"pgid":"1.f","version":"35'6849","reported_seq":6898,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:46.235798+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:38:46.235798+0000","last_peered":"2024-09-05T16:38:46.235798+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:38:46.235798+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6849,"log_dups_size":0,"ondisk_log_size":6849,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49602560,"num_objects":6055,"num_object_clones":0,"num_object_copies":24220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2261,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6055,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6849,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3794}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'6672","reported_seq":6743,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:46.555650+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:38:46.555650+0000","last_peered":"2024-09-05T16:38:46.555650+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:38:46.555650+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6672,"log_dups_size":0,"ondisk_log_size":6672,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48182432,"num_objects":5882,"num_object_clones":0,"num_object_copies":23528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2268,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5882,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6670,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3616}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'6745","reported_seq":6828,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:46.588306+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:38:46.588306+0000","last_peered":"2024-09-05T16:38:46.588306+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:38:46.588306+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6745,"log_dups_size":0,"ondisk_log_size":6745,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49207912,"num_objects":6008,"num_object_clones":0,"num_object_copies":24032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6008,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6744,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3759}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'6865","reported_seq":11588,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-05T16:38:45.730564+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:38:45.730564+0000","last_peered":"2024-09-05T16:38:45.730564+0000","last_clean":"2024-09-05T16:38:45.730564+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:38:45.730564+0000","last_undegraded":"2024-09-05T16:38:45.730564+0000","last_fullsized":"2024-09-05T16:38:45.730564+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":6865,"log_dups_size":0,"ondisk_log_size":6865,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49717248,"num_objects":6069,"num_object_clones":0,"num_object_copies":24276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6069,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1165,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'6701","reported_seq":6766,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:46.072775+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:38:46.072775+0000","last_peered":"2024-09-05T16:38:46.072775+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:38:46.072775+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6701,"log_dups_size":0,"ondisk_log_size":6701,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48280736,"num_objects":5894,"num_object_clones":0,"num_object_copies":23576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2161,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5894,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6699,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3735}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'6771","reported_seq":6822,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:46.161865+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:38:46.161865+0000","last_peered":"2024-09-05T16:38:46.161865+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:38:46.161865+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6771,"log_dups_size":0,"ondisk_log_size":6771,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49160192,"num_objects":6001,"num_object_clones":0,"num_object_copies":24004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2248,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6001,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6771,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3753}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'6684","reported_seq":6758,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:46.283734+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:38:46.283734+0000","last_peered":"2024-09-05T16:38:46.283734+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:38:46.283734+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6684,"log_dups_size":0,"ondisk_log_size":6684,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48968864,"num_objects":5978,"num_object_clones":0,"num_object_copies":23912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5978,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6682,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3833}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'6774","reported_seq":6845,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:46.464329+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:38:46.464329+0000","last_peered":"2024-09-05T16:38:46.464329+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:38:46.464329+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6774,"log_dups_size":0,"ondisk_log_size":6774,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49190048,"num_objects":6005,"num_object_clones":0,"num_object_copies":24020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6005,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6772,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3767}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'6781","reported_seq":6840,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:46.167975+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:38:46.167975+0000","last_peered":"2024-09-05T16:38:46.167975+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:38:46.167975+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6781,"log_dups_size":0,"ondisk_log_size":6781,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49088832,"num_objects":5993,"num_object_clones":0,"num_object_copies":23972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5993,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6778,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3732}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'6816","reported_seq":6908,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:46.634548+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:38:46.634548+0000","last_peered":"2024-09-05T16:38:46.634548+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:38:46.634548+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6816,"log_dups_size":0,"ondisk_log_size":6816,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49099936,"num_objects":5994,"num_object_clones":0,"num_object_copies":23976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5994,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6814,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3737}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'6953","reported_seq":7017,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:46.147498+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:38:46.147498+0000","last_peered":"2024-09-05T16:38:46.147498+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:38:46.147498+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6953,"log_dups_size":0,"ondisk_log_size":6953,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50540272,"num_objects":6170,"num_object_clones":0,"num_object_copies":24680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6170,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6950,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3861}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'6757","reported_seq":6798,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:46.065207+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:38:46.065207+0000","last_peered":"2024-09-05T16:38:46.065207+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:38:46.065207+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6757,"log_dups_size":0,"ondisk_log_size":6757,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49166928,"num_objects":6002,"num_object_clones":0,"num_object_copies":24008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6002,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6756,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3738}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'6798","reported_seq":8480,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:46.701702+0000","last_change":"2024-09-05T16:37:30.634570+0000","last_active":"2024-09-05T16:38:46.701702+0000","last_peered":"2024-09-05T16:38:46.701702+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:38:46.701702+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6798,"log_dups_size":0,"ondisk_log_size":6798,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49240656,"num_objects":6011,"num_object_clones":0,"num_object_copies":24044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1493,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6011,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6797,"num_write_kb":51231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":802,"num_bytes_recovered":6561792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4518}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'6964","reported_seq":7025,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:46.076076+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:38:46.076076+0000","last_peered":"2024-09-05T16:38:46.076076+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:38:46.076076+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6964,"log_dups_size":0,"ondisk_log_size":6964,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50476192,"num_objects":6162,"num_object_clones":0,"num_object_copies":24648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2357,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6162,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6962,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3807}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'6808","reported_seq":6896,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:46.717422+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:38:46.717422+0000","last_peered":"2024-09-05T16:38:46.717422+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:38:46.717422+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6808,"log_dups_size":0,"ondisk_log_size":6808,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49353888,"num_objects":6025,"num_object_clones":0,"num_object_copies":24100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6025,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6806,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3708}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'6795","reported_seq":6868,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:46.533847+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:38:46.533847+0000","last_peered":"2024-09-05T16:38:46.533847+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:38:46.533847+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6795,"log_dups_size":0,"ondisk_log_size":6795,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48985248,"num_objects":5980,"num_object_clones":0,"num_object_copies":23920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2237,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5980,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6793,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3745}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":788261944,"num_objects":96229,"num_object_clones":0,"num_object_copies":384916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33081,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":96229,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":103008,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":851,"num_bytes_recovered":6561792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1462857728,"data_stored":1462857728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":108733,"ondisk_log_size":108733,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084346,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":756480,"kb_used_data":386020,"kb_used_omap":4,"kb_used_meta":370427,"kb_avail":93615360,"statfs":{"total":96636764160,"available":95862128640,"internally_reserved":0,"allocated":395284480,"data_stored":395092439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":379317971},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,1,2,1,6,0,0,0,0,0,3],"upper_bound":8192},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986223,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":545188,"kb_used_data":274704,"kb_used_omap":4,"kb_used_meta":270459,"kb_avail":93826652,"statfs":{"total":96636764160,"available":96078491648,"internally_reserved":0,"allocated":281296896,"data_stored":281104855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":276950738},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,7,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673029,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":777244,"kb_used_data":385100,"kb_used_omap":1,"kb_used_meta":392126,"kb_avail":93594596,"statfs":{"total":96636764160,"available":95840866304,"internally_reserved":0,"allocated":394342400,"data_stored":394150359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":401537477},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2,5,3,3,1,0,0,0,0,0,2],"upper_bound":16384},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182460,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":760152,"kb_used_data":385608,"kb_used_omap":4,"kb_used_meta":374523,"kb_avail":93611688,"statfs":{"total":96636764160,"available":95858368512,"internally_reserved":0,"allocated":394862592,"data_stored":394670551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":383512269},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,0,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":393891840,"data_stored":393891840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":280928256,"data_stored":280928256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":393781248,"data_stored":393781248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":394256384,"data_stored":394256384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:38:47.613 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-05T16:38:47.613 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'6849", 'reported_seq': 6898, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:46.235798+0000', 'last_change': '2024-09-05T16:34:56.833497+0000', 'last_active': '2024-09-05T16:38:46.235798+0000', 'last_peered': '2024-09-05T16:38:46.235798+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.828289+0000', 'last_became_peered': '2024-09-05T16:34:56.828289+0000', 'last_unstale': '2024-09-05T16:38:46.235798+0000', 'last_undegraded': '2024-09-05T16:34:56.612160+0000', 'last_fullsized': '2024-09-05T16:34:56.557632+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6849, 'log_dups_size': 0, 'ondisk_log_size': 6849, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49602560, 'num_objects': 6055, 'num_object_clones': 0, 'num_object_copies': 24220, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2261, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6055, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6849, 'num_write_kb': 51616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3794}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:38:47.613 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-05T16:38:47.614 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "35'6672", 'reported_seq': 6743, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:46.555650+0000', 'last_change': '2024-09-05T16:34:56.825031+0000', 'last_active': '2024-09-05T16:38:46.555650+0000', 'last_peered': '2024-09-05T16:38:46.555650+0000', 'last_clean': '2024-09-05T16:33:53.200762+0000', 'last_became_active': '2024-09-05T16:34:56.820292+0000', 'last_became_peered': '2024-09-05T16:34:56.820292+0000', 'last_unstale': '2024-09-05T16:38:46.555650+0000', 'last_undegraded': '2024-09-05T16:34:56.605776+0000', 'last_fullsized': '2024-09-05T16:34:56.559383+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6672, 'log_dups_size': 0, 'ondisk_log_size': 6672, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48182432, 'num_objects': 5882, 'num_object_clones': 0, 'num_object_copies': 23528, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2268, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5882, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6670, 'num_write_kb': 50212, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3616}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:38:47.614 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:38:47.614 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'6745", 'reported_seq': 6828, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:46.588306+0000', 'last_change': '2024-09-05T16:34:56.867900+0000', 'last_active': '2024-09-05T16:38:46.588306+0000', 'last_peered': '2024-09-05T16:38:46.588306+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:38:46.588306+0000', 'last_undegraded': '2024-09-05T16:34:56.590972+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6745, 'log_dups_size': 0, 'ondisk_log_size': 6745, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49207912, 'num_objects': 6008, 'num_object_clones': 0, 'num_object_copies': 24032, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6008, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 6744, 'num_write_kb': 50999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3759}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:38:47.614 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:38:47.614 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'6701", 'reported_seq': 6766, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:46.072775+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:38:46.072775+0000', 'last_peered': '2024-09-05T16:38:46.072775+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:38:46.072775+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6701, 'log_dups_size': 0, 'ondisk_log_size': 6701, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48280736, 'num_objects': 5894, 'num_object_clones': 0, 'num_object_copies': 23576, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2161, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5894, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6699, 'num_write_kb': 50373, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3735}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:38:47.614 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:38:47.614 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'6771", 'reported_seq': 6822, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:46.161865+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:38:46.161865+0000', 'last_peered': '2024-09-05T16:38:46.161865+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:38:46.161865+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6771, 'log_dups_size': 0, 'ondisk_log_size': 6771, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49160192, 'num_objects': 6001, 'num_object_clones': 0, 'num_object_copies': 24004, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2248, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6001, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6771, 'num_write_kb': 51088, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3753}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:38:47.614 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:38:47.614 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'6684", 'reported_seq': 6758, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:46.283734+0000', 'last_change': '2024-09-05T16:34:56.903436+0000', 'last_active': '2024-09-05T16:38:46.283734+0000', 'last_peered': '2024-09-05T16:38:46.283734+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:38:46.283734+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6684, 'log_dups_size': 0, 'ondisk_log_size': 6684, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48968864, 'num_objects': 5978, 'num_object_clones': 0, 'num_object_copies': 23912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2147, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5978, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6682, 'num_write_kb': 50636, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3833}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:38:47.614 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-05T16:38:47.614 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "35'6774", 'reported_seq': 6845, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:46.464329+0000', 'last_change': '2024-09-05T16:34:56.883319+0000', 'last_active': '2024-09-05T16:38:46.464329+0000', 'last_peered': '2024-09-05T16:38:46.464329+0000', 'last_clean': '2024-09-05T16:33:53.505305+0000', 'last_became_active': '2024-09-05T16:34:56.873389+0000', 'last_became_peered': '2024-09-05T16:34:56.873389+0000', 'last_unstale': '2024-09-05T16:38:46.464329+0000', 'last_undegraded': '2024-09-05T16:34:56.602567+0000', 'last_fullsized': '2024-09-05T16:34:56.559958+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6774, 'log_dups_size': 0, 'ondisk_log_size': 6774, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49190048, 'num_objects': 6005, 'num_object_clones': 0, 'num_object_copies': 24020, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2240, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6005, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6772, 'num_write_kb': 51108, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3767}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:38:47.615 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:38:47.615 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'6781", 'reported_seq': 6840, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:46.167975+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:38:46.167975+0000', 'last_peered': '2024-09-05T16:38:46.167975+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:38:46.167975+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6781, 'log_dups_size': 0, 'ondisk_log_size': 6781, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49088832, 'num_objects': 5993, 'num_object_clones': 0, 'num_object_copies': 23972, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2264, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5993, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6778, 'num_write_kb': 51081, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3732}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:38:47.615 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:38:47.615 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'6816", 'reported_seq': 6908, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:46.634548+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:38:46.634548+0000', 'last_peered': '2024-09-05T16:38:46.634548+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:38:46.634548+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6816, 'log_dups_size': 0, 'ondisk_log_size': 6816, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49099936, 'num_objects': 5994, 'num_object_clones': 0, 'num_object_copies': 23976, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5994, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6814, 'num_write_kb': 51230, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3737}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:38:47.615 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:38:47.615 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'6953", 'reported_seq': 7017, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:46.147498+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:38:46.147498+0000', 'last_peered': '2024-09-05T16:38:46.147498+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:38:46.147498+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6953, 'log_dups_size': 0, 'ondisk_log_size': 6953, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50540272, 'num_objects': 6170, 'num_object_clones': 0, 'num_object_copies': 24680, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2312, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6170, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6950, 'num_write_kb': 52476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3861}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:38:47.615 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:38:47.615 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'6757", 'reported_seq': 6798, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:46.065207+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:38:46.065207+0000', 'last_peered': '2024-09-05T16:38:46.065207+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:38:46.065207+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6757, 'log_dups_size': 0, 'ondisk_log_size': 6757, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49166928, 'num_objects': 6002, 'num_object_clones': 0, 'num_object_copies': 24008, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2265, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6002, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6756, 'num_write_kb': 51031, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3738}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:38:47.615 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-05T16:38:47.615 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'6798", 'reported_seq': 8480, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:46.701702+0000', 'last_change': '2024-09-05T16:37:30.634570+0000', 'last_active': '2024-09-05T16:38:46.701702+0000', 'last_peered': '2024-09-05T16:38:46.701702+0000', 'last_clean': '2024-09-05T16:33:45.082039+0000', 'last_became_active': '2024-09-05T16:34:56.792386+0000', 'last_became_peered': '2024-09-05T16:34:56.792386+0000', 'last_unstale': '2024-09-05T16:38:46.701702+0000', 'last_undegraded': '2024-09-05T16:34:56.600595+0000', 'last_fullsized': '2024-09-05T16:34:56.558203+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6798, 'log_dups_size': 0, 'ondisk_log_size': 6798, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49240656, 'num_objects': 6011, 'num_object_clones': 0, 'num_object_copies': 24044, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1493, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6011, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6797, 'num_write_kb': 51231, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 802, 'num_bytes_recovered': 6561792, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 4518}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:38:47.615 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:38:47.616 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'6964", 'reported_seq': 7025, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:46.076076+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:38:46.076076+0000', 'last_peered': '2024-09-05T16:38:46.076076+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:38:46.076076+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6964, 'log_dups_size': 0, 'ondisk_log_size': 6964, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50476192, 'num_objects': 6162, 'num_object_clones': 0, 'num_object_copies': 24648, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2357, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6162, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6962, 'num_write_kb': 52500, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3807}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:38:47.616 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:38:47.616 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'6808", 'reported_seq': 6896, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:46.717422+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:38:46.717422+0000', 'last_peered': '2024-09-05T16:38:46.717422+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:38:46.717422+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6808, 'log_dups_size': 0, 'ondisk_log_size': 6808, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49353888, 'num_objects': 6025, 'num_object_clones': 0, 'num_object_copies': 24100, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2319, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6025, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6806, 'num_write_kb': 51325, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3708}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:38:47.616 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:38:47.616 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'6795", 'reported_seq': 6868, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:46.533847+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:38:46.533847+0000', 'last_peered': '2024-09-05T16:38:46.533847+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:38:46.533847+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6795, 'log_dups_size': 0, 'ondisk_log_size': 6795, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48985248, 'num_objects': 5980, 'num_object_clones': 0, 'num_object_copies': 23920, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2237, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5980, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6793, 'num_write_kb': 51092, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3745}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:38:47.616 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:38:48.375 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:38:48.375 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 33081/384916 objects degraded (8.594%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":331,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":96229,"data_bytes":788261944,"bytes_used":2907201536,"bytes_avail":383639855104,"bytes_total":386547056640,"degraded_objects":33081,"degraded_total":384916,"degraded_ratio":0.085943426617755558,"recovering_objects_per_sec":15,"recovering_bytes_per_sec":113277,"recovering_keys_per_sec":0,"num_objects_recovered":191,"num_bytes_recovered":1359872,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":328},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:38:48.409 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:38:48.409 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:38:49.074 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:38:49.074 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:38:49.104 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":192,"stamp":"2024-09-05T16:38:48.891980+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":788261944,"num_objects":96229,"num_object_clones":0,"num_object_copies":384916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33081,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":96229,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":103008,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":851,"num_bytes_recovered":6561792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":108733,"ondisk_log_size":108733,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2839064,"kb_used_data":1431432,"kb_used_omap":14,"kb_used_meta":1407537,"kb_avail":374648296,"statfs":{"total":386547056640,"available":383639855104,"internally_reserved":0,"allocated":1465786368,"data_stored":1465018204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1441318455},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,4,7,14,11,11,1,0,0,0,0,3,2],"upper_bound":16384},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21487616,"num_objects":-2623,"num_object_clones":0,"num_object_copies":-10492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-123,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2623,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2623,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":123,"num_bytes_recovered":901120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004655"},"pg_stats":[{"pgid":"1.f","version":"35'6849","reported_seq":6898,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:46.235798+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:38:46.235798+0000","last_peered":"2024-09-05T16:38:46.235798+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:38:46.235798+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6849,"log_dups_size":0,"ondisk_log_size":6849,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49602560,"num_objects":6055,"num_object_clones":0,"num_object_copies":24220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2261,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6055,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6849,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3794}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'6672","reported_seq":6743,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:46.555650+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:38:46.555650+0000","last_peered":"2024-09-05T16:38:46.555650+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:38:46.555650+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6672,"log_dups_size":0,"ondisk_log_size":6672,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48182432,"num_objects":5882,"num_object_clones":0,"num_object_copies":23528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2268,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5882,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6670,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3616}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'6745","reported_seq":6828,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:46.588306+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:38:46.588306+0000","last_peered":"2024-09-05T16:38:46.588306+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:38:46.588306+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6745,"log_dups_size":0,"ondisk_log_size":6745,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49207912,"num_objects":6008,"num_object_clones":0,"num_object_copies":24032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6008,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6744,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3759}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'6865","reported_seq":11588,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-05T16:38:45.730564+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:38:45.730564+0000","last_peered":"2024-09-05T16:38:45.730564+0000","last_clean":"2024-09-05T16:38:45.730564+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:38:45.730564+0000","last_undegraded":"2024-09-05T16:38:45.730564+0000","last_fullsized":"2024-09-05T16:38:45.730564+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":6865,"log_dups_size":0,"ondisk_log_size":6865,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49717248,"num_objects":6069,"num_object_clones":0,"num_object_copies":24276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6069,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1165,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'6701","reported_seq":6766,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:46.072775+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:38:46.072775+0000","last_peered":"2024-09-05T16:38:46.072775+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:38:46.072775+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6701,"log_dups_size":0,"ondisk_log_size":6701,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48280736,"num_objects":5894,"num_object_clones":0,"num_object_copies":23576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2161,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5894,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6699,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3735}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'6771","reported_seq":6822,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:46.161865+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:38:46.161865+0000","last_peered":"2024-09-05T16:38:46.161865+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:38:46.161865+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6771,"log_dups_size":0,"ondisk_log_size":6771,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49160192,"num_objects":6001,"num_object_clones":0,"num_object_copies":24004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2248,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6001,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6771,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3753}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'6684","reported_seq":6758,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:46.283734+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:38:46.283734+0000","last_peered":"2024-09-05T16:38:46.283734+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:38:46.283734+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6684,"log_dups_size":0,"ondisk_log_size":6684,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48968864,"num_objects":5978,"num_object_clones":0,"num_object_copies":23912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5978,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6682,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3833}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'6774","reported_seq":6845,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:46.464329+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:38:46.464329+0000","last_peered":"2024-09-05T16:38:46.464329+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:38:46.464329+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6774,"log_dups_size":0,"ondisk_log_size":6774,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49190048,"num_objects":6005,"num_object_clones":0,"num_object_copies":24020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6005,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6772,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3767}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'6781","reported_seq":6840,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:46.167975+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:38:46.167975+0000","last_peered":"2024-09-05T16:38:46.167975+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:38:46.167975+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6781,"log_dups_size":0,"ondisk_log_size":6781,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49088832,"num_objects":5993,"num_object_clones":0,"num_object_copies":23972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5993,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6778,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3732}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'6816","reported_seq":6908,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:46.634548+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:38:46.634548+0000","last_peered":"2024-09-05T16:38:46.634548+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:38:46.634548+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6816,"log_dups_size":0,"ondisk_log_size":6816,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49099936,"num_objects":5994,"num_object_clones":0,"num_object_copies":23976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5994,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6814,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3737}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'6953","reported_seq":7017,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:46.147498+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:38:46.147498+0000","last_peered":"2024-09-05T16:38:46.147498+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:38:46.147498+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6953,"log_dups_size":0,"ondisk_log_size":6953,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50540272,"num_objects":6170,"num_object_clones":0,"num_object_copies":24680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6170,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6950,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3861}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'6757","reported_seq":6798,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:46.065207+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:38:46.065207+0000","last_peered":"2024-09-05T16:38:46.065207+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:38:46.065207+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6757,"log_dups_size":0,"ondisk_log_size":6757,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49166928,"num_objects":6002,"num_object_clones":0,"num_object_copies":24008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6002,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6756,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3738}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'6798","reported_seq":8480,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:46.701702+0000","last_change":"2024-09-05T16:37:30.634570+0000","last_active":"2024-09-05T16:38:46.701702+0000","last_peered":"2024-09-05T16:38:46.701702+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:38:46.701702+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6798,"log_dups_size":0,"ondisk_log_size":6798,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49240656,"num_objects":6011,"num_object_clones":0,"num_object_copies":24044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1493,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6011,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6797,"num_write_kb":51231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":802,"num_bytes_recovered":6561792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4518}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'6964","reported_seq":7025,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:46.076076+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:38:46.076076+0000","last_peered":"2024-09-05T16:38:46.076076+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:38:46.076076+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6964,"log_dups_size":0,"ondisk_log_size":6964,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50476192,"num_objects":6162,"num_object_clones":0,"num_object_copies":24648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2357,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6162,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6962,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3807}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'6808","reported_seq":6896,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:46.717422+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:38:46.717422+0000","last_peered":"2024-09-05T16:38:46.717422+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:38:46.717422+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6808,"log_dups_size":0,"ondisk_log_size":6808,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49353888,"num_objects":6025,"num_object_clones":0,"num_object_copies":24100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6025,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6806,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3708}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'6795","reported_seq":6868,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:46.533847+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:38:46.533847+0000","last_peered":"2024-09-05T16:38:46.533847+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:38:46.533847+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6795,"log_dups_size":0,"ondisk_log_size":6795,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48985248,"num_objects":5980,"num_object_clones":0,"num_object_copies":23920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2237,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5980,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6793,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3745}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":788261944,"num_objects":96229,"num_object_clones":0,"num_object_copies":384916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33081,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":96229,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":103008,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":851,"num_bytes_recovered":6561792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1462857728,"data_stored":1462857728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":108733,"ondisk_log_size":108733,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084346,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":756480,"kb_used_data":386020,"kb_used_omap":4,"kb_used_meta":370427,"kb_avail":93615360,"statfs":{"total":96636764160,"available":95862128640,"internally_reserved":0,"allocated":395284480,"data_stored":395092439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":379317971},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,1,2,1,6,0,0,0,0,0,3],"upper_bound":8192},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986223,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":545188,"kb_used_data":274704,"kb_used_omap":4,"kb_used_meta":270459,"kb_avail":93826652,"statfs":{"total":96636764160,"available":96078491648,"internally_reserved":0,"allocated":281296896,"data_stored":281104855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":276950738},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,7,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673029,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":777244,"kb_used_data":385100,"kb_used_omap":1,"kb_used_meta":392126,"kb_avail":93594596,"statfs":{"total":96636764160,"available":95840866304,"internally_reserved":0,"allocated":394342400,"data_stored":394150359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":401537477},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2,5,3,3,1,0,0,0,0,0,2],"upper_bound":16384},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182460,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":760152,"kb_used_data":385608,"kb_used_omap":4,"kb_used_meta":374523,"kb_avail":93611688,"statfs":{"total":96636764160,"available":95858368512,"internally_reserved":0,"allocated":394862592,"data_stored":394670551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":383512269},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,0,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":393891840,"data_stored":393891840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":280928256,"data_stored":280928256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":393781248,"data_stored":393781248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":394256384,"data_stored":394256384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:38:52.106 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:38:52.736 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:38:52.736 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:38:52.758 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":193,"stamp":"2024-09-05T16:38:50.892627+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":787631160,"num_objects":96152,"num_object_clones":0,"num_object_copies":384608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33081,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":96152,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":103085,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":851,"num_bytes_recovered":6561792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":108810,"ondisk_log_size":108810,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2842860,"kb_used_data":1431068,"kb_used_omap":14,"kb_used_meta":1411697,"kb_avail":374644500,"statfs":{"total":386547056640,"available":383635968000,"internally_reserved":0,"allocated":1465413632,"data_stored":1464645468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1445578295},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,3,4,3,8,12,16,1,0,0,0,0,3,2],"upper_bound":16384},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22118400,"num_objects":-2700,"num_object_clones":0,"num_object_copies":-10800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-123,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2700,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2700,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":123,"num_bytes_recovered":901120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004386"},"pg_stats":[{"pgid":"1.f","version":"35'6849","reported_seq":6898,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:46.235798+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:38:46.235798+0000","last_peered":"2024-09-05T16:38:46.235798+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:38:46.235798+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6849,"log_dups_size":0,"ondisk_log_size":6849,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49602560,"num_objects":6055,"num_object_clones":0,"num_object_copies":24220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2261,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6055,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6849,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3794}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'6672","reported_seq":6743,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:46.555650+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:38:46.555650+0000","last_peered":"2024-09-05T16:38:46.555650+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:38:46.555650+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6672,"log_dups_size":0,"ondisk_log_size":6672,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48182432,"num_objects":5882,"num_object_clones":0,"num_object_copies":23528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2268,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5882,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6670,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3616}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'6745","reported_seq":6828,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:46.588306+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:38:46.588306+0000","last_peered":"2024-09-05T16:38:46.588306+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:38:46.588306+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6745,"log_dups_size":0,"ondisk_log_size":6745,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49207912,"num_objects":6008,"num_object_clones":0,"num_object_copies":24032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6008,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6744,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3759}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'6942","reported_seq":11665,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-05T16:38:50.761711+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:38:50.761711+0000","last_peered":"2024-09-05T16:38:50.761711+0000","last_clean":"2024-09-05T16:38:50.761711+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:38:50.761711+0000","last_undegraded":"2024-09-05T16:38:50.761711+0000","last_fullsized":"2024-09-05T16:38:50.761711+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":6942,"log_dups_size":0,"ondisk_log_size":6942,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49086464,"num_objects":5992,"num_object_clones":0,"num_object_copies":23968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5992,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1242,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'6701","reported_seq":6766,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:46.072775+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:38:46.072775+0000","last_peered":"2024-09-05T16:38:46.072775+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:38:46.072775+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6701,"log_dups_size":0,"ondisk_log_size":6701,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48280736,"num_objects":5894,"num_object_clones":0,"num_object_copies":23576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2161,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5894,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6699,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3735}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'6771","reported_seq":6822,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:46.161865+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:38:46.161865+0000","last_peered":"2024-09-05T16:38:46.161865+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:38:46.161865+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6771,"log_dups_size":0,"ondisk_log_size":6771,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49160192,"num_objects":6001,"num_object_clones":0,"num_object_copies":24004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2248,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6001,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6771,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3753}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'6684","reported_seq":6758,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:46.283734+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:38:46.283734+0000","last_peered":"2024-09-05T16:38:46.283734+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:38:46.283734+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6684,"log_dups_size":0,"ondisk_log_size":6684,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48968864,"num_objects":5978,"num_object_clones":0,"num_object_copies":23912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5978,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6682,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3833}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'6774","reported_seq":6845,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:46.464329+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:38:46.464329+0000","last_peered":"2024-09-05T16:38:46.464329+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:38:46.464329+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6774,"log_dups_size":0,"ondisk_log_size":6774,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49190048,"num_objects":6005,"num_object_clones":0,"num_object_copies":24020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6005,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6772,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3767}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'6781","reported_seq":6840,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:46.167975+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:38:46.167975+0000","last_peered":"2024-09-05T16:38:46.167975+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:38:46.167975+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6781,"log_dups_size":0,"ondisk_log_size":6781,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49088832,"num_objects":5993,"num_object_clones":0,"num_object_copies":23972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5993,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6778,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3732}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'6816","reported_seq":6908,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:46.634548+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:38:46.634548+0000","last_peered":"2024-09-05T16:38:46.634548+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:38:46.634548+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6816,"log_dups_size":0,"ondisk_log_size":6816,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49099936,"num_objects":5994,"num_object_clones":0,"num_object_copies":23976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5994,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6814,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3737}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'6953","reported_seq":7017,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:46.147498+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:38:46.147498+0000","last_peered":"2024-09-05T16:38:46.147498+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:38:46.147498+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6953,"log_dups_size":0,"ondisk_log_size":6953,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50540272,"num_objects":6170,"num_object_clones":0,"num_object_copies":24680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6170,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6950,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3861}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'6757","reported_seq":6798,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:46.065207+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:38:46.065207+0000","last_peered":"2024-09-05T16:38:46.065207+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:38:46.065207+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6757,"log_dups_size":0,"ondisk_log_size":6757,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49166928,"num_objects":6002,"num_object_clones":0,"num_object_copies":24008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6002,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6756,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3738}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'6798","reported_seq":8480,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:46.701702+0000","last_change":"2024-09-05T16:37:30.634570+0000","last_active":"2024-09-05T16:38:46.701702+0000","last_peered":"2024-09-05T16:38:46.701702+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:38:46.701702+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6798,"log_dups_size":0,"ondisk_log_size":6798,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49240656,"num_objects":6011,"num_object_clones":0,"num_object_copies":24044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1493,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6011,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6797,"num_write_kb":51231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":802,"num_bytes_recovered":6561792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4518}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'6964","reported_seq":7025,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:46.076076+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:38:46.076076+0000","last_peered":"2024-09-05T16:38:46.076076+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:38:46.076076+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6964,"log_dups_size":0,"ondisk_log_size":6964,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50476192,"num_objects":6162,"num_object_clones":0,"num_object_copies":24648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2357,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6162,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6962,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3807}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'6808","reported_seq":6896,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:46.717422+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:38:46.717422+0000","last_peered":"2024-09-05T16:38:46.717422+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:38:46.717422+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6808,"log_dups_size":0,"ondisk_log_size":6808,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49353888,"num_objects":6025,"num_object_clones":0,"num_object_copies":24100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6025,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6806,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3708}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'6795","reported_seq":6868,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:46.533847+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:38:46.533847+0000","last_peered":"2024-09-05T16:38:46.533847+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:38:46.533847+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6795,"log_dups_size":0,"ondisk_log_size":6795,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48985248,"num_objects":5980,"num_object_clones":0,"num_object_copies":23920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2237,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5980,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6793,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3745}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":787631160,"num_objects":96152,"num_object_clones":0,"num_object_copies":384608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33081,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":96152,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":103085,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":851,"num_bytes_recovered":6561792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1462493184,"data_stored":1462493184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":108810,"ondisk_log_size":108810,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084346,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":756480,"kb_used_data":386020,"kb_used_omap":4,"kb_used_meta":370427,"kb_avail":93615360,"statfs":{"total":96636764160,"available":95862128640,"internally_reserved":0,"allocated":395284480,"data_stored":395092439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":379317971},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,1,2,1,6,0,0,0,0,0,3],"upper_bound":8192},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986224,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":548984,"kb_used_data":274340,"kb_used_omap":4,"kb_used_meta":274619,"kb_avail":93822856,"statfs":{"total":96636764160,"available":96074604544,"internally_reserved":0,"allocated":280924160,"data_stored":280732119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":281210578},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,1,3,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673029,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":777244,"kb_used_data":385100,"kb_used_omap":1,"kb_used_meta":392126,"kb_avail":93594596,"statfs":{"total":96636764160,"available":95840866304,"internally_reserved":0,"allocated":394342400,"data_stored":394150359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":401537477},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2,5,3,3,1,0,0,0,0,0,2],"upper_bound":16384},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182460,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":760152,"kb_used_data":385608,"kb_used_omap":4,"kb_used_meta":374523,"kb_avail":93611688,"statfs":{"total":96636764160,"available":95858368512,"internally_reserved":0,"allocated":394862592,"data_stored":394670551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":383512269},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,0,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":393891840,"data_stored":393891840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":280563712,"data_stored":280563712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":393781248,"data_stored":393781248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":394256384,"data_stored":394256384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:38:52.760 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:38:53.318 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:38:53.318 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:38:53.340 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":194,"stamp":"2024-09-05T16:38:52.893536+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":777931832,"num_objects":94968,"num_object_clones":0,"num_object_copies":379872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33020,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":94968,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":104269,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":912,"num_bytes_recovered":7061504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":109994,"ondisk_log_size":109994,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2876308,"kb_used_data":1415928,"kb_used_omap":14,"kb_used_meta":1460273,"kb_avail":374611052,"statfs":{"total":386547056640,"available":383601717248,"internally_reserved":0,"allocated":1449910272,"data_stored":1449142108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1495320119},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,3,1,3,12,6,12,0,0,0,0,0,0,3,2],"upper_bound":32768},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-31064064,"num_objects":-3792,"num_object_clones":0,"num_object_copies":-15168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-184,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3792,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3792,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":184,"num_bytes_recovered":1400832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004415"},"pg_stats":[{"pgid":"1.f","version":"35'6921","reported_seq":6970,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:51.218341+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:38:51.218341+0000","last_peered":"2024-09-05T16:38:51.218341+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:38:51.218341+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6921,"log_dups_size":0,"ondisk_log_size":6921,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49012736,"num_objects":5983,"num_object_clones":0,"num_object_copies":23932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2261,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5983,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6921,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3722}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'6756","reported_seq":6827,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:51.499235+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:38:51.499235+0000","last_peered":"2024-09-05T16:38:51.499235+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:38:51.499235+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6756,"log_dups_size":0,"ondisk_log_size":6756,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47494304,"num_objects":5798,"num_object_clones":0,"num_object_copies":23192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2268,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6754,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3532}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'6835","reported_seq":6918,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:51.714273+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:38:51.714273+0000","last_peered":"2024-09-05T16:38:51.714273+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:38:51.714273+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6835,"log_dups_size":0,"ondisk_log_size":6835,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48470632,"num_objects":5918,"num_object_clones":0,"num_object_copies":23672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5918,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6834,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3669}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'6942","reported_seq":11665,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-05T16:38:50.761711+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:38:50.761711+0000","last_peered":"2024-09-05T16:38:50.761711+0000","last_clean":"2024-09-05T16:38:50.761711+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:38:50.761711+0000","last_undegraded":"2024-09-05T16:38:50.761711+0000","last_fullsized":"2024-09-05T16:38:50.761711+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":6942,"log_dups_size":0,"ondisk_log_size":6942,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49086464,"num_objects":5992,"num_object_clones":0,"num_object_copies":23968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5992,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1242,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'6775","reported_seq":6840,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:50.971880+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:38:50.971880+0000","last_peered":"2024-09-05T16:38:50.971880+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:38:50.971880+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6775,"log_dups_size":0,"ondisk_log_size":6775,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47674528,"num_objects":5820,"num_object_clones":0,"num_object_copies":23280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2161,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5820,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6773,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3661}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'6838","reported_seq":6889,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:51.192289+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:38:51.192289+0000","last_peered":"2024-09-05T16:38:51.192289+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:38:51.192289+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6838,"log_dups_size":0,"ondisk_log_size":6838,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48611328,"num_objects":5934,"num_object_clones":0,"num_object_copies":23736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2248,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5934,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6838,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3686}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'6762","reported_seq":6836,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:51.292923+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:38:51.292923+0000","last_peered":"2024-09-05T16:38:51.292923+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:38:51.292923+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6762,"log_dups_size":0,"ondisk_log_size":6762,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48329888,"num_objects":5900,"num_object_clones":0,"num_object_copies":23600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5900,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6760,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3755}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'6845","reported_seq":6916,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:51.368778+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:38:51.368778+0000","last_peered":"2024-09-05T16:38:51.368778+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:38:51.368778+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6845,"log_dups_size":0,"ondisk_log_size":6845,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48608416,"num_objects":5934,"num_object_clones":0,"num_object_copies":23736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5934,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6843,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3696}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'6854","reported_seq":6915,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:51.134158+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:38:51.134158+0000","last_peered":"2024-09-05T16:38:51.134158+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:38:51.134158+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6854,"log_dups_size":0,"ondisk_log_size":6854,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48490816,"num_objects":5920,"num_object_clones":0,"num_object_copies":23680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5920,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6851,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3659}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'6897","reported_seq":6989,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:51.723039+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:38:51.723039+0000","last_peered":"2024-09-05T16:38:51.723039+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:38:51.723039+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6897,"log_dups_size":0,"ondisk_log_size":6897,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48436384,"num_objects":5913,"num_object_clones":0,"num_object_copies":23652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5913,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6895,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3656}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'7054","reported_seq":7119,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:51.120603+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:38:51.120603+0000","last_peered":"2024-09-05T16:38:51.120603+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:38:51.120603+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7054,"log_dups_size":0,"ondisk_log_size":7054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49712880,"num_objects":6069,"num_object_clones":0,"num_object_copies":24276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6069,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7051,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3760}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'6833","reported_seq":6874,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:51.196989+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:38:51.196989+0000","last_peered":"2024-09-05T16:38:51.196989+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:38:51.196989+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6833,"log_dups_size":0,"ondisk_log_size":6833,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48544336,"num_objects":5926,"num_object_clones":0,"num_object_copies":23704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6832,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3662}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'6876","reported_seq":8680,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:51.706047+0000","last_change":"2024-09-05T16:37:30.634570+0000","last_active":"2024-09-05T16:38:51.706047+0000","last_peered":"2024-09-05T16:38:51.706047+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:38:51.706047+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6876,"log_dups_size":0,"ondisk_log_size":6876,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48601680,"num_objects":5933,"num_object_clones":0,"num_object_copies":23732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1432,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5933,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6875,"num_write_kb":51231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":863,"num_bytes_recovered":7061504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4501}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'7054","reported_seq":7115,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:51.281884+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:38:51.281884+0000","last_peered":"2024-09-05T16:38:51.281884+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:38:51.281884+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7054,"log_dups_size":0,"ondisk_log_size":7054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49738912,"num_objects":6072,"num_object_clones":0,"num_object_copies":24288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2357,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6072,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7052,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3717}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'6883","reported_seq":6972,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:51.697036+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:38:51.697036+0000","last_peered":"2024-09-05T16:38:51.697036+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:38:51.697036+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6883,"log_dups_size":0,"ondisk_log_size":6883,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48739488,"num_objects":5950,"num_object_clones":0,"num_object_copies":23800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5950,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6881,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3633}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'6869","reported_seq":6942,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:51.583929+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:38:51.583929+0000","last_peered":"2024-09-05T16:38:51.583929+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:38:51.583929+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6869,"log_dups_size":0,"ondisk_log_size":6869,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48379040,"num_objects":5906,"num_object_clones":0,"num_object_copies":23624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2237,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5906,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6867,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3671}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":777931832,"num_objects":94968,"num_object_clones":0,"num_object_copies":379872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33020,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":94968,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":104269,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":912,"num_bytes_recovered":7061504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1446924288,"data_stored":1446924288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":109994,"ondisk_log_size":109994,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084347,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":773716,"kb_used_data":381048,"kb_used_omap":4,"kb_used_meta":392635,"kb_avail":93598124,"statfs":{"total":96636764160,"available":95844478976,"internally_reserved":0,"allocated":390193152,"data_stored":390001111,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":402058963},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,5,0,3,0,0,0,0,0,0,3],"upper_bound":16384},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986224,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":548984,"kb_used_data":274340,"kb_used_omap":4,"kb_used_meta":274619,"kb_avail":93822856,"statfs":{"total":96636764160,"available":96074604544,"internally_reserved":0,"allocated":280924160,"data_stored":280732119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":281210578},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,1,3,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673030,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":776256,"kb_used_data":379948,"kb_used_omap":1,"kb_used_meta":396286,"kb_avail":93595584,"statfs":{"total":96636764160,"available":95841878016,"internally_reserved":0,"allocated":389066752,"data_stored":388874711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":405797317},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,0,3,2,0,0,0,0,0,0,0,0,2],"upper_bound":32768},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182461,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":777352,"kb_used_data":380592,"kb_used_omap":4,"kb_used_meta":396731,"kb_avail":93594488,"statfs":{"total":96636764160,"available":95840755712,"internally_reserved":0,"allocated":389726208,"data_stored":389534167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":406253261},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,3,1,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":388694016,"data_stored":388694016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":280563712,"data_stored":280563712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":388579328,"data_stored":388579328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":389087232,"data_stored":389087232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:38:53.342 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-05T16:38:53.342 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'6921", 'reported_seq': 6970, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:51.218341+0000', 'last_change': '2024-09-05T16:34:56.833497+0000', 'last_active': '2024-09-05T16:38:51.218341+0000', 'last_peered': '2024-09-05T16:38:51.218341+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.828289+0000', 'last_became_peered': '2024-09-05T16:34:56.828289+0000', 'last_unstale': '2024-09-05T16:38:51.218341+0000', 'last_undegraded': '2024-09-05T16:34:56.612160+0000', 'last_fullsized': '2024-09-05T16:34:56.557632+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6921, 'log_dups_size': 0, 'ondisk_log_size': 6921, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49012736, 'num_objects': 5983, 'num_object_clones': 0, 'num_object_copies': 23932, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2261, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5983, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6921, 'num_write_kb': 51616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3722}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:38:53.342 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-05T16:38:53.343 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "35'6756", 'reported_seq': 6827, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:51.499235+0000', 'last_change': '2024-09-05T16:34:56.825031+0000', 'last_active': '2024-09-05T16:38:51.499235+0000', 'last_peered': '2024-09-05T16:38:51.499235+0000', 'last_clean': '2024-09-05T16:33:53.200762+0000', 'last_became_active': '2024-09-05T16:34:56.820292+0000', 'last_became_peered': '2024-09-05T16:34:56.820292+0000', 'last_unstale': '2024-09-05T16:38:51.499235+0000', 'last_undegraded': '2024-09-05T16:34:56.605776+0000', 'last_fullsized': '2024-09-05T16:34:56.559383+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6756, 'log_dups_size': 0, 'ondisk_log_size': 6756, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47494304, 'num_objects': 5798, 'num_object_clones': 0, 'num_object_copies': 23192, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2268, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5798, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6754, 'num_write_kb': 50212, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3532}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:38:53.343 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:38:53.343 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'6835", 'reported_seq': 6918, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:51.714273+0000', 'last_change': '2024-09-05T16:34:56.867900+0000', 'last_active': '2024-09-05T16:38:51.714273+0000', 'last_peered': '2024-09-05T16:38:51.714273+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:38:51.714273+0000', 'last_undegraded': '2024-09-05T16:34:56.590972+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6835, 'log_dups_size': 0, 'ondisk_log_size': 6835, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48470632, 'num_objects': 5918, 'num_object_clones': 0, 'num_object_copies': 23672, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5918, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 6834, 'num_write_kb': 50999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3669}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:38:53.343 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:38:53.344 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'6775", 'reported_seq': 6840, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:50.971880+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:38:50.971880+0000', 'last_peered': '2024-09-05T16:38:50.971880+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:38:50.971880+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6775, 'log_dups_size': 0, 'ondisk_log_size': 6775, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47674528, 'num_objects': 5820, 'num_object_clones': 0, 'num_object_copies': 23280, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2161, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5820, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6773, 'num_write_kb': 50373, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3661}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:38:53.344 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:38:53.344 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'6838", 'reported_seq': 6889, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:51.192289+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:38:51.192289+0000', 'last_peered': '2024-09-05T16:38:51.192289+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:38:51.192289+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6838, 'log_dups_size': 0, 'ondisk_log_size': 6838, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48611328, 'num_objects': 5934, 'num_object_clones': 0, 'num_object_copies': 23736, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2248, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5934, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6838, 'num_write_kb': 51088, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3686}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:38:53.344 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:38:53.344 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'6762", 'reported_seq': 6836, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:51.292923+0000', 'last_change': '2024-09-05T16:34:56.903436+0000', 'last_active': '2024-09-05T16:38:51.292923+0000', 'last_peered': '2024-09-05T16:38:51.292923+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:38:51.292923+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6762, 'log_dups_size': 0, 'ondisk_log_size': 6762, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48329888, 'num_objects': 5900, 'num_object_clones': 0, 'num_object_copies': 23600, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2147, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5900, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6760, 'num_write_kb': 50636, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3755}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:38:53.345 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-05T16:38:53.345 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "35'6845", 'reported_seq': 6916, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:51.368778+0000', 'last_change': '2024-09-05T16:34:56.883319+0000', 'last_active': '2024-09-05T16:38:51.368778+0000', 'last_peered': '2024-09-05T16:38:51.368778+0000', 'last_clean': '2024-09-05T16:33:53.505305+0000', 'last_became_active': '2024-09-05T16:34:56.873389+0000', 'last_became_peered': '2024-09-05T16:34:56.873389+0000', 'last_unstale': '2024-09-05T16:38:51.368778+0000', 'last_undegraded': '2024-09-05T16:34:56.602567+0000', 'last_fullsized': '2024-09-05T16:34:56.559958+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6845, 'log_dups_size': 0, 'ondisk_log_size': 6845, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48608416, 'num_objects': 5934, 'num_object_clones': 0, 'num_object_copies': 23736, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2240, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5934, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6843, 'num_write_kb': 51108, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3696}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:38:53.345 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:38:53.345 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'6854", 'reported_seq': 6915, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:51.134158+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:38:51.134158+0000', 'last_peered': '2024-09-05T16:38:51.134158+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:38:51.134158+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6854, 'log_dups_size': 0, 'ondisk_log_size': 6854, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48490816, 'num_objects': 5920, 'num_object_clones': 0, 'num_object_copies': 23680, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2264, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5920, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6851, 'num_write_kb': 51081, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3659}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:38:53.346 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:38:53.346 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'6897", 'reported_seq': 6989, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:51.723039+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:38:51.723039+0000', 'last_peered': '2024-09-05T16:38:51.723039+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:38:51.723039+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6897, 'log_dups_size': 0, 'ondisk_log_size': 6897, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48436384, 'num_objects': 5913, 'num_object_clones': 0, 'num_object_copies': 23652, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5913, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6895, 'num_write_kb': 51230, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3656}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:38:53.346 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:38:53.346 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'7054", 'reported_seq': 7119, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:51.120603+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:38:51.120603+0000', 'last_peered': '2024-09-05T16:38:51.120603+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:38:51.120603+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7054, 'log_dups_size': 0, 'ondisk_log_size': 7054, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49712880, 'num_objects': 6069, 'num_object_clones': 0, 'num_object_copies': 24276, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2312, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6069, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7051, 'num_write_kb': 52476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3760}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:38:53.347 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:38:53.347 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'6833", 'reported_seq': 6874, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:51.196989+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:38:51.196989+0000', 'last_peered': '2024-09-05T16:38:51.196989+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:38:51.196989+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6833, 'log_dups_size': 0, 'ondisk_log_size': 6833, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48544336, 'num_objects': 5926, 'num_object_clones': 0, 'num_object_copies': 23704, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2265, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5926, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6832, 'num_write_kb': 51031, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3662}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:38:53.348 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-05T16:38:53.348 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'6876", 'reported_seq': 8680, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:51.706047+0000', 'last_change': '2024-09-05T16:37:30.634570+0000', 'last_active': '2024-09-05T16:38:51.706047+0000', 'last_peered': '2024-09-05T16:38:51.706047+0000', 'last_clean': '2024-09-05T16:33:45.082039+0000', 'last_became_active': '2024-09-05T16:34:56.792386+0000', 'last_became_peered': '2024-09-05T16:34:56.792386+0000', 'last_unstale': '2024-09-05T16:38:51.706047+0000', 'last_undegraded': '2024-09-05T16:34:56.600595+0000', 'last_fullsized': '2024-09-05T16:34:56.558203+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6876, 'log_dups_size': 0, 'ondisk_log_size': 6876, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48601680, 'num_objects': 5933, 'num_object_clones': 0, 'num_object_copies': 23732, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1432, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5933, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6875, 'num_write_kb': 51231, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 863, 'num_bytes_recovered': 7061504, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 4501}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:38:53.348 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:38:53.349 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'7054", 'reported_seq': 7115, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:51.281884+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:38:51.281884+0000', 'last_peered': '2024-09-05T16:38:51.281884+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:38:51.281884+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7054, 'log_dups_size': 0, 'ondisk_log_size': 7054, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49738912, 'num_objects': 6072, 'num_object_clones': 0, 'num_object_copies': 24288, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2357, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6072, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7052, 'num_write_kb': 52500, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3717}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:38:53.349 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:38:53.349 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'6883", 'reported_seq': 6972, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:51.697036+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:38:51.697036+0000', 'last_peered': '2024-09-05T16:38:51.697036+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:38:51.697036+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6883, 'log_dups_size': 0, 'ondisk_log_size': 6883, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48739488, 'num_objects': 5950, 'num_object_clones': 0, 'num_object_copies': 23800, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2319, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5950, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6881, 'num_write_kb': 51325, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3633}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:38:53.349 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:38:53.349 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'6869", 'reported_seq': 6942, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:51.583929+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:38:51.583929+0000', 'last_peered': '2024-09-05T16:38:51.583929+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:38:51.583929+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6869, 'log_dups_size': 0, 'ondisk_log_size': 6869, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48379040, 'num_objects': 5906, 'num_object_clones': 0, 'num_object_copies': 23624, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2237, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5906, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6867, 'num_write_kb': 51092, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3671}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:38:53.350 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:38:54.080 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:38:54.081 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 33020/379872 objects degraded (8.692%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":337,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":94968,"data_bytes":777931832,"bytes_used":2945339392,"bytes_avail":383601717248,"bytes_total":386547056640,"degraded_objects":33020,"degraded_total":379872,"degraded_ratio":0.086924016510824695,"recovering_objects_per_sec":15,"recovering_bytes_per_sec":116693,"recovering_keys_per_sec":0,"num_objects_recovered":184,"num_bytes_recovered":1400832,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":315},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:38:54.113 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:38:54.113 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:38:54.719 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:38:54.719 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:38:54.754 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":194,"stamp":"2024-09-05T16:38:52.893536+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":777931832,"num_objects":94968,"num_object_clones":0,"num_object_copies":379872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33020,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":94968,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":104269,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":912,"num_bytes_recovered":7061504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":109994,"ondisk_log_size":109994,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2876308,"kb_used_data":1415928,"kb_used_omap":14,"kb_used_meta":1460273,"kb_avail":374611052,"statfs":{"total":386547056640,"available":383601717248,"internally_reserved":0,"allocated":1449910272,"data_stored":1449142108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1495320119},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,3,1,3,12,6,12,0,0,0,0,0,0,3,2],"upper_bound":32768},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-31064064,"num_objects":-3792,"num_object_clones":0,"num_object_copies":-15168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-184,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3792,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3792,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":184,"num_bytes_recovered":1400832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004415"},"pg_stats":[{"pgid":"1.f","version":"35'6921","reported_seq":6970,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:51.218341+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:38:51.218341+0000","last_peered":"2024-09-05T16:38:51.218341+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:38:51.218341+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6921,"log_dups_size":0,"ondisk_log_size":6921,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49012736,"num_objects":5983,"num_object_clones":0,"num_object_copies":23932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2261,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5983,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6921,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3722}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'6756","reported_seq":6827,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:51.499235+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:38:51.499235+0000","last_peered":"2024-09-05T16:38:51.499235+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:38:51.499235+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6756,"log_dups_size":0,"ondisk_log_size":6756,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47494304,"num_objects":5798,"num_object_clones":0,"num_object_copies":23192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2268,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6754,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3532}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'6835","reported_seq":6918,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:51.714273+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:38:51.714273+0000","last_peered":"2024-09-05T16:38:51.714273+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:38:51.714273+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6835,"log_dups_size":0,"ondisk_log_size":6835,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48470632,"num_objects":5918,"num_object_clones":0,"num_object_copies":23672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5918,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6834,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3669}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'6942","reported_seq":11665,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-05T16:38:50.761711+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:38:50.761711+0000","last_peered":"2024-09-05T16:38:50.761711+0000","last_clean":"2024-09-05T16:38:50.761711+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:38:50.761711+0000","last_undegraded":"2024-09-05T16:38:50.761711+0000","last_fullsized":"2024-09-05T16:38:50.761711+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":6942,"log_dups_size":0,"ondisk_log_size":6942,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49086464,"num_objects":5992,"num_object_clones":0,"num_object_copies":23968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5992,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1242,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'6775","reported_seq":6840,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:50.971880+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:38:50.971880+0000","last_peered":"2024-09-05T16:38:50.971880+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:38:50.971880+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6775,"log_dups_size":0,"ondisk_log_size":6775,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47674528,"num_objects":5820,"num_object_clones":0,"num_object_copies":23280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2161,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5820,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6773,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3661}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'6838","reported_seq":6889,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:51.192289+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:38:51.192289+0000","last_peered":"2024-09-05T16:38:51.192289+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:38:51.192289+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6838,"log_dups_size":0,"ondisk_log_size":6838,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48611328,"num_objects":5934,"num_object_clones":0,"num_object_copies":23736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2248,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5934,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6838,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3686}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'6762","reported_seq":6836,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:51.292923+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:38:51.292923+0000","last_peered":"2024-09-05T16:38:51.292923+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:38:51.292923+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6762,"log_dups_size":0,"ondisk_log_size":6762,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48329888,"num_objects":5900,"num_object_clones":0,"num_object_copies":23600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5900,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6760,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3755}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'6845","reported_seq":6916,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:51.368778+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:38:51.368778+0000","last_peered":"2024-09-05T16:38:51.368778+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:38:51.368778+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6845,"log_dups_size":0,"ondisk_log_size":6845,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48608416,"num_objects":5934,"num_object_clones":0,"num_object_copies":23736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5934,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6843,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3696}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'6854","reported_seq":6915,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:51.134158+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:38:51.134158+0000","last_peered":"2024-09-05T16:38:51.134158+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:38:51.134158+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6854,"log_dups_size":0,"ondisk_log_size":6854,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48490816,"num_objects":5920,"num_object_clones":0,"num_object_copies":23680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5920,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6851,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3659}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'6897","reported_seq":6989,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:51.723039+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:38:51.723039+0000","last_peered":"2024-09-05T16:38:51.723039+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:38:51.723039+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6897,"log_dups_size":0,"ondisk_log_size":6897,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48436384,"num_objects":5913,"num_object_clones":0,"num_object_copies":23652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5913,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6895,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3656}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'7054","reported_seq":7119,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:51.120603+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:38:51.120603+0000","last_peered":"2024-09-05T16:38:51.120603+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:38:51.120603+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7054,"log_dups_size":0,"ondisk_log_size":7054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49712880,"num_objects":6069,"num_object_clones":0,"num_object_copies":24276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6069,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7051,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3760}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'6833","reported_seq":6874,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:51.196989+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:38:51.196989+0000","last_peered":"2024-09-05T16:38:51.196989+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:38:51.196989+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6833,"log_dups_size":0,"ondisk_log_size":6833,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48544336,"num_objects":5926,"num_object_clones":0,"num_object_copies":23704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6832,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3662}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'6876","reported_seq":8680,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:51.706047+0000","last_change":"2024-09-05T16:37:30.634570+0000","last_active":"2024-09-05T16:38:51.706047+0000","last_peered":"2024-09-05T16:38:51.706047+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:38:51.706047+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6876,"log_dups_size":0,"ondisk_log_size":6876,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48601680,"num_objects":5933,"num_object_clones":0,"num_object_copies":23732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1432,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5933,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6875,"num_write_kb":51231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":863,"num_bytes_recovered":7061504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4501}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'7054","reported_seq":7115,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:51.281884+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:38:51.281884+0000","last_peered":"2024-09-05T16:38:51.281884+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:38:51.281884+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7054,"log_dups_size":0,"ondisk_log_size":7054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49738912,"num_objects":6072,"num_object_clones":0,"num_object_copies":24288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2357,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6072,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7052,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3717}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'6883","reported_seq":6972,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:51.697036+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:38:51.697036+0000","last_peered":"2024-09-05T16:38:51.697036+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:38:51.697036+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6883,"log_dups_size":0,"ondisk_log_size":6883,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48739488,"num_objects":5950,"num_object_clones":0,"num_object_copies":23800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5950,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6881,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3633}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'6869","reported_seq":6942,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:51.583929+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:38:51.583929+0000","last_peered":"2024-09-05T16:38:51.583929+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:38:51.583929+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6869,"log_dups_size":0,"ondisk_log_size":6869,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48379040,"num_objects":5906,"num_object_clones":0,"num_object_copies":23624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2237,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5906,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6867,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3671}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":777931832,"num_objects":94968,"num_object_clones":0,"num_object_copies":379872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33020,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":94968,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":104269,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":912,"num_bytes_recovered":7061504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1446924288,"data_stored":1446924288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":109994,"ondisk_log_size":109994,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084347,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":773716,"kb_used_data":381048,"kb_used_omap":4,"kb_used_meta":392635,"kb_avail":93598124,"statfs":{"total":96636764160,"available":95844478976,"internally_reserved":0,"allocated":390193152,"data_stored":390001111,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":402058963},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,5,0,3,0,0,0,0,0,0,3],"upper_bound":16384},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986224,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":548984,"kb_used_data":274340,"kb_used_omap":4,"kb_used_meta":274619,"kb_avail":93822856,"statfs":{"total":96636764160,"available":96074604544,"internally_reserved":0,"allocated":280924160,"data_stored":280732119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":281210578},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,1,3,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673030,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":776256,"kb_used_data":379948,"kb_used_omap":1,"kb_used_meta":396286,"kb_avail":93595584,"statfs":{"total":96636764160,"available":95841878016,"internally_reserved":0,"allocated":389066752,"data_stored":388874711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":405797317},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,0,3,2,0,0,0,0,0,0,0,0,2],"upper_bound":32768},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182461,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":777352,"kb_used_data":380592,"kb_used_omap":4,"kb_used_meta":396731,"kb_avail":93594488,"statfs":{"total":96636764160,"available":95840755712,"internally_reserved":0,"allocated":389726208,"data_stored":389534167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":406253261},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,3,1,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":388694016,"data_stored":388694016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":280563712,"data_stored":280563712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":388579328,"data_stored":388579328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":389087232,"data_stored":389087232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:38:57.757 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:38:58.336 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:38:58.337 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:38:58.366 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":196,"stamp":"2024-09-05T16:38:56.895031+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":767740984,"num_objects":93724,"num_object_clones":0,"num_object_copies":374896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32947,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":93724,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":105513,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":985,"num_bytes_recovered":7585792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111238,"ondisk_log_size":111238,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2869232,"kb_used_data":1400540,"kb_used_omap":14,"kb_used_meta":1468593,"kb_avail":374618128,"statfs":{"total":386547056640,"available":383608963072,"internally_reserved":0,"allocated":1434152960,"data_stored":1433384796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1503839799},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,1,3,13,5,8,3,2,0,0,1,0,0,0,0,2],"upper_bound":65536},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-31178752,"num_objects":-3806,"num_object_clones":0,"num_object_copies":-15224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-196,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3806,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3806,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":196,"num_bytes_recovered":1507328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004377"},"pg_stats":[{"pgid":"1.f","version":"35'6998","reported_seq":7051,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:56.225912+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:38:56.225912+0000","last_peered":"2024-09-05T16:38:56.225912+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:38:56.225912+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6998,"log_dups_size":0,"ondisk_log_size":6998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48381952,"num_objects":5906,"num_object_clones":0,"num_object_copies":23624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5906,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6998,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3647}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'6839","reported_seq":6910,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:56.417765+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:38:56.417765+0000","last_peered":"2024-09-05T16:38:56.417765+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:38:56.417765+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6839,"log_dups_size":0,"ondisk_log_size":6839,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46814368,"num_objects":5715,"num_object_clones":0,"num_object_copies":22860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2268,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5715,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6837,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3449}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'6924","reported_seq":7007,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:56.620417+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:38:56.620417+0000","last_peered":"2024-09-05T16:38:56.620417+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:38:56.620417+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6924,"log_dups_size":0,"ondisk_log_size":6924,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47741544,"num_objects":5829,"num_object_clones":0,"num_object_copies":23316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5829,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6923,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3580}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'7025","reported_seq":11748,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-05T16:38:55.829768+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:38:55.829768+0000","last_peered":"2024-09-05T16:38:55.829768+0000","last_clean":"2024-09-05T16:38:55.829768+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:38:55.829768+0000","last_undegraded":"2024-09-05T16:38:55.829768+0000","last_fullsized":"2024-09-05T16:38:55.829768+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":7025,"log_dups_size":0,"ondisk_log_size":7025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48406528,"num_objects":5909,"num_object_clones":0,"num_object_copies":23636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5909,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1325,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'6853","reported_seq":6918,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:56.291256+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:38:56.291256+0000","last_peered":"2024-09-05T16:38:56.291256+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:38:56.291256+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6853,"log_dups_size":0,"ondisk_log_size":6853,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47035552,"num_objects":5742,"num_object_clones":0,"num_object_copies":22968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2161,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5742,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6851,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3583}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'6920","reported_seq":6975,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:56.317923+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:38:56.317923+0000","last_peered":"2024-09-05T16:38:56.317923+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:38:56.317923+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6920,"log_dups_size":0,"ondisk_log_size":6920,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47939584,"num_objects":5852,"num_object_clones":0,"num_object_copies":23408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2246,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5852,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6920,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3606}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'6830","reported_seq":6904,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:56.409734+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:38:56.409734+0000","last_peered":"2024-09-05T16:38:56.409734+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:38:56.409734+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6830,"log_dups_size":0,"ondisk_log_size":6830,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47772832,"num_objects":5832,"num_object_clones":0,"num_object_copies":23328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5832,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6828,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3687}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'6927","reported_seq":6998,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:56.579349+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:38:56.579349+0000","last_peered":"2024-09-05T16:38:56.579349+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:38:56.579349+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6927,"log_dups_size":0,"ondisk_log_size":6927,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47936672,"num_objects":5852,"num_object_clones":0,"num_object_copies":23408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5852,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6925,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3614}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'6924","reported_seq":6985,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:56.268845+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:38:56.268845+0000","last_peered":"2024-09-05T16:38:56.268845+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:38:56.268845+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6924,"log_dups_size":0,"ondisk_log_size":6924,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47917376,"num_objects":5850,"num_object_clones":0,"num_object_copies":23400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5850,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6921,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3589}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'6977","reported_seq":7069,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:56.729551+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:38:56.729551+0000","last_peered":"2024-09-05T16:38:56.729551+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:38:56.729551+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6977,"log_dups_size":0,"ondisk_log_size":6977,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47781024,"num_objects":5833,"num_object_clones":0,"num_object_copies":23332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5833,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6975,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3576}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'7132","reported_seq":7199,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:56.291797+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:38:56.291797+0000","last_peered":"2024-09-05T16:38:56.291797+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:38:56.291797+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7132,"log_dups_size":0,"ondisk_log_size":7132,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49073904,"num_objects":5991,"num_object_clones":0,"num_object_copies":23964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2311,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5991,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7129,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3683}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'6897","reported_seq":6944,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:56.241029+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:38:56.241029+0000","last_peered":"2024-09-05T16:38:56.241029+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:38:56.241029+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6897,"log_dups_size":0,"ondisk_log_size":6897,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48020048,"num_objects":5862,"num_object_clones":0,"num_object_copies":23448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5862,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6896,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3601}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'6956","reported_seq":8888,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:56.745453+0000","last_change":"2024-09-05T16:37:30.634570+0000","last_active":"2024-09-05T16:38:56.745453+0000","last_peered":"2024-09-05T16:38:56.745453+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:38:56.745453+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6956,"log_dups_size":0,"ondisk_log_size":6956,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47946320,"num_objects":5853,"num_object_clones":0,"num_object_copies":23412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5853,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6955,"num_write_kb":51231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":927,"num_bytes_recovered":7585792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4485}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'7131","reported_seq":7194,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:56.131932+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:38:56.131932+0000","last_peered":"2024-09-05T16:38:56.131932+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:38:56.131932+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7131,"log_dups_size":0,"ondisk_log_size":7131,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49108128,"num_objects":5995,"num_object_clones":0,"num_object_copies":23980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2356,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5995,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7129,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3641}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'6955","reported_seq":7044,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:56.670746+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:38:56.670746+0000","last_peered":"2024-09-05T16:38:56.670746+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:38:56.670746+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6955,"log_dups_size":0,"ondisk_log_size":6955,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48149664,"num_objects":5878,"num_object_clones":0,"num_object_copies":23512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5878,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6953,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3561}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'6950","reported_seq":7023,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:56.616320+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:38:56.616320+0000","last_peered":"2024-09-05T16:38:56.616320+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:38:56.616320+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6950,"log_dups_size":0,"ondisk_log_size":6950,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47715488,"num_objects":5825,"num_object_clones":0,"num_object_copies":23300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2237,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5825,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6948,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3590}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":767740984,"num_objects":93724,"num_object_clones":0,"num_object_copies":374896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32947,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":93724,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":105513,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":985,"num_bytes_recovered":7585792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1431216128,"data_stored":1431216128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111238,"ondisk_log_size":111238,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084348,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":772844,"kb_used_data":376020,"kb_used_omap":4,"kb_used_meta":396795,"kb_avail":93598996,"statfs":{"total":96636764160,"available":95845371904,"internally_reserved":0,"allocated":385044480,"data_stored":384852439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":406318803},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,0,2,7,0,2,2,2,0,0,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986225,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":536108,"kb_used_data":273940,"kb_used_omap":4,"kb_used_meta":262139,"kb_avail":93835732,"statfs":{"total":96636764160,"available":96087789568,"internally_reserved":0,"allocated":280514560,"data_stored":280322519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":268431058},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673031,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":779516,"kb_used_data":374892,"kb_used_omap":1,"kb_used_meta":404606,"kb_avail":93592324,"statfs":{"total":96636764160,"available":95838539776,"internally_reserved":0,"allocated":383889408,"data_stored":383697367,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":414316997},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,1,2,0,0,0,0,0,0,0,0,0,2],"upper_bound":65536},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182462,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":780764,"kb_used_data":375688,"kb_used_omap":4,"kb_used_meta":405051,"kb_avail":93591076,"statfs":{"total":96636764160,"available":95837261824,"internally_reserved":0,"allocated":384704512,"data_stored":384512471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":414772941},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":383639552,"data_stored":383639552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":280129536,"data_stored":280129536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":383475712,"data_stored":383475712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":383971328,"data_stored":383971328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:38:58.367 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:38:58.930 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:38:58.931 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:38:58.954 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":197,"stamp":"2024-09-05T16:38:58.897142+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":767740984,"num_objects":93724,"num_object_clones":0,"num_object_copies":374896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32947,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":93724,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":105513,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":985,"num_bytes_recovered":7585792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111238,"ondisk_log_size":111238,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2869232,"kb_used_data":1400540,"kb_used_omap":14,"kb_used_meta":1468593,"kb_avail":374618128,"statfs":{"total":386547056640,"available":383608963072,"internally_reserved":0,"allocated":1434152960,"data_stored":1433384796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1503839799},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,1,3,13,5,8,3,2,0,0,1,0,0,0,0,2],"upper_bound":65536},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20520960,"num_objects":-2505,"num_object_clones":0,"num_object_copies":-10020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2505,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2505,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":134,"num_bytes_recovered":1024000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005844"},"pg_stats":[{"pgid":"1.f","version":"35'6998","reported_seq":7051,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:56.225912+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:38:56.225912+0000","last_peered":"2024-09-05T16:38:56.225912+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:38:56.225912+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6998,"log_dups_size":0,"ondisk_log_size":6998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48381952,"num_objects":5906,"num_object_clones":0,"num_object_copies":23624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5906,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6998,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3647}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'6839","reported_seq":6910,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:56.417765+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:38:56.417765+0000","last_peered":"2024-09-05T16:38:56.417765+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:38:56.417765+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6839,"log_dups_size":0,"ondisk_log_size":6839,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46814368,"num_objects":5715,"num_object_clones":0,"num_object_copies":22860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2268,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5715,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6837,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3449}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'6924","reported_seq":7007,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:56.620417+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:38:56.620417+0000","last_peered":"2024-09-05T16:38:56.620417+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:38:56.620417+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6924,"log_dups_size":0,"ondisk_log_size":6924,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47741544,"num_objects":5829,"num_object_clones":0,"num_object_copies":23316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5829,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6923,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3580}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'7025","reported_seq":11748,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-05T16:38:55.829768+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:38:55.829768+0000","last_peered":"2024-09-05T16:38:55.829768+0000","last_clean":"2024-09-05T16:38:55.829768+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:38:55.829768+0000","last_undegraded":"2024-09-05T16:38:55.829768+0000","last_fullsized":"2024-09-05T16:38:55.829768+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":7025,"log_dups_size":0,"ondisk_log_size":7025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48406528,"num_objects":5909,"num_object_clones":0,"num_object_copies":23636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5909,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1325,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'6853","reported_seq":6918,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:56.291256+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:38:56.291256+0000","last_peered":"2024-09-05T16:38:56.291256+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:38:56.291256+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6853,"log_dups_size":0,"ondisk_log_size":6853,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47035552,"num_objects":5742,"num_object_clones":0,"num_object_copies":22968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2161,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5742,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6851,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3583}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'6920","reported_seq":6975,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:56.317923+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:38:56.317923+0000","last_peered":"2024-09-05T16:38:56.317923+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:38:56.317923+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6920,"log_dups_size":0,"ondisk_log_size":6920,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47939584,"num_objects":5852,"num_object_clones":0,"num_object_copies":23408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2246,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5852,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6920,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3606}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'6830","reported_seq":6904,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:56.409734+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:38:56.409734+0000","last_peered":"2024-09-05T16:38:56.409734+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:38:56.409734+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6830,"log_dups_size":0,"ondisk_log_size":6830,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47772832,"num_objects":5832,"num_object_clones":0,"num_object_copies":23328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5832,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6828,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3687}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'6927","reported_seq":6998,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:56.579349+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:38:56.579349+0000","last_peered":"2024-09-05T16:38:56.579349+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:38:56.579349+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6927,"log_dups_size":0,"ondisk_log_size":6927,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47936672,"num_objects":5852,"num_object_clones":0,"num_object_copies":23408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5852,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6925,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3614}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'6924","reported_seq":6985,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:56.268845+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:38:56.268845+0000","last_peered":"2024-09-05T16:38:56.268845+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:38:56.268845+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6924,"log_dups_size":0,"ondisk_log_size":6924,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47917376,"num_objects":5850,"num_object_clones":0,"num_object_copies":23400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5850,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6921,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3589}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'6977","reported_seq":7069,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:56.729551+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:38:56.729551+0000","last_peered":"2024-09-05T16:38:56.729551+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:38:56.729551+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6977,"log_dups_size":0,"ondisk_log_size":6977,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47781024,"num_objects":5833,"num_object_clones":0,"num_object_copies":23332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5833,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6975,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3576}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'7132","reported_seq":7199,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:56.291797+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:38:56.291797+0000","last_peered":"2024-09-05T16:38:56.291797+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:38:56.291797+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7132,"log_dups_size":0,"ondisk_log_size":7132,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49073904,"num_objects":5991,"num_object_clones":0,"num_object_copies":23964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2311,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5991,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7129,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3683}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'6897","reported_seq":6944,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:56.241029+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:38:56.241029+0000","last_peered":"2024-09-05T16:38:56.241029+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:38:56.241029+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6897,"log_dups_size":0,"ondisk_log_size":6897,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48020048,"num_objects":5862,"num_object_clones":0,"num_object_copies":23448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5862,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6896,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3601}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'6956","reported_seq":8888,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:56.745453+0000","last_change":"2024-09-05T16:37:30.634570+0000","last_active":"2024-09-05T16:38:56.745453+0000","last_peered":"2024-09-05T16:38:56.745453+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:38:56.745453+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6956,"log_dups_size":0,"ondisk_log_size":6956,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47946320,"num_objects":5853,"num_object_clones":0,"num_object_copies":23412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5853,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6955,"num_write_kb":51231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":927,"num_bytes_recovered":7585792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4485}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'7131","reported_seq":7194,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:56.131932+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:38:56.131932+0000","last_peered":"2024-09-05T16:38:56.131932+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:38:56.131932+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7131,"log_dups_size":0,"ondisk_log_size":7131,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49108128,"num_objects":5995,"num_object_clones":0,"num_object_copies":23980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2356,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5995,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7129,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3641}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'6955","reported_seq":7044,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:56.670746+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:38:56.670746+0000","last_peered":"2024-09-05T16:38:56.670746+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:38:56.670746+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6955,"log_dups_size":0,"ondisk_log_size":6955,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48149664,"num_objects":5878,"num_object_clones":0,"num_object_copies":23512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5878,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6953,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3561}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'6950","reported_seq":7023,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:56.616320+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:38:56.616320+0000","last_peered":"2024-09-05T16:38:56.616320+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:38:56.616320+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6950,"log_dups_size":0,"ondisk_log_size":6950,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47715488,"num_objects":5825,"num_object_clones":0,"num_object_copies":23300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2237,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5825,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6948,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3590}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":767740984,"num_objects":93724,"num_object_clones":0,"num_object_copies":374896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32947,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":93724,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":105513,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":985,"num_bytes_recovered":7585792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1431216128,"data_stored":1431216128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111238,"ondisk_log_size":111238,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084348,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":772844,"kb_used_data":376020,"kb_used_omap":4,"kb_used_meta":396795,"kb_avail":93598996,"statfs":{"total":96636764160,"available":95845371904,"internally_reserved":0,"allocated":385044480,"data_stored":384852439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":406318803},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,0,2,7,0,2,2,2,0,0,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986225,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":536108,"kb_used_data":273940,"kb_used_omap":4,"kb_used_meta":262139,"kb_avail":93835732,"statfs":{"total":96636764160,"available":96087789568,"internally_reserved":0,"allocated":280514560,"data_stored":280322519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":268431058},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673031,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":779516,"kb_used_data":374892,"kb_used_omap":1,"kb_used_meta":404606,"kb_avail":93592324,"statfs":{"total":96636764160,"available":95838539776,"internally_reserved":0,"allocated":383889408,"data_stored":383697367,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":414316997},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,1,2,0,0,0,0,0,0,0,0,0,2],"upper_bound":65536},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182462,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":780764,"kb_used_data":375688,"kb_used_omap":4,"kb_used_meta":405051,"kb_avail":93591076,"statfs":{"total":96636764160,"available":95837261824,"internally_reserved":0,"allocated":384704512,"data_stored":384512471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":414772941},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":383639552,"data_stored":383639552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":280129536,"data_stored":280129536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":383475712,"data_stored":383475712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":383971328,"data_stored":383971328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:38:58.955 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-05T16:38:58.955 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'6998", 'reported_seq': 7051, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:56.225912+0000', 'last_change': '2024-09-05T16:34:56.833497+0000', 'last_active': '2024-09-05T16:38:56.225912+0000', 'last_peered': '2024-09-05T16:38:56.225912+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.828289+0000', 'last_became_peered': '2024-09-05T16:34:56.828289+0000', 'last_unstale': '2024-09-05T16:38:56.225912+0000', 'last_undegraded': '2024-09-05T16:34:56.612160+0000', 'last_fullsized': '2024-09-05T16:34:56.557632+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6998, 'log_dups_size': 0, 'ondisk_log_size': 6998, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48381952, 'num_objects': 5906, 'num_object_clones': 0, 'num_object_copies': 23624, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5906, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6998, 'num_write_kb': 51616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3647}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:38:58.955 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-05T16:38:58.955 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "35'6839", 'reported_seq': 6910, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:56.417765+0000', 'last_change': '2024-09-05T16:34:56.825031+0000', 'last_active': '2024-09-05T16:38:56.417765+0000', 'last_peered': '2024-09-05T16:38:56.417765+0000', 'last_clean': '2024-09-05T16:33:53.200762+0000', 'last_became_active': '2024-09-05T16:34:56.820292+0000', 'last_became_peered': '2024-09-05T16:34:56.820292+0000', 'last_unstale': '2024-09-05T16:38:56.417765+0000', 'last_undegraded': '2024-09-05T16:34:56.605776+0000', 'last_fullsized': '2024-09-05T16:34:56.559383+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6839, 'log_dups_size': 0, 'ondisk_log_size': 6839, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46814368, 'num_objects': 5715, 'num_object_clones': 0, 'num_object_copies': 22860, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2268, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5715, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6837, 'num_write_kb': 50212, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3449}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:38:58.956 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:38:58.956 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'6924", 'reported_seq': 7007, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:56.620417+0000', 'last_change': '2024-09-05T16:34:56.867900+0000', 'last_active': '2024-09-05T16:38:56.620417+0000', 'last_peered': '2024-09-05T16:38:56.620417+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:38:56.620417+0000', 'last_undegraded': '2024-09-05T16:34:56.590972+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6924, 'log_dups_size': 0, 'ondisk_log_size': 6924, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47741544, 'num_objects': 5829, 'num_object_clones': 0, 'num_object_copies': 23316, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5829, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 6923, 'num_write_kb': 50999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3580}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:38:58.956 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:38:58.956 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'6853", 'reported_seq': 6918, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:56.291256+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:38:56.291256+0000', 'last_peered': '2024-09-05T16:38:56.291256+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:38:56.291256+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6853, 'log_dups_size': 0, 'ondisk_log_size': 6853, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47035552, 'num_objects': 5742, 'num_object_clones': 0, 'num_object_copies': 22968, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2161, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5742, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6851, 'num_write_kb': 50373, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3583}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:38:58.956 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:38:58.956 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'6920", 'reported_seq': 6975, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:56.317923+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:38:56.317923+0000', 'last_peered': '2024-09-05T16:38:56.317923+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:38:56.317923+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6920, 'log_dups_size': 0, 'ondisk_log_size': 6920, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47939584, 'num_objects': 5852, 'num_object_clones': 0, 'num_object_copies': 23408, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2246, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5852, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6920, 'num_write_kb': 51088, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3606}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:38:58.956 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:38:58.956 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'6830", 'reported_seq': 6904, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:56.409734+0000', 'last_change': '2024-09-05T16:34:56.903436+0000', 'last_active': '2024-09-05T16:38:56.409734+0000', 'last_peered': '2024-09-05T16:38:56.409734+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:38:56.409734+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6830, 'log_dups_size': 0, 'ondisk_log_size': 6830, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47772832, 'num_objects': 5832, 'num_object_clones': 0, 'num_object_copies': 23328, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2147, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5832, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6828, 'num_write_kb': 50636, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3687}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:38:58.956 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-05T16:38:58.957 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "35'6927", 'reported_seq': 6998, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:56.579349+0000', 'last_change': '2024-09-05T16:34:56.883319+0000', 'last_active': '2024-09-05T16:38:56.579349+0000', 'last_peered': '2024-09-05T16:38:56.579349+0000', 'last_clean': '2024-09-05T16:33:53.505305+0000', 'last_became_active': '2024-09-05T16:34:56.873389+0000', 'last_became_peered': '2024-09-05T16:34:56.873389+0000', 'last_unstale': '2024-09-05T16:38:56.579349+0000', 'last_undegraded': '2024-09-05T16:34:56.602567+0000', 'last_fullsized': '2024-09-05T16:34:56.559958+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6927, 'log_dups_size': 0, 'ondisk_log_size': 6927, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47936672, 'num_objects': 5852, 'num_object_clones': 0, 'num_object_copies': 23408, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2240, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5852, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6925, 'num_write_kb': 51108, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3614}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:38:58.957 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:38:58.957 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'6924", 'reported_seq': 6985, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:56.268845+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:38:56.268845+0000', 'last_peered': '2024-09-05T16:38:56.268845+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:38:56.268845+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6924, 'log_dups_size': 0, 'ondisk_log_size': 6924, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47917376, 'num_objects': 5850, 'num_object_clones': 0, 'num_object_copies': 23400, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2264, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5850, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6921, 'num_write_kb': 51081, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3589}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:38:58.957 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:38:58.957 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'6977", 'reported_seq': 7069, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:56.729551+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:38:56.729551+0000', 'last_peered': '2024-09-05T16:38:56.729551+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:38:56.729551+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6977, 'log_dups_size': 0, 'ondisk_log_size': 6977, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47781024, 'num_objects': 5833, 'num_object_clones': 0, 'num_object_copies': 23332, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5833, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6975, 'num_write_kb': 51230, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3576}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:38:58.957 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:38:58.957 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'7132", 'reported_seq': 7199, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:56.291797+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:38:56.291797+0000', 'last_peered': '2024-09-05T16:38:56.291797+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:38:56.291797+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7132, 'log_dups_size': 0, 'ondisk_log_size': 7132, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49073904, 'num_objects': 5991, 'num_object_clones': 0, 'num_object_copies': 23964, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2311, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5991, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7129, 'num_write_kb': 52476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 6, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3683}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:38:58.957 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:38:58.957 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'6897", 'reported_seq': 6944, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:56.241029+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:38:56.241029+0000', 'last_peered': '2024-09-05T16:38:56.241029+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:38:56.241029+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6897, 'log_dups_size': 0, 'ondisk_log_size': 6897, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48020048, 'num_objects': 5862, 'num_object_clones': 0, 'num_object_copies': 23448, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2262, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5862, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6896, 'num_write_kb': 51031, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3601}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:38:58.958 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-05T16:38:58.958 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'6956", 'reported_seq': 8888, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:56.745453+0000', 'last_change': '2024-09-05T16:37:30.634570+0000', 'last_active': '2024-09-05T16:38:56.745453+0000', 'last_peered': '2024-09-05T16:38:56.745453+0000', 'last_clean': '2024-09-05T16:33:45.082039+0000', 'last_became_active': '2024-09-05T16:34:56.792386+0000', 'last_became_peered': '2024-09-05T16:34:56.792386+0000', 'last_unstale': '2024-09-05T16:38:56.745453+0000', 'last_undegraded': '2024-09-05T16:34:56.600595+0000', 'last_fullsized': '2024-09-05T16:34:56.558203+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6956, 'log_dups_size': 0, 'ondisk_log_size': 6956, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47946320, 'num_objects': 5853, 'num_object_clones': 0, 'num_object_copies': 23412, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1368, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5853, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6955, 'num_write_kb': 51231, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 927, 'num_bytes_recovered': 7585792, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 4485}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:38:58.958 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:38:58.958 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'7131", 'reported_seq': 7194, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:56.131932+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:38:56.131932+0000', 'last_peered': '2024-09-05T16:38:56.131932+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:38:56.131932+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7131, 'log_dups_size': 0, 'ondisk_log_size': 7131, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49108128, 'num_objects': 5995, 'num_object_clones': 0, 'num_object_copies': 23980, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2356, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5995, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7129, 'num_write_kb': 52500, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3641}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:38:58.958 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:38:58.958 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'6955", 'reported_seq': 7044, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:56.670746+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:38:56.670746+0000', 'last_peered': '2024-09-05T16:38:56.670746+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:38:56.670746+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6955, 'log_dups_size': 0, 'ondisk_log_size': 6955, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48149664, 'num_objects': 5878, 'num_object_clones': 0, 'num_object_copies': 23512, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2319, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5878, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6953, 'num_write_kb': 51325, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3561}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:38:58.958 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:38:58.958 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'6950", 'reported_seq': 7023, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:38:56.616320+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:38:56.616320+0000', 'last_peered': '2024-09-05T16:38:56.616320+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:38:56.616320+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6950, 'log_dups_size': 0, 'ondisk_log_size': 6950, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47715488, 'num_objects': 5825, 'num_object_clones': 0, 'num_object_copies': 23300, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2237, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5825, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6948, 'num_write_kb': 51092, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3590}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:38:58.959 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:38:59.648 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:38:59.648 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 32947/374896 objects degraded (8.788%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":342,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":93724,"data_bytes":767740984,"bytes_used":2938093568,"bytes_avail":383608963072,"bytes_total":386547056640,"degraded_objects":32947,"degraded_total":374896,"degraded_ratio":0.087883039562972123,"recovering_objects_per_sec":11,"recovering_bytes_per_sec":85291,"recovering_keys_per_sec":0,"num_objects_recovered":134,"num_bytes_recovered":1024000,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":208},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:38:59.678 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:38:59.678 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:39:00.327 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:39:00.328 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:39:00.354 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":197,"stamp":"2024-09-05T16:38:58.897142+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":767740984,"num_objects":93724,"num_object_clones":0,"num_object_copies":374896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32947,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":93724,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":105513,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":985,"num_bytes_recovered":7585792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111238,"ondisk_log_size":111238,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2869232,"kb_used_data":1400540,"kb_used_omap":14,"kb_used_meta":1468593,"kb_avail":374618128,"statfs":{"total":386547056640,"available":383608963072,"internally_reserved":0,"allocated":1434152960,"data_stored":1433384796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1503839799},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,1,3,13,5,8,3,2,0,0,1,0,0,0,0,2],"upper_bound":65536},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20520960,"num_objects":-2505,"num_object_clones":0,"num_object_copies":-10020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2505,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2505,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":134,"num_bytes_recovered":1024000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005844"},"pg_stats":[{"pgid":"1.f","version":"35'6998","reported_seq":7051,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:56.225912+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:38:56.225912+0000","last_peered":"2024-09-05T16:38:56.225912+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:38:56.225912+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6998,"log_dups_size":0,"ondisk_log_size":6998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48381952,"num_objects":5906,"num_object_clones":0,"num_object_copies":23624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5906,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6998,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3647}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'6839","reported_seq":6910,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:56.417765+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:38:56.417765+0000","last_peered":"2024-09-05T16:38:56.417765+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:38:56.417765+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6839,"log_dups_size":0,"ondisk_log_size":6839,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46814368,"num_objects":5715,"num_object_clones":0,"num_object_copies":22860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2268,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5715,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6837,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3449}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'6924","reported_seq":7007,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:56.620417+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:38:56.620417+0000","last_peered":"2024-09-05T16:38:56.620417+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:38:56.620417+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6924,"log_dups_size":0,"ondisk_log_size":6924,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47741544,"num_objects":5829,"num_object_clones":0,"num_object_copies":23316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5829,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6923,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3580}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'7025","reported_seq":11748,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-05T16:38:55.829768+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:38:55.829768+0000","last_peered":"2024-09-05T16:38:55.829768+0000","last_clean":"2024-09-05T16:38:55.829768+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:38:55.829768+0000","last_undegraded":"2024-09-05T16:38:55.829768+0000","last_fullsized":"2024-09-05T16:38:55.829768+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":7025,"log_dups_size":0,"ondisk_log_size":7025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48406528,"num_objects":5909,"num_object_clones":0,"num_object_copies":23636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5909,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1325,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'6853","reported_seq":6918,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:56.291256+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:38:56.291256+0000","last_peered":"2024-09-05T16:38:56.291256+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:38:56.291256+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6853,"log_dups_size":0,"ondisk_log_size":6853,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47035552,"num_objects":5742,"num_object_clones":0,"num_object_copies":22968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2161,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5742,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6851,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3583}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'6920","reported_seq":6975,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:56.317923+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:38:56.317923+0000","last_peered":"2024-09-05T16:38:56.317923+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:38:56.317923+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6920,"log_dups_size":0,"ondisk_log_size":6920,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47939584,"num_objects":5852,"num_object_clones":0,"num_object_copies":23408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2246,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5852,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6920,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3606}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'6830","reported_seq":6904,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:56.409734+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:38:56.409734+0000","last_peered":"2024-09-05T16:38:56.409734+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:38:56.409734+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6830,"log_dups_size":0,"ondisk_log_size":6830,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47772832,"num_objects":5832,"num_object_clones":0,"num_object_copies":23328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5832,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6828,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3687}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'6927","reported_seq":6998,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:56.579349+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:38:56.579349+0000","last_peered":"2024-09-05T16:38:56.579349+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:38:56.579349+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6927,"log_dups_size":0,"ondisk_log_size":6927,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47936672,"num_objects":5852,"num_object_clones":0,"num_object_copies":23408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5852,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6925,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3614}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'6924","reported_seq":6985,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:56.268845+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:38:56.268845+0000","last_peered":"2024-09-05T16:38:56.268845+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:38:56.268845+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6924,"log_dups_size":0,"ondisk_log_size":6924,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47917376,"num_objects":5850,"num_object_clones":0,"num_object_copies":23400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5850,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6921,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3589}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'6977","reported_seq":7069,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:56.729551+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:38:56.729551+0000","last_peered":"2024-09-05T16:38:56.729551+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:38:56.729551+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6977,"log_dups_size":0,"ondisk_log_size":6977,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47781024,"num_objects":5833,"num_object_clones":0,"num_object_copies":23332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5833,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6975,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3576}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'7132","reported_seq":7199,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:56.291797+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:38:56.291797+0000","last_peered":"2024-09-05T16:38:56.291797+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:38:56.291797+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7132,"log_dups_size":0,"ondisk_log_size":7132,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49073904,"num_objects":5991,"num_object_clones":0,"num_object_copies":23964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2311,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5991,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7129,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3683}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'6897","reported_seq":6944,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:56.241029+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:38:56.241029+0000","last_peered":"2024-09-05T16:38:56.241029+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:38:56.241029+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6897,"log_dups_size":0,"ondisk_log_size":6897,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48020048,"num_objects":5862,"num_object_clones":0,"num_object_copies":23448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5862,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6896,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3601}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'6956","reported_seq":8888,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:56.745453+0000","last_change":"2024-09-05T16:37:30.634570+0000","last_active":"2024-09-05T16:38:56.745453+0000","last_peered":"2024-09-05T16:38:56.745453+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:38:56.745453+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6956,"log_dups_size":0,"ondisk_log_size":6956,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47946320,"num_objects":5853,"num_object_clones":0,"num_object_copies":23412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5853,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6955,"num_write_kb":51231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":927,"num_bytes_recovered":7585792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4485}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'7131","reported_seq":7194,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:56.131932+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:38:56.131932+0000","last_peered":"2024-09-05T16:38:56.131932+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:38:56.131932+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7131,"log_dups_size":0,"ondisk_log_size":7131,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49108128,"num_objects":5995,"num_object_clones":0,"num_object_copies":23980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2356,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5995,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7129,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3641}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'6955","reported_seq":7044,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:56.670746+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:38:56.670746+0000","last_peered":"2024-09-05T16:38:56.670746+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:38:56.670746+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6955,"log_dups_size":0,"ondisk_log_size":6955,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48149664,"num_objects":5878,"num_object_clones":0,"num_object_copies":23512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5878,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6953,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3561}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'6950","reported_seq":7023,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:38:56.616320+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:38:56.616320+0000","last_peered":"2024-09-05T16:38:56.616320+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:38:56.616320+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6950,"log_dups_size":0,"ondisk_log_size":6950,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47715488,"num_objects":5825,"num_object_clones":0,"num_object_copies":23300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2237,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5825,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6948,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3590}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":767740984,"num_objects":93724,"num_object_clones":0,"num_object_copies":374896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32947,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":93724,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":105513,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":985,"num_bytes_recovered":7585792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1431216128,"data_stored":1431216128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111238,"ondisk_log_size":111238,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084348,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":772844,"kb_used_data":376020,"kb_used_omap":4,"kb_used_meta":396795,"kb_avail":93598996,"statfs":{"total":96636764160,"available":95845371904,"internally_reserved":0,"allocated":385044480,"data_stored":384852439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":406318803},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,0,2,7,0,2,2,2,0,0,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986225,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":536108,"kb_used_data":273940,"kb_used_omap":4,"kb_used_meta":262139,"kb_avail":93835732,"statfs":{"total":96636764160,"available":96087789568,"internally_reserved":0,"allocated":280514560,"data_stored":280322519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":268431058},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673031,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":779516,"kb_used_data":374892,"kb_used_omap":1,"kb_used_meta":404606,"kb_avail":93592324,"statfs":{"total":96636764160,"available":95838539776,"internally_reserved":0,"allocated":383889408,"data_stored":383697367,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":414316997},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,1,2,0,0,0,0,0,0,0,0,0,2],"upper_bound":65536},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182462,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":780764,"kb_used_data":375688,"kb_used_omap":4,"kb_used_meta":405051,"kb_avail":93591076,"statfs":{"total":96636764160,"available":95837261824,"internally_reserved":0,"allocated":384704512,"data_stored":384512471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":414772941},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":383639552,"data_stored":383639552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":280129536,"data_stored":280129536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":383475712,"data_stored":383475712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":383971328,"data_stored":383971328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:39:03.356 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:39:03.902 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:39:03.903 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:39:03.937 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":199,"stamp":"2024-09-05T16:39:02.898766+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":757836856,"num_objects":92515,"num_object_clones":0,"num_object_copies":370060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32871,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":92515,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":106722,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1061,"num_bytes_recovered":8183808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112447,"ondisk_log_size":112447,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2841980,"kb_used_data":1385776,"kb_used_omap":14,"kb_used_meta":1456113,"kb_avail":374645380,"statfs":{"total":386547056640,"available":383636869120,"internally_reserved":0,"allocated":1419034624,"data_stored":1418266460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1491060279},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,6,10,15,7,1,0,2,0,0,0,5],"upper_bound":16384},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-29794304,"num_objects":-3637,"num_object_clones":0,"num_object_copies":-14548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-210,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3637,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3637,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":210,"num_bytes_recovered":1622016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006138"},"pg_stats":[{"pgid":"1.f","version":"35'7077","reported_seq":7130,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:01.215972+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:39:01.215972+0000","last_peered":"2024-09-05T16:39:01.215972+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:39:01.215972+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7077,"log_dups_size":0,"ondisk_log_size":7077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47734784,"num_objects":5827,"num_object_clones":0,"num_object_copies":23308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5827,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7077,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3568}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'6930","reported_seq":7001,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:01.633593+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:39:01.633593+0000","last_peered":"2024-09-05T16:39:01.633593+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:39:01.633593+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6930,"log_dups_size":0,"ondisk_log_size":6930,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46068896,"num_objects":5624,"num_object_clones":0,"num_object_copies":22496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2268,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5624,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6928,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3358}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'6997","reported_seq":7084,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:01.799685+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:39:01.799685+0000","last_peered":"2024-09-05T16:39:01.799685+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:39:01.799685+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6997,"log_dups_size":0,"ondisk_log_size":6997,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47143528,"num_objects":5756,"num_object_clones":0,"num_object_copies":23024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2248,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5756,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6996,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3509}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'7095","reported_seq":11818,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-05T16:39:00.817064+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:39:00.817064+0000","last_peered":"2024-09-05T16:39:00.817064+0000","last_clean":"2024-09-05T16:39:00.817064+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:39:00.817064+0000","last_undegraded":"2024-09-05T16:39:00.817064+0000","last_fullsized":"2024-09-05T16:39:00.817064+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":7095,"log_dups_size":0,"ondisk_log_size":7095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47833088,"num_objects":5839,"num_object_clones":0,"num_object_copies":23356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5839,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1395,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'6933","reported_seq":6998,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:01.302239+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:39:01.302239+0000","last_peered":"2024-09-05T16:39:01.302239+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:39:01.302239+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6933,"log_dups_size":0,"ondisk_log_size":6933,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46380192,"num_objects":5662,"num_object_clones":0,"num_object_copies":22648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2161,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5662,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6931,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3503}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'7001","reported_seq":7056,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:01.251936+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:39:01.251936+0000","last_peered":"2024-09-05T16:39:01.251936+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:39:01.251936+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7001,"log_dups_size":0,"ondisk_log_size":7001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47276032,"num_objects":5771,"num_object_clones":0,"num_object_copies":23084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2246,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5771,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7001,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3525}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'6885","reported_seq":6959,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:01.452641+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:39:01.452641+0000","last_peered":"2024-09-05T16:39:01.452641+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:39:01.452641+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6885,"log_dups_size":0,"ondisk_log_size":6885,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47322272,"num_objects":5777,"num_object_clones":0,"num_object_copies":23108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5777,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6883,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3632}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'7003","reported_seq":7074,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:01.614673+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:39:01.614673+0000","last_peered":"2024-09-05T16:39:01.614673+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:39:01.614673+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7003,"log_dups_size":0,"ondisk_log_size":7003,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47314080,"num_objects":5776,"num_object_clones":0,"num_object_copies":23104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5776,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7001,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3538}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'7002","reported_seq":7063,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:01.249906+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:39:01.249906+0000","last_peered":"2024-09-05T16:39:01.249906+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:39:01.249906+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7002,"log_dups_size":0,"ondisk_log_size":7002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47278400,"num_objects":5772,"num_object_clones":0,"num_object_copies":23088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6999,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3511}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'7058","reported_seq":7150,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:01.535032+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:39:01.535032+0000","last_peered":"2024-09-05T16:39:01.535032+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:39:01.535032+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7058,"log_dups_size":0,"ondisk_log_size":7058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47117472,"num_objects":5752,"num_object_clones":0,"num_object_copies":23008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7056,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3495}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'7207","reported_seq":7274,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:01.159349+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:39:01.159349+0000","last_peered":"2024-09-05T16:39:01.159349+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:39:01.159349+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7207,"log_dups_size":0,"ondisk_log_size":7207,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48459504,"num_objects":5916,"num_object_clones":0,"num_object_copies":23664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2311,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5916,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7204,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3608}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'6961","reported_seq":7008,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:01.232574+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:39:01.232574+0000","last_peered":"2024-09-05T16:39:01.232574+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:39:01.232574+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6961,"log_dups_size":0,"ondisk_log_size":6961,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47495760,"num_objects":5798,"num_object_clones":0,"num_object_copies":23192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6960,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3537}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'7035","reported_seq":9113,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:01.815670+0000","last_change":"2024-09-05T16:37:30.634570+0000","last_active":"2024-09-05T16:39:01.815670+0000","last_peered":"2024-09-05T16:39:01.815670+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:39:01.815670+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7035,"log_dups_size":0,"ondisk_log_size":7035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47299152,"num_objects":5774,"num_object_clones":0,"num_object_copies":23096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5774,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7034,"num_write_kb":51231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1000,"num_bytes_recovered":8183808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4479}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'7207","reported_seq":7270,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:01.268896+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:39:01.268896+0000","last_peered":"2024-09-05T16:39:01.268896+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:39:01.268896+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7207,"log_dups_size":0,"ondisk_log_size":7207,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48485536,"num_objects":5919,"num_object_clones":0,"num_object_copies":23676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2356,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5919,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7205,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3565}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'7044","reported_seq":7135,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:01.807863+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:39:01.807863+0000","last_peered":"2024-09-05T16:39:01.807863+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:39:01.807863+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7044,"log_dups_size":0,"ondisk_log_size":7044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47420576,"num_objects":5789,"num_object_clones":0,"num_object_copies":23156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2318,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5789,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7042,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3473}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'7012","reported_seq":7085,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:01.651992+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:39:01.651992+0000","last_peered":"2024-09-05T16:39:01.651992+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:39:01.651992+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7012,"log_dups_size":0,"ondisk_log_size":7012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47207584,"num_objects":5763,"num_object_clones":0,"num_object_copies":23052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2237,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5763,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7010,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3528}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":757836856,"num_objects":92515,"num_object_clones":0,"num_object_copies":370060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32871,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":92515,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":106722,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1061,"num_bytes_recovered":8183808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1416069120,"data_stored":1416069120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112447,"ondisk_log_size":112447,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084349,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":772204,"kb_used_data":371228,"kb_used_omap":4,"kb_used_meta":400955,"kb_avail":93599636,"statfs":{"total":96636764160,"available":95846027264,"internally_reserved":0,"allocated":380137472,"data_stored":379945431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":410578643},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,1,1,2,5,0,0,0,0,0,0,5],"upper_bound":16384},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986226,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":544084,"kb_used_data":273604,"kb_used_omap":4,"kb_used_meta":270459,"kb_avail":93827756,"statfs":{"total":96636764160,"available":96079622144,"internally_reserved":0,"allocated":280170496,"data_stored":279978455,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":276950738},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673032,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":762204,"kb_used_data":370060,"kb_used_omap":1,"kb_used_meta":392126,"kb_avail":93609636,"statfs":{"total":96636764160,"available":95856267264,"internally_reserved":0,"allocated":378941440,"data_stored":378749399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":401537477},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,1,7,0,1,0,2],"upper_bound":1024},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182463,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":763488,"kb_used_data":370884,"kb_used_omap":4,"kb_used_meta":392571,"kb_avail":93608352,"statfs":{"total":96636764160,"available":95854952448,"internally_reserved":0,"allocated":379785216,"data_stored":379593175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":401993421},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,5,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":378703872,"data_stored":378703872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":279830528,"data_stored":279830528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":378515456,"data_stored":378515456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":379019264,"data_stored":379019264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:39:03.938 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:39:04.566 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:39:04.567 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:39:04.601 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":199,"stamp":"2024-09-05T16:39:02.898766+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":757836856,"num_objects":92515,"num_object_clones":0,"num_object_copies":370060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32871,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":92515,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":106722,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1061,"num_bytes_recovered":8183808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112447,"ondisk_log_size":112447,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2841980,"kb_used_data":1385776,"kb_used_omap":14,"kb_used_meta":1456113,"kb_avail":374645380,"statfs":{"total":386547056640,"available":383636869120,"internally_reserved":0,"allocated":1419034624,"data_stored":1418266460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1491060279},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,6,10,15,7,1,0,2,0,0,0,5],"upper_bound":16384},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-29794304,"num_objects":-3637,"num_object_clones":0,"num_object_copies":-14548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-210,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3637,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3637,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":210,"num_bytes_recovered":1622016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006138"},"pg_stats":[{"pgid":"1.f","version":"35'7077","reported_seq":7130,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:01.215972+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:39:01.215972+0000","last_peered":"2024-09-05T16:39:01.215972+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:39:01.215972+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7077,"log_dups_size":0,"ondisk_log_size":7077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47734784,"num_objects":5827,"num_object_clones":0,"num_object_copies":23308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5827,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7077,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3568}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'6930","reported_seq":7001,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:01.633593+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:39:01.633593+0000","last_peered":"2024-09-05T16:39:01.633593+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:39:01.633593+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6930,"log_dups_size":0,"ondisk_log_size":6930,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46068896,"num_objects":5624,"num_object_clones":0,"num_object_copies":22496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2268,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5624,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6928,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3358}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'6997","reported_seq":7084,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:01.799685+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:39:01.799685+0000","last_peered":"2024-09-05T16:39:01.799685+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:39:01.799685+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6997,"log_dups_size":0,"ondisk_log_size":6997,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47143528,"num_objects":5756,"num_object_clones":0,"num_object_copies":23024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2248,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5756,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6996,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3509}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'7095","reported_seq":11818,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-05T16:39:00.817064+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:39:00.817064+0000","last_peered":"2024-09-05T16:39:00.817064+0000","last_clean":"2024-09-05T16:39:00.817064+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:39:00.817064+0000","last_undegraded":"2024-09-05T16:39:00.817064+0000","last_fullsized":"2024-09-05T16:39:00.817064+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":7095,"log_dups_size":0,"ondisk_log_size":7095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47833088,"num_objects":5839,"num_object_clones":0,"num_object_copies":23356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5839,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1395,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'6933","reported_seq":6998,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:01.302239+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:39:01.302239+0000","last_peered":"2024-09-05T16:39:01.302239+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:39:01.302239+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6933,"log_dups_size":0,"ondisk_log_size":6933,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46380192,"num_objects":5662,"num_object_clones":0,"num_object_copies":22648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2161,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5662,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6931,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3503}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'7001","reported_seq":7056,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:01.251936+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:39:01.251936+0000","last_peered":"2024-09-05T16:39:01.251936+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:39:01.251936+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7001,"log_dups_size":0,"ondisk_log_size":7001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47276032,"num_objects":5771,"num_object_clones":0,"num_object_copies":23084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2246,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5771,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7001,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3525}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'6885","reported_seq":6959,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:01.452641+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:39:01.452641+0000","last_peered":"2024-09-05T16:39:01.452641+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:39:01.452641+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6885,"log_dups_size":0,"ondisk_log_size":6885,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47322272,"num_objects":5777,"num_object_clones":0,"num_object_copies":23108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5777,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6883,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3632}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'7003","reported_seq":7074,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:01.614673+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:39:01.614673+0000","last_peered":"2024-09-05T16:39:01.614673+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:39:01.614673+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7003,"log_dups_size":0,"ondisk_log_size":7003,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47314080,"num_objects":5776,"num_object_clones":0,"num_object_copies":23104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5776,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7001,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3538}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'7002","reported_seq":7063,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:01.249906+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:39:01.249906+0000","last_peered":"2024-09-05T16:39:01.249906+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:39:01.249906+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7002,"log_dups_size":0,"ondisk_log_size":7002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47278400,"num_objects":5772,"num_object_clones":0,"num_object_copies":23088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6999,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3511}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'7058","reported_seq":7150,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:01.535032+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:39:01.535032+0000","last_peered":"2024-09-05T16:39:01.535032+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:39:01.535032+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7058,"log_dups_size":0,"ondisk_log_size":7058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47117472,"num_objects":5752,"num_object_clones":0,"num_object_copies":23008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7056,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3495}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'7207","reported_seq":7274,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:01.159349+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:39:01.159349+0000","last_peered":"2024-09-05T16:39:01.159349+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:39:01.159349+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7207,"log_dups_size":0,"ondisk_log_size":7207,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48459504,"num_objects":5916,"num_object_clones":0,"num_object_copies":23664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2311,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5916,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7204,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3608}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'6961","reported_seq":7008,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:01.232574+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:39:01.232574+0000","last_peered":"2024-09-05T16:39:01.232574+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:39:01.232574+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6961,"log_dups_size":0,"ondisk_log_size":6961,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47495760,"num_objects":5798,"num_object_clones":0,"num_object_copies":23192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6960,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3537}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'7035","reported_seq":9113,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:01.815670+0000","last_change":"2024-09-05T16:37:30.634570+0000","last_active":"2024-09-05T16:39:01.815670+0000","last_peered":"2024-09-05T16:39:01.815670+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:39:01.815670+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7035,"log_dups_size":0,"ondisk_log_size":7035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47299152,"num_objects":5774,"num_object_clones":0,"num_object_copies":23096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5774,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7034,"num_write_kb":51231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1000,"num_bytes_recovered":8183808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4479}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'7207","reported_seq":7270,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:01.268896+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:39:01.268896+0000","last_peered":"2024-09-05T16:39:01.268896+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:39:01.268896+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7207,"log_dups_size":0,"ondisk_log_size":7207,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48485536,"num_objects":5919,"num_object_clones":0,"num_object_copies":23676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2356,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5919,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7205,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3565}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'7044","reported_seq":7135,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:01.807863+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:39:01.807863+0000","last_peered":"2024-09-05T16:39:01.807863+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:39:01.807863+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7044,"log_dups_size":0,"ondisk_log_size":7044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47420576,"num_objects":5789,"num_object_clones":0,"num_object_copies":23156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2318,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5789,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7042,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3473}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'7012","reported_seq":7085,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:01.651992+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:39:01.651992+0000","last_peered":"2024-09-05T16:39:01.651992+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:39:01.651992+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7012,"log_dups_size":0,"ondisk_log_size":7012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47207584,"num_objects":5763,"num_object_clones":0,"num_object_copies":23052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2237,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5763,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7010,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3528}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":757836856,"num_objects":92515,"num_object_clones":0,"num_object_copies":370060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32871,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":92515,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":106722,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1061,"num_bytes_recovered":8183808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1416069120,"data_stored":1416069120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112447,"ondisk_log_size":112447,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084349,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":772204,"kb_used_data":371228,"kb_used_omap":4,"kb_used_meta":400955,"kb_avail":93599636,"statfs":{"total":96636764160,"available":95846027264,"internally_reserved":0,"allocated":380137472,"data_stored":379945431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":410578643},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,1,1,2,5,0,0,0,0,0,0,5],"upper_bound":16384},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986226,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":544084,"kb_used_data":273604,"kb_used_omap":4,"kb_used_meta":270459,"kb_avail":93827756,"statfs":{"total":96636764160,"available":96079622144,"internally_reserved":0,"allocated":280170496,"data_stored":279978455,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":276950738},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673032,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":762204,"kb_used_data":370060,"kb_used_omap":1,"kb_used_meta":392126,"kb_avail":93609636,"statfs":{"total":96636764160,"available":95856267264,"internally_reserved":0,"allocated":378941440,"data_stored":378749399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":401537477},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,1,7,0,1,0,2],"upper_bound":1024},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182463,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":763488,"kb_used_data":370884,"kb_used_omap":4,"kb_used_meta":392571,"kb_avail":93608352,"statfs":{"total":96636764160,"available":95854952448,"internally_reserved":0,"allocated":379785216,"data_stored":379593175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":401993421},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,5,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":378703872,"data_stored":378703872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":279830528,"data_stored":279830528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":378515456,"data_stored":378515456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":379019264,"data_stored":379019264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:39:04.602 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-05T16:39:04.602 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'7077", 'reported_seq': 7130, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:01.215972+0000', 'last_change': '2024-09-05T16:34:56.833497+0000', 'last_active': '2024-09-05T16:39:01.215972+0000', 'last_peered': '2024-09-05T16:39:01.215972+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.828289+0000', 'last_became_peered': '2024-09-05T16:34:56.828289+0000', 'last_unstale': '2024-09-05T16:39:01.215972+0000', 'last_undegraded': '2024-09-05T16:34:56.612160+0000', 'last_fullsized': '2024-09-05T16:34:56.557632+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7077, 'log_dups_size': 0, 'ondisk_log_size': 7077, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47734784, 'num_objects': 5827, 'num_object_clones': 0, 'num_object_copies': 23308, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5827, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7077, 'num_write_kb': 51616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3568}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:39:04.603 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-05T16:39:04.603 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "35'6930", 'reported_seq': 7001, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:01.633593+0000', 'last_change': '2024-09-05T16:34:56.825031+0000', 'last_active': '2024-09-05T16:39:01.633593+0000', 'last_peered': '2024-09-05T16:39:01.633593+0000', 'last_clean': '2024-09-05T16:33:53.200762+0000', 'last_became_active': '2024-09-05T16:34:56.820292+0000', 'last_became_peered': '2024-09-05T16:34:56.820292+0000', 'last_unstale': '2024-09-05T16:39:01.633593+0000', 'last_undegraded': '2024-09-05T16:34:56.605776+0000', 'last_fullsized': '2024-09-05T16:34:56.559383+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6930, 'log_dups_size': 0, 'ondisk_log_size': 6930, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46068896, 'num_objects': 5624, 'num_object_clones': 0, 'num_object_copies': 22496, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2268, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5624, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6928, 'num_write_kb': 50212, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3358}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:39:04.603 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:39:04.603 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'6997", 'reported_seq': 7084, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:01.799685+0000', 'last_change': '2024-09-05T16:34:56.867900+0000', 'last_active': '2024-09-05T16:39:01.799685+0000', 'last_peered': '2024-09-05T16:39:01.799685+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:39:01.799685+0000', 'last_undegraded': '2024-09-05T16:34:56.590972+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6997, 'log_dups_size': 0, 'ondisk_log_size': 6997, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47143528, 'num_objects': 5756, 'num_object_clones': 0, 'num_object_copies': 23024, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2248, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5756, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 6996, 'num_write_kb': 50999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3509}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:39:04.603 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:39:04.603 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'6933", 'reported_seq': 6998, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:01.302239+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:39:01.302239+0000', 'last_peered': '2024-09-05T16:39:01.302239+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:39:01.302239+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6933, 'log_dups_size': 0, 'ondisk_log_size': 6933, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46380192, 'num_objects': 5662, 'num_object_clones': 0, 'num_object_copies': 22648, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2161, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5662, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6931, 'num_write_kb': 50373, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3503}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:39:04.603 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:39:04.603 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'7001", 'reported_seq': 7056, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:01.251936+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:39:01.251936+0000', 'last_peered': '2024-09-05T16:39:01.251936+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:39:01.251936+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7001, 'log_dups_size': 0, 'ondisk_log_size': 7001, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47276032, 'num_objects': 5771, 'num_object_clones': 0, 'num_object_copies': 23084, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2246, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5771, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7001, 'num_write_kb': 51088, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3525}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:39:04.603 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:39:04.603 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'6885", 'reported_seq': 6959, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:01.452641+0000', 'last_change': '2024-09-05T16:34:56.903436+0000', 'last_active': '2024-09-05T16:39:01.452641+0000', 'last_peered': '2024-09-05T16:39:01.452641+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:39:01.452641+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6885, 'log_dups_size': 0, 'ondisk_log_size': 6885, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47322272, 'num_objects': 5777, 'num_object_clones': 0, 'num_object_copies': 23108, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2147, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5777, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6883, 'num_write_kb': 50636, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3632}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:39:04.604 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-05T16:39:04.604 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "35'7003", 'reported_seq': 7074, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:01.614673+0000', 'last_change': '2024-09-05T16:34:56.883319+0000', 'last_active': '2024-09-05T16:39:01.614673+0000', 'last_peered': '2024-09-05T16:39:01.614673+0000', 'last_clean': '2024-09-05T16:33:53.505305+0000', 'last_became_active': '2024-09-05T16:34:56.873389+0000', 'last_became_peered': '2024-09-05T16:34:56.873389+0000', 'last_unstale': '2024-09-05T16:39:01.614673+0000', 'last_undegraded': '2024-09-05T16:34:56.602567+0000', 'last_fullsized': '2024-09-05T16:34:56.559958+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7003, 'log_dups_size': 0, 'ondisk_log_size': 7003, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47314080, 'num_objects': 5776, 'num_object_clones': 0, 'num_object_copies': 23104, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2240, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5776, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7001, 'num_write_kb': 51108, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3538}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:39:04.604 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:39:04.604 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'7002", 'reported_seq': 7063, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:01.249906+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:39:01.249906+0000', 'last_peered': '2024-09-05T16:39:01.249906+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:39:01.249906+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7002, 'log_dups_size': 0, 'ondisk_log_size': 7002, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47278400, 'num_objects': 5772, 'num_object_clones': 0, 'num_object_copies': 23088, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2264, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5772, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6999, 'num_write_kb': 51081, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3511}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:39:04.604 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:39:04.604 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'7058", 'reported_seq': 7150, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:01.535032+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:39:01.535032+0000', 'last_peered': '2024-09-05T16:39:01.535032+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:39:01.535032+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7058, 'log_dups_size': 0, 'ondisk_log_size': 7058, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47117472, 'num_objects': 5752, 'num_object_clones': 0, 'num_object_copies': 23008, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5752, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7056, 'num_write_kb': 51230, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3495}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:39:04.604 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:39:04.604 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'7207", 'reported_seq': 7274, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:01.159349+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:39:01.159349+0000', 'last_peered': '2024-09-05T16:39:01.159349+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:39:01.159349+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7207, 'log_dups_size': 0, 'ondisk_log_size': 7207, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48459504, 'num_objects': 5916, 'num_object_clones': 0, 'num_object_copies': 23664, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2311, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5916, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7204, 'num_write_kb': 52476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 6, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3608}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:39:04.604 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:39:04.604 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'6961", 'reported_seq': 7008, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:01.232574+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:39:01.232574+0000', 'last_peered': '2024-09-05T16:39:01.232574+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:39:01.232574+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6961, 'log_dups_size': 0, 'ondisk_log_size': 6961, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47495760, 'num_objects': 5798, 'num_object_clones': 0, 'num_object_copies': 23192, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2262, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5798, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6960, 'num_write_kb': 51031, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3537}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:39:04.604 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-05T16:39:04.605 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'7035", 'reported_seq': 9113, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:01.815670+0000', 'last_change': '2024-09-05T16:37:30.634570+0000', 'last_active': '2024-09-05T16:39:01.815670+0000', 'last_peered': '2024-09-05T16:39:01.815670+0000', 'last_clean': '2024-09-05T16:33:45.082039+0000', 'last_became_active': '2024-09-05T16:34:56.792386+0000', 'last_became_peered': '2024-09-05T16:34:56.792386+0000', 'last_unstale': '2024-09-05T16:39:01.815670+0000', 'last_undegraded': '2024-09-05T16:34:56.600595+0000', 'last_fullsized': '2024-09-05T16:34:56.558203+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7035, 'log_dups_size': 0, 'ondisk_log_size': 7035, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47299152, 'num_objects': 5774, 'num_object_clones': 0, 'num_object_copies': 23096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1295, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5774, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7034, 'num_write_kb': 51231, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1000, 'num_bytes_recovered': 8183808, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 4479}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:39:04.605 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:39:04.605 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'7207", 'reported_seq': 7270, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:01.268896+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:39:01.268896+0000', 'last_peered': '2024-09-05T16:39:01.268896+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:39:01.268896+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7207, 'log_dups_size': 0, 'ondisk_log_size': 7207, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48485536, 'num_objects': 5919, 'num_object_clones': 0, 'num_object_copies': 23676, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2356, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5919, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7205, 'num_write_kb': 52500, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3565}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:39:04.605 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:39:04.605 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'7044", 'reported_seq': 7135, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:01.807863+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:39:01.807863+0000', 'last_peered': '2024-09-05T16:39:01.807863+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:39:01.807863+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7044, 'log_dups_size': 0, 'ondisk_log_size': 7044, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47420576, 'num_objects': 5789, 'num_object_clones': 0, 'num_object_copies': 23156, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2318, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5789, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7042, 'num_write_kb': 51325, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 6, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3473}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:39:04.605 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:39:04.605 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'7012", 'reported_seq': 7085, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:01.651992+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:39:01.651992+0000', 'last_peered': '2024-09-05T16:39:01.651992+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:39:01.651992+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7012, 'log_dups_size': 0, 'ondisk_log_size': 7012, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47207584, 'num_objects': 5763, 'num_object_clones': 0, 'num_object_copies': 23052, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2237, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5763, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7010, 'num_write_kb': 51092, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3528}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:39:04.605 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:39:05.331 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:39:05.331 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 32871/370060 objects degraded (8.883%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":348,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":92515,"data_bytes":757836856,"bytes_used":2910187520,"bytes_avail":383636869120,"bytes_total":386547056640,"degraded_objects":32871,"degraded_total":370060,"degraded_ratio":0.088826136302221262,"recovering_objects_per_sec":12,"recovering_bytes_per_sec":93475,"recovering_keys_per_sec":0,"num_objects_recovered":149,"num_bytes_recovered":1122304,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":204},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:39:05.353 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:39:05.354 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:39:05.942 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:39:05.943 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:39:05.971 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":200,"stamp":"2024-09-05T16:39:04.899937+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":757836856,"num_objects":92515,"num_object_clones":0,"num_object_copies":370060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32871,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":92515,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":106722,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1061,"num_bytes_recovered":8183808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112447,"ondisk_log_size":112447,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2841980,"kb_used_data":1385776,"kb_used_omap":14,"kb_used_meta":1456113,"kb_avail":374645380,"statfs":{"total":386547056640,"available":383636869120,"internally_reserved":0,"allocated":1419034624,"data_stored":1418266460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1491060279},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,6,10,15,7,1,0,2,0,0,0,5],"upper_bound":16384},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20094976,"num_objects":-2453,"num_object_clones":0,"num_object_copies":-9812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2453,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":149,"num_bytes_recovered":1122304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006400"},"pg_stats":[{"pgid":"1.f","version":"35'7077","reported_seq":7130,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:01.215972+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:39:01.215972+0000","last_peered":"2024-09-05T16:39:01.215972+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:39:01.215972+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7077,"log_dups_size":0,"ondisk_log_size":7077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47734784,"num_objects":5827,"num_object_clones":0,"num_object_copies":23308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5827,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7077,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3568}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'6930","reported_seq":7001,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:01.633593+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:39:01.633593+0000","last_peered":"2024-09-05T16:39:01.633593+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:39:01.633593+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6930,"log_dups_size":0,"ondisk_log_size":6930,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46068896,"num_objects":5624,"num_object_clones":0,"num_object_copies":22496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2268,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5624,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6928,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3358}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'6997","reported_seq":7084,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:01.799685+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:39:01.799685+0000","last_peered":"2024-09-05T16:39:01.799685+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:39:01.799685+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6997,"log_dups_size":0,"ondisk_log_size":6997,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47143528,"num_objects":5756,"num_object_clones":0,"num_object_copies":23024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2248,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5756,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6996,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3509}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'7095","reported_seq":11818,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-05T16:39:00.817064+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:39:00.817064+0000","last_peered":"2024-09-05T16:39:00.817064+0000","last_clean":"2024-09-05T16:39:00.817064+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:39:00.817064+0000","last_undegraded":"2024-09-05T16:39:00.817064+0000","last_fullsized":"2024-09-05T16:39:00.817064+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":7095,"log_dups_size":0,"ondisk_log_size":7095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47833088,"num_objects":5839,"num_object_clones":0,"num_object_copies":23356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5839,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1395,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'6933","reported_seq":6998,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:01.302239+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:39:01.302239+0000","last_peered":"2024-09-05T16:39:01.302239+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:39:01.302239+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6933,"log_dups_size":0,"ondisk_log_size":6933,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46380192,"num_objects":5662,"num_object_clones":0,"num_object_copies":22648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2161,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5662,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6931,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3503}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'7001","reported_seq":7056,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:01.251936+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:39:01.251936+0000","last_peered":"2024-09-05T16:39:01.251936+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:39:01.251936+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7001,"log_dups_size":0,"ondisk_log_size":7001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47276032,"num_objects":5771,"num_object_clones":0,"num_object_copies":23084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2246,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5771,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7001,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3525}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'6885","reported_seq":6959,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:01.452641+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:39:01.452641+0000","last_peered":"2024-09-05T16:39:01.452641+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:39:01.452641+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6885,"log_dups_size":0,"ondisk_log_size":6885,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47322272,"num_objects":5777,"num_object_clones":0,"num_object_copies":23108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5777,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6883,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3632}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'7003","reported_seq":7074,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:01.614673+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:39:01.614673+0000","last_peered":"2024-09-05T16:39:01.614673+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:39:01.614673+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7003,"log_dups_size":0,"ondisk_log_size":7003,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47314080,"num_objects":5776,"num_object_clones":0,"num_object_copies":23104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5776,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7001,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3538}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'7002","reported_seq":7063,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:01.249906+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:39:01.249906+0000","last_peered":"2024-09-05T16:39:01.249906+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:39:01.249906+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7002,"log_dups_size":0,"ondisk_log_size":7002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47278400,"num_objects":5772,"num_object_clones":0,"num_object_copies":23088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6999,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3511}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'7058","reported_seq":7150,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:01.535032+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:39:01.535032+0000","last_peered":"2024-09-05T16:39:01.535032+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:39:01.535032+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7058,"log_dups_size":0,"ondisk_log_size":7058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47117472,"num_objects":5752,"num_object_clones":0,"num_object_copies":23008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7056,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3495}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'7207","reported_seq":7274,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:01.159349+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:39:01.159349+0000","last_peered":"2024-09-05T16:39:01.159349+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:39:01.159349+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7207,"log_dups_size":0,"ondisk_log_size":7207,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48459504,"num_objects":5916,"num_object_clones":0,"num_object_copies":23664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2311,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5916,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7204,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3608}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'6961","reported_seq":7008,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:01.232574+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:39:01.232574+0000","last_peered":"2024-09-05T16:39:01.232574+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:39:01.232574+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6961,"log_dups_size":0,"ondisk_log_size":6961,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47495760,"num_objects":5798,"num_object_clones":0,"num_object_copies":23192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6960,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3537}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'7035","reported_seq":9113,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:01.815670+0000","last_change":"2024-09-05T16:37:30.634570+0000","last_active":"2024-09-05T16:39:01.815670+0000","last_peered":"2024-09-05T16:39:01.815670+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:39:01.815670+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7035,"log_dups_size":0,"ondisk_log_size":7035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47299152,"num_objects":5774,"num_object_clones":0,"num_object_copies":23096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5774,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7034,"num_write_kb":51231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1000,"num_bytes_recovered":8183808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4479}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'7207","reported_seq":7270,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:01.268896+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:39:01.268896+0000","last_peered":"2024-09-05T16:39:01.268896+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:39:01.268896+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7207,"log_dups_size":0,"ondisk_log_size":7207,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48485536,"num_objects":5919,"num_object_clones":0,"num_object_copies":23676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2356,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5919,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7205,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3565}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'7044","reported_seq":7135,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:01.807863+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:39:01.807863+0000","last_peered":"2024-09-05T16:39:01.807863+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:39:01.807863+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7044,"log_dups_size":0,"ondisk_log_size":7044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47420576,"num_objects":5789,"num_object_clones":0,"num_object_copies":23156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2318,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5789,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7042,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3473}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'7012","reported_seq":7085,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:01.651992+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:39:01.651992+0000","last_peered":"2024-09-05T16:39:01.651992+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:39:01.651992+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7012,"log_dups_size":0,"ondisk_log_size":7012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47207584,"num_objects":5763,"num_object_clones":0,"num_object_copies":23052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2237,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5763,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7010,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3528}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":757836856,"num_objects":92515,"num_object_clones":0,"num_object_copies":370060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32871,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":92515,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":106722,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1061,"num_bytes_recovered":8183808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1416069120,"data_stored":1416069120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112447,"ondisk_log_size":112447,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084349,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":772204,"kb_used_data":371228,"kb_used_omap":4,"kb_used_meta":400955,"kb_avail":93599636,"statfs":{"total":96636764160,"available":95846027264,"internally_reserved":0,"allocated":380137472,"data_stored":379945431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":410578643},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,1,1,2,5,0,0,0,0,0,0,5],"upper_bound":16384},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986226,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":544084,"kb_used_data":273604,"kb_used_omap":4,"kb_used_meta":270459,"kb_avail":93827756,"statfs":{"total":96636764160,"available":96079622144,"internally_reserved":0,"allocated":280170496,"data_stored":279978455,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":276950738},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673032,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":762204,"kb_used_data":370060,"kb_used_omap":1,"kb_used_meta":392126,"kb_avail":93609636,"statfs":{"total":96636764160,"available":95856267264,"internally_reserved":0,"allocated":378941440,"data_stored":378749399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":401537477},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,1,7,0,1,0,2],"upper_bound":1024},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182463,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":763488,"kb_used_data":370884,"kb_used_omap":4,"kb_used_meta":392571,"kb_avail":93608352,"statfs":{"total":96636764160,"available":95854952448,"internally_reserved":0,"allocated":379785216,"data_stored":379593175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":401993421},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,5,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":378703872,"data_stored":378703872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":279830528,"data_stored":279830528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":378515456,"data_stored":378515456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":379019264,"data_stored":379019264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:39:08.973 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:39:09.550 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:39:09.551 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:39:09.571 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":202,"stamp":"2024-09-05T16:39:08.901620+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":748366904,"num_objects":91359,"num_object_clones":0,"num_object_copies":365436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32786,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91359,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":107878,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1146,"num_bytes_recovered":8830976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":113603,"ondisk_log_size":113603,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2850140,"kb_used_data":1371728,"kb_used_omap":14,"kb_used_meta":1478321,"kb_avail":374637220,"statfs":{"total":386547056640,"available":383628513280,"internally_reserved":0,"allocated":1404649472,"data_stored":1403881308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1513801271},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,8,1,6,6,5,0,0,0,0,4,0,0,5],"upper_bound":32768},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19374080,"num_objects":-2365,"num_object_clones":0,"num_object_copies":-9460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-161,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2365,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":161,"num_bytes_recovered":1245184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006589"},"pg_stats":[{"pgid":"1.f","version":"35'7159","reported_seq":7212,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:06.225428+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:39:06.225428+0000","last_peered":"2024-09-05T16:39:06.225428+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:39:06.225428+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7159,"log_dups_size":0,"ondisk_log_size":7159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47063040,"num_objects":5745,"num_object_clones":0,"num_object_copies":22980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5745,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7159,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3486}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'7001","reported_seq":7076,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:06.505696+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:39:06.505696+0000","last_peered":"2024-09-05T16:39:06.505696+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:39:06.505696+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7001,"log_dups_size":0,"ondisk_log_size":7001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45487264,"num_objects":5553,"num_object_clones":0,"num_object_copies":22212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5553,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6999,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3289}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'7077","reported_seq":7164,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:06.871603+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:39:06.871603+0000","last_peered":"2024-09-05T16:39:06.871603+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:39:06.871603+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7077,"log_dups_size":0,"ondisk_log_size":7077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46488168,"num_objects":5676,"num_object_clones":0,"num_object_copies":22704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2248,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5676,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7076,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3429}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'7161","reported_seq":11884,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-05T16:39:05.737080+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:39:05.737080+0000","last_peered":"2024-09-05T16:39:05.737080+0000","last_clean":"2024-09-05T16:39:05.737080+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:39:05.737080+0000","last_undegraded":"2024-09-05T16:39:05.737080+0000","last_fullsized":"2024-09-05T16:39:05.737080+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":7161,"log_dups_size":0,"ondisk_log_size":7161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47292416,"num_objects":5773,"num_object_clones":0,"num_object_copies":23092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5773,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1461,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'7001","reported_seq":7066,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:06.108935+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:39:06.108935+0000","last_peered":"2024-09-05T16:39:06.108935+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:39:06.108935+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7001,"log_dups_size":0,"ondisk_log_size":7001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45823136,"num_objects":5594,"num_object_clones":0,"num_object_copies":22376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2161,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5594,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6999,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3435}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'7064","reported_seq":7119,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:06.182125+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:39:06.182125+0000","last_peered":"2024-09-05T16:39:06.182125+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:39:06.182125+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7064,"log_dups_size":0,"ondisk_log_size":7064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46759936,"num_objects":5708,"num_object_clones":0,"num_object_copies":22832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2246,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5708,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7064,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3462}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'6957","reported_seq":7033,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:06.526593+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:39:06.526593+0000","last_peered":"2024-09-05T16:39:06.526593+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:39:06.526593+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6957,"log_dups_size":0,"ondisk_log_size":6957,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46732448,"num_objects":5705,"num_object_clones":0,"num_object_copies":22820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2146,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5705,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6955,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3561}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'7084","reported_seq":7155,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:06.608969+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:39:06.608969+0000","last_peered":"2024-09-05T16:39:06.608969+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:39:06.608969+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7084,"log_dups_size":0,"ondisk_log_size":7084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46650528,"num_objects":5695,"num_object_clones":0,"num_object_copies":22780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5695,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7082,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3457}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'7065","reported_seq":7126,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:06.276451+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:39:06.276451+0000","last_peered":"2024-09-05T16:39:06.276451+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:39:06.276451+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7065,"log_dups_size":0,"ondisk_log_size":7065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46762304,"num_objects":5709,"num_object_clones":0,"num_object_copies":22836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5709,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7062,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3448}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'7135","reported_seq":7231,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:06.780322+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:39:06.780322+0000","last_peered":"2024-09-05T16:39:06.780322+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:39:06.780322+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7135,"log_dups_size":0,"ondisk_log_size":7135,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46486688,"num_objects":5675,"num_object_clones":0,"num_object_copies":22700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5675,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7133,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3420}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'7273","reported_seq":7340,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:06.300796+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:39:06.300796+0000","last_peered":"2024-09-05T16:39:06.300796+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:39:06.300796+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7273,"log_dups_size":0,"ondisk_log_size":7273,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47918832,"num_objects":5850,"num_object_clones":0,"num_object_copies":23400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2311,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5850,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7270,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3542}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'7035","reported_seq":7082,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:06.259000+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:39:06.259000+0000","last_peered":"2024-09-05T16:39:06.259000+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:39:06.259000+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7035,"log_dups_size":0,"ondisk_log_size":7035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46889552,"num_objects":5724,"num_object_clones":0,"num_object_copies":22896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7034,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3463}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'7109","reported_seq":9345,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:06.811784+0000","last_change":"2024-09-05T16:37:30.634570+0000","last_active":"2024-09-05T16:39:06.811784+0000","last_peered":"2024-09-05T16:39:06.811784+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:39:06.811784+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7109,"log_dups_size":0,"ondisk_log_size":7109,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46692944,"num_objects":5700,"num_object_clones":0,"num_object_copies":22800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1216,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5700,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7108,"num_write_kb":51231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1079,"num_bytes_recovered":8830976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4484}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'7282","reported_seq":7345,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:06.262504+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:39:06.262504+0000","last_peered":"2024-09-05T16:39:06.262504+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:39:06.262504+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7282,"log_dups_size":0,"ondisk_log_size":7282,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47871136,"num_objects":5844,"num_object_clones":0,"num_object_copies":23376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2356,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5844,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7280,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3490}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'7124","reported_seq":7217,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:06.820318+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:39:06.820318+0000","last_peered":"2024-09-05T16:39:06.820318+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:39:06.820318+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7124,"log_dups_size":0,"ondisk_log_size":7124,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46765216,"num_objects":5709,"num_object_clones":0,"num_object_copies":22836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5709,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7122,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3394}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'7076","reported_seq":7149,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:06.526233+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:39:06.526233+0000","last_peered":"2024-09-05T16:39:06.526233+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:39:06.526233+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7076,"log_dups_size":0,"ondisk_log_size":7076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46683296,"num_objects":5699,"num_object_clones":0,"num_object_copies":22796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2237,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5699,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7074,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3464}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":748366904,"num_objects":91359,"num_object_clones":0,"num_object_copies":365436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32786,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91359,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":107878,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1146,"num_bytes_recovered":8830976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1401593856,"data_stored":1401593856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":113603,"ondisk_log_size":113603,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084350,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":759248,"kb_used_data":366580,"kb_used_omap":4,"kb_used_meta":392635,"kb_avail":93612592,"statfs":{"total":96636764160,"available":95859294208,"internally_reserved":0,"allocated":375377920,"data_stored":375185879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":402058963},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,2,1,3,0,0,0,0,0,0,0,5],"upper_bound":32768},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986227,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":566052,"kb_used_data":273364,"kb_used_omap":4,"kb_used_meta":292667,"kb_avail":93805788,"statfs":{"total":96636764160,"available":96057126912,"internally_reserved":0,"allocated":279924736,"data_stored":279732695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":299691730},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673033,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":761824,"kb_used_data":365520,"kb_used_omap":1,"kb_used_meta":396286,"kb_avail":93610016,"statfs":{"total":96636764160,"available":95856656384,"internally_reserved":0,"allocated":374292480,"data_stored":374100439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":405797317},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,2,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182464,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":763016,"kb_used_data":366264,"kb_used_omap":4,"kb_used_meta":396731,"kb_avail":93608824,"statfs":{"total":96636764160,"available":95855435776,"internally_reserved":0,"allocated":375054336,"data_stored":374862295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":406253261},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,0,1,0,1,0,0,0,0,4],"upper_bound":4096},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":373977088,"data_stored":373977088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":279580672,"data_stored":279580672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":373743616,"data_stored":373743616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":374292480,"data_stored":374292480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:39:09.572 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:39:10.188 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:39:10.189 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:39:10.214 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":202,"stamp":"2024-09-05T16:39:08.901620+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":748366904,"num_objects":91359,"num_object_clones":0,"num_object_copies":365436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32786,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91359,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":107878,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1146,"num_bytes_recovered":8830976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":113603,"ondisk_log_size":113603,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2850140,"kb_used_data":1371728,"kb_used_omap":14,"kb_used_meta":1478321,"kb_avail":374637220,"statfs":{"total":386547056640,"available":383628513280,"internally_reserved":0,"allocated":1404649472,"data_stored":1403881308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1513801271},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,8,1,6,6,5,0,0,0,0,4,0,0,5],"upper_bound":32768},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19374080,"num_objects":-2365,"num_object_clones":0,"num_object_copies":-9460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-161,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2365,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":161,"num_bytes_recovered":1245184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006589"},"pg_stats":[{"pgid":"1.f","version":"35'7159","reported_seq":7212,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:06.225428+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:39:06.225428+0000","last_peered":"2024-09-05T16:39:06.225428+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:39:06.225428+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7159,"log_dups_size":0,"ondisk_log_size":7159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47063040,"num_objects":5745,"num_object_clones":0,"num_object_copies":22980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5745,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7159,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3486}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'7001","reported_seq":7076,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:06.505696+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:39:06.505696+0000","last_peered":"2024-09-05T16:39:06.505696+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:39:06.505696+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7001,"log_dups_size":0,"ondisk_log_size":7001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45487264,"num_objects":5553,"num_object_clones":0,"num_object_copies":22212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5553,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6999,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3289}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'7077","reported_seq":7164,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:06.871603+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:39:06.871603+0000","last_peered":"2024-09-05T16:39:06.871603+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:39:06.871603+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7077,"log_dups_size":0,"ondisk_log_size":7077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46488168,"num_objects":5676,"num_object_clones":0,"num_object_copies":22704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2248,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5676,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7076,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3429}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'7161","reported_seq":11884,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-05T16:39:05.737080+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:39:05.737080+0000","last_peered":"2024-09-05T16:39:05.737080+0000","last_clean":"2024-09-05T16:39:05.737080+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:39:05.737080+0000","last_undegraded":"2024-09-05T16:39:05.737080+0000","last_fullsized":"2024-09-05T16:39:05.737080+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":7161,"log_dups_size":0,"ondisk_log_size":7161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47292416,"num_objects":5773,"num_object_clones":0,"num_object_copies":23092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5773,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1461,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'7001","reported_seq":7066,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:06.108935+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:39:06.108935+0000","last_peered":"2024-09-05T16:39:06.108935+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:39:06.108935+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7001,"log_dups_size":0,"ondisk_log_size":7001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45823136,"num_objects":5594,"num_object_clones":0,"num_object_copies":22376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2161,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5594,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6999,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3435}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'7064","reported_seq":7119,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:06.182125+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:39:06.182125+0000","last_peered":"2024-09-05T16:39:06.182125+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:39:06.182125+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7064,"log_dups_size":0,"ondisk_log_size":7064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46759936,"num_objects":5708,"num_object_clones":0,"num_object_copies":22832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2246,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5708,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7064,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3462}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'6957","reported_seq":7033,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:06.526593+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:39:06.526593+0000","last_peered":"2024-09-05T16:39:06.526593+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:39:06.526593+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6957,"log_dups_size":0,"ondisk_log_size":6957,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46732448,"num_objects":5705,"num_object_clones":0,"num_object_copies":22820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2146,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5705,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6955,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3561}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'7084","reported_seq":7155,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:06.608969+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:39:06.608969+0000","last_peered":"2024-09-05T16:39:06.608969+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:39:06.608969+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7084,"log_dups_size":0,"ondisk_log_size":7084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46650528,"num_objects":5695,"num_object_clones":0,"num_object_copies":22780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5695,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7082,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3457}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'7065","reported_seq":7126,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:06.276451+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:39:06.276451+0000","last_peered":"2024-09-05T16:39:06.276451+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:39:06.276451+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7065,"log_dups_size":0,"ondisk_log_size":7065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46762304,"num_objects":5709,"num_object_clones":0,"num_object_copies":22836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5709,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7062,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3448}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'7135","reported_seq":7231,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:06.780322+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:39:06.780322+0000","last_peered":"2024-09-05T16:39:06.780322+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:39:06.780322+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7135,"log_dups_size":0,"ondisk_log_size":7135,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46486688,"num_objects":5675,"num_object_clones":0,"num_object_copies":22700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5675,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7133,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3420}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'7273","reported_seq":7340,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:06.300796+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:39:06.300796+0000","last_peered":"2024-09-05T16:39:06.300796+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:39:06.300796+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7273,"log_dups_size":0,"ondisk_log_size":7273,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47918832,"num_objects":5850,"num_object_clones":0,"num_object_copies":23400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2311,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5850,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7270,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3542}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'7035","reported_seq":7082,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:06.259000+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:39:06.259000+0000","last_peered":"2024-09-05T16:39:06.259000+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:39:06.259000+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7035,"log_dups_size":0,"ondisk_log_size":7035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46889552,"num_objects":5724,"num_object_clones":0,"num_object_copies":22896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7034,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3463}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'7109","reported_seq":9345,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:06.811784+0000","last_change":"2024-09-05T16:37:30.634570+0000","last_active":"2024-09-05T16:39:06.811784+0000","last_peered":"2024-09-05T16:39:06.811784+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:39:06.811784+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7109,"log_dups_size":0,"ondisk_log_size":7109,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46692944,"num_objects":5700,"num_object_clones":0,"num_object_copies":22800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1216,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5700,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7108,"num_write_kb":51231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1079,"num_bytes_recovered":8830976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4484}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'7282","reported_seq":7345,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:06.262504+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:39:06.262504+0000","last_peered":"2024-09-05T16:39:06.262504+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:39:06.262504+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7282,"log_dups_size":0,"ondisk_log_size":7282,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47871136,"num_objects":5844,"num_object_clones":0,"num_object_copies":23376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2356,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5844,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7280,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3490}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'7124","reported_seq":7217,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:06.820318+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:39:06.820318+0000","last_peered":"2024-09-05T16:39:06.820318+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:39:06.820318+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7124,"log_dups_size":0,"ondisk_log_size":7124,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46765216,"num_objects":5709,"num_object_clones":0,"num_object_copies":22836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5709,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7122,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3394}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'7076","reported_seq":7149,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:06.526233+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:39:06.526233+0000","last_peered":"2024-09-05T16:39:06.526233+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:39:06.526233+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7076,"log_dups_size":0,"ondisk_log_size":7076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46683296,"num_objects":5699,"num_object_clones":0,"num_object_copies":22796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2237,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5699,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7074,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3464}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":748366904,"num_objects":91359,"num_object_clones":0,"num_object_copies":365436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32786,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91359,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":107878,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1146,"num_bytes_recovered":8830976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1401593856,"data_stored":1401593856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":113603,"ondisk_log_size":113603,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084350,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":759248,"kb_used_data":366580,"kb_used_omap":4,"kb_used_meta":392635,"kb_avail":93612592,"statfs":{"total":96636764160,"available":95859294208,"internally_reserved":0,"allocated":375377920,"data_stored":375185879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":402058963},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,2,1,3,0,0,0,0,0,0,0,5],"upper_bound":32768},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986227,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":566052,"kb_used_data":273364,"kb_used_omap":4,"kb_used_meta":292667,"kb_avail":93805788,"statfs":{"total":96636764160,"available":96057126912,"internally_reserved":0,"allocated":279924736,"data_stored":279732695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":299691730},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673033,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":761824,"kb_used_data":365520,"kb_used_omap":1,"kb_used_meta":396286,"kb_avail":93610016,"statfs":{"total":96636764160,"available":95856656384,"internally_reserved":0,"allocated":374292480,"data_stored":374100439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":405797317},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,2,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182464,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":763016,"kb_used_data":366264,"kb_used_omap":4,"kb_used_meta":396731,"kb_avail":93608824,"statfs":{"total":96636764160,"available":95855435776,"internally_reserved":0,"allocated":375054336,"data_stored":374862295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":406253261},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,0,1,0,1,0,0,0,0,4],"upper_bound":4096},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":373977088,"data_stored":373977088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":279580672,"data_stored":279580672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":373743616,"data_stored":373743616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":374292480,"data_stored":374292480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:39:10.215 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-05T16:39:10.215 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'7159", 'reported_seq': 7212, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:06.225428+0000', 'last_change': '2024-09-05T16:34:56.833497+0000', 'last_active': '2024-09-05T16:39:06.225428+0000', 'last_peered': '2024-09-05T16:39:06.225428+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.828289+0000', 'last_became_peered': '2024-09-05T16:34:56.828289+0000', 'last_unstale': '2024-09-05T16:39:06.225428+0000', 'last_undegraded': '2024-09-05T16:34:56.612160+0000', 'last_fullsized': '2024-09-05T16:34:56.557632+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7159, 'log_dups_size': 0, 'ondisk_log_size': 7159, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47063040, 'num_objects': 5745, 'num_object_clones': 0, 'num_object_copies': 22980, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5745, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7159, 'num_write_kb': 51616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3486}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:39:10.215 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-05T16:39:10.215 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "35'7001", 'reported_seq': 7076, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:06.505696+0000', 'last_change': '2024-09-05T16:34:56.825031+0000', 'last_active': '2024-09-05T16:39:06.505696+0000', 'last_peered': '2024-09-05T16:39:06.505696+0000', 'last_clean': '2024-09-05T16:33:53.200762+0000', 'last_became_active': '2024-09-05T16:34:56.820292+0000', 'last_became_peered': '2024-09-05T16:34:56.820292+0000', 'last_unstale': '2024-09-05T16:39:06.505696+0000', 'last_undegraded': '2024-09-05T16:34:56.605776+0000', 'last_fullsized': '2024-09-05T16:34:56.559383+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7001, 'log_dups_size': 0, 'ondisk_log_size': 7001, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45487264, 'num_objects': 5553, 'num_object_clones': 0, 'num_object_copies': 22212, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2266, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5553, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6999, 'num_write_kb': 50212, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3289}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:39:10.216 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:39:10.216 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'7077", 'reported_seq': 7164, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:06.871603+0000', 'last_change': '2024-09-05T16:34:56.867900+0000', 'last_active': '2024-09-05T16:39:06.871603+0000', 'last_peered': '2024-09-05T16:39:06.871603+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:39:06.871603+0000', 'last_undegraded': '2024-09-05T16:34:56.590972+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7077, 'log_dups_size': 0, 'ondisk_log_size': 7077, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46488168, 'num_objects': 5676, 'num_object_clones': 0, 'num_object_copies': 22704, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2248, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5676, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 7076, 'num_write_kb': 50999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3429}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:39:10.216 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:39:10.216 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'7001", 'reported_seq': 7066, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:06.108935+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:39:06.108935+0000', 'last_peered': '2024-09-05T16:39:06.108935+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:39:06.108935+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7001, 'log_dups_size': 0, 'ondisk_log_size': 7001, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45823136, 'num_objects': 5594, 'num_object_clones': 0, 'num_object_copies': 22376, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2161, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5594, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6999, 'num_write_kb': 50373, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3435}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:39:10.216 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:39:10.216 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'7064", 'reported_seq': 7119, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:06.182125+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:39:06.182125+0000', 'last_peered': '2024-09-05T16:39:06.182125+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:39:06.182125+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7064, 'log_dups_size': 0, 'ondisk_log_size': 7064, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46759936, 'num_objects': 5708, 'num_object_clones': 0, 'num_object_copies': 22832, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2246, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5708, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7064, 'num_write_kb': 51088, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3462}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:39:10.216 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:39:10.216 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'6957", 'reported_seq': 7033, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:06.526593+0000', 'last_change': '2024-09-05T16:34:56.903436+0000', 'last_active': '2024-09-05T16:39:06.526593+0000', 'last_peered': '2024-09-05T16:39:06.526593+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:39:06.526593+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 6957, 'log_dups_size': 0, 'ondisk_log_size': 6957, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46732448, 'num_objects': 5705, 'num_object_clones': 0, 'num_object_copies': 22820, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2146, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5705, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6955, 'num_write_kb': 50636, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3561}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:39:10.216 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-05T16:39:10.216 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "35'7084", 'reported_seq': 7155, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:06.608969+0000', 'last_change': '2024-09-05T16:34:56.883319+0000', 'last_active': '2024-09-05T16:39:06.608969+0000', 'last_peered': '2024-09-05T16:39:06.608969+0000', 'last_clean': '2024-09-05T16:33:53.505305+0000', 'last_became_active': '2024-09-05T16:34:56.873389+0000', 'last_became_peered': '2024-09-05T16:34:56.873389+0000', 'last_unstale': '2024-09-05T16:39:06.608969+0000', 'last_undegraded': '2024-09-05T16:34:56.602567+0000', 'last_fullsized': '2024-09-05T16:34:56.559958+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7084, 'log_dups_size': 0, 'ondisk_log_size': 7084, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46650528, 'num_objects': 5695, 'num_object_clones': 0, 'num_object_copies': 22780, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2240, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5695, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7082, 'num_write_kb': 51108, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3457}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:39:10.217 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:39:10.217 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'7065", 'reported_seq': 7126, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:06.276451+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:39:06.276451+0000', 'last_peered': '2024-09-05T16:39:06.276451+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:39:06.276451+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7065, 'log_dups_size': 0, 'ondisk_log_size': 7065, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46762304, 'num_objects': 5709, 'num_object_clones': 0, 'num_object_copies': 22836, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2264, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5709, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7062, 'num_write_kb': 51081, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3448}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:39:10.217 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:39:10.217 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'7135", 'reported_seq': 7231, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:06.780322+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:39:06.780322+0000', 'last_peered': '2024-09-05T16:39:06.780322+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:39:06.780322+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7135, 'log_dups_size': 0, 'ondisk_log_size': 7135, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46486688, 'num_objects': 5675, 'num_object_clones': 0, 'num_object_copies': 22700, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5675, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7133, 'num_write_kb': 51230, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3420}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:39:10.217 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:39:10.217 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'7273", 'reported_seq': 7340, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:06.300796+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:39:06.300796+0000', 'last_peered': '2024-09-05T16:39:06.300796+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:39:06.300796+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7273, 'log_dups_size': 0, 'ondisk_log_size': 7273, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47918832, 'num_objects': 5850, 'num_object_clones': 0, 'num_object_copies': 23400, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2311, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5850, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7270, 'num_write_kb': 52476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 6, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3542}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:39:10.217 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:39:10.217 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'7035", 'reported_seq': 7082, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:06.259000+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:39:06.259000+0000', 'last_peered': '2024-09-05T16:39:06.259000+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:39:06.259000+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7035, 'log_dups_size': 0, 'ondisk_log_size': 7035, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46889552, 'num_objects': 5724, 'num_object_clones': 0, 'num_object_copies': 22896, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2262, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5724, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7034, 'num_write_kb': 51031, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3463}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:39:10.217 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-05T16:39:10.217 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'7109", 'reported_seq': 9345, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:06.811784+0000', 'last_change': '2024-09-05T16:37:30.634570+0000', 'last_active': '2024-09-05T16:39:06.811784+0000', 'last_peered': '2024-09-05T16:39:06.811784+0000', 'last_clean': '2024-09-05T16:33:45.082039+0000', 'last_became_active': '2024-09-05T16:34:56.792386+0000', 'last_became_peered': '2024-09-05T16:34:56.792386+0000', 'last_unstale': '2024-09-05T16:39:06.811784+0000', 'last_undegraded': '2024-09-05T16:34:56.600595+0000', 'last_fullsized': '2024-09-05T16:34:56.558203+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7109, 'log_dups_size': 0, 'ondisk_log_size': 7109, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46692944, 'num_objects': 5700, 'num_object_clones': 0, 'num_object_copies': 22800, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1216, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5700, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7108, 'num_write_kb': 51231, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1079, 'num_bytes_recovered': 8830976, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 4484}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:39:10.217 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:39:10.218 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'7282", 'reported_seq': 7345, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:06.262504+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:39:06.262504+0000', 'last_peered': '2024-09-05T16:39:06.262504+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:39:06.262504+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7282, 'log_dups_size': 0, 'ondisk_log_size': 7282, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47871136, 'num_objects': 5844, 'num_object_clones': 0, 'num_object_copies': 23376, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2356, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5844, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7280, 'num_write_kb': 52500, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3490}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:39:10.218 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:39:10.218 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'7124", 'reported_seq': 7217, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:06.820318+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:39:06.820318+0000', 'last_peered': '2024-09-05T16:39:06.820318+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:39:06.820318+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7124, 'log_dups_size': 0, 'ondisk_log_size': 7124, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46765216, 'num_objects': 5709, 'num_object_clones': 0, 'num_object_copies': 22836, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2317, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5709, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7122, 'num_write_kb': 51325, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3394}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:39:10.218 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:39:10.218 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'7076", 'reported_seq': 7149, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:06.526233+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:39:06.526233+0000', 'last_peered': '2024-09-05T16:39:06.526233+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:39:06.526233+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7076, 'log_dups_size': 0, 'ondisk_log_size': 7076, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46683296, 'num_objects': 5699, 'num_object_clones': 0, 'num_object_copies': 22796, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2237, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5699, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7074, 'num_write_kb': 51092, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3464}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:39:10.218 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:39:10.970 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:39:10.970 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 32786/365436 objects degraded (8.972%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":354,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":91359,"data_bytes":748366904,"bytes_used":2918543360,"bytes_avail":383628513280,"bytes_total":386547056640,"degraded_objects":32786,"degraded_total":365436,"degraded_ratio":0.089717488151140015,"recovering_objects_per_sec":13,"recovering_bytes_per_sec":103708,"recovering_keys_per_sec":0,"num_objects_recovered":161,"num_bytes_recovered":1245184,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":196},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:39:10.994 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:39:10.995 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:39:11.591 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:39:11.591 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:39:11.621 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":203,"stamp":"2024-09-05T16:39:10.902485+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":748366904,"num_objects":91359,"num_object_clones":0,"num_object_copies":365436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32786,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91359,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":107878,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1146,"num_bytes_recovered":8830976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":113603,"ondisk_log_size":113603,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2850140,"kb_used_data":1371728,"kb_used_omap":14,"kb_used_meta":1478321,"kb_avail":374637220,"statfs":{"total":386547056640,"available":383628513280,"internally_reserved":0,"allocated":1404649472,"data_stored":1403881308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1513801271},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,8,1,6,6,5,0,0,0,0,4,0,0,5],"upper_bound":32768},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19374080,"num_objects":-2365,"num_object_clones":0,"num_object_copies":-9460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-161,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2365,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":161,"num_bytes_recovered":1245184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005343"},"pg_stats":[{"pgid":"1.f","version":"35'7159","reported_seq":7212,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:06.225428+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:39:06.225428+0000","last_peered":"2024-09-05T16:39:06.225428+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:39:06.225428+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7159,"log_dups_size":0,"ondisk_log_size":7159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47063040,"num_objects":5745,"num_object_clones":0,"num_object_copies":22980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5745,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7159,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3486}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'7001","reported_seq":7076,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:06.505696+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:39:06.505696+0000","last_peered":"2024-09-05T16:39:06.505696+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:39:06.505696+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7001,"log_dups_size":0,"ondisk_log_size":7001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45487264,"num_objects":5553,"num_object_clones":0,"num_object_copies":22212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5553,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6999,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3289}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'7077","reported_seq":7164,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:06.871603+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:39:06.871603+0000","last_peered":"2024-09-05T16:39:06.871603+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:39:06.871603+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7077,"log_dups_size":0,"ondisk_log_size":7077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46488168,"num_objects":5676,"num_object_clones":0,"num_object_copies":22704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2248,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5676,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7076,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3429}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'7161","reported_seq":11884,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-05T16:39:05.737080+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:39:05.737080+0000","last_peered":"2024-09-05T16:39:05.737080+0000","last_clean":"2024-09-05T16:39:05.737080+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:39:05.737080+0000","last_undegraded":"2024-09-05T16:39:05.737080+0000","last_fullsized":"2024-09-05T16:39:05.737080+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":7161,"log_dups_size":0,"ondisk_log_size":7161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47292416,"num_objects":5773,"num_object_clones":0,"num_object_copies":23092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5773,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1461,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'7001","reported_seq":7066,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:06.108935+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:39:06.108935+0000","last_peered":"2024-09-05T16:39:06.108935+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:39:06.108935+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7001,"log_dups_size":0,"ondisk_log_size":7001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45823136,"num_objects":5594,"num_object_clones":0,"num_object_copies":22376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2161,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5594,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6999,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3435}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'7064","reported_seq":7119,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:06.182125+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:39:06.182125+0000","last_peered":"2024-09-05T16:39:06.182125+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:39:06.182125+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7064,"log_dups_size":0,"ondisk_log_size":7064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46759936,"num_objects":5708,"num_object_clones":0,"num_object_copies":22832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2246,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5708,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7064,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3462}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'6957","reported_seq":7033,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:06.526593+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:39:06.526593+0000","last_peered":"2024-09-05T16:39:06.526593+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:39:06.526593+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":6957,"log_dups_size":0,"ondisk_log_size":6957,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46732448,"num_objects":5705,"num_object_clones":0,"num_object_copies":22820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2146,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5705,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6955,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3561}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'7084","reported_seq":7155,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:06.608969+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:39:06.608969+0000","last_peered":"2024-09-05T16:39:06.608969+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:39:06.608969+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7084,"log_dups_size":0,"ondisk_log_size":7084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46650528,"num_objects":5695,"num_object_clones":0,"num_object_copies":22780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5695,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7082,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3457}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'7065","reported_seq":7126,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:06.276451+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:39:06.276451+0000","last_peered":"2024-09-05T16:39:06.276451+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:39:06.276451+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7065,"log_dups_size":0,"ondisk_log_size":7065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46762304,"num_objects":5709,"num_object_clones":0,"num_object_copies":22836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5709,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7062,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3448}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'7135","reported_seq":7231,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:06.780322+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:39:06.780322+0000","last_peered":"2024-09-05T16:39:06.780322+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:39:06.780322+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7135,"log_dups_size":0,"ondisk_log_size":7135,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46486688,"num_objects":5675,"num_object_clones":0,"num_object_copies":22700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5675,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7133,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3420}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'7273","reported_seq":7340,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:06.300796+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:39:06.300796+0000","last_peered":"2024-09-05T16:39:06.300796+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:39:06.300796+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7273,"log_dups_size":0,"ondisk_log_size":7273,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47918832,"num_objects":5850,"num_object_clones":0,"num_object_copies":23400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2311,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5850,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7270,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3542}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'7035","reported_seq":7082,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:06.259000+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:39:06.259000+0000","last_peered":"2024-09-05T16:39:06.259000+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:39:06.259000+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7035,"log_dups_size":0,"ondisk_log_size":7035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46889552,"num_objects":5724,"num_object_clones":0,"num_object_copies":22896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7034,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3463}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'7109","reported_seq":9345,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:06.811784+0000","last_change":"2024-09-05T16:37:30.634570+0000","last_active":"2024-09-05T16:39:06.811784+0000","last_peered":"2024-09-05T16:39:06.811784+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:39:06.811784+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7109,"log_dups_size":0,"ondisk_log_size":7109,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46692944,"num_objects":5700,"num_object_clones":0,"num_object_copies":22800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1216,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5700,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7108,"num_write_kb":51231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1079,"num_bytes_recovered":8830976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4484}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'7282","reported_seq":7345,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:06.262504+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:39:06.262504+0000","last_peered":"2024-09-05T16:39:06.262504+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:39:06.262504+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7282,"log_dups_size":0,"ondisk_log_size":7282,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47871136,"num_objects":5844,"num_object_clones":0,"num_object_copies":23376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2356,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5844,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7280,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3490}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'7124","reported_seq":7217,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:06.820318+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:39:06.820318+0000","last_peered":"2024-09-05T16:39:06.820318+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:39:06.820318+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7124,"log_dups_size":0,"ondisk_log_size":7124,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46765216,"num_objects":5709,"num_object_clones":0,"num_object_copies":22836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5709,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7122,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3394}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'7076","reported_seq":7149,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:06.526233+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:39:06.526233+0000","last_peered":"2024-09-05T16:39:06.526233+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:39:06.526233+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7076,"log_dups_size":0,"ondisk_log_size":7076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46683296,"num_objects":5699,"num_object_clones":0,"num_object_copies":22796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2237,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5699,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7074,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3464}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":748366904,"num_objects":91359,"num_object_clones":0,"num_object_copies":365436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32786,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91359,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":107878,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1146,"num_bytes_recovered":8830976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1401593856,"data_stored":1401593856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":113603,"ondisk_log_size":113603,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084350,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":759248,"kb_used_data":366580,"kb_used_omap":4,"kb_used_meta":392635,"kb_avail":93612592,"statfs":{"total":96636764160,"available":95859294208,"internally_reserved":0,"allocated":375377920,"data_stored":375185879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":402058963},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,2,1,3,0,0,0,0,0,0,0,5],"upper_bound":32768},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986227,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":566052,"kb_used_data":273364,"kb_used_omap":4,"kb_used_meta":292667,"kb_avail":93805788,"statfs":{"total":96636764160,"available":96057126912,"internally_reserved":0,"allocated":279924736,"data_stored":279732695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":299691730},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673033,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":761824,"kb_used_data":365520,"kb_used_omap":1,"kb_used_meta":396286,"kb_avail":93610016,"statfs":{"total":96636764160,"available":95856656384,"internally_reserved":0,"allocated":374292480,"data_stored":374100439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":405797317},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,2,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182464,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":763016,"kb_used_data":366264,"kb_used_omap":4,"kb_used_meta":396731,"kb_avail":93608824,"statfs":{"total":96636764160,"available":95855435776,"internally_reserved":0,"allocated":375054336,"data_stored":374862295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":406253261},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,0,1,0,1,0,0,0,0,4],"upper_bound":4096},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":373977088,"data_stored":373977088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":279580672,"data_stored":279580672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":373743616,"data_stored":373743616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":374292480,"data_stored":374292480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:39:14.625 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:39:15.257 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:39:15.257 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:39:15.276 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":205,"stamp":"2024-09-05T16:39:14.904875+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":738569272,"num_objects":90163,"num_object_clones":0,"num_object_copies":360652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32695,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":90163,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":109074,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1237,"num_bytes_recovered":9445376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":114799,"ondisk_log_size":114799,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2835668,"kb_used_data":1357244,"kb_used_omap":14,"kb_used_meta":1478321,"kb_avail":374651692,"statfs":{"total":386547056640,"available":383643332608,"internally_reserved":0,"allocated":1389817856,"data_stored":1389049692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1513801271},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,3,1,8,13,5,6,0,0,0,6,0,0,4],"upper_bound":16384},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19267584,"num_objects":-2352,"num_object_clones":0,"num_object_copies":-9408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-176,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2352,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2352,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":176,"num_bytes_recovered":1261568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006109"},"pg_stats":[{"pgid":"1.f","version":"35'7233","reported_seq":7289,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:11.416688+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:39:11.416688+0000","last_peered":"2024-09-05T16:39:11.416688+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:39:11.416688+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7233,"log_dups_size":0,"ondisk_log_size":7233,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46456832,"num_objects":5671,"num_object_clones":0,"num_object_copies":22684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5671,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7233,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3414}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'7079","reported_seq":7152,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:11.686741+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:39:11.686741+0000","last_peered":"2024-09-05T16:39:11.686741+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:39:11.686741+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7079,"log_dups_size":0,"ondisk_log_size":7079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44848288,"num_objects":5475,"num_object_clones":0,"num_object_copies":21900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5475,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7077,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3211}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'7154","reported_seq":7241,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:11.924964+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:39:11.924964+0000","last_peered":"2024-09-05T16:39:11.924964+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:39:11.924964+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7154,"log_dups_size":0,"ondisk_log_size":7154,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45857384,"num_objects":5599,"num_object_clones":0,"num_object_copies":22396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2248,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5599,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7153,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3352}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'7237","reported_seq":11960,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-05T16:39:10.836474+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:39:10.836474+0000","last_peered":"2024-09-05T16:39:10.836474+0000","last_clean":"2024-09-05T16:39:10.836474+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:39:10.836474+0000","last_undegraded":"2024-09-05T16:39:10.836474+0000","last_fullsized":"2024-09-05T16:39:10.836474+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":7237,"log_dups_size":0,"ondisk_log_size":7237,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46669824,"num_objects":5697,"num_object_clones":0,"num_object_copies":22788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5697,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1537,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'7069","reported_seq":7140,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:11.410214+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:39:11.410214+0000","last_peered":"2024-09-05T16:39:11.410214+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:39:11.410214+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7069,"log_dups_size":0,"ondisk_log_size":7069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45266080,"num_objects":5526,"num_object_clones":0,"num_object_copies":22104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5526,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7067,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3370}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'7121","reported_seq":7180,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:11.342659+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:39:11.342659+0000","last_peered":"2024-09-05T16:39:11.342659+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:39:11.342659+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7121,"log_dups_size":0,"ondisk_log_size":7121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46292992,"num_objects":5651,"num_object_clones":0,"num_object_copies":22604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2244,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5651,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7121,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3407}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'7022","reported_seq":7097,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:11.537143+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:39:11.537143+0000","last_peered":"2024-09-05T16:39:11.537143+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:39:11.537143+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7022,"log_dups_size":0,"ondisk_log_size":7022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46199968,"num_objects":5640,"num_object_clones":0,"num_object_copies":22560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2146,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5640,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7020,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3496}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'7161","reported_seq":7232,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:11.630329+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:39:11.630329+0000","last_peered":"2024-09-05T16:39:11.630329+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:39:11.630329+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7161,"log_dups_size":0,"ondisk_log_size":7161,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46019744,"num_objects":5618,"num_object_clones":0,"num_object_copies":22472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5618,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7159,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3380}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'7137","reported_seq":7206,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:10.894605+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:39:10.894605+0000","last_peered":"2024-09-05T16:39:10.894605+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:39:10.894605+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7137,"log_dups_size":0,"ondisk_log_size":7137,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46172480,"num_objects":5637,"num_object_clones":0,"num_object_copies":22548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5637,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7134,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3380}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'7202","reported_seq":7298,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:11.926771+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:39:11.926771+0000","last_peered":"2024-09-05T16:39:11.926771+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:39:11.926771+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7202,"log_dups_size":0,"ondisk_log_size":7202,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45937824,"num_objects":5608,"num_object_clones":0,"num_object_copies":22432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5608,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7200,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3353}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'7354","reported_seq":7425,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:11.393760+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:39:11.393760+0000","last_peered":"2024-09-05T16:39:11.393760+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:39:11.393760+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7354,"log_dups_size":0,"ondisk_log_size":7354,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47255280,"num_objects":5769,"num_object_clones":0,"num_object_copies":23076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5769,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7351,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3463}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'7121","reported_seq":7172,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:11.310711+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:39:11.310711+0000","last_peered":"2024-09-05T16:39:11.310711+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:39:11.310711+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7121,"log_dups_size":0,"ondisk_log_size":7121,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46185040,"num_objects":5638,"num_object_clones":0,"num_object_copies":22552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5638,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7120,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3379}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'7173","reported_seq":9559,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:11.876472+0000","last_change":"2024-09-05T16:37:30.634570+0000","last_active":"2024-09-05T16:39:11.876472+0000","last_peered":"2024-09-05T16:39:11.876472+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:39:11.876472+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7173,"log_dups_size":0,"ondisk_log_size":7173,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46168656,"num_objects":5636,"num_object_clones":0,"num_object_copies":22544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1141,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7172,"num_write_kb":51231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1154,"num_bytes_recovered":9445376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4495}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'7368","reported_seq":7433,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:11.406238+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:39:11.406238+0000","last_peered":"2024-09-05T16:39:11.406238+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:39:11.406238+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7368,"log_dups_size":0,"ondisk_log_size":7368,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47166624,"num_objects":5758,"num_object_clones":0,"num_object_copies":23032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2355,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5758,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7366,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3405}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'7214","reported_seq":7307,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:11.813458+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:39:11.813458+0000","last_peered":"2024-09-05T16:39:11.813458+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:39:11.813458+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7214,"log_dups_size":0,"ondisk_log_size":7214,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46027936,"num_objects":5619,"num_object_clones":0,"num_object_copies":22476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5619,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7212,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3304}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'7154","reported_seq":7227,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:11.663739+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:39:11.663739+0000","last_peered":"2024-09-05T16:39:11.663739+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:39:11.663739+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7154,"log_dups_size":0,"ondisk_log_size":7154,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46044320,"num_objects":5621,"num_object_clones":0,"num_object_copies":22484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2237,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5621,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7152,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3386}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":738569272,"num_objects":90163,"num_object_clones":0,"num_object_copies":360652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32695,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":90163,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":109074,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1237,"num_bytes_recovered":9445376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1386602496,"data_stored":1386602496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":114799,"ondisk_log_size":114799,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084351,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":758668,"kb_used_data":361844,"kb_used_omap":4,"kb_used_meta":396795,"kb_avail":93613172,"statfs":{"total":96636764160,"available":95859888128,"internally_reserved":0,"allocated":370528256,"data_stored":370336215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":406318803},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,4,3,4,4,0,0,0,6],"upper_bound":2048},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986228,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":553284,"kb_used_data":273076,"kb_used_omap":4,"kb_used_meta":280187,"kb_avail":93818556,"statfs":{"total":96636764160,"available":96070201344,"internally_reserved":0,"allocated":279629824,"data_stored":279437783,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":286912210},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,1,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673034,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":761232,"kb_used_data":360752,"kb_used_omap":1,"kb_used_meta":400446,"kb_avail":93610608,"statfs":{"total":96636764160,"available":95857262592,"internally_reserved":0,"allocated":369410048,"data_stored":369218007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":410057157},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182465,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":762484,"kb_used_data":361572,"kb_used_omap":4,"kb_used_meta":400891,"kb_avail":93609356,"statfs":{"total":96636764160,"available":95855980544,"internally_reserved":0,"allocated":370249728,"data_stored":370057687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":410513101},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,3,6,1,2,0,0,0,0,0,0,4],"upper_bound":16384},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":369082368,"data_stored":369082368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":279310336,"data_stored":279310336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":368881664,"data_stored":368881664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":369328128,"data_stored":369328128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:39:15.277 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:39:15.914 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:39:15.915 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:39:15.963 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":205,"stamp":"2024-09-05T16:39:14.904875+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":738569272,"num_objects":90163,"num_object_clones":0,"num_object_copies":360652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32695,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":90163,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":109074,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1237,"num_bytes_recovered":9445376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":114799,"ondisk_log_size":114799,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2835668,"kb_used_data":1357244,"kb_used_omap":14,"kb_used_meta":1478321,"kb_avail":374651692,"statfs":{"total":386547056640,"available":383643332608,"internally_reserved":0,"allocated":1389817856,"data_stored":1389049692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1513801271},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,3,1,8,13,5,6,0,0,0,6,0,0,4],"upper_bound":16384},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19267584,"num_objects":-2352,"num_object_clones":0,"num_object_copies":-9408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-176,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2352,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2352,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":176,"num_bytes_recovered":1261568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006109"},"pg_stats":[{"pgid":"1.f","version":"35'7233","reported_seq":7289,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:11.416688+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:39:11.416688+0000","last_peered":"2024-09-05T16:39:11.416688+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:39:11.416688+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7233,"log_dups_size":0,"ondisk_log_size":7233,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46456832,"num_objects":5671,"num_object_clones":0,"num_object_copies":22684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5671,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7233,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3414}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'7079","reported_seq":7152,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:11.686741+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:39:11.686741+0000","last_peered":"2024-09-05T16:39:11.686741+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:39:11.686741+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7079,"log_dups_size":0,"ondisk_log_size":7079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44848288,"num_objects":5475,"num_object_clones":0,"num_object_copies":21900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5475,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7077,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3211}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'7154","reported_seq":7241,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:11.924964+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:39:11.924964+0000","last_peered":"2024-09-05T16:39:11.924964+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:39:11.924964+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7154,"log_dups_size":0,"ondisk_log_size":7154,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45857384,"num_objects":5599,"num_object_clones":0,"num_object_copies":22396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2248,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5599,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7153,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3352}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'7237","reported_seq":11960,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-05T16:39:10.836474+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:39:10.836474+0000","last_peered":"2024-09-05T16:39:10.836474+0000","last_clean":"2024-09-05T16:39:10.836474+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:39:10.836474+0000","last_undegraded":"2024-09-05T16:39:10.836474+0000","last_fullsized":"2024-09-05T16:39:10.836474+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":7237,"log_dups_size":0,"ondisk_log_size":7237,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46669824,"num_objects":5697,"num_object_clones":0,"num_object_copies":22788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5697,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1537,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'7069","reported_seq":7140,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:11.410214+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:39:11.410214+0000","last_peered":"2024-09-05T16:39:11.410214+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:39:11.410214+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7069,"log_dups_size":0,"ondisk_log_size":7069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45266080,"num_objects":5526,"num_object_clones":0,"num_object_copies":22104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5526,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7067,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3370}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'7121","reported_seq":7180,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:11.342659+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:39:11.342659+0000","last_peered":"2024-09-05T16:39:11.342659+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:39:11.342659+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7121,"log_dups_size":0,"ondisk_log_size":7121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46292992,"num_objects":5651,"num_object_clones":0,"num_object_copies":22604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2244,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5651,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7121,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3407}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'7022","reported_seq":7097,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:11.537143+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:39:11.537143+0000","last_peered":"2024-09-05T16:39:11.537143+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:39:11.537143+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7022,"log_dups_size":0,"ondisk_log_size":7022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46199968,"num_objects":5640,"num_object_clones":0,"num_object_copies":22560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2146,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5640,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7020,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3496}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'7161","reported_seq":7232,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:11.630329+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:39:11.630329+0000","last_peered":"2024-09-05T16:39:11.630329+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:39:11.630329+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7161,"log_dups_size":0,"ondisk_log_size":7161,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46019744,"num_objects":5618,"num_object_clones":0,"num_object_copies":22472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5618,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7159,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3380}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'7137","reported_seq":7206,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:10.894605+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:39:10.894605+0000","last_peered":"2024-09-05T16:39:10.894605+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:39:10.894605+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7137,"log_dups_size":0,"ondisk_log_size":7137,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46172480,"num_objects":5637,"num_object_clones":0,"num_object_copies":22548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5637,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7134,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3380}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'7202","reported_seq":7298,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:11.926771+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:39:11.926771+0000","last_peered":"2024-09-05T16:39:11.926771+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:39:11.926771+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7202,"log_dups_size":0,"ondisk_log_size":7202,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45937824,"num_objects":5608,"num_object_clones":0,"num_object_copies":22432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5608,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7200,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3353}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'7354","reported_seq":7425,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:11.393760+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:39:11.393760+0000","last_peered":"2024-09-05T16:39:11.393760+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:39:11.393760+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7354,"log_dups_size":0,"ondisk_log_size":7354,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47255280,"num_objects":5769,"num_object_clones":0,"num_object_copies":23076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5769,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7351,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3463}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'7121","reported_seq":7172,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:11.310711+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:39:11.310711+0000","last_peered":"2024-09-05T16:39:11.310711+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:39:11.310711+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7121,"log_dups_size":0,"ondisk_log_size":7121,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46185040,"num_objects":5638,"num_object_clones":0,"num_object_copies":22552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5638,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7120,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3379}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'7173","reported_seq":9559,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:11.876472+0000","last_change":"2024-09-05T16:37:30.634570+0000","last_active":"2024-09-05T16:39:11.876472+0000","last_peered":"2024-09-05T16:39:11.876472+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:39:11.876472+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7173,"log_dups_size":0,"ondisk_log_size":7173,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46168656,"num_objects":5636,"num_object_clones":0,"num_object_copies":22544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1141,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7172,"num_write_kb":51231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1154,"num_bytes_recovered":9445376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4495}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'7368","reported_seq":7433,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:11.406238+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:39:11.406238+0000","last_peered":"2024-09-05T16:39:11.406238+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:39:11.406238+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7368,"log_dups_size":0,"ondisk_log_size":7368,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47166624,"num_objects":5758,"num_object_clones":0,"num_object_copies":23032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2355,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5758,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7366,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3405}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'7214","reported_seq":7307,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:11.813458+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:39:11.813458+0000","last_peered":"2024-09-05T16:39:11.813458+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:39:11.813458+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7214,"log_dups_size":0,"ondisk_log_size":7214,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46027936,"num_objects":5619,"num_object_clones":0,"num_object_copies":22476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5619,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7212,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3304}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'7154","reported_seq":7227,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:11.663739+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:39:11.663739+0000","last_peered":"2024-09-05T16:39:11.663739+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:39:11.663739+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7154,"log_dups_size":0,"ondisk_log_size":7154,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46044320,"num_objects":5621,"num_object_clones":0,"num_object_copies":22484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2237,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5621,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7152,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3386}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":738569272,"num_objects":90163,"num_object_clones":0,"num_object_copies":360652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32695,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":90163,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":109074,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1237,"num_bytes_recovered":9445376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1386602496,"data_stored":1386602496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":114799,"ondisk_log_size":114799,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084351,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":758668,"kb_used_data":361844,"kb_used_omap":4,"kb_used_meta":396795,"kb_avail":93613172,"statfs":{"total":96636764160,"available":95859888128,"internally_reserved":0,"allocated":370528256,"data_stored":370336215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":406318803},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,4,3,4,4,0,0,0,6],"upper_bound":2048},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986228,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":553284,"kb_used_data":273076,"kb_used_omap":4,"kb_used_meta":280187,"kb_avail":93818556,"statfs":{"total":96636764160,"available":96070201344,"internally_reserved":0,"allocated":279629824,"data_stored":279437783,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":286912210},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,1,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673034,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":761232,"kb_used_data":360752,"kb_used_omap":1,"kb_used_meta":400446,"kb_avail":93610608,"statfs":{"total":96636764160,"available":95857262592,"internally_reserved":0,"allocated":369410048,"data_stored":369218007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":410057157},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182465,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":762484,"kb_used_data":361572,"kb_used_omap":4,"kb_used_meta":400891,"kb_avail":93609356,"statfs":{"total":96636764160,"available":95855980544,"internally_reserved":0,"allocated":370249728,"data_stored":370057687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":410513101},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,3,6,1,2,0,0,0,0,0,0,4],"upper_bound":16384},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":369082368,"data_stored":369082368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":279310336,"data_stored":279310336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":368881664,"data_stored":368881664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":369328128,"data_stored":369328128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:39:15.964 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-05T16:39:15.964 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'7233", 'reported_seq': 7289, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:11.416688+0000', 'last_change': '2024-09-05T16:34:56.833497+0000', 'last_active': '2024-09-05T16:39:11.416688+0000', 'last_peered': '2024-09-05T16:39:11.416688+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.828289+0000', 'last_became_peered': '2024-09-05T16:34:56.828289+0000', 'last_unstale': '2024-09-05T16:39:11.416688+0000', 'last_undegraded': '2024-09-05T16:34:56.612160+0000', 'last_fullsized': '2024-09-05T16:34:56.557632+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7233, 'log_dups_size': 0, 'ondisk_log_size': 7233, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46456832, 'num_objects': 5671, 'num_object_clones': 0, 'num_object_copies': 22684, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5671, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7233, 'num_write_kb': 51616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3414}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:39:15.964 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-05T16:39:15.965 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "35'7079", 'reported_seq': 7152, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:11.686741+0000', 'last_change': '2024-09-05T16:34:56.825031+0000', 'last_active': '2024-09-05T16:39:11.686741+0000', 'last_peered': '2024-09-05T16:39:11.686741+0000', 'last_clean': '2024-09-05T16:33:53.200762+0000', 'last_became_active': '2024-09-05T16:34:56.820292+0000', 'last_became_peered': '2024-09-05T16:34:56.820292+0000', 'last_unstale': '2024-09-05T16:39:11.686741+0000', 'last_undegraded': '2024-09-05T16:34:56.605776+0000', 'last_fullsized': '2024-09-05T16:34:56.559383+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7079, 'log_dups_size': 0, 'ondisk_log_size': 7079, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44848288, 'num_objects': 5475, 'num_object_clones': 0, 'num_object_copies': 21900, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2266, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5475, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7077, 'num_write_kb': 50212, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3211}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:39:15.965 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:39:15.965 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'7154", 'reported_seq': 7241, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:11.924964+0000', 'last_change': '2024-09-05T16:34:56.867900+0000', 'last_active': '2024-09-05T16:39:11.924964+0000', 'last_peered': '2024-09-05T16:39:11.924964+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:39:11.924964+0000', 'last_undegraded': '2024-09-05T16:34:56.590972+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7154, 'log_dups_size': 0, 'ondisk_log_size': 7154, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45857384, 'num_objects': 5599, 'num_object_clones': 0, 'num_object_copies': 22396, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2248, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5599, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 7153, 'num_write_kb': 50999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3352}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:39:15.965 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:39:15.965 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'7069", 'reported_seq': 7140, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:11.410214+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:39:11.410214+0000', 'last_peered': '2024-09-05T16:39:11.410214+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:39:11.410214+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7069, 'log_dups_size': 0, 'ondisk_log_size': 7069, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45266080, 'num_objects': 5526, 'num_object_clones': 0, 'num_object_copies': 22104, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2158, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5526, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7067, 'num_write_kb': 50373, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 8, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3370}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:39:15.965 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:39:15.965 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'7121", 'reported_seq': 7180, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:11.342659+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:39:11.342659+0000', 'last_peered': '2024-09-05T16:39:11.342659+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:39:11.342659+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7121, 'log_dups_size': 0, 'ondisk_log_size': 7121, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46292992, 'num_objects': 5651, 'num_object_clones': 0, 'num_object_copies': 22604, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2244, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5651, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7121, 'num_write_kb': 51088, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 6, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3407}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:39:15.965 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:39:15.965 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'7022", 'reported_seq': 7097, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:11.537143+0000', 'last_change': '2024-09-05T16:34:56.903436+0000', 'last_active': '2024-09-05T16:39:11.537143+0000', 'last_peered': '2024-09-05T16:39:11.537143+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:39:11.537143+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7022, 'log_dups_size': 0, 'ondisk_log_size': 7022, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46199968, 'num_objects': 5640, 'num_object_clones': 0, 'num_object_copies': 22560, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2146, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5640, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7020, 'num_write_kb': 50636, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3496}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:39:15.966 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-05T16:39:15.966 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "35'7161", 'reported_seq': 7232, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:11.630329+0000', 'last_change': '2024-09-05T16:34:56.883319+0000', 'last_active': '2024-09-05T16:39:11.630329+0000', 'last_peered': '2024-09-05T16:39:11.630329+0000', 'last_clean': '2024-09-05T16:33:53.505305+0000', 'last_became_active': '2024-09-05T16:34:56.873389+0000', 'last_became_peered': '2024-09-05T16:34:56.873389+0000', 'last_unstale': '2024-09-05T16:39:11.630329+0000', 'last_undegraded': '2024-09-05T16:34:56.602567+0000', 'last_fullsized': '2024-09-05T16:34:56.559958+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7161, 'log_dups_size': 0, 'ondisk_log_size': 7161, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46019744, 'num_objects': 5618, 'num_object_clones': 0, 'num_object_copies': 22472, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2240, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5618, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7159, 'num_write_kb': 51108, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3380}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:39:15.966 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:39:15.966 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'7137", 'reported_seq': 7206, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:10.894605+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:39:10.894605+0000', 'last_peered': '2024-09-05T16:39:10.894605+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:39:10.894605+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7137, 'log_dups_size': 0, 'ondisk_log_size': 7137, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46172480, 'num_objects': 5637, 'num_object_clones': 0, 'num_object_copies': 22548, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2260, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5637, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7134, 'num_write_kb': 51081, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3380}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:39:15.966 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:39:15.966 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'7202", 'reported_seq': 7298, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:11.926771+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:39:11.926771+0000', 'last_peered': '2024-09-05T16:39:11.926771+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:39:11.926771+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7202, 'log_dups_size': 0, 'ondisk_log_size': 7202, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45937824, 'num_objects': 5608, 'num_object_clones': 0, 'num_object_copies': 22432, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5608, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7200, 'num_write_kb': 51230, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3353}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:39:15.966 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:39:15.966 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'7354", 'reported_seq': 7425, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:11.393760+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:39:11.393760+0000', 'last_peered': '2024-09-05T16:39:11.393760+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:39:11.393760+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7354, 'log_dups_size': 0, 'ondisk_log_size': 7354, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47255280, 'num_objects': 5769, 'num_object_clones': 0, 'num_object_copies': 23076, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2309, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5769, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7351, 'num_write_kb': 52476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 8, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3463}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:39:15.966 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:39:15.966 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'7121", 'reported_seq': 7172, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:11.310711+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:39:11.310711+0000', 'last_peered': '2024-09-05T16:39:11.310711+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:39:11.310711+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7121, 'log_dups_size': 0, 'ondisk_log_size': 7121, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46185040, 'num_objects': 5638, 'num_object_clones': 0, 'num_object_copies': 22552, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2260, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5638, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7120, 'num_write_kb': 51031, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 6, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3379}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:39:15.966 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-05T16:39:15.967 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'7173", 'reported_seq': 9559, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:11.876472+0000', 'last_change': '2024-09-05T16:37:30.634570+0000', 'last_active': '2024-09-05T16:39:11.876472+0000', 'last_peered': '2024-09-05T16:39:11.876472+0000', 'last_clean': '2024-09-05T16:33:45.082039+0000', 'last_became_active': '2024-09-05T16:34:56.792386+0000', 'last_became_peered': '2024-09-05T16:34:56.792386+0000', 'last_unstale': '2024-09-05T16:39:11.876472+0000', 'last_undegraded': '2024-09-05T16:34:56.600595+0000', 'last_fullsized': '2024-09-05T16:34:56.558203+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7173, 'log_dups_size': 0, 'ondisk_log_size': 7173, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46168656, 'num_objects': 5636, 'num_object_clones': 0, 'num_object_copies': 22544, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1141, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5636, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7172, 'num_write_kb': 51231, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1154, 'num_bytes_recovered': 9445376, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 4495}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:39:15.967 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:39:15.967 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'7368", 'reported_seq': 7433, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:11.406238+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:39:11.406238+0000', 'last_peered': '2024-09-05T16:39:11.406238+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:39:11.406238+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7368, 'log_dups_size': 0, 'ondisk_log_size': 7368, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47166624, 'num_objects': 5758, 'num_object_clones': 0, 'num_object_copies': 23032, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2355, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5758, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7366, 'num_write_kb': 52500, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 6, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3405}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:39:15.967 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:39:15.967 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'7214", 'reported_seq': 7307, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:11.813458+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:39:11.813458+0000', 'last_peered': '2024-09-05T16:39:11.813458+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:39:11.813458+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7214, 'log_dups_size': 0, 'ondisk_log_size': 7214, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46027936, 'num_objects': 5619, 'num_object_clones': 0, 'num_object_copies': 22476, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2317, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5619, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7212, 'num_write_kb': 51325, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3304}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:39:15.967 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:39:15.967 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'7154", 'reported_seq': 7227, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:11.663739+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:39:11.663739+0000', 'last_peered': '2024-09-05T16:39:11.663739+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:39:11.663739+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7154, 'log_dups_size': 0, 'ondisk_log_size': 7154, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46044320, 'num_objects': 5621, 'num_object_clones': 0, 'num_object_copies': 22484, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2237, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5621, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7152, 'num_write_kb': 51092, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3386}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:39:15.967 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:39:16.698 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:39:16.698 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 32695/360652 objects degraded (9.066%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":359,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":90163,"data_bytes":738569272,"bytes_used":2903724032,"bytes_avail":383643332608,"bytes_total":386547056640,"degraded_objects":32695,"degraded_total":360652,"degraded_ratio":0.090655257699943453,"recovering_objects_per_sec":14,"recovering_bytes_per_sec":105077,"recovering_keys_per_sec":0,"num_objects_recovered":176,"num_bytes_recovered":1261568,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":195},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:39:16.730 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:39:16.730 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:39:17.341 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:39:17.342 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:39:17.365 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":206,"stamp":"2024-09-05T16:39:16.905560+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":731376696,"num_objects":89285,"num_object_clones":0,"num_object_copies":357140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32669,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89285,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":109952,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1263,"num_bytes_recovered":9445376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115677,"ondisk_log_size":115677,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2821616,"kb_used_data":1347360,"kb_used_omap":14,"kb_used_meta":1474161,"kb_avail":374665744,"statfs":{"total":386547056640,"available":383657721856,"internally_reserved":0,"allocated":1379696640,"data_stored":1378928476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1509541431},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,1,2,7,10,11,11,1,0,0,0,0,0,0,4],"upper_bound":32768},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-26460160,"num_objects":-3230,"num_object_clones":0,"num_object_copies":-12920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3230,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3230,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":202,"num_bytes_recovered":1261568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005623"},"pg_stats":[{"pgid":"1.f","version":"35'7316","reported_seq":7381,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:16.266443+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:39:16.266443+0000","last_peered":"2024-09-05T16:39:16.266443+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:39:16.266443+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7316,"log_dups_size":0,"ondisk_log_size":7316,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45776896,"num_objects":5588,"num_object_clones":0,"num_object_copies":22352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2253,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5588,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7316,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'7148","reported_seq":7227,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:16.679280+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:39:16.679280+0000","last_peered":"2024-09-05T16:39:16.679280+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:39:16.679280+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7148,"log_dups_size":0,"ondisk_log_size":7148,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44283040,"num_objects":5406,"num_object_clones":0,"num_object_copies":21624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5406,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7146,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3145}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'7154","reported_seq":7241,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:11.924964+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:39:11.924964+0000","last_peered":"2024-09-05T16:39:11.924964+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:39:11.924964+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7154,"log_dups_size":0,"ondisk_log_size":7154,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45857384,"num_objects":5599,"num_object_clones":0,"num_object_copies":22396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2248,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5599,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7153,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3352}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'7318","reported_seq":12041,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-05T16:39:15.789291+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:39:15.789291+0000","last_peered":"2024-09-05T16:39:15.789291+0000","last_clean":"2024-09-05T16:39:15.789291+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:39:15.789291+0000","last_undegraded":"2024-09-05T16:39:15.789291+0000","last_fullsized":"2024-09-05T16:39:15.789291+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":7318,"log_dups_size":0,"ondisk_log_size":7318,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46006272,"num_objects":5616,"num_object_clones":0,"num_object_copies":22464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5616,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1618,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'7132","reported_seq":7205,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:16.030839+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:39:16.030839+0000","last_peered":"2024-09-05T16:39:16.030839+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:39:16.030839+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7132,"log_dups_size":0,"ondisk_log_size":7132,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44749984,"num_objects":5463,"num_object_clones":0,"num_object_copies":21852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2157,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5463,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7130,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3308}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'7198","reported_seq":7261,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:16.380886+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:39:16.380886+0000","last_peered":"2024-09-05T16:39:16.380886+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:39:16.380886+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7198,"log_dups_size":0,"ondisk_log_size":7198,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45662208,"num_objects":5574,"num_object_clones":0,"num_object_copies":22296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2242,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5574,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7198,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3332}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'7082","reported_seq":7164,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:16.594040+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:39:16.594040+0000","last_peered":"2024-09-05T16:39:16.594040+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:39:16.594040+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7082,"log_dups_size":0,"ondisk_log_size":7082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45708448,"num_objects":5580,"num_object_clones":0,"num_object_copies":22320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5580,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7080,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3439}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'7234","reported_seq":7309,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:16.554269+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:39:16.554269+0000","last_peered":"2024-09-05T16:39:16.554269+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:39:16.554269+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7234,"log_dups_size":0,"ondisk_log_size":7234,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45421728,"num_objects":5545,"num_object_clones":0,"num_object_copies":22180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2238,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5545,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7232,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3309}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'7210","reported_seq":7279,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:16.318469+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:39:16.318469+0000","last_peered":"2024-09-05T16:39:16.318469+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:39:16.318469+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7210,"log_dups_size":0,"ondisk_log_size":7210,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45574464,"num_objects":5564,"num_object_clones":0,"num_object_copies":22256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5564,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7207,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3307}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'7202","reported_seq":7298,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:11.926771+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:39:11.926771+0000","last_peered":"2024-09-05T16:39:11.926771+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:39:11.926771+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7202,"log_dups_size":0,"ondisk_log_size":7202,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45937824,"num_objects":5608,"num_object_clones":0,"num_object_copies":22432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5608,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7200,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3353}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'7435","reported_seq":7510,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:16.452716+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:39:16.452716+0000","last_peered":"2024-09-05T16:39:16.452716+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:39:16.452716+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7435,"log_dups_size":0,"ondisk_log_size":7435,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46591728,"num_objects":5688,"num_object_clones":0,"num_object_copies":22752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2307,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7432,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3384}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'7192","reported_seq":7253,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:16.264406+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:39:16.264406+0000","last_peered":"2024-09-05T16:39:16.264406+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:39:16.264406+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7192,"log_dups_size":0,"ondisk_log_size":7192,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45603408,"num_objects":5567,"num_object_clones":0,"num_object_copies":22268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2255,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5567,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7191,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3313}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'7173","reported_seq":9559,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:11.876472+0000","last_change":"2024-09-05T16:37:30.634570+0000","last_active":"2024-09-05T16:39:11.876472+0000","last_peered":"2024-09-05T16:39:11.876472+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:39:11.876472+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7173,"log_dups_size":0,"ondisk_log_size":7173,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46168656,"num_objects":5636,"num_object_clones":0,"num_object_copies":22544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1141,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7172,"num_write_kb":51231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1154,"num_bytes_recovered":9445376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4495}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'7436","reported_seq":7503,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:16.432292+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:39:16.432292+0000","last_peered":"2024-09-05T16:39:16.432292+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:39:16.432292+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7436,"log_dups_size":0,"ondisk_log_size":7436,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46609568,"num_objects":5690,"num_object_clones":0,"num_object_copies":22760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2354,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5690,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7434,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3338}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'7214","reported_seq":7307,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:11.813458+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:39:11.813458+0000","last_peered":"2024-09-05T16:39:11.813458+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:39:11.813458+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7214,"log_dups_size":0,"ondisk_log_size":7214,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46027936,"num_objects":5619,"num_object_clones":0,"num_object_copies":22476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5619,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7212,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3304}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'7233","reported_seq":7312,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:16.563651+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:39:16.563651+0000","last_peered":"2024-09-05T16:39:16.563651+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:39:16.563651+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7233,"log_dups_size":0,"ondisk_log_size":7233,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45397152,"num_objects":5542,"num_object_clones":0,"num_object_copies":22168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5542,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7231,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3310}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":731376696,"num_objects":89285,"num_object_clones":0,"num_object_copies":357140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32669,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89285,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":109952,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1263,"num_bytes_recovered":9445376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1376612352,"data_stored":1376612352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115677,"ondisk_log_size":115677,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084352,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":758124,"kb_used_data":357148,"kb_used_omap":4,"kb_used_meta":400955,"kb_avail":93613716,"statfs":{"total":96636764160,"available":95860445184,"internally_reserved":0,"allocated":365719552,"data_stored":365527511,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":410578643},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,6,4,8,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986229,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":557064,"kb_used_data":272696,"kb_used_omap":4,"kb_used_meta":284347,"kb_avail":93814776,"statfs":{"total":96636764160,"available":96066330624,"internally_reserved":0,"allocated":279240704,"data_stored":279048663,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":291172050},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,2,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673034,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":761232,"kb_used_data":360752,"kb_used_omap":1,"kb_used_meta":400446,"kb_avail":93610608,"statfs":{"total":96636764160,"available":95857262592,"internally_reserved":0,"allocated":369410048,"data_stored":369218007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":410057157},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182466,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":745196,"kb_used_data":356764,"kb_used_omap":4,"kb_used_meta":388411,"kb_avail":93626644,"statfs":{"total":96636764160,"available":95873683456,"internally_reserved":0,"allocated":365326336,"data_stored":365134295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":397733581},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,1,0,1,3,3,0,0,0,0,0,0,0,4],"upper_bound":32768},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":364314624,"data_stored":364314624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":278880256,"data_stored":278880256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":368881664,"data_stored":368881664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":364535808,"data_stored":364535808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:39:20.368 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:39:20.952 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:39:20.952 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:39:20.973 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":208,"stamp":"2024-09-05T16:39:20.907602+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":728861752,"num_objects":88978,"num_object_clones":0,"num_object_copies":355912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32609,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":88978,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":110259,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1323,"num_bytes_recovered":9936896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115984,"ondisk_log_size":115984,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2804532,"kb_used_data":1342760,"kb_used_omap":14,"kb_used_meta":1461681,"kb_avail":374682828,"statfs":{"total":386547056640,"available":383675215872,"internally_reserved":0,"allocated":1374986240,"data_stored":1374218076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1496761911},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,1,2,10,10,13,11,1,0,0,0,0,0,0,4],"upper_bound":32768},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19505152,"num_objects":-2381,"num_object_clones":0,"num_object_copies":-9524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-177,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2381,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2381,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":177,"num_bytes_recovered":1105920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005982"},"pg_stats":[{"pgid":"1.f","version":"35'7316","reported_seq":7381,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:16.266443+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:39:16.266443+0000","last_peered":"2024-09-05T16:39:16.266443+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:39:16.266443+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7316,"log_dups_size":0,"ondisk_log_size":7316,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45776896,"num_objects":5588,"num_object_clones":0,"num_object_copies":22352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2253,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5588,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7316,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'7148","reported_seq":7227,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:16.679280+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:39:16.679280+0000","last_peered":"2024-09-05T16:39:16.679280+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:39:16.679280+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7148,"log_dups_size":0,"ondisk_log_size":7148,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44283040,"num_objects":5406,"num_object_clones":0,"num_object_copies":21624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5406,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7146,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3145}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'7237","reported_seq":7324,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:16.702799+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:39:16.702799+0000","last_peered":"2024-09-05T16:39:16.702799+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:39:16.702799+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7237,"log_dups_size":0,"ondisk_log_size":7237,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45177448,"num_objects":5516,"num_object_clones":0,"num_object_copies":22064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2248,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5516,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7236,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3269}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'7318","reported_seq":12041,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-05T16:39:15.789291+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:39:15.789291+0000","last_peered":"2024-09-05T16:39:15.789291+0000","last_clean":"2024-09-05T16:39:15.789291+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:39:15.789291+0000","last_undegraded":"2024-09-05T16:39:15.789291+0000","last_fullsized":"2024-09-05T16:39:15.789291+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":7318,"log_dups_size":0,"ondisk_log_size":7318,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46006272,"num_objects":5616,"num_object_clones":0,"num_object_copies":22464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5616,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1618,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'7132","reported_seq":7205,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:16.030839+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:39:16.030839+0000","last_peered":"2024-09-05T16:39:16.030839+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:39:16.030839+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7132,"log_dups_size":0,"ondisk_log_size":7132,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44749984,"num_objects":5463,"num_object_clones":0,"num_object_copies":21852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2157,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5463,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7130,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3308}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'7198","reported_seq":7261,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:16.380886+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:39:16.380886+0000","last_peered":"2024-09-05T16:39:16.380886+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:39:16.380886+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7198,"log_dups_size":0,"ondisk_log_size":7198,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45662208,"num_objects":5574,"num_object_clones":0,"num_object_copies":22296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2242,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5574,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7198,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3332}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'7082","reported_seq":7164,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:16.594040+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:39:16.594040+0000","last_peered":"2024-09-05T16:39:16.594040+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:39:16.594040+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7082,"log_dups_size":0,"ondisk_log_size":7082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45708448,"num_objects":5580,"num_object_clones":0,"num_object_copies":22320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5580,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7080,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3439}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'7234","reported_seq":7309,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:16.554269+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:39:16.554269+0000","last_peered":"2024-09-05T16:39:16.554269+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:39:16.554269+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7234,"log_dups_size":0,"ondisk_log_size":7234,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45421728,"num_objects":5545,"num_object_clones":0,"num_object_copies":22180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2238,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5545,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7232,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3309}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'7210","reported_seq":7279,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:16.318469+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:39:16.318469+0000","last_peered":"2024-09-05T16:39:16.318469+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:39:16.318469+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7210,"log_dups_size":0,"ondisk_log_size":7210,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45574464,"num_objects":5564,"num_object_clones":0,"num_object_copies":22256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5564,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7207,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3307}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'7277","reported_seq":7372,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:16.966299+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:39:16.966299+0000","last_peered":"2024-09-05T16:39:16.966299+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:39:16.966299+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7277,"log_dups_size":0,"ondisk_log_size":7277,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45323424,"num_objects":5533,"num_object_clones":0,"num_object_copies":22132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7275,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3278}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'7435","reported_seq":7510,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:16.452716+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:39:16.452716+0000","last_peered":"2024-09-05T16:39:16.452716+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:39:16.452716+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7435,"log_dups_size":0,"ondisk_log_size":7435,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46591728,"num_objects":5688,"num_object_clones":0,"num_object_copies":22752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2307,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7432,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3384}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'7192","reported_seq":7253,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:16.264406+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:39:16.264406+0000","last_peered":"2024-09-05T16:39:16.264406+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:39:16.264406+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7192,"log_dups_size":0,"ondisk_log_size":7192,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45603408,"num_objects":5567,"num_object_clones":0,"num_object_copies":22268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2255,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5567,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7191,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3313}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'7256","reported_seq":9761,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:17.003364+0000","last_change":"2024-09-05T16:37:30.634570+0000","last_active":"2024-09-05T16:39:17.003364+0000","last_peered":"2024-09-05T16:39:17.003364+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:39:17.003364+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7256,"log_dups_size":0,"ondisk_log_size":7256,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45488720,"num_objects":5553,"num_object_clones":0,"num_object_copies":22212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1081,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5553,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7255,"num_write_kb":51231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1214,"num_bytes_recovered":9936896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4472}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'7436","reported_seq":7503,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:16.432292+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:39:16.432292+0000","last_peered":"2024-09-05T16:39:16.432292+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:39:16.432292+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7436,"log_dups_size":0,"ondisk_log_size":7436,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46609568,"num_objects":5690,"num_object_clones":0,"num_object_copies":22760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2354,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5690,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7434,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3338}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'7280","reported_seq":7373,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:16.910679+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:39:16.910679+0000","last_peered":"2024-09-05T16:39:16.910679+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:39:16.910679+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7280,"log_dups_size":0,"ondisk_log_size":7280,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45487264,"num_objects":5553,"num_object_clones":0,"num_object_copies":22212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5553,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7278,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3238}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'7233","reported_seq":7312,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:16.563651+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:39:16.563651+0000","last_peered":"2024-09-05T16:39:16.563651+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:39:16.563651+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7233,"log_dups_size":0,"ondisk_log_size":7233,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45397152,"num_objects":5542,"num_object_clones":0,"num_object_copies":22168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5542,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7231,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3310}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":728861752,"num_objects":88978,"num_object_clones":0,"num_object_copies":355912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32609,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":88978,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":110259,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1323,"num_bytes_recovered":9936896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1371807744,"data_stored":1371807744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115984,"ondisk_log_size":115984,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084352,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":758124,"kb_used_data":357148,"kb_used_omap":4,"kb_used_meta":400955,"kb_avail":93613716,"statfs":{"total":96636764160,"available":95860445184,"internally_reserved":0,"allocated":365719552,"data_stored":365527511,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":410578643},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,6,4,8,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986229,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":557064,"kb_used_data":272696,"kb_used_omap":4,"kb_used_meta":284347,"kb_avail":93814776,"statfs":{"total":96636764160,"available":96066330624,"internally_reserved":0,"allocated":279240704,"data_stored":279048663,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":291172050},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,2,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673035,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744148,"kb_used_data":356152,"kb_used_omap":1,"kb_used_meta":387966,"kb_avail":93627692,"statfs":{"total":96636764160,"available":95874756608,"internally_reserved":0,"allocated":364699648,"data_stored":364507607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":397277637},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,3,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182466,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":745196,"kb_used_data":356764,"kb_used_omap":4,"kb_used_meta":388411,"kb_avail":93626644,"statfs":{"total":96636764160,"available":95873683456,"internally_reserved":0,"allocated":365326336,"data_stored":365134295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":397733581},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,1,0,1,3,3,0,0,0,0,0,0,0,4],"upper_bound":32768},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":364314624,"data_stored":364314624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":278880256,"data_stored":278880256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":364077056,"data_stored":364077056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":364535808,"data_stored":364535808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:39:20.974 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:39:21.556 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:39:21.557 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:39:21.585 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":208,"stamp":"2024-09-05T16:39:20.907602+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":728861752,"num_objects":88978,"num_object_clones":0,"num_object_copies":355912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32609,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":88978,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":110259,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1323,"num_bytes_recovered":9936896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115984,"ondisk_log_size":115984,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2804532,"kb_used_data":1342760,"kb_used_omap":14,"kb_used_meta":1461681,"kb_avail":374682828,"statfs":{"total":386547056640,"available":383675215872,"internally_reserved":0,"allocated":1374986240,"data_stored":1374218076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1496761911},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,1,2,10,10,13,11,1,0,0,0,0,0,0,4],"upper_bound":32768},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19505152,"num_objects":-2381,"num_object_clones":0,"num_object_copies":-9524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-177,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2381,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2381,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":177,"num_bytes_recovered":1105920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005982"},"pg_stats":[{"pgid":"1.f","version":"35'7316","reported_seq":7381,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:16.266443+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:39:16.266443+0000","last_peered":"2024-09-05T16:39:16.266443+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:39:16.266443+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7316,"log_dups_size":0,"ondisk_log_size":7316,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45776896,"num_objects":5588,"num_object_clones":0,"num_object_copies":22352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2253,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5588,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7316,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'7148","reported_seq":7227,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:16.679280+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:39:16.679280+0000","last_peered":"2024-09-05T16:39:16.679280+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:39:16.679280+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7148,"log_dups_size":0,"ondisk_log_size":7148,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44283040,"num_objects":5406,"num_object_clones":0,"num_object_copies":21624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5406,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7146,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3145}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'7237","reported_seq":7324,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:16.702799+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:39:16.702799+0000","last_peered":"2024-09-05T16:39:16.702799+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:39:16.702799+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7237,"log_dups_size":0,"ondisk_log_size":7237,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45177448,"num_objects":5516,"num_object_clones":0,"num_object_copies":22064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2248,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5516,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7236,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3269}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'7318","reported_seq":12041,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-05T16:39:15.789291+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:39:15.789291+0000","last_peered":"2024-09-05T16:39:15.789291+0000","last_clean":"2024-09-05T16:39:15.789291+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:39:15.789291+0000","last_undegraded":"2024-09-05T16:39:15.789291+0000","last_fullsized":"2024-09-05T16:39:15.789291+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":7318,"log_dups_size":0,"ondisk_log_size":7318,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46006272,"num_objects":5616,"num_object_clones":0,"num_object_copies":22464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5616,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1618,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'7132","reported_seq":7205,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:16.030839+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:39:16.030839+0000","last_peered":"2024-09-05T16:39:16.030839+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:39:16.030839+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7132,"log_dups_size":0,"ondisk_log_size":7132,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44749984,"num_objects":5463,"num_object_clones":0,"num_object_copies":21852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2157,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5463,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7130,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3308}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'7198","reported_seq":7261,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:16.380886+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:39:16.380886+0000","last_peered":"2024-09-05T16:39:16.380886+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:39:16.380886+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7198,"log_dups_size":0,"ondisk_log_size":7198,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45662208,"num_objects":5574,"num_object_clones":0,"num_object_copies":22296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2242,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5574,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7198,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3332}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'7082","reported_seq":7164,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:16.594040+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:39:16.594040+0000","last_peered":"2024-09-05T16:39:16.594040+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:39:16.594040+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7082,"log_dups_size":0,"ondisk_log_size":7082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45708448,"num_objects":5580,"num_object_clones":0,"num_object_copies":22320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5580,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7080,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3439}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'7234","reported_seq":7309,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:16.554269+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:39:16.554269+0000","last_peered":"2024-09-05T16:39:16.554269+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:39:16.554269+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7234,"log_dups_size":0,"ondisk_log_size":7234,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45421728,"num_objects":5545,"num_object_clones":0,"num_object_copies":22180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2238,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5545,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7232,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3309}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'7210","reported_seq":7279,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:16.318469+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:39:16.318469+0000","last_peered":"2024-09-05T16:39:16.318469+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:39:16.318469+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7210,"log_dups_size":0,"ondisk_log_size":7210,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45574464,"num_objects":5564,"num_object_clones":0,"num_object_copies":22256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5564,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7207,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3307}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'7277","reported_seq":7372,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:16.966299+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:39:16.966299+0000","last_peered":"2024-09-05T16:39:16.966299+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:39:16.966299+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7277,"log_dups_size":0,"ondisk_log_size":7277,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45323424,"num_objects":5533,"num_object_clones":0,"num_object_copies":22132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7275,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3278}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'7435","reported_seq":7510,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:16.452716+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:39:16.452716+0000","last_peered":"2024-09-05T16:39:16.452716+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:39:16.452716+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7435,"log_dups_size":0,"ondisk_log_size":7435,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46591728,"num_objects":5688,"num_object_clones":0,"num_object_copies":22752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2307,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7432,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3384}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'7192","reported_seq":7253,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:16.264406+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:39:16.264406+0000","last_peered":"2024-09-05T16:39:16.264406+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:39:16.264406+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7192,"log_dups_size":0,"ondisk_log_size":7192,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45603408,"num_objects":5567,"num_object_clones":0,"num_object_copies":22268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2255,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5567,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7191,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3313}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'7256","reported_seq":9761,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:17.003364+0000","last_change":"2024-09-05T16:37:30.634570+0000","last_active":"2024-09-05T16:39:17.003364+0000","last_peered":"2024-09-05T16:39:17.003364+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:39:17.003364+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7256,"log_dups_size":0,"ondisk_log_size":7256,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45488720,"num_objects":5553,"num_object_clones":0,"num_object_copies":22212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1081,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5553,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7255,"num_write_kb":51231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1214,"num_bytes_recovered":9936896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4472}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'7436","reported_seq":7503,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:16.432292+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:39:16.432292+0000","last_peered":"2024-09-05T16:39:16.432292+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:39:16.432292+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7436,"log_dups_size":0,"ondisk_log_size":7436,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46609568,"num_objects":5690,"num_object_clones":0,"num_object_copies":22760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2354,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5690,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7434,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3338}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'7280","reported_seq":7373,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:16.910679+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:39:16.910679+0000","last_peered":"2024-09-05T16:39:16.910679+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:39:16.910679+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7280,"log_dups_size":0,"ondisk_log_size":7280,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45487264,"num_objects":5553,"num_object_clones":0,"num_object_copies":22212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5553,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7278,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3238}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'7233","reported_seq":7312,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:16.563651+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:39:16.563651+0000","last_peered":"2024-09-05T16:39:16.563651+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:39:16.563651+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7233,"log_dups_size":0,"ondisk_log_size":7233,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45397152,"num_objects":5542,"num_object_clones":0,"num_object_copies":22168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5542,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7231,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3310}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":728861752,"num_objects":88978,"num_object_clones":0,"num_object_copies":355912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32609,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":88978,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":110259,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1323,"num_bytes_recovered":9936896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1371807744,"data_stored":1371807744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115984,"ondisk_log_size":115984,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084352,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":758124,"kb_used_data":357148,"kb_used_omap":4,"kb_used_meta":400955,"kb_avail":93613716,"statfs":{"total":96636764160,"available":95860445184,"internally_reserved":0,"allocated":365719552,"data_stored":365527511,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":410578643},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,6,4,8,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986229,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":557064,"kb_used_data":272696,"kb_used_omap":4,"kb_used_meta":284347,"kb_avail":93814776,"statfs":{"total":96636764160,"available":96066330624,"internally_reserved":0,"allocated":279240704,"data_stored":279048663,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":291172050},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,2,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673035,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744148,"kb_used_data":356152,"kb_used_omap":1,"kb_used_meta":387966,"kb_avail":93627692,"statfs":{"total":96636764160,"available":95874756608,"internally_reserved":0,"allocated":364699648,"data_stored":364507607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":397277637},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,3,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182466,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":745196,"kb_used_data":356764,"kb_used_omap":4,"kb_used_meta":388411,"kb_avail":93626644,"statfs":{"total":96636764160,"available":95873683456,"internally_reserved":0,"allocated":365326336,"data_stored":365134295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":397733581},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,1,0,1,3,3,0,0,0,0,0,0,0,4],"upper_bound":32768},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":364314624,"data_stored":364314624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":278880256,"data_stored":278880256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":364077056,"data_stored":364077056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":364535808,"data_stored":364535808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:39:21.586 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-05T16:39:21.586 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'7316", 'reported_seq': 7381, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:16.266443+0000', 'last_change': '2024-09-05T16:34:56.833497+0000', 'last_active': '2024-09-05T16:39:16.266443+0000', 'last_peered': '2024-09-05T16:39:16.266443+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.828289+0000', 'last_became_peered': '2024-09-05T16:34:56.828289+0000', 'last_unstale': '2024-09-05T16:39:16.266443+0000', 'last_undegraded': '2024-09-05T16:34:56.612160+0000', 'last_fullsized': '2024-09-05T16:34:56.557632+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7316, 'log_dups_size': 0, 'ondisk_log_size': 7316, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45776896, 'num_objects': 5588, 'num_object_clones': 0, 'num_object_copies': 22352, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2253, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5588, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7316, 'num_write_kb': 51616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:39:21.586 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-05T16:39:21.586 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "35'7148", 'reported_seq': 7227, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:16.679280+0000', 'last_change': '2024-09-05T16:34:56.825031+0000', 'last_active': '2024-09-05T16:39:16.679280+0000', 'last_peered': '2024-09-05T16:39:16.679280+0000', 'last_clean': '2024-09-05T16:33:53.200762+0000', 'last_became_active': '2024-09-05T16:34:56.820292+0000', 'last_became_peered': '2024-09-05T16:34:56.820292+0000', 'last_unstale': '2024-09-05T16:39:16.679280+0000', 'last_undegraded': '2024-09-05T16:34:56.605776+0000', 'last_fullsized': '2024-09-05T16:34:56.559383+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7148, 'log_dups_size': 0, 'ondisk_log_size': 7148, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44283040, 'num_objects': 5406, 'num_object_clones': 0, 'num_object_copies': 21624, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2263, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5406, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7146, 'num_write_kb': 50212, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 8, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3145}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:39:21.586 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:39:21.586 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'7237", 'reported_seq': 7324, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:16.702799+0000', 'last_change': '2024-09-05T16:34:56.867900+0000', 'last_active': '2024-09-05T16:39:16.702799+0000', 'last_peered': '2024-09-05T16:39:16.702799+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:39:16.702799+0000', 'last_undegraded': '2024-09-05T16:34:56.590972+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7237, 'log_dups_size': 0, 'ondisk_log_size': 7237, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45177448, 'num_objects': 5516, 'num_object_clones': 0, 'num_object_copies': 22064, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2248, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5516, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 7236, 'num_write_kb': 50999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3269}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:39:21.587 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:39:21.587 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'7132", 'reported_seq': 7205, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:16.030839+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:39:16.030839+0000', 'last_peered': '2024-09-05T16:39:16.030839+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:39:16.030839+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7132, 'log_dups_size': 0, 'ondisk_log_size': 7132, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44749984, 'num_objects': 5463, 'num_object_clones': 0, 'num_object_copies': 21852, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2157, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5463, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7130, 'num_write_kb': 50373, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3308}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:39:21.587 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:39:21.587 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'7198", 'reported_seq': 7261, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:16.380886+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:39:16.380886+0000', 'last_peered': '2024-09-05T16:39:16.380886+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:39:16.380886+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7198, 'log_dups_size': 0, 'ondisk_log_size': 7198, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45662208, 'num_objects': 5574, 'num_object_clones': 0, 'num_object_copies': 22296, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2242, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5574, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7198, 'num_write_kb': 51088, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 8, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3332}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:39:21.587 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:39:21.587 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'7082", 'reported_seq': 7164, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:16.594040+0000', 'last_change': '2024-09-05T16:34:56.903436+0000', 'last_active': '2024-09-05T16:39:16.594040+0000', 'last_peered': '2024-09-05T16:39:16.594040+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:39:16.594040+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7082, 'log_dups_size': 0, 'ondisk_log_size': 7082, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45708448, 'num_objects': 5580, 'num_object_clones': 0, 'num_object_copies': 22320, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2143, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5580, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7080, 'num_write_kb': 50636, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 8, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3439}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:39:21.587 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-05T16:39:21.587 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "35'7234", 'reported_seq': 7309, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:16.554269+0000', 'last_change': '2024-09-05T16:34:56.883319+0000', 'last_active': '2024-09-05T16:39:16.554269+0000', 'last_peered': '2024-09-05T16:39:16.554269+0000', 'last_clean': '2024-09-05T16:33:53.505305+0000', 'last_became_active': '2024-09-05T16:34:56.873389+0000', 'last_became_peered': '2024-09-05T16:34:56.873389+0000', 'last_unstale': '2024-09-05T16:39:16.554269+0000', 'last_undegraded': '2024-09-05T16:34:56.602567+0000', 'last_fullsized': '2024-09-05T16:34:56.559958+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7234, 'log_dups_size': 0, 'ondisk_log_size': 7234, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45421728, 'num_objects': 5545, 'num_object_clones': 0, 'num_object_copies': 22180, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2238, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5545, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7232, 'num_write_kb': 51108, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3309}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:39:21.587 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:39:21.587 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'7210", 'reported_seq': 7279, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:16.318469+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:39:16.318469+0000', 'last_peered': '2024-09-05T16:39:16.318469+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:39:16.318469+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7210, 'log_dups_size': 0, 'ondisk_log_size': 7210, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45574464, 'num_objects': 5564, 'num_object_clones': 0, 'num_object_copies': 22256, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2260, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5564, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7207, 'num_write_kb': 51081, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3307}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:39:21.587 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:39:21.588 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'7277", 'reported_seq': 7372, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:16.966299+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:39:16.966299+0000', 'last_peered': '2024-09-05T16:39:16.966299+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:39:16.966299+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7277, 'log_dups_size': 0, 'ondisk_log_size': 7277, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45323424, 'num_objects': 5533, 'num_object_clones': 0, 'num_object_copies': 22132, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5533, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7275, 'num_write_kb': 51230, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3278}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:39:21.588 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:39:21.588 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'7435", 'reported_seq': 7510, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:16.452716+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:39:16.452716+0000', 'last_peered': '2024-09-05T16:39:16.452716+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:39:16.452716+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7435, 'log_dups_size': 0, 'ondisk_log_size': 7435, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46591728, 'num_objects': 5688, 'num_object_clones': 0, 'num_object_copies': 22752, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2307, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5688, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7432, 'num_write_kb': 52476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3384}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:39:21.588 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:39:21.588 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'7192", 'reported_seq': 7253, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:16.264406+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:39:16.264406+0000', 'last_peered': '2024-09-05T16:39:16.264406+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:39:16.264406+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7192, 'log_dups_size': 0, 'ondisk_log_size': 7192, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45603408, 'num_objects': 5567, 'num_object_clones': 0, 'num_object_copies': 22268, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2255, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5567, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7191, 'num_write_kb': 51031, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3313}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:39:21.588 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-05T16:39:21.588 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'7256", 'reported_seq': 9761, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:17.003364+0000', 'last_change': '2024-09-05T16:37:30.634570+0000', 'last_active': '2024-09-05T16:39:17.003364+0000', 'last_peered': '2024-09-05T16:39:17.003364+0000', 'last_clean': '2024-09-05T16:33:45.082039+0000', 'last_became_active': '2024-09-05T16:34:56.792386+0000', 'last_became_peered': '2024-09-05T16:34:56.792386+0000', 'last_unstale': '2024-09-05T16:39:17.003364+0000', 'last_undegraded': '2024-09-05T16:34:56.600595+0000', 'last_fullsized': '2024-09-05T16:34:56.558203+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7256, 'log_dups_size': 0, 'ondisk_log_size': 7256, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45488720, 'num_objects': 5553, 'num_object_clones': 0, 'num_object_copies': 22212, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1081, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5553, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7255, 'num_write_kb': 51231, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1214, 'num_bytes_recovered': 9936896, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 4472}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:39:21.588 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:39:21.588 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'7436", 'reported_seq': 7503, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:16.432292+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:39:16.432292+0000', 'last_peered': '2024-09-05T16:39:16.432292+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:39:16.432292+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7436, 'log_dups_size': 0, 'ondisk_log_size': 7436, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46609568, 'num_objects': 5690, 'num_object_clones': 0, 'num_object_copies': 22760, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2354, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5690, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7434, 'num_write_kb': 52500, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3338}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:39:21.588 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:39:21.588 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'7280", 'reported_seq': 7373, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:16.910679+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:39:16.910679+0000', 'last_peered': '2024-09-05T16:39:16.910679+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:39:16.910679+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7280, 'log_dups_size': 0, 'ondisk_log_size': 7280, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45487264, 'num_objects': 5553, 'num_object_clones': 0, 'num_object_copies': 22212, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2317, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5553, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7278, 'num_write_kb': 51325, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3238}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:39:21.589 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:39:21.589 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'7233", 'reported_seq': 7312, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:16.563651+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:39:16.563651+0000', 'last_peered': '2024-09-05T16:39:16.563651+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:39:16.563651+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7233, 'log_dups_size': 0, 'ondisk_log_size': 7233, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45397152, 'num_objects': 5542, 'num_object_clones': 0, 'num_object_copies': 22168, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2234, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5542, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7231, 'num_write_kb': 51092, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3310}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:39:21.589 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:39:22.323 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:39:22.323 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 32609/355912 objects degraded (9.162%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":365,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":88978,"data_bytes":728861752,"bytes_used":2871840768,"bytes_avail":383675215872,"bytes_total":386547056640,"degraded_objects":32609,"degraded_total":355912,"degraded_ratio":0.091620962485108676,"recovering_objects_per_sec":14,"recovering_bytes_per_sec":92114,"recovering_keys_per_sec":0,"num_objects_recovered":177,"num_bytes_recovered":1105920,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":198},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:39:22.348 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:39:22.349 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:39:22.931 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:39:22.931 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:39:22.953 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":209,"stamp":"2024-09-05T16:39:22.908099+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":718965816,"num_objects":87770,"num_object_clones":0,"num_object_copies":351080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32541,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":87770,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":111467,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1391,"num_bytes_recovered":10420224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117192,"ondisk_log_size":117192,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2788756,"kb_used_data":1326996,"kb_used_omap":14,"kb_used_meta":1461681,"kb_avail":374698604,"statfs":{"total":386547056640,"available":383691370496,"internally_reserved":0,"allocated":1358843904,"data_stored":1358075740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1496761911},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,6,12,7,16,3,0,0,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-29401088,"num_objects":-3589,"num_object_clones":0,"num_object_copies":-14356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-245,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3589,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3589,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":245,"num_bytes_recovered":1589248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005614"},"pg_stats":[{"pgid":"1.f","version":"35'7395","reported_seq":7462,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:21.467083+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:39:21.467083+0000","last_peered":"2024-09-05T16:39:21.467083+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:39:21.467083+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7395,"log_dups_size":0,"ondisk_log_size":7395,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45129728,"num_objects":5509,"num_object_clones":0,"num_object_copies":22036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5509,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7395,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3257}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'7228","reported_seq":7313,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:21.606543+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:39:21.606543+0000","last_peered":"2024-09-05T16:39:21.606543+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:39:21.606543+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7228,"log_dups_size":0,"ondisk_log_size":7228,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43627680,"num_objects":5326,"num_object_clones":0,"num_object_copies":21304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2261,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5326,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7226,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3067}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'7314","reported_seq":7401,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:21.936963+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:39:21.936963+0000","last_peered":"2024-09-05T16:39:21.936963+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:39:21.936963+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7314,"log_dups_size":0,"ondisk_log_size":7314,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44546664,"num_objects":5439,"num_object_clones":0,"num_object_copies":21756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2248,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5439,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7313,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3192}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'7392","reported_seq":12115,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-05T16:39:20.961407+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:39:20.961407+0000","last_peered":"2024-09-05T16:39:20.961407+0000","last_clean":"2024-09-05T16:39:20.961407+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:39:20.961407+0000","last_undegraded":"2024-09-05T16:39:20.961407+0000","last_fullsized":"2024-09-05T16:39:20.961407+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":7392,"log_dups_size":0,"ondisk_log_size":7392,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45400064,"num_objects":5542,"num_object_clones":0,"num_object_copies":22168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5542,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1692,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'7200","reported_seq":7273,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:21.526869+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:39:21.526869+0000","last_peered":"2024-09-05T16:39:21.526869+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:39:21.526869+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7200,"log_dups_size":0,"ondisk_log_size":7200,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44192928,"num_objects":5395,"num_object_clones":0,"num_object_copies":21580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2157,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5395,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7198,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3240}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'7269","reported_seq":7334,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:21.464773+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:39:21.464773+0000","last_peered":"2024-09-05T16:39:21.464773+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:39:21.464773+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7269,"log_dups_size":0,"ondisk_log_size":7269,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45080576,"num_objects":5503,"num_object_clones":0,"num_object_copies":22012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7269,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3262}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'7136","reported_seq":7220,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:21.306238+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:39:21.306238+0000","last_peered":"2024-09-05T16:39:21.306238+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:39:21.306238+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7136,"log_dups_size":0,"ondisk_log_size":7136,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45266080,"num_objects":5526,"num_object_clones":0,"num_object_copies":22104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2142,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5526,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7134,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3386}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'7323","reported_seq":7400,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:21.605157+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:39:21.605157+0000","last_peered":"2024-09-05T16:39:21.605157+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:39:21.605157+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7323,"log_dups_size":0,"ondisk_log_size":7323,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44692640,"num_objects":5456,"num_object_clones":0,"num_object_copies":21824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2237,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5456,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7321,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3221}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'7282","reported_seq":7351,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:21.175581+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:39:21.175581+0000","last_peered":"2024-09-05T16:39:21.175581+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:39:21.175581+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7282,"log_dups_size":0,"ondisk_log_size":7282,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44984640,"num_objects":5492,"num_object_clones":0,"num_object_copies":21968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5492,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7279,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3235}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'7341","reported_seq":7437,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:21.929814+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:39:21.929814+0000","last_peered":"2024-09-05T16:39:21.929814+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:39:21.929814+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7341,"log_dups_size":0,"ondisk_log_size":7341,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44799136,"num_objects":5469,"num_object_clones":0,"num_object_copies":21876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5469,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7339,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3214}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'7516","reported_seq":7591,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:21.467464+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:39:21.467464+0000","last_peered":"2024-09-05T16:39:21.467464+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:39:21.467464+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7516,"log_dups_size":0,"ondisk_log_size":7516,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45928176,"num_objects":5607,"num_object_clones":0,"num_object_copies":22428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2307,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5607,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7513,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3303}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'7257","reported_seq":7318,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:21.390648+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:39:21.390648+0000","last_peered":"2024-09-05T16:39:21.390648+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:39:21.390648+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7257,"log_dups_size":0,"ondisk_log_size":7257,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45070928,"num_objects":5502,"num_object_clones":0,"num_object_copies":22008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2255,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5502,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7256,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3248}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'7338","reported_seq":9962,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:21.997625+0000","last_change":"2024-09-05T16:37:30.634570+0000","last_active":"2024-09-05T16:39:21.997625+0000","last_peered":"2024-09-05T16:39:21.997625+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:39:21.997625+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7338,"log_dups_size":0,"ondisk_log_size":7338,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44816976,"num_objects":5471,"num_object_clones":0,"num_object_copies":21884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1022,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5471,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7337,"num_write_kb":51231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1273,"num_bytes_recovered":10420224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4449}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'7525","reported_seq":7594,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:21.514724+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:39:21.514724+0000","last_peered":"2024-09-05T16:39:21.514724+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:39:21.514724+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7525,"log_dups_size":0,"ondisk_log_size":7525,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45880480,"num_objects":5601,"num_object_clones":0,"num_object_copies":22404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2353,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5601,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7523,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3250}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'7356","reported_seq":7449,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:21.545674+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:39:21.545674+0000","last_peered":"2024-09-05T16:39:21.545674+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:39:21.545674+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7356,"log_dups_size":0,"ondisk_log_size":7356,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44864672,"num_objects":5477,"num_object_clones":0,"num_object_copies":21908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5477,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7354,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3162}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'7320","reported_seq":7403,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:21.553762+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:39:21.553762+0000","last_peered":"2024-09-05T16:39:21.553762+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:39:21.553762+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7320,"log_dups_size":0,"ondisk_log_size":7320,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44684448,"num_objects":5455,"num_object_clones":0,"num_object_copies":21820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5455,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7318,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3225}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":718965816,"num_objects":87770,"num_object_clones":0,"num_object_copies":351080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32541,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":87770,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":111467,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1391,"num_bytes_recovered":10420224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1356496896,"data_stored":1356496896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117192,"ondisk_log_size":117192,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084353,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":739864,"kb_used_data":351368,"kb_used_omap":4,"kb_used_meta":388475,"kb_avail":93631976,"statfs":{"total":96636764160,"available":95879143424,"internally_reserved":0,"allocated":359800832,"data_stored":359608791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":397799123},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,6,4,8,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986230,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":560876,"kb_used_data":272348,"kb_used_omap":4,"kb_used_meta":288507,"kb_avail":93810964,"statfs":{"total":96636764160,"available":96062427136,"internally_reserved":0,"allocated":278884352,"data_stored":278692311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":295431890},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673036,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":743432,"kb_used_data":351288,"kb_used_omap":1,"kb_used_meta":392126,"kb_avail":93628408,"statfs":{"total":96636764160,"available":95875489792,"internally_reserved":0,"allocated":359718912,"data_stored":359526871,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":401537477},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,3,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182467,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744584,"kb_used_data":351992,"kb_used_omap":4,"kb_used_meta":392571,"kb_avail":93627256,"statfs":{"total":96636764160,"available":95874310144,"internally_reserved":0,"allocated":360439808,"data_stored":360247767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":401993421},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,6,2,0,0,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":359378944,"data_stored":359378944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":278507520,"data_stored":278507520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":359124992,"data_stored":359124992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":359485440,"data_stored":359485440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:39:25.955 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:39:26.565 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:39:26.566 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:39:26.586 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":210,"stamp":"2024-09-05T16:39:24.908860+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":718965816,"num_objects":87770,"num_object_clones":0,"num_object_copies":351080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32541,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":87770,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":111467,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1391,"num_bytes_recovered":10420224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117192,"ondisk_log_size":117192,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2788756,"kb_used_data":1326996,"kb_used_omap":14,"kb_used_meta":1461681,"kb_avail":374698604,"statfs":{"total":386547056640,"available":383691370496,"internally_reserved":0,"allocated":1358843904,"data_stored":1358075740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1496761911},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,6,12,7,16,3,0,0,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19603456,"num_objects":-2393,"num_object_clones":0,"num_object_copies":-9572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2393,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2393,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":154,"num_bytes_recovered":974848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005627"},"pg_stats":[{"pgid":"1.f","version":"35'7395","reported_seq":7462,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:21.467083+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:39:21.467083+0000","last_peered":"2024-09-05T16:39:21.467083+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:39:21.467083+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7395,"log_dups_size":0,"ondisk_log_size":7395,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45129728,"num_objects":5509,"num_object_clones":0,"num_object_copies":22036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5509,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7395,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3257}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'7228","reported_seq":7313,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:21.606543+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:39:21.606543+0000","last_peered":"2024-09-05T16:39:21.606543+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:39:21.606543+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7228,"log_dups_size":0,"ondisk_log_size":7228,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43627680,"num_objects":5326,"num_object_clones":0,"num_object_copies":21304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2261,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5326,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7226,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3067}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'7314","reported_seq":7401,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:21.936963+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:39:21.936963+0000","last_peered":"2024-09-05T16:39:21.936963+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:39:21.936963+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7314,"log_dups_size":0,"ondisk_log_size":7314,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44546664,"num_objects":5439,"num_object_clones":0,"num_object_copies":21756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2248,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5439,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7313,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3192}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'7392","reported_seq":12115,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-05T16:39:20.961407+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:39:20.961407+0000","last_peered":"2024-09-05T16:39:20.961407+0000","last_clean":"2024-09-05T16:39:20.961407+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:39:20.961407+0000","last_undegraded":"2024-09-05T16:39:20.961407+0000","last_fullsized":"2024-09-05T16:39:20.961407+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":7392,"log_dups_size":0,"ondisk_log_size":7392,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45400064,"num_objects":5542,"num_object_clones":0,"num_object_copies":22168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5542,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1692,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'7200","reported_seq":7273,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:21.526869+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:39:21.526869+0000","last_peered":"2024-09-05T16:39:21.526869+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:39:21.526869+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7200,"log_dups_size":0,"ondisk_log_size":7200,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44192928,"num_objects":5395,"num_object_clones":0,"num_object_copies":21580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2157,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5395,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7198,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3240}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'7269","reported_seq":7334,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:21.464773+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:39:21.464773+0000","last_peered":"2024-09-05T16:39:21.464773+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:39:21.464773+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7269,"log_dups_size":0,"ondisk_log_size":7269,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45080576,"num_objects":5503,"num_object_clones":0,"num_object_copies":22012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7269,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3262}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'7136","reported_seq":7220,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:21.306238+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:39:21.306238+0000","last_peered":"2024-09-05T16:39:21.306238+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:39:21.306238+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7136,"log_dups_size":0,"ondisk_log_size":7136,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45266080,"num_objects":5526,"num_object_clones":0,"num_object_copies":22104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2142,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5526,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7134,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3386}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'7323","reported_seq":7400,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:21.605157+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:39:21.605157+0000","last_peered":"2024-09-05T16:39:21.605157+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:39:21.605157+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7323,"log_dups_size":0,"ondisk_log_size":7323,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44692640,"num_objects":5456,"num_object_clones":0,"num_object_copies":21824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2237,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5456,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7321,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3221}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'7282","reported_seq":7351,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:21.175581+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:39:21.175581+0000","last_peered":"2024-09-05T16:39:21.175581+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:39:21.175581+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7282,"log_dups_size":0,"ondisk_log_size":7282,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44984640,"num_objects":5492,"num_object_clones":0,"num_object_copies":21968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5492,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7279,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3235}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'7341","reported_seq":7437,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:21.929814+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:39:21.929814+0000","last_peered":"2024-09-05T16:39:21.929814+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:39:21.929814+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7341,"log_dups_size":0,"ondisk_log_size":7341,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44799136,"num_objects":5469,"num_object_clones":0,"num_object_copies":21876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5469,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7339,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3214}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'7516","reported_seq":7591,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:21.467464+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:39:21.467464+0000","last_peered":"2024-09-05T16:39:21.467464+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:39:21.467464+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7516,"log_dups_size":0,"ondisk_log_size":7516,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45928176,"num_objects":5607,"num_object_clones":0,"num_object_copies":22428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2307,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5607,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7513,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3303}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'7257","reported_seq":7318,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:21.390648+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:39:21.390648+0000","last_peered":"2024-09-05T16:39:21.390648+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:39:21.390648+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7257,"log_dups_size":0,"ondisk_log_size":7257,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45070928,"num_objects":5502,"num_object_clones":0,"num_object_copies":22008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2255,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5502,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7256,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3248}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'7338","reported_seq":9962,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:21.997625+0000","last_change":"2024-09-05T16:37:30.634570+0000","last_active":"2024-09-05T16:39:21.997625+0000","last_peered":"2024-09-05T16:39:21.997625+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:39:21.997625+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7338,"log_dups_size":0,"ondisk_log_size":7338,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44816976,"num_objects":5471,"num_object_clones":0,"num_object_copies":21884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1022,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5471,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7337,"num_write_kb":51231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1273,"num_bytes_recovered":10420224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4449}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'7525","reported_seq":7594,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:21.514724+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:39:21.514724+0000","last_peered":"2024-09-05T16:39:21.514724+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:39:21.514724+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7525,"log_dups_size":0,"ondisk_log_size":7525,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45880480,"num_objects":5601,"num_object_clones":0,"num_object_copies":22404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2353,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5601,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7523,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3250}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'7356","reported_seq":7449,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:21.545674+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:39:21.545674+0000","last_peered":"2024-09-05T16:39:21.545674+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:39:21.545674+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7356,"log_dups_size":0,"ondisk_log_size":7356,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44864672,"num_objects":5477,"num_object_clones":0,"num_object_copies":21908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5477,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7354,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3162}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'7320","reported_seq":7403,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:21.553762+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:39:21.553762+0000","last_peered":"2024-09-05T16:39:21.553762+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:39:21.553762+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7320,"log_dups_size":0,"ondisk_log_size":7320,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44684448,"num_objects":5455,"num_object_clones":0,"num_object_copies":21820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5455,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7318,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3225}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":718965816,"num_objects":87770,"num_object_clones":0,"num_object_copies":351080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32541,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":87770,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":111467,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1391,"num_bytes_recovered":10420224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1356496896,"data_stored":1356496896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117192,"ondisk_log_size":117192,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084353,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":739864,"kb_used_data":351368,"kb_used_omap":4,"kb_used_meta":388475,"kb_avail":93631976,"statfs":{"total":96636764160,"available":95879143424,"internally_reserved":0,"allocated":359800832,"data_stored":359608791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":397799123},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,6,4,8,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986230,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":560876,"kb_used_data":272348,"kb_used_omap":4,"kb_used_meta":288507,"kb_avail":93810964,"statfs":{"total":96636764160,"available":96062427136,"internally_reserved":0,"allocated":278884352,"data_stored":278692311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":295431890},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673036,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":743432,"kb_used_data":351288,"kb_used_omap":1,"kb_used_meta":392126,"kb_avail":93628408,"statfs":{"total":96636764160,"available":95875489792,"internally_reserved":0,"allocated":359718912,"data_stored":359526871,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":401537477},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,3,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182467,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744584,"kb_used_data":351992,"kb_used_omap":4,"kb_used_meta":392571,"kb_avail":93627256,"statfs":{"total":96636764160,"available":95874310144,"internally_reserved":0,"allocated":360439808,"data_stored":360247767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":401993421},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,6,2,0,0,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":359378944,"data_stored":359378944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":278507520,"data_stored":278507520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":359124992,"data_stored":359124992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":359485440,"data_stored":359485440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:39:26.588 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:39:27.177 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:39:27.177 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:39:27.198 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":211,"stamp":"2024-09-05T16:39:26.909705+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":711732280,"num_objects":86887,"num_object_clones":0,"num_object_copies":347548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86887,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":112350,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1398,"num_bytes_recovered":10420224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":118075,"ondisk_log_size":118075,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2832568,"kb_used_data":1318052,"kb_used_omap":14,"kb_used_meta":1514417,"kb_avail":374654792,"statfs":{"total":386547056640,"available":383646507008,"internally_reserved":0,"allocated":1349685248,"data_stored":1348917084,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1550763575},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,5,13,7,5,0,0,0,0,0,0,5,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-26836992,"num_objects":-3276,"num_object_clones":0,"num_object_copies":-13104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-161,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3276,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3276,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":161,"num_bytes_recovered":974848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004829"},"pg_stats":[{"pgid":"1.f","version":"35'7462","reported_seq":7529,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:26.466111+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:39:26.466111+0000","last_peered":"2024-09-05T16:39:26.466111+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:39:26.466111+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7462,"log_dups_size":0,"ondisk_log_size":7462,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44580864,"num_objects":5442,"num_object_clones":0,"num_object_copies":21768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5442,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7462,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3190}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'7312","reported_seq":7397,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:26.684251+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:39:26.684251+0000","last_peered":"2024-09-05T16:39:26.684251+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:39:26.684251+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7312,"log_dups_size":0,"ondisk_log_size":7312,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42939552,"num_objects":5242,"num_object_clones":0,"num_object_copies":20968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2261,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5242,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7310,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2983}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'7314","reported_seq":7401,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:21.936963+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:39:21.936963+0000","last_peered":"2024-09-05T16:39:21.936963+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:39:21.936963+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7314,"log_dups_size":0,"ondisk_log_size":7314,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44546664,"num_objects":5439,"num_object_clones":0,"num_object_copies":21756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2248,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5439,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7313,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3192}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'7470","reported_seq":12193,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-05T16:39:25.851554+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:39:25.851554+0000","last_peered":"2024-09-05T16:39:25.851554+0000","last_clean":"2024-09-05T16:39:25.851554+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:39:25.851554+0000","last_undegraded":"2024-09-05T16:39:25.851554+0000","last_fullsized":"2024-09-05T16:39:25.851554+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":7470,"log_dups_size":0,"ondisk_log_size":7470,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44761088,"num_objects":5464,"num_object_clones":0,"num_object_copies":21856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5464,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1770,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'7265","reported_seq":7342,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:26.450514+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:39:26.450514+0000","last_peered":"2024-09-05T16:39:26.450514+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:39:26.450514+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7265,"log_dups_size":0,"ondisk_log_size":7265,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43660448,"num_objects":5330,"num_object_clones":0,"num_object_copies":21320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2155,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5330,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7263,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3177}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'7351","reported_seq":7422,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:26.468186+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:39:26.468186+0000","last_peered":"2024-09-05T16:39:26.468186+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:39:26.468186+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7351,"log_dups_size":0,"ondisk_log_size":7351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44408832,"num_objects":5421,"num_object_clones":0,"num_object_copies":21684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2238,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5421,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7351,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3183}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'7223","reported_seq":7307,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:26.753460+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:39:26.753460+0000","last_peered":"2024-09-05T16:39:26.753460+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:39:26.753460+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7223,"log_dups_size":0,"ondisk_log_size":7223,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44553376,"num_objects":5439,"num_object_clones":0,"num_object_copies":21756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2142,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5439,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7221,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3299}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'7389","reported_seq":7466,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:26.636289+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:39:26.636289+0000","last_peered":"2024-09-05T16:39:26.636289+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:39:26.636289+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7389,"log_dups_size":0,"ondisk_log_size":7389,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44151968,"num_objects":5390,"num_object_clones":0,"num_object_copies":21560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2237,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5390,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7387,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3155}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'7356","reported_seq":7425,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:26.389307+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:39:26.389307+0000","last_peered":"2024-09-05T16:39:26.389307+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:39:26.389307+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7356,"log_dups_size":0,"ondisk_log_size":7356,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44378432,"num_objects":5418,"num_object_clones":0,"num_object_copies":21672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5418,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7353,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3161}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'7341","reported_seq":7437,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:21.929814+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:39:21.929814+0000","last_peered":"2024-09-05T16:39:21.929814+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:39:21.929814+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7341,"log_dups_size":0,"ondisk_log_size":7341,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44799136,"num_objects":5469,"num_object_clones":0,"num_object_copies":21876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5469,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7339,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3214}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'7587","reported_seq":7664,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:26.444859+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:39:26.444859+0000","last_peered":"2024-09-05T16:39:26.444859+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:39:26.444859+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7587,"log_dups_size":0,"ondisk_log_size":7587,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45346544,"num_objects":5536,"num_object_clones":0,"num_object_copies":22144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2306,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5536,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7584,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3233}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'7316","reported_seq":7379,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:26.522006+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:39:26.522006+0000","last_peered":"2024-09-05T16:39:26.522006+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:39:26.522006+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7316,"log_dups_size":0,"ondisk_log_size":7316,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44587600,"num_objects":5443,"num_object_clones":0,"num_object_copies":21772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2254,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5443,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7315,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3190}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'7338","reported_seq":9962,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:21.997625+0000","last_change":"2024-09-05T16:37:30.634570+0000","last_active":"2024-09-05T16:39:21.997625+0000","last_peered":"2024-09-05T16:39:21.997625+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:39:21.997625+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7338,"log_dups_size":0,"ondisk_log_size":7338,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44816976,"num_objects":5471,"num_object_clones":0,"num_object_copies":21884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1022,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5471,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7337,"num_write_kb":51231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1273,"num_bytes_recovered":10420224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4449}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'7607","reported_seq":7676,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:26.336449+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:39:26.336449+0000","last_peered":"2024-09-05T16:39:26.336449+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:39:26.336449+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7607,"log_dups_size":0,"ondisk_log_size":7607,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45208736,"num_objects":5519,"num_object_clones":0,"num_object_copies":22076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2353,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5519,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7605,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3168}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'7356","reported_seq":7449,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:21.545674+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:39:21.545674+0000","last_peered":"2024-09-05T16:39:21.545674+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:39:21.545674+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7356,"log_dups_size":0,"ondisk_log_size":7356,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44864672,"num_objects":5477,"num_object_clones":0,"num_object_copies":21908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5477,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7354,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3162}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'7388","reported_seq":7471,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:26.643826+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:39:26.643826+0000","last_peered":"2024-09-05T16:39:26.643826+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:39:26.643826+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7388,"log_dups_size":0,"ondisk_log_size":7388,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44127392,"num_objects":5387,"num_object_clones":0,"num_object_copies":21548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5387,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7386,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3157}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":711732280,"num_objects":86887,"num_object_clones":0,"num_object_copies":347548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86887,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":112350,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1398,"num_bytes_recovered":10420224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1346347008,"data_stored":1346347008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":118075,"ondisk_log_size":118075,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084354,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":758216,"kb_used_data":347512,"kb_used_omap":4,"kb_used_meta":410683,"kb_avail":93613624,"statfs":{"total":96636764160,"available":95860350976,"internally_reserved":0,"allocated":355852288,"data_stored":355660247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":420540115},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,5,0,1,0,0,0,0,0,0,5],"upper_bound":16384},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986231,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":564640,"kb_used_data":271952,"kb_used_omap":4,"kb_used_meta":292667,"kb_avail":93807200,"statfs":{"total":96636764160,"available":96058572800,"internally_reserved":0,"allocated":278478848,"data_stored":278286807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":299691730},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673036,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":743432,"kb_used_data":351288,"kb_used_omap":1,"kb_used_meta":392126,"kb_avail":93628408,"statfs":{"total":96636764160,"available":95875489792,"internally_reserved":0,"allocated":359718912,"data_stored":359526871,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":401537477},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,3,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182468,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":766280,"kb_used_data":347300,"kb_used_omap":4,"kb_used_meta":418939,"kb_avail":93605560,"statfs":{"total":96636764160,"available":95852093440,"internally_reserved":0,"allocated":355635200,"data_stored":355443159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":428994253},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,3,0,2,0,0,0,0,0,0,0,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":354455552,"data_stored":354455552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":278122496,"data_stored":278122496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":359124992,"data_stored":359124992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":354643968,"data_stored":354643968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:39:27.200 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-05T16:39:27.200 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'7462", 'reported_seq': 7529, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:26.466111+0000', 'last_change': '2024-09-05T16:34:56.833497+0000', 'last_active': '2024-09-05T16:39:26.466111+0000', 'last_peered': '2024-09-05T16:39:26.466111+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.828289+0000', 'last_became_peered': '2024-09-05T16:34:56.828289+0000', 'last_unstale': '2024-09-05T16:39:26.466111+0000', 'last_undegraded': '2024-09-05T16:34:56.612160+0000', 'last_fullsized': '2024-09-05T16:34:56.557632+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7462, 'log_dups_size': 0, 'ondisk_log_size': 7462, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44580864, 'num_objects': 5442, 'num_object_clones': 0, 'num_object_copies': 21768, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2252, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5442, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7462, 'num_write_kb': 51616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3190}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:39:27.200 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-05T16:39:27.200 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "35'7312", 'reported_seq': 7397, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:26.684251+0000', 'last_change': '2024-09-05T16:34:56.825031+0000', 'last_active': '2024-09-05T16:39:26.684251+0000', 'last_peered': '2024-09-05T16:39:26.684251+0000', 'last_clean': '2024-09-05T16:33:53.200762+0000', 'last_became_active': '2024-09-05T16:34:56.820292+0000', 'last_became_peered': '2024-09-05T16:34:56.820292+0000', 'last_unstale': '2024-09-05T16:39:26.684251+0000', 'last_undegraded': '2024-09-05T16:34:56.605776+0000', 'last_fullsized': '2024-09-05T16:34:56.559383+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7312, 'log_dups_size': 0, 'ondisk_log_size': 7312, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42939552, 'num_objects': 5242, 'num_object_clones': 0, 'num_object_copies': 20968, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2261, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5242, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7310, 'num_write_kb': 50212, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2983}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:39:27.200 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:39:27.200 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'7314", 'reported_seq': 7401, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:21.936963+0000', 'last_change': '2024-09-05T16:34:56.867900+0000', 'last_active': '2024-09-05T16:39:21.936963+0000', 'last_peered': '2024-09-05T16:39:21.936963+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:39:21.936963+0000', 'last_undegraded': '2024-09-05T16:34:56.590972+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7314, 'log_dups_size': 0, 'ondisk_log_size': 7314, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44546664, 'num_objects': 5439, 'num_object_clones': 0, 'num_object_copies': 21756, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2248, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5439, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 7313, 'num_write_kb': 50999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3192}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:39:27.201 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:39:27.201 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'7265", 'reported_seq': 7342, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:26.450514+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:39:26.450514+0000', 'last_peered': '2024-09-05T16:39:26.450514+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:39:26.450514+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7265, 'log_dups_size': 0, 'ondisk_log_size': 7265, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43660448, 'num_objects': 5330, 'num_object_clones': 0, 'num_object_copies': 21320, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2155, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5330, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7263, 'num_write_kb': 50373, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3177}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:39:27.201 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:39:27.201 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'7351", 'reported_seq': 7422, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:26.468186+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:39:26.468186+0000', 'last_peered': '2024-09-05T16:39:26.468186+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:39:26.468186+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7351, 'log_dups_size': 0, 'ondisk_log_size': 7351, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44408832, 'num_objects': 5421, 'num_object_clones': 0, 'num_object_copies': 21684, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2238, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5421, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7351, 'num_write_kb': 51088, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3183}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:39:27.201 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:39:27.201 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'7223", 'reported_seq': 7307, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:26.753460+0000', 'last_change': '2024-09-05T16:34:56.903436+0000', 'last_active': '2024-09-05T16:39:26.753460+0000', 'last_peered': '2024-09-05T16:39:26.753460+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:39:26.753460+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7223, 'log_dups_size': 0, 'ondisk_log_size': 7223, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44553376, 'num_objects': 5439, 'num_object_clones': 0, 'num_object_copies': 21756, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2142, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5439, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7221, 'num_write_kb': 50636, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3299}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:39:27.201 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-05T16:39:27.202 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "35'7389", 'reported_seq': 7466, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:26.636289+0000', 'last_change': '2024-09-05T16:34:56.883319+0000', 'last_active': '2024-09-05T16:39:26.636289+0000', 'last_peered': '2024-09-05T16:39:26.636289+0000', 'last_clean': '2024-09-05T16:33:53.505305+0000', 'last_became_active': '2024-09-05T16:34:56.873389+0000', 'last_became_peered': '2024-09-05T16:34:56.873389+0000', 'last_unstale': '2024-09-05T16:39:26.636289+0000', 'last_undegraded': '2024-09-05T16:34:56.602567+0000', 'last_fullsized': '2024-09-05T16:34:56.559958+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7389, 'log_dups_size': 0, 'ondisk_log_size': 7389, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44151968, 'num_objects': 5390, 'num_object_clones': 0, 'num_object_copies': 21560, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2237, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5390, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7387, 'num_write_kb': 51108, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 6, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3155}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:39:27.202 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:39:27.202 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'7356", 'reported_seq': 7425, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:26.389307+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:39:26.389307+0000', 'last_peered': '2024-09-05T16:39:26.389307+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:39:26.389307+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7356, 'log_dups_size': 0, 'ondisk_log_size': 7356, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44378432, 'num_objects': 5418, 'num_object_clones': 0, 'num_object_copies': 21672, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2260, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5418, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7353, 'num_write_kb': 51081, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3161}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:39:27.202 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:39:27.202 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'7341", 'reported_seq': 7437, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:21.929814+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:39:21.929814+0000', 'last_peered': '2024-09-05T16:39:21.929814+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:39:21.929814+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7341, 'log_dups_size': 0, 'ondisk_log_size': 7341, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44799136, 'num_objects': 5469, 'num_object_clones': 0, 'num_object_copies': 21876, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5469, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7339, 'num_write_kb': 51230, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3214}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:39:27.202 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:39:27.202 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'7587", 'reported_seq': 7664, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:26.444859+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:39:26.444859+0000', 'last_peered': '2024-09-05T16:39:26.444859+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:39:26.444859+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7587, 'log_dups_size': 0, 'ondisk_log_size': 7587, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45346544, 'num_objects': 5536, 'num_object_clones': 0, 'num_object_copies': 22144, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2306, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5536, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7584, 'num_write_kb': 52476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3233}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:39:27.202 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:39:27.203 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'7316", 'reported_seq': 7379, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:26.522006+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:39:26.522006+0000', 'last_peered': '2024-09-05T16:39:26.522006+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:39:26.522006+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7316, 'log_dups_size': 0, 'ondisk_log_size': 7316, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44587600, 'num_objects': 5443, 'num_object_clones': 0, 'num_object_copies': 21772, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2254, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5443, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7315, 'num_write_kb': 51031, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3190}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:39:27.203 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-05T16:39:27.203 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'7338", 'reported_seq': 9962, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:21.997625+0000', 'last_change': '2024-09-05T16:37:30.634570+0000', 'last_active': '2024-09-05T16:39:21.997625+0000', 'last_peered': '2024-09-05T16:39:21.997625+0000', 'last_clean': '2024-09-05T16:33:45.082039+0000', 'last_became_active': '2024-09-05T16:34:56.792386+0000', 'last_became_peered': '2024-09-05T16:34:56.792386+0000', 'last_unstale': '2024-09-05T16:39:21.997625+0000', 'last_undegraded': '2024-09-05T16:34:56.600595+0000', 'last_fullsized': '2024-09-05T16:34:56.558203+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7338, 'log_dups_size': 0, 'ondisk_log_size': 7338, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44816976, 'num_objects': 5471, 'num_object_clones': 0, 'num_object_copies': 21884, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1022, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5471, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7337, 'num_write_kb': 51231, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1273, 'num_bytes_recovered': 10420224, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 4449}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:39:27.203 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:39:27.203 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'7607", 'reported_seq': 7676, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:26.336449+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:39:26.336449+0000', 'last_peered': '2024-09-05T16:39:26.336449+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:39:26.336449+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7607, 'log_dups_size': 0, 'ondisk_log_size': 7607, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45208736, 'num_objects': 5519, 'num_object_clones': 0, 'num_object_copies': 22076, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2353, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5519, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7605, 'num_write_kb': 52500, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 8, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3168}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:39:27.203 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:39:27.203 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'7356", 'reported_seq': 7449, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:21.545674+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:39:21.545674+0000', 'last_peered': '2024-09-05T16:39:21.545674+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:39:21.545674+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7356, 'log_dups_size': 0, 'ondisk_log_size': 7356, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44864672, 'num_objects': 5477, 'num_object_clones': 0, 'num_object_copies': 21908, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2317, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5477, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7354, 'num_write_kb': 51325, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3162}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:39:27.204 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:39:27.204 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'7388", 'reported_seq': 7471, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:26.643826+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:39:26.643826+0000', 'last_peered': '2024-09-05T16:39:26.643826+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:39:26.643826+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7388, 'log_dups_size': 0, 'ondisk_log_size': 7388, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44127392, 'num_objects': 5387, 'num_object_clones': 0, 'num_object_copies': 21548, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2232, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5387, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7386, 'num_write_kb': 51092, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3157}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:39:27.204 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:39:27.920 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:39:27.920 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 32534/347548 objects degraded (9.361%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":371,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+clean","count":1}],"num_pgs":16,"num_pools":1,"num_objects":86887,"data_bytes":711732280,"bytes_used":2900549632,"bytes_avail":383646507008,"bytes_total":386547056640,"degraded_objects":32534,"degraded_total":347548,"degraded_ratio":0.093610091267968751,"recovering_objects_per_sec":13,"recovering_bytes_per_sec":81204,"recovering_keys_per_sec":0,"num_objects_recovered":161,"num_bytes_recovered":974848,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":272},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:39:27.951 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:39:27.951 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:39:28.563 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:39:28.564 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:39:28.582 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":211,"stamp":"2024-09-05T16:39:26.909705+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":711732280,"num_objects":86887,"num_object_clones":0,"num_object_copies":347548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86887,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":112350,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1398,"num_bytes_recovered":10420224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":118075,"ondisk_log_size":118075,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2832568,"kb_used_data":1318052,"kb_used_omap":14,"kb_used_meta":1514417,"kb_avail":374654792,"statfs":{"total":386547056640,"available":383646507008,"internally_reserved":0,"allocated":1349685248,"data_stored":1348917084,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1550763575},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,5,13,7,5,0,0,0,0,0,0,5,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-26836992,"num_objects":-3276,"num_object_clones":0,"num_object_copies":-13104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-161,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3276,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3276,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":161,"num_bytes_recovered":974848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004829"},"pg_stats":[{"pgid":"1.f","version":"35'7462","reported_seq":7529,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:26.466111+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:39:26.466111+0000","last_peered":"2024-09-05T16:39:26.466111+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:39:26.466111+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7462,"log_dups_size":0,"ondisk_log_size":7462,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44580864,"num_objects":5442,"num_object_clones":0,"num_object_copies":21768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5442,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7462,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3190}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'7312","reported_seq":7397,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:26.684251+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:39:26.684251+0000","last_peered":"2024-09-05T16:39:26.684251+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:39:26.684251+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7312,"log_dups_size":0,"ondisk_log_size":7312,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42939552,"num_objects":5242,"num_object_clones":0,"num_object_copies":20968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2261,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5242,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7310,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2983}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'7314","reported_seq":7401,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:21.936963+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:39:21.936963+0000","last_peered":"2024-09-05T16:39:21.936963+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:39:21.936963+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7314,"log_dups_size":0,"ondisk_log_size":7314,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44546664,"num_objects":5439,"num_object_clones":0,"num_object_copies":21756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2248,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5439,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7313,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3192}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'7470","reported_seq":12193,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-05T16:39:25.851554+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:39:25.851554+0000","last_peered":"2024-09-05T16:39:25.851554+0000","last_clean":"2024-09-05T16:39:25.851554+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:39:25.851554+0000","last_undegraded":"2024-09-05T16:39:25.851554+0000","last_fullsized":"2024-09-05T16:39:25.851554+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":7470,"log_dups_size":0,"ondisk_log_size":7470,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44761088,"num_objects":5464,"num_object_clones":0,"num_object_copies":21856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5464,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1770,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'7265","reported_seq":7342,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:26.450514+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:39:26.450514+0000","last_peered":"2024-09-05T16:39:26.450514+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:39:26.450514+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7265,"log_dups_size":0,"ondisk_log_size":7265,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43660448,"num_objects":5330,"num_object_clones":0,"num_object_copies":21320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2155,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5330,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7263,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3177}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'7351","reported_seq":7422,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:26.468186+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:39:26.468186+0000","last_peered":"2024-09-05T16:39:26.468186+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:39:26.468186+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7351,"log_dups_size":0,"ondisk_log_size":7351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44408832,"num_objects":5421,"num_object_clones":0,"num_object_copies":21684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2238,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5421,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7351,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3183}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'7223","reported_seq":7307,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:26.753460+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:39:26.753460+0000","last_peered":"2024-09-05T16:39:26.753460+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:39:26.753460+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7223,"log_dups_size":0,"ondisk_log_size":7223,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44553376,"num_objects":5439,"num_object_clones":0,"num_object_copies":21756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2142,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5439,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7221,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3299}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'7389","reported_seq":7466,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:26.636289+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:39:26.636289+0000","last_peered":"2024-09-05T16:39:26.636289+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:39:26.636289+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7389,"log_dups_size":0,"ondisk_log_size":7389,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44151968,"num_objects":5390,"num_object_clones":0,"num_object_copies":21560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2237,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5390,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7387,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3155}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'7356","reported_seq":7425,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:26.389307+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:39:26.389307+0000","last_peered":"2024-09-05T16:39:26.389307+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:39:26.389307+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7356,"log_dups_size":0,"ondisk_log_size":7356,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44378432,"num_objects":5418,"num_object_clones":0,"num_object_copies":21672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5418,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7353,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3161}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'7341","reported_seq":7437,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:21.929814+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:39:21.929814+0000","last_peered":"2024-09-05T16:39:21.929814+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:39:21.929814+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7341,"log_dups_size":0,"ondisk_log_size":7341,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44799136,"num_objects":5469,"num_object_clones":0,"num_object_copies":21876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5469,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7339,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3214}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'7587","reported_seq":7664,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:26.444859+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:39:26.444859+0000","last_peered":"2024-09-05T16:39:26.444859+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:39:26.444859+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7587,"log_dups_size":0,"ondisk_log_size":7587,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45346544,"num_objects":5536,"num_object_clones":0,"num_object_copies":22144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2306,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5536,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7584,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3233}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'7316","reported_seq":7379,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:26.522006+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:39:26.522006+0000","last_peered":"2024-09-05T16:39:26.522006+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:39:26.522006+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7316,"log_dups_size":0,"ondisk_log_size":7316,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44587600,"num_objects":5443,"num_object_clones":0,"num_object_copies":21772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2254,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5443,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7315,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3190}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'7338","reported_seq":9962,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:21.997625+0000","last_change":"2024-09-05T16:37:30.634570+0000","last_active":"2024-09-05T16:39:21.997625+0000","last_peered":"2024-09-05T16:39:21.997625+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:39:21.997625+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7338,"log_dups_size":0,"ondisk_log_size":7338,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44816976,"num_objects":5471,"num_object_clones":0,"num_object_copies":21884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1022,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5471,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7337,"num_write_kb":51231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1273,"num_bytes_recovered":10420224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4449}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'7607","reported_seq":7676,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:26.336449+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:39:26.336449+0000","last_peered":"2024-09-05T16:39:26.336449+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:39:26.336449+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7607,"log_dups_size":0,"ondisk_log_size":7607,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45208736,"num_objects":5519,"num_object_clones":0,"num_object_copies":22076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2353,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5519,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7605,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3168}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'7356","reported_seq":7449,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:21.545674+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:39:21.545674+0000","last_peered":"2024-09-05T16:39:21.545674+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:39:21.545674+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7356,"log_dups_size":0,"ondisk_log_size":7356,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44864672,"num_objects":5477,"num_object_clones":0,"num_object_copies":21908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5477,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7354,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3162}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'7388","reported_seq":7471,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:26.643826+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:39:26.643826+0000","last_peered":"2024-09-05T16:39:26.643826+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:39:26.643826+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7388,"log_dups_size":0,"ondisk_log_size":7388,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44127392,"num_objects":5387,"num_object_clones":0,"num_object_copies":21548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5387,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7386,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3157}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":711732280,"num_objects":86887,"num_object_clones":0,"num_object_copies":347548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86887,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":112350,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1398,"num_bytes_recovered":10420224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1346347008,"data_stored":1346347008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":118075,"ondisk_log_size":118075,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084354,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":758216,"kb_used_data":347512,"kb_used_omap":4,"kb_used_meta":410683,"kb_avail":93613624,"statfs":{"total":96636764160,"available":95860350976,"internally_reserved":0,"allocated":355852288,"data_stored":355660247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":420540115},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,5,0,1,0,0,0,0,0,0,5],"upper_bound":16384},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986231,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":564640,"kb_used_data":271952,"kb_used_omap":4,"kb_used_meta":292667,"kb_avail":93807200,"statfs":{"total":96636764160,"available":96058572800,"internally_reserved":0,"allocated":278478848,"data_stored":278286807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":299691730},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673036,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":743432,"kb_used_data":351288,"kb_used_omap":1,"kb_used_meta":392126,"kb_avail":93628408,"statfs":{"total":96636764160,"available":95875489792,"internally_reserved":0,"allocated":359718912,"data_stored":359526871,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":401537477},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,3,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182468,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":766280,"kb_used_data":347300,"kb_used_omap":4,"kb_used_meta":418939,"kb_avail":93605560,"statfs":{"total":96636764160,"available":95852093440,"internally_reserved":0,"allocated":355635200,"data_stored":355443159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":428994253},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,3,0,2,0,0,0,0,0,0,0,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":354455552,"data_stored":354455552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":278122496,"data_stored":278122496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":359124992,"data_stored":359124992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":354643968,"data_stored":354643968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:39:31.584 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:39:32.196 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:39:32.197 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:39:32.219 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":213,"stamp":"2024-09-05T16:39:30.911790+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":709127224,"num_objects":86569,"num_object_clones":0,"num_object_copies":346276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32467,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86569,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":112668,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1465,"num_bytes_recovered":10969088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":118393,"ondisk_log_size":118393,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2850008,"kb_used_data":1313284,"kb_used_omap":14,"kb_used_meta":1536625,"kb_avail":374637352,"statfs":{"total":386547056640,"available":383628648448,"internally_reserved":0,"allocated":1344802816,"data_stored":1344034652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1573504567},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,2,3,15,5,4,4,0,0,0,0,2,5,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19734528,"num_objects":-2409,"num_object_clones":0,"num_object_copies":-9636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-142,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2409,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2409,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":142,"num_bytes_recovered":1032192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004692"},"pg_stats":[{"pgid":"1.f","version":"35'7462","reported_seq":7529,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:26.466111+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:39:26.466111+0000","last_peered":"2024-09-05T16:39:26.466111+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:39:26.466111+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7462,"log_dups_size":0,"ondisk_log_size":7462,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44580864,"num_objects":5442,"num_object_clones":0,"num_object_copies":21768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5442,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7462,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3190}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'7312","reported_seq":7397,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:26.684251+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:39:26.684251+0000","last_peered":"2024-09-05T16:39:26.684251+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:39:26.684251+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7312,"log_dups_size":0,"ondisk_log_size":7312,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42939552,"num_objects":5242,"num_object_clones":0,"num_object_copies":20968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2261,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5242,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7310,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2983}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'7397","reported_seq":7484,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:26.955085+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:39:26.955085+0000","last_peered":"2024-09-05T16:39:26.955085+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:39:26.955085+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7397,"log_dups_size":0,"ondisk_log_size":7397,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43866728,"num_objects":5356,"num_object_clones":0,"num_object_copies":21424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2248,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5356,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7396,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3109}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'7470","reported_seq":12193,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-05T16:39:25.851554+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:39:25.851554+0000","last_peered":"2024-09-05T16:39:25.851554+0000","last_clean":"2024-09-05T16:39:25.851554+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:39:25.851554+0000","last_undegraded":"2024-09-05T16:39:25.851554+0000","last_fullsized":"2024-09-05T16:39:25.851554+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":7470,"log_dups_size":0,"ondisk_log_size":7470,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44761088,"num_objects":5464,"num_object_clones":0,"num_object_copies":21856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5464,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1770,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'7265","reported_seq":7342,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:26.450514+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:39:26.450514+0000","last_peered":"2024-09-05T16:39:26.450514+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:39:26.450514+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7265,"log_dups_size":0,"ondisk_log_size":7265,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43660448,"num_objects":5330,"num_object_clones":0,"num_object_copies":21320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2155,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5330,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7263,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3177}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'7351","reported_seq":7422,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:26.468186+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:39:26.468186+0000","last_peered":"2024-09-05T16:39:26.468186+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:39:26.468186+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7351,"log_dups_size":0,"ondisk_log_size":7351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44408832,"num_objects":5421,"num_object_clones":0,"num_object_copies":21684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2238,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5421,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7351,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3183}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'7223","reported_seq":7307,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:26.753460+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:39:26.753460+0000","last_peered":"2024-09-05T16:39:26.753460+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:39:26.753460+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7223,"log_dups_size":0,"ondisk_log_size":7223,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44553376,"num_objects":5439,"num_object_clones":0,"num_object_copies":21756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2142,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5439,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7221,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3299}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'7389","reported_seq":7466,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:26.636289+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:39:26.636289+0000","last_peered":"2024-09-05T16:39:26.636289+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:39:26.636289+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7389,"log_dups_size":0,"ondisk_log_size":7389,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44151968,"num_objects":5390,"num_object_clones":0,"num_object_copies":21560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2237,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5390,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7387,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3155}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'7356","reported_seq":7425,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:26.389307+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:39:26.389307+0000","last_peered":"2024-09-05T16:39:26.389307+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:39:26.389307+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7356,"log_dups_size":0,"ondisk_log_size":7356,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44378432,"num_objects":5418,"num_object_clones":0,"num_object_copies":21672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5418,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7353,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3161}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'7425","reported_seq":7521,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:27.031607+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:39:27.031607+0000","last_peered":"2024-09-05T16:39:27.031607+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:39:27.031607+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7425,"log_dups_size":0,"ondisk_log_size":7425,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44111008,"num_objects":5385,"num_object_clones":0,"num_object_copies":21540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5385,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7423,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3130}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'7587","reported_seq":7664,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:26.444859+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:39:26.444859+0000","last_peered":"2024-09-05T16:39:26.444859+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:39:26.444859+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7587,"log_dups_size":0,"ondisk_log_size":7587,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45346544,"num_objects":5536,"num_object_clones":0,"num_object_copies":22144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2306,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5536,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7584,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3233}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'7316","reported_seq":7379,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:26.522006+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:39:26.522006+0000","last_peered":"2024-09-05T16:39:26.522006+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:39:26.522006+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7316,"log_dups_size":0,"ondisk_log_size":7316,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44587600,"num_objects":5443,"num_object_clones":0,"num_object_copies":21772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2254,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5443,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7315,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3190}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'7417","reported_seq":10172,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:26.982667+0000","last_change":"2024-09-05T16:37:30.634570+0000","last_active":"2024-09-05T16:39:26.982667+0000","last_peered":"2024-09-05T16:39:26.982667+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:39:26.982667+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7417,"log_dups_size":0,"ondisk_log_size":7417,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44169808,"num_objects":5392,"num_object_clones":0,"num_object_copies":21568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5392,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7416,"num_write_kb":51231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1340,"num_bytes_recovered":10969088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4437}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'7607","reported_seq":7676,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:26.336449+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:39:26.336449+0000","last_peered":"2024-09-05T16:39:26.336449+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:39:26.336449+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7607,"log_dups_size":0,"ondisk_log_size":7607,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45208736,"num_objects":5519,"num_object_clones":0,"num_object_copies":22076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2353,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5519,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7605,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3168}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'7428","reported_seq":7521,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:26.976973+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:39:26.976973+0000","last_peered":"2024-09-05T16:39:26.976973+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:39:26.976973+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7428,"log_dups_size":0,"ondisk_log_size":7428,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44274848,"num_objects":5405,"num_object_clones":0,"num_object_copies":21620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5405,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7426,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3090}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'7388","reported_seq":7471,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:26.643826+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:39:26.643826+0000","last_peered":"2024-09-05T16:39:26.643826+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:39:26.643826+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7388,"log_dups_size":0,"ondisk_log_size":7388,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44127392,"num_objects":5387,"num_object_clones":0,"num_object_copies":21548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5387,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7386,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3157}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":709127224,"num_objects":86569,"num_object_clones":0,"num_object_copies":346276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32467,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86569,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":112668,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1465,"num_bytes_recovered":10969088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1341468672,"data_stored":1341468672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":118393,"ondisk_log_size":118393,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084354,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":758216,"kb_used_data":347512,"kb_used_omap":4,"kb_used_meta":410683,"kb_avail":93613624,"statfs":{"total":96636764160,"available":95860350976,"internally_reserved":0,"allocated":355852288,"data_stored":355660247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":420540115},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,5,0,1,0,0,0,0,0,0,5],"upper_bound":16384},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986231,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":564640,"kb_used_data":271952,"kb_used_omap":4,"kb_used_meta":292667,"kb_avail":93807200,"statfs":{"total":96636764160,"available":96058572800,"internally_reserved":0,"allocated":278478848,"data_stored":278286807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":299691730},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673037,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":760872,"kb_used_data":346520,"kb_used_omap":1,"kb_used_meta":414334,"kb_avail":93610968,"statfs":{"total":96636764160,"available":95857631232,"internally_reserved":0,"allocated":354836480,"data_stored":354644439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":424278469},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,0,0,5,0,1,4,0,0,0,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182468,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":766280,"kb_used_data":347300,"kb_used_omap":4,"kb_used_meta":418939,"kb_avail":93605560,"statfs":{"total":96636764160,"available":95852093440,"internally_reserved":0,"allocated":355635200,"data_stored":355443159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":428994253},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,3,0,2,0,0,0,0,0,0,0,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":354455552,"data_stored":354455552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":278122496,"data_stored":278122496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":354246656,"data_stored":354246656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":354643968,"data_stored":354643968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:39:32.220 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:39:32.807 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:39:32.808 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:39:32.827 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":213,"stamp":"2024-09-05T16:39:30.911790+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":709127224,"num_objects":86569,"num_object_clones":0,"num_object_copies":346276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32467,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86569,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":112668,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1465,"num_bytes_recovered":10969088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":118393,"ondisk_log_size":118393,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2850008,"kb_used_data":1313284,"kb_used_omap":14,"kb_used_meta":1536625,"kb_avail":374637352,"statfs":{"total":386547056640,"available":383628648448,"internally_reserved":0,"allocated":1344802816,"data_stored":1344034652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1573504567},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,2,3,15,5,4,4,0,0,0,0,2,5,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19734528,"num_objects":-2409,"num_object_clones":0,"num_object_copies":-9636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-142,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2409,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2409,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":142,"num_bytes_recovered":1032192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004692"},"pg_stats":[{"pgid":"1.f","version":"35'7462","reported_seq":7529,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:26.466111+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:39:26.466111+0000","last_peered":"2024-09-05T16:39:26.466111+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:39:26.466111+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7462,"log_dups_size":0,"ondisk_log_size":7462,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44580864,"num_objects":5442,"num_object_clones":0,"num_object_copies":21768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5442,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7462,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3190}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'7312","reported_seq":7397,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:26.684251+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:39:26.684251+0000","last_peered":"2024-09-05T16:39:26.684251+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:39:26.684251+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7312,"log_dups_size":0,"ondisk_log_size":7312,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42939552,"num_objects":5242,"num_object_clones":0,"num_object_copies":20968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2261,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5242,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7310,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2983}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'7397","reported_seq":7484,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:26.955085+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:39:26.955085+0000","last_peered":"2024-09-05T16:39:26.955085+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:39:26.955085+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7397,"log_dups_size":0,"ondisk_log_size":7397,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43866728,"num_objects":5356,"num_object_clones":0,"num_object_copies":21424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2248,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5356,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7396,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3109}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'7470","reported_seq":12193,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-05T16:39:25.851554+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:39:25.851554+0000","last_peered":"2024-09-05T16:39:25.851554+0000","last_clean":"2024-09-05T16:39:25.851554+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:39:25.851554+0000","last_undegraded":"2024-09-05T16:39:25.851554+0000","last_fullsized":"2024-09-05T16:39:25.851554+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":7470,"log_dups_size":0,"ondisk_log_size":7470,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44761088,"num_objects":5464,"num_object_clones":0,"num_object_copies":21856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5464,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1770,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'7265","reported_seq":7342,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:26.450514+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:39:26.450514+0000","last_peered":"2024-09-05T16:39:26.450514+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:39:26.450514+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7265,"log_dups_size":0,"ondisk_log_size":7265,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43660448,"num_objects":5330,"num_object_clones":0,"num_object_copies":21320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2155,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5330,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7263,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3177}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'7351","reported_seq":7422,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:26.468186+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:39:26.468186+0000","last_peered":"2024-09-05T16:39:26.468186+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:39:26.468186+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7351,"log_dups_size":0,"ondisk_log_size":7351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44408832,"num_objects":5421,"num_object_clones":0,"num_object_copies":21684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2238,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5421,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7351,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3183}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'7223","reported_seq":7307,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:26.753460+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:39:26.753460+0000","last_peered":"2024-09-05T16:39:26.753460+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:39:26.753460+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7223,"log_dups_size":0,"ondisk_log_size":7223,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44553376,"num_objects":5439,"num_object_clones":0,"num_object_copies":21756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2142,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5439,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7221,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3299}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'7389","reported_seq":7466,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:26.636289+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:39:26.636289+0000","last_peered":"2024-09-05T16:39:26.636289+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:39:26.636289+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7389,"log_dups_size":0,"ondisk_log_size":7389,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44151968,"num_objects":5390,"num_object_clones":0,"num_object_copies":21560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2237,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5390,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7387,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3155}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'7356","reported_seq":7425,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:26.389307+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:39:26.389307+0000","last_peered":"2024-09-05T16:39:26.389307+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:39:26.389307+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7356,"log_dups_size":0,"ondisk_log_size":7356,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44378432,"num_objects":5418,"num_object_clones":0,"num_object_copies":21672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5418,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7353,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3161}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'7425","reported_seq":7521,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:27.031607+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:39:27.031607+0000","last_peered":"2024-09-05T16:39:27.031607+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:39:27.031607+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7425,"log_dups_size":0,"ondisk_log_size":7425,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44111008,"num_objects":5385,"num_object_clones":0,"num_object_copies":21540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5385,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7423,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3130}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'7587","reported_seq":7664,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:26.444859+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:39:26.444859+0000","last_peered":"2024-09-05T16:39:26.444859+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:39:26.444859+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7587,"log_dups_size":0,"ondisk_log_size":7587,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45346544,"num_objects":5536,"num_object_clones":0,"num_object_copies":22144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2306,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5536,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7584,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3233}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'7316","reported_seq":7379,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:26.522006+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:39:26.522006+0000","last_peered":"2024-09-05T16:39:26.522006+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:39:26.522006+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7316,"log_dups_size":0,"ondisk_log_size":7316,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44587600,"num_objects":5443,"num_object_clones":0,"num_object_copies":21772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2254,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5443,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7315,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3190}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'7417","reported_seq":10172,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:26.982667+0000","last_change":"2024-09-05T16:37:30.634570+0000","last_active":"2024-09-05T16:39:26.982667+0000","last_peered":"2024-09-05T16:39:26.982667+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:39:26.982667+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7417,"log_dups_size":0,"ondisk_log_size":7417,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44169808,"num_objects":5392,"num_object_clones":0,"num_object_copies":21568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5392,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7416,"num_write_kb":51231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1340,"num_bytes_recovered":10969088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4437}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'7607","reported_seq":7676,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:26.336449+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:39:26.336449+0000","last_peered":"2024-09-05T16:39:26.336449+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:39:26.336449+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7607,"log_dups_size":0,"ondisk_log_size":7607,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45208736,"num_objects":5519,"num_object_clones":0,"num_object_copies":22076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2353,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5519,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7605,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3168}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'7428","reported_seq":7521,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:26.976973+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:39:26.976973+0000","last_peered":"2024-09-05T16:39:26.976973+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:39:26.976973+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7428,"log_dups_size":0,"ondisk_log_size":7428,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44274848,"num_objects":5405,"num_object_clones":0,"num_object_copies":21620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5405,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7426,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3090}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'7388","reported_seq":7471,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:26.643826+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:39:26.643826+0000","last_peered":"2024-09-05T16:39:26.643826+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:39:26.643826+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7388,"log_dups_size":0,"ondisk_log_size":7388,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44127392,"num_objects":5387,"num_object_clones":0,"num_object_copies":21548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5387,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7386,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3157}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":709127224,"num_objects":86569,"num_object_clones":0,"num_object_copies":346276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32467,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86569,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":112668,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1465,"num_bytes_recovered":10969088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1341468672,"data_stored":1341468672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":118393,"ondisk_log_size":118393,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084354,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":758216,"kb_used_data":347512,"kb_used_omap":4,"kb_used_meta":410683,"kb_avail":93613624,"statfs":{"total":96636764160,"available":95860350976,"internally_reserved":0,"allocated":355852288,"data_stored":355660247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":420540115},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,5,0,1,0,0,0,0,0,0,5],"upper_bound":16384},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986231,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":564640,"kb_used_data":271952,"kb_used_omap":4,"kb_used_meta":292667,"kb_avail":93807200,"statfs":{"total":96636764160,"available":96058572800,"internally_reserved":0,"allocated":278478848,"data_stored":278286807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":299691730},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673037,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":760872,"kb_used_data":346520,"kb_used_omap":1,"kb_used_meta":414334,"kb_avail":93610968,"statfs":{"total":96636764160,"available":95857631232,"internally_reserved":0,"allocated":354836480,"data_stored":354644439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":424278469},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,0,0,5,0,1,4,0,0,0,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182468,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":766280,"kb_used_data":347300,"kb_used_omap":4,"kb_used_meta":418939,"kb_avail":93605560,"statfs":{"total":96636764160,"available":95852093440,"internally_reserved":0,"allocated":355635200,"data_stored":355443159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":428994253},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,3,0,2,0,0,0,0,0,0,0,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":354455552,"data_stored":354455552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":278122496,"data_stored":278122496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":354246656,"data_stored":354246656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":354643968,"data_stored":354643968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:39:32.828 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-05T16:39:32.828 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'7462", 'reported_seq': 7529, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:26.466111+0000', 'last_change': '2024-09-05T16:34:56.833497+0000', 'last_active': '2024-09-05T16:39:26.466111+0000', 'last_peered': '2024-09-05T16:39:26.466111+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.828289+0000', 'last_became_peered': '2024-09-05T16:34:56.828289+0000', 'last_unstale': '2024-09-05T16:39:26.466111+0000', 'last_undegraded': '2024-09-05T16:34:56.612160+0000', 'last_fullsized': '2024-09-05T16:34:56.557632+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7462, 'log_dups_size': 0, 'ondisk_log_size': 7462, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44580864, 'num_objects': 5442, 'num_object_clones': 0, 'num_object_copies': 21768, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2252, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5442, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7462, 'num_write_kb': 51616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3190}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:39:32.829 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-05T16:39:32.829 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "35'7312", 'reported_seq': 7397, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:26.684251+0000', 'last_change': '2024-09-05T16:34:56.825031+0000', 'last_active': '2024-09-05T16:39:26.684251+0000', 'last_peered': '2024-09-05T16:39:26.684251+0000', 'last_clean': '2024-09-05T16:33:53.200762+0000', 'last_became_active': '2024-09-05T16:34:56.820292+0000', 'last_became_peered': '2024-09-05T16:34:56.820292+0000', 'last_unstale': '2024-09-05T16:39:26.684251+0000', 'last_undegraded': '2024-09-05T16:34:56.605776+0000', 'last_fullsized': '2024-09-05T16:34:56.559383+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7312, 'log_dups_size': 0, 'ondisk_log_size': 7312, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42939552, 'num_objects': 5242, 'num_object_clones': 0, 'num_object_copies': 20968, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2261, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5242, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7310, 'num_write_kb': 50212, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2983}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:39:32.829 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:39:32.829 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'7397", 'reported_seq': 7484, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:26.955085+0000', 'last_change': '2024-09-05T16:34:56.867900+0000', 'last_active': '2024-09-05T16:39:26.955085+0000', 'last_peered': '2024-09-05T16:39:26.955085+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:39:26.955085+0000', 'last_undegraded': '2024-09-05T16:34:56.590972+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7397, 'log_dups_size': 0, 'ondisk_log_size': 7397, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43866728, 'num_objects': 5356, 'num_object_clones': 0, 'num_object_copies': 21424, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2248, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5356, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 7396, 'num_write_kb': 50999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3109}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:39:32.829 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:39:32.829 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'7265", 'reported_seq': 7342, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:26.450514+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:39:26.450514+0000', 'last_peered': '2024-09-05T16:39:26.450514+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:39:26.450514+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7265, 'log_dups_size': 0, 'ondisk_log_size': 7265, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43660448, 'num_objects': 5330, 'num_object_clones': 0, 'num_object_copies': 21320, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2155, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5330, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7263, 'num_write_kb': 50373, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3177}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:39:32.829 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:39:32.829 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'7351", 'reported_seq': 7422, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:26.468186+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:39:26.468186+0000', 'last_peered': '2024-09-05T16:39:26.468186+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:39:26.468186+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7351, 'log_dups_size': 0, 'ondisk_log_size': 7351, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44408832, 'num_objects': 5421, 'num_object_clones': 0, 'num_object_copies': 21684, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2238, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5421, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7351, 'num_write_kb': 51088, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3183}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:39:32.829 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:39:32.830 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'7223", 'reported_seq': 7307, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:26.753460+0000', 'last_change': '2024-09-05T16:34:56.903436+0000', 'last_active': '2024-09-05T16:39:26.753460+0000', 'last_peered': '2024-09-05T16:39:26.753460+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:39:26.753460+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7223, 'log_dups_size': 0, 'ondisk_log_size': 7223, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44553376, 'num_objects': 5439, 'num_object_clones': 0, 'num_object_copies': 21756, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2142, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5439, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7221, 'num_write_kb': 50636, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3299}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:39:32.830 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-05T16:39:32.830 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "35'7389", 'reported_seq': 7466, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:26.636289+0000', 'last_change': '2024-09-05T16:34:56.883319+0000', 'last_active': '2024-09-05T16:39:26.636289+0000', 'last_peered': '2024-09-05T16:39:26.636289+0000', 'last_clean': '2024-09-05T16:33:53.505305+0000', 'last_became_active': '2024-09-05T16:34:56.873389+0000', 'last_became_peered': '2024-09-05T16:34:56.873389+0000', 'last_unstale': '2024-09-05T16:39:26.636289+0000', 'last_undegraded': '2024-09-05T16:34:56.602567+0000', 'last_fullsized': '2024-09-05T16:34:56.559958+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7389, 'log_dups_size': 0, 'ondisk_log_size': 7389, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44151968, 'num_objects': 5390, 'num_object_clones': 0, 'num_object_copies': 21560, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2237, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5390, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7387, 'num_write_kb': 51108, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 6, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3155}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:39:32.830 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:39:32.830 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'7356", 'reported_seq': 7425, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:26.389307+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:39:26.389307+0000', 'last_peered': '2024-09-05T16:39:26.389307+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:39:26.389307+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7356, 'log_dups_size': 0, 'ondisk_log_size': 7356, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44378432, 'num_objects': 5418, 'num_object_clones': 0, 'num_object_copies': 21672, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2260, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5418, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7353, 'num_write_kb': 51081, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3161}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:39:32.830 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:39:32.830 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'7425", 'reported_seq': 7521, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:27.031607+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:39:27.031607+0000', 'last_peered': '2024-09-05T16:39:27.031607+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:39:27.031607+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7425, 'log_dups_size': 0, 'ondisk_log_size': 7425, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44111008, 'num_objects': 5385, 'num_object_clones': 0, 'num_object_copies': 21540, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5385, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7423, 'num_write_kb': 51230, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3130}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:39:32.830 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:39:32.830 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'7587", 'reported_seq': 7664, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:26.444859+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:39:26.444859+0000', 'last_peered': '2024-09-05T16:39:26.444859+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:39:26.444859+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7587, 'log_dups_size': 0, 'ondisk_log_size': 7587, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45346544, 'num_objects': 5536, 'num_object_clones': 0, 'num_object_copies': 22144, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2306, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5536, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7584, 'num_write_kb': 52476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3233}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:39:32.830 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:39:32.831 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'7316", 'reported_seq': 7379, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:26.522006+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:39:26.522006+0000', 'last_peered': '2024-09-05T16:39:26.522006+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:39:26.522006+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7316, 'log_dups_size': 0, 'ondisk_log_size': 7316, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44587600, 'num_objects': 5443, 'num_object_clones': 0, 'num_object_copies': 21772, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2254, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5443, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7315, 'num_write_kb': 51031, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3190}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:39:32.831 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-05T16:39:32.831 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'7417", 'reported_seq': 10172, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:26.982667+0000', 'last_change': '2024-09-05T16:37:30.634570+0000', 'last_active': '2024-09-05T16:39:26.982667+0000', 'last_peered': '2024-09-05T16:39:26.982667+0000', 'last_clean': '2024-09-05T16:33:45.082039+0000', 'last_became_active': '2024-09-05T16:34:56.792386+0000', 'last_became_peered': '2024-09-05T16:34:56.792386+0000', 'last_unstale': '2024-09-05T16:39:26.982667+0000', 'last_undegraded': '2024-09-05T16:34:56.600595+0000', 'last_fullsized': '2024-09-05T16:34:56.558203+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7417, 'log_dups_size': 0, 'ondisk_log_size': 7417, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44169808, 'num_objects': 5392, 'num_object_clones': 0, 'num_object_copies': 21568, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 955, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5392, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7416, 'num_write_kb': 51231, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1340, 'num_bytes_recovered': 10969088, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 4437}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:39:32.831 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:39:32.831 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'7607", 'reported_seq': 7676, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:26.336449+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:39:26.336449+0000', 'last_peered': '2024-09-05T16:39:26.336449+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:39:26.336449+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7607, 'log_dups_size': 0, 'ondisk_log_size': 7607, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45208736, 'num_objects': 5519, 'num_object_clones': 0, 'num_object_copies': 22076, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2353, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5519, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7605, 'num_write_kb': 52500, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 8, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3168}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:39:32.831 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:39:32.831 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'7428", 'reported_seq': 7521, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:26.976973+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:39:26.976973+0000', 'last_peered': '2024-09-05T16:39:26.976973+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:39:26.976973+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7428, 'log_dups_size': 0, 'ondisk_log_size': 7428, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44274848, 'num_objects': 5405, 'num_object_clones': 0, 'num_object_copies': 21620, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2317, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5405, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7426, 'num_write_kb': 51325, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3090}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:39:32.831 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:39:32.831 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'7388", 'reported_seq': 7471, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:26.643826+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:39:26.643826+0000', 'last_peered': '2024-09-05T16:39:26.643826+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:39:26.643826+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7388, 'log_dups_size': 0, 'ondisk_log_size': 7388, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44127392, 'num_objects': 5387, 'num_object_clones': 0, 'num_object_copies': 21548, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2232, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5387, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7386, 'num_write_kb': 51092, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3157}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:39:32.832 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:39:33.623 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:39:33.623 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 32401/341432 objects degraded (9.490%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":376,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":85358,"data_bytes":699206712,"bytes_used":2925858816,"bytes_avail":383621197824,"bytes_total":386547056640,"degraded_objects":32401,"degraded_total":341432,"degraded_ratio":0.094897373415497088,"recovering_objects_per_sec":17,"recovering_bytes_per_sec":131016,"recovering_keys_per_sec":0,"num_objects_recovered":209,"num_bytes_recovered":1572864,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":301},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:39:33.649 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:39:33.649 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:39:34.204 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:39:34.204 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:39:34.232 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":214,"stamp":"2024-09-05T16:39:32.912655+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":699206712,"num_objects":85358,"num_object_clones":0,"num_object_copies":341432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32401,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":85358,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":113879,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1532,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119604,"ondisk_log_size":119604,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2857284,"kb_used_data":1298368,"kb_used_omap":14,"kb_used_meta":1558833,"kb_avail":374630076,"statfs":{"total":386547056640,"available":383621197824,"internally_reserved":0,"allocated":1329528832,"data_stored":1328760668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1596245559},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,7,9,11,15,11,10,2,0,0,0,0,3,2,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-29655040,"num_objects":-3620,"num_object_clones":0,"num_object_copies":-14480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3620,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3620,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":209,"num_bytes_recovered":1572864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005053"},"pg_stats":[{"pgid":"1.f","version":"35'7546","reported_seq":7613,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:31.500575+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:39:31.500575+0000","last_peered":"2024-09-05T16:39:31.500575+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:39:31.500575+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7546,"log_dups_size":0,"ondisk_log_size":7546,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43892736,"num_objects":5358,"num_object_clones":0,"num_object_copies":21432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5358,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7546,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3106}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'7383","reported_seq":7468,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:31.767677+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:39:31.767677+0000","last_peered":"2024-09-05T16:39:31.767677+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:39:31.767677+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7383,"log_dups_size":0,"ondisk_log_size":7383,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42357920,"num_objects":5171,"num_object_clones":0,"num_object_copies":20684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2261,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5171,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7381,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2912}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'7468","reported_seq":7555,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:31.837670+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:39:31.837670+0000","last_peered":"2024-09-05T16:39:31.837670+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:39:31.837670+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7468,"log_dups_size":0,"ondisk_log_size":7468,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43285096,"num_objects":5285,"num_object_clones":0,"num_object_copies":21140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2248,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5285,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7467,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3038}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'7563","reported_seq":12286,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-05T16:39:31.030616+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:39:31.030616+0000","last_peered":"2024-09-05T16:39:31.030616+0000","last_clean":"2024-09-05T16:39:31.030616+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:39:31.030616+0000","last_undegraded":"2024-09-05T16:39:31.030616+0000","last_fullsized":"2024-09-05T16:39:31.030616+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":7563,"log_dups_size":0,"ondisk_log_size":7563,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43999232,"num_objects":5371,"num_object_clones":0,"num_object_copies":21484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5371,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1863,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'7329","reported_seq":7406,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:31.533662+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:39:31.533662+0000","last_peered":"2024-09-05T16:39:31.533662+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:39:31.533662+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7329,"log_dups_size":0,"ondisk_log_size":7329,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43136160,"num_objects":5266,"num_object_clones":0,"num_object_copies":21064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2155,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5266,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7327,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3113}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'7430","reported_seq":7501,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:31.567711+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:39:31.567711+0000","last_peered":"2024-09-05T16:39:31.567711+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:39:31.567711+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7430,"log_dups_size":0,"ondisk_log_size":7430,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43761664,"num_objects":5342,"num_object_clones":0,"num_object_copies":21368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2238,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5342,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7430,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3104}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'7290","reported_seq":7374,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:31.594193+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:39:31.594193+0000","last_peered":"2024-09-05T16:39:31.594193+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:39:31.594193+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7290,"log_dups_size":0,"ondisk_log_size":7290,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44004512,"num_objects":5372,"num_object_clones":0,"num_object_copies":21488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2142,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5372,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7288,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3232}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'7459","reported_seq":7536,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:31.732147+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:39:31.732147+0000","last_peered":"2024-09-05T16:39:31.732147+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:39:31.732147+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7459,"log_dups_size":0,"ondisk_log_size":7459,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43578528,"num_objects":5320,"num_object_clones":0,"num_object_copies":21280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2237,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5320,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7457,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3085}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'7430","reported_seq":7499,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:31.489110+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:39:31.489110+0000","last_peered":"2024-09-05T16:39:31.489110+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:39:31.489110+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7430,"log_dups_size":0,"ondisk_log_size":7430,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43772224,"num_objects":5344,"num_object_clones":0,"num_object_copies":21376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5344,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7427,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3087}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'7500","reported_seq":7596,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:31.865876+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:39:31.865876+0000","last_peered":"2024-09-05T16:39:31.865876+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:39:31.865876+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7500,"log_dups_size":0,"ondisk_log_size":7500,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43496608,"num_objects":5310,"num_object_clones":0,"num_object_copies":21240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5310,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7498,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3055}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'7663","reported_seq":7740,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:31.457092+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:39:31.457092+0000","last_peered":"2024-09-05T16:39:31.457092+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:39:31.457092+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7663,"log_dups_size":0,"ondisk_log_size":7663,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44723952,"num_objects":5460,"num_object_clones":0,"num_object_copies":21840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2306,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5460,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7660,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3157}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'7394","reported_seq":7457,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:31.536042+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:39:31.536042+0000","last_peered":"2024-09-05T16:39:31.536042+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:39:31.536042+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7394,"log_dups_size":0,"ondisk_log_size":7394,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43948624,"num_objects":5365,"num_object_clones":0,"num_object_copies":21460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2254,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7393,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3112}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'7497","reported_seq":10389,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:32.079037+0000","last_change":"2024-09-05T16:37:30.634570+0000","last_active":"2024-09-05T16:39:32.079037+0000","last_peered":"2024-09-05T16:39:32.079037+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:39:32.079037+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7497,"log_dups_size":0,"ondisk_log_size":7497,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43514448,"num_objects":5312,"num_object_clones":0,"num_object_copies":21248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":889,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5312,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7496,"num_write_kb":51231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1407,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4423}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'7675","reported_seq":7743,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:31.499097+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:39:31.499097+0000","last_peered":"2024-09-05T16:39:31.499097+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:39:31.499097+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7675,"log_dups_size":0,"ondisk_log_size":7675,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44651680,"num_objects":5451,"num_object_clones":0,"num_object_copies":21804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2353,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5451,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7673,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3100}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'7505","reported_seq":7598,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:31.974853+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:39:31.974853+0000","last_peered":"2024-09-05T16:39:31.974853+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:39:31.974853+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7505,"log_dups_size":0,"ondisk_log_size":7505,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43644064,"num_objects":5328,"num_object_clones":0,"num_object_copies":21312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5328,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7503,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3013}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'7472","reported_seq":7555,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:31.807773+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:39:31.807773+0000","last_peered":"2024-09-05T16:39:31.807773+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:39:31.807773+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7472,"log_dups_size":0,"ondisk_log_size":7472,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43439264,"num_objects":5303,"num_object_clones":0,"num_object_copies":21212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5303,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7470,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3073}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":699206712,"num_objects":85358,"num_object_clones":0,"num_object_copies":341432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32401,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":85358,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":113879,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1532,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1326006272,"data_stored":1326006272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119604,"ondisk_log_size":119604,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084355,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":757624,"kb_used_data":342756,"kb_used_omap":4,"kb_used_meta":414843,"kb_avail":93614216,"statfs":{"total":96636764160,"available":95860957184,"internally_reserved":0,"allocated":350982144,"data_stored":350790103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":424799955},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,7,3,2,4,1,0,0,0,0,3],"upper_bound":8192},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986232,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":555892,"kb_used_data":271524,"kb_used_omap":4,"kb_used_meta":284347,"kb_avail":93815948,"statfs":{"total":96636764160,"available":96067530752,"internally_reserved":0,"allocated":278040576,"data_stored":277848535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":291172050},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,4,6],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673038,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":778100,"kb_used_data":341540,"kb_used_omap":1,"kb_used_meta":436542,"kb_avail":93593740,"statfs":{"total":96636764160,"available":95839989760,"internally_reserved":0,"allocated":349736960,"data_stored":349544919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":447019461},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,5,3,0,3,2,3,1,0,0,0,0,0,2],"upper_bound":16384},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182469,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":765668,"kb_used_data":342548,"kb_used_omap":4,"kb_used_meta":423099,"kb_avail":93606172,"statfs":{"total":96636764160,"available":95852720128,"internally_reserved":0,"allocated":350769152,"data_stored":350577111,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":433254093},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,5,2,5,1,3,0,0,0,0,0,0,0,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":349466624,"data_stored":349466624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":277639168,"data_stored":277639168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":349220864,"data_stored":349220864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":349679616,"data_stored":349679616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:39:37.234 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:39:37.857 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:39:37.858 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:39:37.879 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":216,"stamp":"2024-09-05T16:39:36.914378+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":691784760,"num_objects":84452,"num_object_clones":0,"num_object_copies":337808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32395,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84452,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":114785,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1538,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120510,"ondisk_log_size":120510,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2878208,"kb_used_data":1287336,"kb_used_omap":14,"kb_used_meta":1590769,"kb_avail":374609152,"statfs":{"total":386547056640,"available":383599771648,"internally_reserved":0,"allocated":1318232064,"data_stored":1317463900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1628948023},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,10,9,1,12,12,11,5,0,0,2,0,2,2],"upper_bound":16384},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-27181056,"num_objects":-3318,"num_object_clones":0,"num_object_copies":-13272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-146,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3318,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3318,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":147,"num_bytes_recovered":1089536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005518"},"pg_stats":[{"pgid":"1.f","version":"35'7628","reported_seq":7693,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:36.562890+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:39:36.562890+0000","last_peered":"2024-09-05T16:39:36.562890+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:39:36.562890+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7628,"log_dups_size":0,"ondisk_log_size":7628,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43220992,"num_objects":5276,"num_object_clones":0,"num_object_copies":21104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5276,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7628,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3024}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'7449","reported_seq":7534,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:36.818752+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:39:36.818752+0000","last_peered":"2024-09-05T16:39:36.818752+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:39:36.818752+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7449,"log_dups_size":0,"ondisk_log_size":7449,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41817248,"num_objects":5105,"num_object_clones":0,"num_object_copies":20420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2261,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5105,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7447,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2846}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'7468","reported_seq":7555,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:31.837670+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:39:31.837670+0000","last_peered":"2024-09-05T16:39:31.837670+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:39:31.837670+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7468,"log_dups_size":0,"ondisk_log_size":7468,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43285096,"num_objects":5285,"num_object_clones":0,"num_object_copies":21140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2248,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5285,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7467,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3038}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'7631","reported_seq":12354,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-05T16:39:36.003437+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:39:36.003437+0000","last_peered":"2024-09-05T16:39:36.003437+0000","last_clean":"2024-09-05T16:39:36.003437+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:39:36.003437+0000","last_undegraded":"2024-09-05T16:39:36.003437+0000","last_fullsized":"2024-09-05T16:39:36.003437+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":7631,"log_dups_size":0,"ondisk_log_size":7631,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43442176,"num_objects":5303,"num_object_clones":0,"num_object_copies":21212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5303,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1931,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'7409","reported_seq":7486,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:36.627283+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:39:36.627283+0000","last_peered":"2024-09-05T16:39:36.627283+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:39:36.627283+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7409,"log_dups_size":0,"ondisk_log_size":7409,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42480800,"num_objects":5186,"num_object_clones":0,"num_object_copies":20744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2155,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5186,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7407,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3033}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'7498","reported_seq":7571,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:36.529809+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:39:36.529809+0000","last_peered":"2024-09-05T16:39:36.529809+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:39:36.529809+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7498,"log_dups_size":0,"ondisk_log_size":7498,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43204608,"num_objects":5274,"num_object_clones":0,"num_object_copies":21096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2237,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5274,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7498,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3037}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'7371","reported_seq":7455,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:36.807506+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:39:36.807506+0000","last_peered":"2024-09-05T16:39:36.807506+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:39:36.807506+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7371,"log_dups_size":0,"ondisk_log_size":7371,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43340960,"num_objects":5291,"num_object_clones":0,"num_object_copies":21164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2142,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5291,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7369,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3151}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'7529","reported_seq":7605,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:36.802479+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:39:36.802479+0000","last_peered":"2024-09-05T16:39:36.802479+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:39:36.802479+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7529,"log_dups_size":0,"ondisk_log_size":7529,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43005088,"num_objects":5250,"num_object_clones":0,"num_object_copies":21000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2237,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5250,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7527,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3015}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'7514","reported_seq":7585,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:36.458341+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:39:36.458341+0000","last_peered":"2024-09-05T16:39:36.458341+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:39:36.458341+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7514,"log_dups_size":0,"ondisk_log_size":7514,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43084096,"num_objects":5260,"num_object_clones":0,"num_object_copies":21040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5260,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7511,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3004}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'7500","reported_seq":7596,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:31.865876+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:39:31.865876+0000","last_peered":"2024-09-05T16:39:31.865876+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:39:31.865876+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7500,"log_dups_size":0,"ondisk_log_size":7500,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43496608,"num_objects":5310,"num_object_clones":0,"num_object_copies":21240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5310,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7498,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3055}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'7741","reported_seq":7820,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:36.451274+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:39:36.451274+0000","last_peered":"2024-09-05T16:39:36.451274+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:39:36.451274+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7741,"log_dups_size":0,"ondisk_log_size":7741,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44084976,"num_objects":5382,"num_object_clones":0,"num_object_copies":21528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5382,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7738,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3080}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'7483","reported_seq":7548,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:36.470547+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:39:36.470547+0000","last_peered":"2024-09-05T16:39:36.470547+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:39:36.470547+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7483,"log_dups_size":0,"ondisk_log_size":7483,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43219536,"num_objects":5276,"num_object_clones":0,"num_object_copies":21104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2253,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5276,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7482,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3024}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'7497","reported_seq":10389,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:32.079037+0000","last_change":"2024-09-05T16:37:30.634570+0000","last_active":"2024-09-05T16:39:32.079037+0000","last_peered":"2024-09-05T16:39:32.079037+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:39:32.079037+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7497,"log_dups_size":0,"ondisk_log_size":7497,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43514448,"num_objects":5312,"num_object_clones":0,"num_object_copies":21248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":889,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5312,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7496,"num_write_kb":51231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1407,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4423}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'7750","reported_seq":7819,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:36.619616+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:39:36.619616+0000","last_peered":"2024-09-05T16:39:36.619616+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:39:36.619616+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7750,"log_dups_size":0,"ondisk_log_size":7750,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44037280,"num_objects":5376,"num_object_clones":0,"num_object_copies":21504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2353,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5376,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7748,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3025}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'7505","reported_seq":7598,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:31.974853+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:39:31.974853+0000","last_peered":"2024-09-05T16:39:31.974853+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:39:31.974853+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7505,"log_dups_size":0,"ondisk_log_size":7505,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43644064,"num_objects":5328,"num_object_clones":0,"num_object_copies":21312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5328,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7503,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3013}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'7537","reported_seq":7624,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:36.761216+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:39:36.761216+0000","last_peered":"2024-09-05T16:39:36.761216+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:39:36.761216+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7537,"log_dups_size":0,"ondisk_log_size":7537,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42906784,"num_objects":5238,"num_object_clones":0,"num_object_copies":20952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5238,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7535,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3010}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":691784760,"num_objects":84452,"num_object_clones":0,"num_object_copies":337808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32395,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84452,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":114785,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1538,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1315815424,"data_stored":1315815424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120510,"ondisk_log_size":120510,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084356,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":774908,"kb_used_data":337832,"kb_used_omap":4,"kb_used_meta":437051,"kb_avail":93596932,"statfs":{"total":96636764160,"available":95843258368,"internally_reserved":0,"allocated":345939968,"data_stored":345747927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":447540947},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,4,0,1,4,3,3,4,0,0,2],"upper_bound":2048},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986233,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":559720,"kb_used_data":271192,"kb_used_omap":4,"kb_used_meta":288507,"kb_avail":93812120,"statfs":{"total":96636764160,"available":96063610880,"internally_reserved":0,"allocated":277700608,"data_stored":277508567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":295431890},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2,1,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673038,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":778100,"kb_used_data":341540,"kb_used_omap":1,"kb_used_meta":436542,"kb_avail":93593740,"statfs":{"total":96636764160,"available":95839989760,"internally_reserved":0,"allocated":349736960,"data_stored":349544919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":447019461},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,5,3,0,3,2,3,1,0,0,0,0,0,2],"upper_bound":16384},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182470,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":765480,"kb_used_data":336772,"kb_used_omap":4,"kb_used_meta":428667,"kb_avail":93606360,"statfs":{"total":96636764160,"available":95852912640,"internally_reserved":0,"allocated":344854528,"data_stored":344662487,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":438955725},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,5,0,3,6,2,0,0,0,0,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":344461312,"data_stored":344461312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":277397504,"data_stored":277397504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":349220864,"data_stored":349220864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":344735744,"data_stored":344735744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:39:37.881 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:39:38.479 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:39:38.479 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:39:38.499 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":216,"stamp":"2024-09-05T16:39:36.914378+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":691784760,"num_objects":84452,"num_object_clones":0,"num_object_copies":337808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32395,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84452,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":114785,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1538,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120510,"ondisk_log_size":120510,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2878208,"kb_used_data":1287336,"kb_used_omap":14,"kb_used_meta":1590769,"kb_avail":374609152,"statfs":{"total":386547056640,"available":383599771648,"internally_reserved":0,"allocated":1318232064,"data_stored":1317463900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1628948023},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,10,9,1,12,12,11,5,0,0,2,0,2,2],"upper_bound":16384},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-27181056,"num_objects":-3318,"num_object_clones":0,"num_object_copies":-13272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-146,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3318,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3318,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":147,"num_bytes_recovered":1089536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005518"},"pg_stats":[{"pgid":"1.f","version":"35'7628","reported_seq":7693,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:36.562890+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:39:36.562890+0000","last_peered":"2024-09-05T16:39:36.562890+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:39:36.562890+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7628,"log_dups_size":0,"ondisk_log_size":7628,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43220992,"num_objects":5276,"num_object_clones":0,"num_object_copies":21104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5276,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7628,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3024}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'7449","reported_seq":7534,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:36.818752+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:39:36.818752+0000","last_peered":"2024-09-05T16:39:36.818752+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:39:36.818752+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7449,"log_dups_size":0,"ondisk_log_size":7449,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41817248,"num_objects":5105,"num_object_clones":0,"num_object_copies":20420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2261,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5105,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7447,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2846}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'7468","reported_seq":7555,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:31.837670+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:39:31.837670+0000","last_peered":"2024-09-05T16:39:31.837670+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:39:31.837670+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7468,"log_dups_size":0,"ondisk_log_size":7468,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43285096,"num_objects":5285,"num_object_clones":0,"num_object_copies":21140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2248,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5285,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7467,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3038}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'7631","reported_seq":12354,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-05T16:39:36.003437+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:39:36.003437+0000","last_peered":"2024-09-05T16:39:36.003437+0000","last_clean":"2024-09-05T16:39:36.003437+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:39:36.003437+0000","last_undegraded":"2024-09-05T16:39:36.003437+0000","last_fullsized":"2024-09-05T16:39:36.003437+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":7631,"log_dups_size":0,"ondisk_log_size":7631,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43442176,"num_objects":5303,"num_object_clones":0,"num_object_copies":21212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5303,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1931,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'7409","reported_seq":7486,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:36.627283+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:39:36.627283+0000","last_peered":"2024-09-05T16:39:36.627283+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:39:36.627283+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7409,"log_dups_size":0,"ondisk_log_size":7409,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42480800,"num_objects":5186,"num_object_clones":0,"num_object_copies":20744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2155,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5186,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7407,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3033}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'7498","reported_seq":7571,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:36.529809+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:39:36.529809+0000","last_peered":"2024-09-05T16:39:36.529809+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:39:36.529809+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7498,"log_dups_size":0,"ondisk_log_size":7498,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43204608,"num_objects":5274,"num_object_clones":0,"num_object_copies":21096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2237,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5274,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7498,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3037}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'7371","reported_seq":7455,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:36.807506+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:39:36.807506+0000","last_peered":"2024-09-05T16:39:36.807506+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:39:36.807506+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7371,"log_dups_size":0,"ondisk_log_size":7371,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43340960,"num_objects":5291,"num_object_clones":0,"num_object_copies":21164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2142,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5291,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7369,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3151}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'7529","reported_seq":7605,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:36.802479+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:39:36.802479+0000","last_peered":"2024-09-05T16:39:36.802479+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:39:36.802479+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7529,"log_dups_size":0,"ondisk_log_size":7529,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43005088,"num_objects":5250,"num_object_clones":0,"num_object_copies":21000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2237,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5250,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7527,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3015}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'7514","reported_seq":7585,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:36.458341+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:39:36.458341+0000","last_peered":"2024-09-05T16:39:36.458341+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:39:36.458341+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7514,"log_dups_size":0,"ondisk_log_size":7514,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43084096,"num_objects":5260,"num_object_clones":0,"num_object_copies":21040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5260,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7511,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3004}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'7500","reported_seq":7596,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:31.865876+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:39:31.865876+0000","last_peered":"2024-09-05T16:39:31.865876+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:39:31.865876+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7500,"log_dups_size":0,"ondisk_log_size":7500,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43496608,"num_objects":5310,"num_object_clones":0,"num_object_copies":21240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5310,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7498,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3055}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'7741","reported_seq":7820,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:36.451274+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:39:36.451274+0000","last_peered":"2024-09-05T16:39:36.451274+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:39:36.451274+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7741,"log_dups_size":0,"ondisk_log_size":7741,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44084976,"num_objects":5382,"num_object_clones":0,"num_object_copies":21528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5382,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7738,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3080}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'7483","reported_seq":7548,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:36.470547+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:39:36.470547+0000","last_peered":"2024-09-05T16:39:36.470547+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:39:36.470547+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7483,"log_dups_size":0,"ondisk_log_size":7483,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43219536,"num_objects":5276,"num_object_clones":0,"num_object_copies":21104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2253,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5276,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7482,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3024}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'7497","reported_seq":10389,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:32.079037+0000","last_change":"2024-09-05T16:37:30.634570+0000","last_active":"2024-09-05T16:39:32.079037+0000","last_peered":"2024-09-05T16:39:32.079037+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:39:32.079037+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7497,"log_dups_size":0,"ondisk_log_size":7497,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43514448,"num_objects":5312,"num_object_clones":0,"num_object_copies":21248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":889,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5312,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7496,"num_write_kb":51231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1407,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4423}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'7750","reported_seq":7819,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:36.619616+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:39:36.619616+0000","last_peered":"2024-09-05T16:39:36.619616+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:39:36.619616+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7750,"log_dups_size":0,"ondisk_log_size":7750,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44037280,"num_objects":5376,"num_object_clones":0,"num_object_copies":21504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2353,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5376,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7748,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3025}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'7505","reported_seq":7598,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:31.974853+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:39:31.974853+0000","last_peered":"2024-09-05T16:39:31.974853+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:39:31.974853+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7505,"log_dups_size":0,"ondisk_log_size":7505,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43644064,"num_objects":5328,"num_object_clones":0,"num_object_copies":21312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5328,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7503,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3013}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'7537","reported_seq":7624,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:36.761216+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:39:36.761216+0000","last_peered":"2024-09-05T16:39:36.761216+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:39:36.761216+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7537,"log_dups_size":0,"ondisk_log_size":7537,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42906784,"num_objects":5238,"num_object_clones":0,"num_object_copies":20952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5238,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7535,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3010}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":691784760,"num_objects":84452,"num_object_clones":0,"num_object_copies":337808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32395,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84452,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":114785,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1538,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1315815424,"data_stored":1315815424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120510,"ondisk_log_size":120510,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084356,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":774908,"kb_used_data":337832,"kb_used_omap":4,"kb_used_meta":437051,"kb_avail":93596932,"statfs":{"total":96636764160,"available":95843258368,"internally_reserved":0,"allocated":345939968,"data_stored":345747927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":447540947},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,4,0,1,4,3,3,4,0,0,2],"upper_bound":2048},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986233,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":559720,"kb_used_data":271192,"kb_used_omap":4,"kb_used_meta":288507,"kb_avail":93812120,"statfs":{"total":96636764160,"available":96063610880,"internally_reserved":0,"allocated":277700608,"data_stored":277508567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":295431890},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2,1,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673038,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":778100,"kb_used_data":341540,"kb_used_omap":1,"kb_used_meta":436542,"kb_avail":93593740,"statfs":{"total":96636764160,"available":95839989760,"internally_reserved":0,"allocated":349736960,"data_stored":349544919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":447019461},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,5,3,0,3,2,3,1,0,0,0,0,0,2],"upper_bound":16384},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182470,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":765480,"kb_used_data":336772,"kb_used_omap":4,"kb_used_meta":428667,"kb_avail":93606360,"statfs":{"total":96636764160,"available":95852912640,"internally_reserved":0,"allocated":344854528,"data_stored":344662487,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":438955725},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,5,0,3,6,2,0,0,0,0,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":344461312,"data_stored":344461312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":277397504,"data_stored":277397504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":349220864,"data_stored":349220864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":344735744,"data_stored":344735744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:39:38.500 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-05T16:39:38.500 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'7628", 'reported_seq': 7693, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:36.562890+0000', 'last_change': '2024-09-05T16:34:56.833497+0000', 'last_active': '2024-09-05T16:39:36.562890+0000', 'last_peered': '2024-09-05T16:39:36.562890+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.828289+0000', 'last_became_peered': '2024-09-05T16:34:56.828289+0000', 'last_unstale': '2024-09-05T16:39:36.562890+0000', 'last_undegraded': '2024-09-05T16:34:56.612160+0000', 'last_fullsized': '2024-09-05T16:34:56.557632+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7628, 'log_dups_size': 0, 'ondisk_log_size': 7628, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43220992, 'num_objects': 5276, 'num_object_clones': 0, 'num_object_copies': 21104, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2252, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5276, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7628, 'num_write_kb': 51616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3024}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:39:38.501 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-05T16:39:38.501 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "35'7449", 'reported_seq': 7534, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:36.818752+0000', 'last_change': '2024-09-05T16:34:56.825031+0000', 'last_active': '2024-09-05T16:39:36.818752+0000', 'last_peered': '2024-09-05T16:39:36.818752+0000', 'last_clean': '2024-09-05T16:33:53.200762+0000', 'last_became_active': '2024-09-05T16:34:56.820292+0000', 'last_became_peered': '2024-09-05T16:34:56.820292+0000', 'last_unstale': '2024-09-05T16:39:36.818752+0000', 'last_undegraded': '2024-09-05T16:34:56.605776+0000', 'last_fullsized': '2024-09-05T16:34:56.559383+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7449, 'log_dups_size': 0, 'ondisk_log_size': 7449, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41817248, 'num_objects': 5105, 'num_object_clones': 0, 'num_object_copies': 20420, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2261, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5105, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7447, 'num_write_kb': 50212, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2846}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:39:38.501 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:39:38.501 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'7468", 'reported_seq': 7555, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:31.837670+0000', 'last_change': '2024-09-05T16:34:56.867900+0000', 'last_active': '2024-09-05T16:39:31.837670+0000', 'last_peered': '2024-09-05T16:39:31.837670+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:39:31.837670+0000', 'last_undegraded': '2024-09-05T16:34:56.590972+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7468, 'log_dups_size': 0, 'ondisk_log_size': 7468, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43285096, 'num_objects': 5285, 'num_object_clones': 0, 'num_object_copies': 21140, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2248, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5285, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 7467, 'num_write_kb': 50999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3038}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:39:38.501 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:39:38.501 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'7409", 'reported_seq': 7486, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:36.627283+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:39:36.627283+0000', 'last_peered': '2024-09-05T16:39:36.627283+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:39:36.627283+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7409, 'log_dups_size': 0, 'ondisk_log_size': 7409, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42480800, 'num_objects': 5186, 'num_object_clones': 0, 'num_object_copies': 20744, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2155, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5186, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7407, 'num_write_kb': 50373, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3033}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:39:38.501 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:39:38.501 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'7498", 'reported_seq': 7571, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:36.529809+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:39:36.529809+0000', 'last_peered': '2024-09-05T16:39:36.529809+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:39:36.529809+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7498, 'log_dups_size': 0, 'ondisk_log_size': 7498, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43204608, 'num_objects': 5274, 'num_object_clones': 0, 'num_object_copies': 21096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2237, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5274, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7498, 'num_write_kb': 51088, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3037}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:39:38.501 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:39:38.501 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'7371", 'reported_seq': 7455, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:36.807506+0000', 'last_change': '2024-09-05T16:34:56.903436+0000', 'last_active': '2024-09-05T16:39:36.807506+0000', 'last_peered': '2024-09-05T16:39:36.807506+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:39:36.807506+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7371, 'log_dups_size': 0, 'ondisk_log_size': 7371, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43340960, 'num_objects': 5291, 'num_object_clones': 0, 'num_object_copies': 21164, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2142, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5291, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7369, 'num_write_kb': 50636, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3151}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:39:38.502 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-05T16:39:38.502 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "35'7529", 'reported_seq': 7605, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:36.802479+0000', 'last_change': '2024-09-05T16:34:56.883319+0000', 'last_active': '2024-09-05T16:39:36.802479+0000', 'last_peered': '2024-09-05T16:39:36.802479+0000', 'last_clean': '2024-09-05T16:33:53.505305+0000', 'last_became_active': '2024-09-05T16:34:56.873389+0000', 'last_became_peered': '2024-09-05T16:34:56.873389+0000', 'last_unstale': '2024-09-05T16:39:36.802479+0000', 'last_undegraded': '2024-09-05T16:34:56.602567+0000', 'last_fullsized': '2024-09-05T16:34:56.559958+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7529, 'log_dups_size': 0, 'ondisk_log_size': 7529, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43005088, 'num_objects': 5250, 'num_object_clones': 0, 'num_object_copies': 21000, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2237, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5250, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7527, 'num_write_kb': 51108, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 6, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3015}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:39:38.502 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:39:38.502 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'7514", 'reported_seq': 7585, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:36.458341+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:39:36.458341+0000', 'last_peered': '2024-09-05T16:39:36.458341+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:39:36.458341+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7514, 'log_dups_size': 0, 'ondisk_log_size': 7514, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43084096, 'num_objects': 5260, 'num_object_clones': 0, 'num_object_copies': 21040, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5260, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7511, 'num_write_kb': 51081, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 8, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3004}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:39:38.502 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:39:38.502 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'7500", 'reported_seq': 7596, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:31.865876+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:39:31.865876+0000', 'last_peered': '2024-09-05T16:39:31.865876+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:39:31.865876+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7500, 'log_dups_size': 0, 'ondisk_log_size': 7500, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43496608, 'num_objects': 5310, 'num_object_clones': 0, 'num_object_copies': 21240, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5310, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7498, 'num_write_kb': 51230, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3055}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:39:38.502 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:39:38.502 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'7741", 'reported_seq': 7820, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:36.451274+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:39:36.451274+0000', 'last_peered': '2024-09-05T16:39:36.451274+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:39:36.451274+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7741, 'log_dups_size': 0, 'ondisk_log_size': 7741, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44084976, 'num_objects': 5382, 'num_object_clones': 0, 'num_object_copies': 21528, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2305, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5382, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7738, 'num_write_kb': 52476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3080}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:39:38.502 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:39:38.502 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'7483", 'reported_seq': 7548, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:36.470547+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:39:36.470547+0000', 'last_peered': '2024-09-05T16:39:36.470547+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:39:36.470547+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7483, 'log_dups_size': 0, 'ondisk_log_size': 7483, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43219536, 'num_objects': 5276, 'num_object_clones': 0, 'num_object_copies': 21104, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2253, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5276, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7482, 'num_write_kb': 51031, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3024}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:39:38.502 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-05T16:39:38.503 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'7497", 'reported_seq': 10389, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:32.079037+0000', 'last_change': '2024-09-05T16:37:30.634570+0000', 'last_active': '2024-09-05T16:39:32.079037+0000', 'last_peered': '2024-09-05T16:39:32.079037+0000', 'last_clean': '2024-09-05T16:33:45.082039+0000', 'last_became_active': '2024-09-05T16:34:56.792386+0000', 'last_became_peered': '2024-09-05T16:34:56.792386+0000', 'last_unstale': '2024-09-05T16:39:32.079037+0000', 'last_undegraded': '2024-09-05T16:34:56.600595+0000', 'last_fullsized': '2024-09-05T16:34:56.558203+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7497, 'log_dups_size': 0, 'ondisk_log_size': 7497, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43514448, 'num_objects': 5312, 'num_object_clones': 0, 'num_object_copies': 21248, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 889, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5312, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7496, 'num_write_kb': 51231, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1407, 'num_bytes_recovered': 11509760, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 4423}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:39:38.503 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:39:38.503 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'7750", 'reported_seq': 7819, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:36.619616+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:39:36.619616+0000', 'last_peered': '2024-09-05T16:39:36.619616+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:39:36.619616+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7750, 'log_dups_size': 0, 'ondisk_log_size': 7750, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44037280, 'num_objects': 5376, 'num_object_clones': 0, 'num_object_copies': 21504, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2353, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5376, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7748, 'num_write_kb': 52500, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 8, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3025}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:39:38.503 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:39:38.503 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'7505", 'reported_seq': 7598, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:31.974853+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:39:31.974853+0000', 'last_peered': '2024-09-05T16:39:31.974853+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:39:31.974853+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7505, 'log_dups_size': 0, 'ondisk_log_size': 7505, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43644064, 'num_objects': 5328, 'num_object_clones': 0, 'num_object_copies': 21312, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2317, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5328, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7503, 'num_write_kb': 51325, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3013}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:39:38.503 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:39:38.503 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'7537", 'reported_seq': 7624, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:36.761216+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:39:36.761216+0000', 'last_peered': '2024-09-05T16:39:36.761216+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:39:36.761216+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7537, 'log_dups_size': 0, 'ondisk_log_size': 7537, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42906784, 'num_objects': 5238, 'num_object_clones': 0, 'num_object_copies': 20952, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2230, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5238, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7535, 'num_write_kb': 51092, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3010}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:39:38.503 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:39:39.252 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:39:39.252 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 32322/336560 objects degraded (9.604%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":382,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":84140,"data_bytes":689228856,"bytes_used":2933743616,"bytes_avail":383613313024,"bytes_total":386547056640,"degraded_objects":32322,"degraded_total":336560,"degraded_ratio":0.096036367958164975,"recovering_objects_per_sec":17,"recovering_bytes_per_sec":139199,"recovering_keys_per_sec":0,"num_objects_recovered":213,"num_bytes_recovered":1671168,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":228},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:39:39.277 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:39:39.277 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:39:39.840 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:39:39.843 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:39:39.875 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":217,"stamp":"2024-09-05T16:39:38.915254+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":689228856,"num_objects":84140,"num_object_clones":0,"num_object_copies":336560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84140,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":115097,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1611,"num_bytes_recovered":12091392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120822,"ondisk_log_size":120822,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2864984,"kb_used_data":1282424,"kb_used_omap":14,"kb_used_meta":1582449,"kb_avail":374622376,"statfs":{"total":386547056640,"available":383613313024,"internally_reserved":0,"allocated":1313202176,"data_stored":1312434012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1620428343},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[7,5,6,2,11,11,9,4,0,0,2,0,2,0,2],"upper_bound":32768},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22503424,"num_objects":-2747,"num_object_clones":0,"num_object_copies":-10988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2747,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2747,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":213,"num_bytes_recovered":1671168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005549"},"pg_stats":[{"pgid":"1.f","version":"35'7628","reported_seq":7693,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:36.562890+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:39:36.562890+0000","last_peered":"2024-09-05T16:39:36.562890+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:39:36.562890+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7628,"log_dups_size":0,"ondisk_log_size":7628,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43220992,"num_objects":5276,"num_object_clones":0,"num_object_copies":21104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5276,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7628,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3024}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'7449","reported_seq":7534,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:36.818752+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:39:36.818752+0000","last_peered":"2024-09-05T16:39:36.818752+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:39:36.818752+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7449,"log_dups_size":0,"ondisk_log_size":7449,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41817248,"num_objects":5105,"num_object_clones":0,"num_object_copies":20420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2261,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5105,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7447,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2846}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'7548","reported_seq":7638,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:37.130978+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:39:37.130978+0000","last_peered":"2024-09-05T16:39:37.130978+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:39:37.130978+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7548,"log_dups_size":0,"ondisk_log_size":7548,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42629736,"num_objects":5205,"num_object_clones":0,"num_object_copies":20820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2246,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5205,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7547,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2960}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'7631","reported_seq":12354,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-05T16:39:36.003437+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:39:36.003437+0000","last_peered":"2024-09-05T16:39:36.003437+0000","last_clean":"2024-09-05T16:39:36.003437+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:39:36.003437+0000","last_undegraded":"2024-09-05T16:39:36.003437+0000","last_fullsized":"2024-09-05T16:39:36.003437+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":7631,"log_dups_size":0,"ondisk_log_size":7631,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43442176,"num_objects":5303,"num_object_clones":0,"num_object_copies":21212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5303,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1931,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'7409","reported_seq":7486,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:36.627283+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:39:36.627283+0000","last_peered":"2024-09-05T16:39:36.627283+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:39:36.627283+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7409,"log_dups_size":0,"ondisk_log_size":7409,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42480800,"num_objects":5186,"num_object_clones":0,"num_object_copies":20744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2155,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5186,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7407,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3033}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'7498","reported_seq":7571,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:36.529809+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:39:36.529809+0000","last_peered":"2024-09-05T16:39:36.529809+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:39:36.529809+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7498,"log_dups_size":0,"ondisk_log_size":7498,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43204608,"num_objects":5274,"num_object_clones":0,"num_object_copies":21096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2237,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5274,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7498,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3037}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'7371","reported_seq":7455,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:36.807506+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:39:36.807506+0000","last_peered":"2024-09-05T16:39:36.807506+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:39:36.807506+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7371,"log_dups_size":0,"ondisk_log_size":7371,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43340960,"num_objects":5291,"num_object_clones":0,"num_object_copies":21164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2142,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5291,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7369,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3151}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'7529","reported_seq":7605,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:36.802479+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:39:36.802479+0000","last_peered":"2024-09-05T16:39:36.802479+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:39:36.802479+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7529,"log_dups_size":0,"ondisk_log_size":7529,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43005088,"num_objects":5250,"num_object_clones":0,"num_object_copies":21000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2237,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5250,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7527,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3015}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'7514","reported_seq":7585,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:36.458341+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:39:36.458341+0000","last_peered":"2024-09-05T16:39:36.458341+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:39:36.458341+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7514,"log_dups_size":0,"ondisk_log_size":7514,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43084096,"num_objects":5260,"num_object_clones":0,"num_object_copies":21040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5260,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7511,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3004}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'7582","reported_seq":7677,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:37.087829+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:39:37.087829+0000","last_peered":"2024-09-05T16:39:37.087829+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:39:37.087829+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7582,"log_dups_size":0,"ondisk_log_size":7582,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42824864,"num_objects":5228,"num_object_clones":0,"num_object_copies":20912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5228,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7580,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2973}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'7741","reported_seq":7820,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:36.451274+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:39:36.451274+0000","last_peered":"2024-09-05T16:39:36.451274+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:39:36.451274+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7741,"log_dups_size":0,"ondisk_log_size":7741,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44084976,"num_objects":5382,"num_object_clones":0,"num_object_copies":21528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5382,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7738,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3080}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'7483","reported_seq":7548,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:36.470547+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:39:36.470547+0000","last_peered":"2024-09-05T16:39:36.470547+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:39:36.470547+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7483,"log_dups_size":0,"ondisk_log_size":7483,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43219536,"num_objects":5276,"num_object_clones":0,"num_object_copies":21104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2253,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5276,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7482,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3024}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'7573","reported_seq":10607,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:37.109061+0000","last_change":"2024-09-05T16:37:30.634570+0000","last_active":"2024-09-05T16:39:37.109061+0000","last_peered":"2024-09-05T16:39:37.109061+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:39:37.109061+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7573,"log_dups_size":0,"ondisk_log_size":7573,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42891856,"num_objects":5236,"num_object_clones":0,"num_object_copies":20944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":818,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7572,"num_write_kb":51231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1478,"num_bytes_recovered":12091392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4418}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'7750","reported_seq":7819,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:36.619616+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:39:36.619616+0000","last_peered":"2024-09-05T16:39:36.619616+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:39:36.619616+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7750,"log_dups_size":0,"ondisk_log_size":7750,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44037280,"num_objects":5376,"num_object_clones":0,"num_object_copies":21504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2353,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5376,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7748,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3025}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'7579","reported_seq":7672,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:36.827411+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:39:36.827411+0000","last_peered":"2024-09-05T16:39:36.827411+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:39:36.827411+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7579,"log_dups_size":0,"ondisk_log_size":7579,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43037856,"num_objects":5254,"num_object_clones":0,"num_object_copies":21016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5254,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7577,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2939}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'7537","reported_seq":7624,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:36.761216+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:39:36.761216+0000","last_peered":"2024-09-05T16:39:36.761216+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:39:36.761216+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7537,"log_dups_size":0,"ondisk_log_size":7537,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42906784,"num_objects":5238,"num_object_clones":0,"num_object_copies":20952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5238,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7535,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3010}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":689228856,"num_objects":84140,"num_object_clones":0,"num_object_copies":336560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84140,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":115097,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1611,"num_bytes_recovered":12091392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1310814208,"data_stored":1310814208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120822,"ondisk_log_size":120822,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084356,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":774908,"kb_used_data":337832,"kb_used_omap":4,"kb_used_meta":437051,"kb_avail":93596932,"statfs":{"total":96636764160,"available":95843258368,"internally_reserved":0,"allocated":345939968,"data_stored":345747927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":447540947},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,4,0,1,4,3,3,4,0,0,2],"upper_bound":2048},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986233,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":559720,"kb_used_data":271192,"kb_used_omap":4,"kb_used_meta":288507,"kb_avail":93812120,"statfs":{"total":96636764160,"available":96063610880,"internally_reserved":0,"allocated":277700608,"data_stored":277508567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":295431890},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2,1,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673039,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":764876,"kb_used_data":336628,"kb_used_omap":1,"kb_used_meta":428222,"kb_avail":93606964,"statfs":{"total":96636764160,"available":95853531136,"internally_reserved":0,"allocated":344707072,"data_stored":344515031,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":438499781},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,2,1,1,0,0,0,0,0,0,0,2],"upper_bound":32768},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182470,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":765480,"kb_used_data":336772,"kb_used_omap":4,"kb_used_meta":428667,"kb_avail":93606360,"statfs":{"total":96636764160,"available":95852912640,"internally_reserved":0,"allocated":344854528,"data_stored":344662487,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":438955725},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,5,0,3,6,2,0,0,0,0,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":344461312,"data_stored":344461312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":277397504,"data_stored":277397504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":344219648,"data_stored":344219648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":344735744,"data_stored":344735744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:39:42.877 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:39:43.463 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:39:43.464 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:39:43.494 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":219,"stamp":"2024-09-05T16:39:42.916654+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":679283768,"num_objects":82926,"num_object_clones":0,"num_object_copies":331704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32256,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82926,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":116311,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1677,"num_bytes_recovered":12541952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":122036,"ondisk_log_size":122036,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2868120,"kb_used_data":1267524,"kb_used_omap":14,"kb_used_meta":1600497,"kb_avail":374619240,"statfs":{"total":386547056640,"available":383610101760,"internally_reserved":0,"allocated":1297944576,"data_stored":1297176412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1638909495},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,5,11,16,15,3,2,0,0,0,0,3],"upper_bound":8192},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-29843456,"num_objects":-3643,"num_object_clones":0,"num_object_copies":-14572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-211,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3643,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3643,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":212,"num_bytes_recovered":1572864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004864"},"pg_stats":[{"pgid":"1.f","version":"35'7720","reported_seq":7793,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:41.524179+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:39:41.524179+0000","last_peered":"2024-09-05T16:39:41.524179+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:39:41.524179+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7720,"log_dups_size":0,"ondisk_log_size":7720,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42467328,"num_objects":5184,"num_object_clones":0,"num_object_copies":20736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2249,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5184,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7720,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2935}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'7523","reported_seq":7608,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:41.759007+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:39:41.759007+0000","last_peered":"2024-09-05T16:39:41.759007+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:39:41.759007+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7523,"log_dups_size":0,"ondisk_log_size":7523,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41211040,"num_objects":5031,"num_object_clones":0,"num_object_copies":20124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2261,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5031,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7521,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2772}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'7619","reported_seq":7710,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:42.060606+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:39:42.060606+0000","last_peered":"2024-09-05T16:39:42.060606+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:39:42.060606+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7619,"log_dups_size":0,"ondisk_log_size":7619,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42048104,"num_objects":5134,"num_object_clones":0,"num_object_copies":20536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2246,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5134,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7618,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2889}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'7705","reported_seq":12429,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-05T16:39:40.913303+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:39:40.913303+0000","last_peered":"2024-09-05T16:39:40.913303+0000","last_clean":"2024-09-05T16:39:40.913303+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:39:40.913303+0000","last_undegraded":"2024-09-05T16:39:40.913303+0000","last_fullsized":"2024-09-05T16:39:40.913303+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":7705,"log_dups_size":0,"ondisk_log_size":7705,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42835968,"num_objects":5229,"num_object_clones":0,"num_object_copies":20916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5229,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2005,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'7471","reported_seq":7550,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:41.635749+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:39:41.635749+0000","last_peered":"2024-09-05T16:39:41.635749+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:39:41.635749+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7471,"log_dups_size":0,"ondisk_log_size":7471,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41972896,"num_objects":5124,"num_object_clones":0,"num_object_copies":20496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5124,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7469,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2972}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'7557","reported_seq":7630,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:41.496833+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:39:41.496833+0000","last_peered":"2024-09-05T16:39:41.496833+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:39:41.496833+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7557,"log_dups_size":0,"ondisk_log_size":7557,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42721280,"num_objects":5215,"num_object_clones":0,"num_object_copies":20860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2237,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5215,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7557,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2978}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'7442","reported_seq":7526,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:41.560337+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:39:41.560337+0000","last_peered":"2024-09-05T16:39:41.560337+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:39:41.560337+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7442,"log_dups_size":0,"ondisk_log_size":7442,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42759328,"num_objects":5220,"num_object_clones":0,"num_object_copies":20880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2142,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5220,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7440,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3080}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'7595","reported_seq":7672,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:41.890680+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:39:41.890680+0000","last_peered":"2024-09-05T16:39:41.890680+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:39:41.890680+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7595,"log_dups_size":0,"ondisk_log_size":7595,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42464416,"num_objects":5184,"num_object_clones":0,"num_object_copies":20736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2237,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5184,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7593,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2949}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'7598","reported_seq":7669,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:41.540439+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:39:41.540439+0000","last_peered":"2024-09-05T16:39:41.540439+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:39:41.540439+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7598,"log_dups_size":0,"ondisk_log_size":7598,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42395968,"num_objects":5176,"num_object_clones":0,"num_object_copies":20704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5176,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7595,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2920}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'7658","reported_seq":7752,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:41.931947+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:39:41.931947+0000","last_peered":"2024-09-05T16:39:41.931947+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:39:41.931947+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7658,"log_dups_size":0,"ondisk_log_size":7658,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42202272,"num_objects":5152,"num_object_clones":0,"num_object_copies":20608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5152,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7656,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2897}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'7829","reported_seq":7909,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:41.640614+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:39:41.640614+0000","last_peered":"2024-09-05T16:39:41.640614+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:39:41.640614+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7829,"log_dups_size":0,"ondisk_log_size":7829,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43364080,"num_objects":5294,"num_object_clones":0,"num_object_copies":21176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5294,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7826,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2993}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'7576","reported_seq":7641,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:41.612575+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:39:41.612575+0000","last_peered":"2024-09-05T16:39:41.612575+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:39:41.612575+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7576,"log_dups_size":0,"ondisk_log_size":7576,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42457680,"num_objects":5183,"num_object_clones":0,"num_object_copies":20732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2253,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5183,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7575,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2931}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'7667","reported_seq":10811,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:42.123713+0000","last_change":"2024-09-05T16:37:30.634570+0000","last_active":"2024-09-05T16:39:42.123713+0000","last_peered":"2024-09-05T16:39:42.123713+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:39:42.123713+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7667,"log_dups_size":0,"ondisk_log_size":7667,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42121808,"num_objects":5142,"num_object_clones":0,"num_object_copies":20568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":763,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5142,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7666,"num_write_kb":51231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1533,"num_bytes_recovered":12541952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4379}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'7820","reported_seq":7899,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:41.522121+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:39:41.522121+0000","last_peered":"2024-09-05T16:39:41.522121+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:39:41.522121+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7820,"log_dups_size":0,"ondisk_log_size":7820,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43463840,"num_objects":5306,"num_object_clones":0,"num_object_copies":21224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2348,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5306,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7818,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2960}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'7647","reported_seq":7742,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:42.099928+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:39:42.099928+0000","last_peered":"2024-09-05T16:39:42.099928+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:39:42.099928+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7647,"log_dups_size":0,"ondisk_log_size":7647,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42480800,"num_objects":5186,"num_object_clones":0,"num_object_copies":20744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5186,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7645,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2872}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'7609","reported_seq":7696,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:41.755202+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:39:41.755202+0000","last_peered":"2024-09-05T16:39:41.755202+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:39:41.755202+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7609,"log_dups_size":0,"ondisk_log_size":7609,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42316960,"num_objects":5166,"num_object_clones":0,"num_object_copies":20664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5166,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7607,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2938}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":679283768,"num_objects":82926,"num_object_clones":0,"num_object_copies":331704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32256,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82926,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":116311,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1677,"num_bytes_recovered":12541952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1295360000,"data_stored":1295360000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":122036,"ondisk_log_size":122036,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084357,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":757536,"kb_used_data":332944,"kb_used_omap":4,"kb_used_meta":424571,"kb_avail":93614304,"statfs":{"total":96636764160,"available":95861047296,"internally_reserved":0,"allocated":340934656,"data_stored":340742615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":434761427},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,4,6,4,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986234,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":581528,"kb_used_data":270780,"kb_used_omap":4,"kb_used_meta":310715,"kb_avail":93790312,"statfs":{"total":96636764160,"available":96041279488,"internally_reserved":0,"allocated":277278720,"data_stored":277086679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":318172882},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,3,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673040,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":764236,"kb_used_data":331836,"kb_used_omap":1,"kb_used_meta":432382,"kb_avail":93607604,"statfs":{"total":96636764160,"available":95854186496,"internally_reserved":0,"allocated":339800064,"data_stored":339608023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":442759621},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2,3,1,0,0,0,0,0,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182471,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":764820,"kb_used_data":331964,"kb_used_omap":4,"kb_used_meta":432827,"kb_avail":93607020,"statfs":{"total":96636764160,"available":95853588480,"internally_reserved":0,"allocated":339931136,"data_stored":339739095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":443215565},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,1,5,5,0,1,0,0,0,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":339476480,"data_stored":339476480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":276910080,"data_stored":276910080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":339234816,"data_stored":339234816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":339738624,"data_stored":339738624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:39:43.496 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:39:44.124 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:39:44.125 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:39:44.149 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":219,"stamp":"2024-09-05T16:39:42.916654+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":679283768,"num_objects":82926,"num_object_clones":0,"num_object_copies":331704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32256,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82926,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":116311,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1677,"num_bytes_recovered":12541952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":122036,"ondisk_log_size":122036,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2868120,"kb_used_data":1267524,"kb_used_omap":14,"kb_used_meta":1600497,"kb_avail":374619240,"statfs":{"total":386547056640,"available":383610101760,"internally_reserved":0,"allocated":1297944576,"data_stored":1297176412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1638909495},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,5,11,16,15,3,2,0,0,0,0,3],"upper_bound":8192},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-29843456,"num_objects":-3643,"num_object_clones":0,"num_object_copies":-14572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-211,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3643,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3643,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":212,"num_bytes_recovered":1572864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004864"},"pg_stats":[{"pgid":"1.f","version":"35'7720","reported_seq":7793,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:41.524179+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:39:41.524179+0000","last_peered":"2024-09-05T16:39:41.524179+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:39:41.524179+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7720,"log_dups_size":0,"ondisk_log_size":7720,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42467328,"num_objects":5184,"num_object_clones":0,"num_object_copies":20736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2249,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5184,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7720,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2935}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'7523","reported_seq":7608,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:41.759007+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:39:41.759007+0000","last_peered":"2024-09-05T16:39:41.759007+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:39:41.759007+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7523,"log_dups_size":0,"ondisk_log_size":7523,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41211040,"num_objects":5031,"num_object_clones":0,"num_object_copies":20124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2261,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5031,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7521,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2772}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'7619","reported_seq":7710,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:42.060606+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:39:42.060606+0000","last_peered":"2024-09-05T16:39:42.060606+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:39:42.060606+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7619,"log_dups_size":0,"ondisk_log_size":7619,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42048104,"num_objects":5134,"num_object_clones":0,"num_object_copies":20536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2246,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5134,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7618,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2889}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'7705","reported_seq":12429,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-05T16:39:40.913303+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:39:40.913303+0000","last_peered":"2024-09-05T16:39:40.913303+0000","last_clean":"2024-09-05T16:39:40.913303+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:39:40.913303+0000","last_undegraded":"2024-09-05T16:39:40.913303+0000","last_fullsized":"2024-09-05T16:39:40.913303+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":7705,"log_dups_size":0,"ondisk_log_size":7705,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42835968,"num_objects":5229,"num_object_clones":0,"num_object_copies":20916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5229,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2005,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'7471","reported_seq":7550,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:41.635749+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:39:41.635749+0000","last_peered":"2024-09-05T16:39:41.635749+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:39:41.635749+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7471,"log_dups_size":0,"ondisk_log_size":7471,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41972896,"num_objects":5124,"num_object_clones":0,"num_object_copies":20496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5124,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7469,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2972}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'7557","reported_seq":7630,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:41.496833+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:39:41.496833+0000","last_peered":"2024-09-05T16:39:41.496833+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:39:41.496833+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7557,"log_dups_size":0,"ondisk_log_size":7557,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42721280,"num_objects":5215,"num_object_clones":0,"num_object_copies":20860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2237,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5215,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7557,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2978}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'7442","reported_seq":7526,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:41.560337+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:39:41.560337+0000","last_peered":"2024-09-05T16:39:41.560337+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:39:41.560337+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7442,"log_dups_size":0,"ondisk_log_size":7442,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42759328,"num_objects":5220,"num_object_clones":0,"num_object_copies":20880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2142,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5220,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7440,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3080}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'7595","reported_seq":7672,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:41.890680+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:39:41.890680+0000","last_peered":"2024-09-05T16:39:41.890680+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:39:41.890680+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7595,"log_dups_size":0,"ondisk_log_size":7595,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42464416,"num_objects":5184,"num_object_clones":0,"num_object_copies":20736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2237,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5184,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7593,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2949}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'7598","reported_seq":7669,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:41.540439+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:39:41.540439+0000","last_peered":"2024-09-05T16:39:41.540439+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:39:41.540439+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7598,"log_dups_size":0,"ondisk_log_size":7598,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42395968,"num_objects":5176,"num_object_clones":0,"num_object_copies":20704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5176,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7595,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2920}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'7658","reported_seq":7752,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:41.931947+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:39:41.931947+0000","last_peered":"2024-09-05T16:39:41.931947+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:39:41.931947+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7658,"log_dups_size":0,"ondisk_log_size":7658,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42202272,"num_objects":5152,"num_object_clones":0,"num_object_copies":20608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5152,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7656,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2897}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'7829","reported_seq":7909,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:41.640614+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:39:41.640614+0000","last_peered":"2024-09-05T16:39:41.640614+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:39:41.640614+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7829,"log_dups_size":0,"ondisk_log_size":7829,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43364080,"num_objects":5294,"num_object_clones":0,"num_object_copies":21176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5294,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7826,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2993}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'7576","reported_seq":7641,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:41.612575+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:39:41.612575+0000","last_peered":"2024-09-05T16:39:41.612575+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:39:41.612575+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7576,"log_dups_size":0,"ondisk_log_size":7576,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42457680,"num_objects":5183,"num_object_clones":0,"num_object_copies":20732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2253,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5183,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7575,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2931}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'7667","reported_seq":10811,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:42.123713+0000","last_change":"2024-09-05T16:37:30.634570+0000","last_active":"2024-09-05T16:39:42.123713+0000","last_peered":"2024-09-05T16:39:42.123713+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:39:42.123713+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7667,"log_dups_size":0,"ondisk_log_size":7667,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42121808,"num_objects":5142,"num_object_clones":0,"num_object_copies":20568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":763,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5142,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7666,"num_write_kb":51231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1533,"num_bytes_recovered":12541952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4379}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'7820","reported_seq":7899,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:41.522121+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:39:41.522121+0000","last_peered":"2024-09-05T16:39:41.522121+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:39:41.522121+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7820,"log_dups_size":0,"ondisk_log_size":7820,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43463840,"num_objects":5306,"num_object_clones":0,"num_object_copies":21224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2348,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5306,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7818,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2960}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'7647","reported_seq":7742,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:42.099928+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:39:42.099928+0000","last_peered":"2024-09-05T16:39:42.099928+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:39:42.099928+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7647,"log_dups_size":0,"ondisk_log_size":7647,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42480800,"num_objects":5186,"num_object_clones":0,"num_object_copies":20744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5186,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7645,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2872}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'7609","reported_seq":7696,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:41.755202+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:39:41.755202+0000","last_peered":"2024-09-05T16:39:41.755202+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:39:41.755202+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7609,"log_dups_size":0,"ondisk_log_size":7609,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42316960,"num_objects":5166,"num_object_clones":0,"num_object_copies":20664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5166,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7607,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2938}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":679283768,"num_objects":82926,"num_object_clones":0,"num_object_copies":331704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32256,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82926,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":116311,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1677,"num_bytes_recovered":12541952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1295360000,"data_stored":1295360000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":122036,"ondisk_log_size":122036,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084357,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":757536,"kb_used_data":332944,"kb_used_omap":4,"kb_used_meta":424571,"kb_avail":93614304,"statfs":{"total":96636764160,"available":95861047296,"internally_reserved":0,"allocated":340934656,"data_stored":340742615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":434761427},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,4,6,4,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986234,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":581528,"kb_used_data":270780,"kb_used_omap":4,"kb_used_meta":310715,"kb_avail":93790312,"statfs":{"total":96636764160,"available":96041279488,"internally_reserved":0,"allocated":277278720,"data_stored":277086679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":318172882},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,3,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673040,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":764236,"kb_used_data":331836,"kb_used_omap":1,"kb_used_meta":432382,"kb_avail":93607604,"statfs":{"total":96636764160,"available":95854186496,"internally_reserved":0,"allocated":339800064,"data_stored":339608023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":442759621},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2,3,1,0,0,0,0,0,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182471,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":764820,"kb_used_data":331964,"kb_used_omap":4,"kb_used_meta":432827,"kb_avail":93607020,"statfs":{"total":96636764160,"available":95853588480,"internally_reserved":0,"allocated":339931136,"data_stored":339739095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":443215565},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,1,5,5,0,1,0,0,0,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":339476480,"data_stored":339476480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":276910080,"data_stored":276910080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":339234816,"data_stored":339234816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":339738624,"data_stored":339738624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:39:44.150 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-05T16:39:44.150 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'7720", 'reported_seq': 7793, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:41.524179+0000', 'last_change': '2024-09-05T16:34:56.833497+0000', 'last_active': '2024-09-05T16:39:41.524179+0000', 'last_peered': '2024-09-05T16:39:41.524179+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.828289+0000', 'last_became_peered': '2024-09-05T16:34:56.828289+0000', 'last_unstale': '2024-09-05T16:39:41.524179+0000', 'last_undegraded': '2024-09-05T16:34:56.612160+0000', 'last_fullsized': '2024-09-05T16:34:56.557632+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7720, 'log_dups_size': 0, 'ondisk_log_size': 7720, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42467328, 'num_objects': 5184, 'num_object_clones': 0, 'num_object_copies': 20736, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2249, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5184, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7720, 'num_write_kb': 51616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2935}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:39:44.150 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-05T16:39:44.150 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "35'7523", 'reported_seq': 7608, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:41.759007+0000', 'last_change': '2024-09-05T16:34:56.825031+0000', 'last_active': '2024-09-05T16:39:41.759007+0000', 'last_peered': '2024-09-05T16:39:41.759007+0000', 'last_clean': '2024-09-05T16:33:53.200762+0000', 'last_became_active': '2024-09-05T16:34:56.820292+0000', 'last_became_peered': '2024-09-05T16:34:56.820292+0000', 'last_unstale': '2024-09-05T16:39:41.759007+0000', 'last_undegraded': '2024-09-05T16:34:56.605776+0000', 'last_fullsized': '2024-09-05T16:34:56.559383+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7523, 'log_dups_size': 0, 'ondisk_log_size': 7523, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41211040, 'num_objects': 5031, 'num_object_clones': 0, 'num_object_copies': 20124, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2261, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5031, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7521, 'num_write_kb': 50212, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2772}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:39:44.151 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:39:44.151 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'7619", 'reported_seq': 7710, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:42.060606+0000', 'last_change': '2024-09-05T16:34:56.867900+0000', 'last_active': '2024-09-05T16:39:42.060606+0000', 'last_peered': '2024-09-05T16:39:42.060606+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:39:42.060606+0000', 'last_undegraded': '2024-09-05T16:34:56.590972+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7619, 'log_dups_size': 0, 'ondisk_log_size': 7619, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42048104, 'num_objects': 5134, 'num_object_clones': 0, 'num_object_copies': 20536, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2246, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5134, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 7618, 'num_write_kb': 50999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 6, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2889}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:39:44.151 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:39:44.151 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'7471", 'reported_seq': 7550, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:41.635749+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:39:41.635749+0000', 'last_peered': '2024-09-05T16:39:41.635749+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:39:41.635749+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7471, 'log_dups_size': 0, 'ondisk_log_size': 7471, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41972896, 'num_objects': 5124, 'num_object_clones': 0, 'num_object_copies': 20496, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2154, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5124, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7469, 'num_write_kb': 50373, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2972}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:39:44.151 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:39:44.151 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'7557", 'reported_seq': 7630, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:41.496833+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:39:41.496833+0000', 'last_peered': '2024-09-05T16:39:41.496833+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:39:41.496833+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7557, 'log_dups_size': 0, 'ondisk_log_size': 7557, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42721280, 'num_objects': 5215, 'num_object_clones': 0, 'num_object_copies': 20860, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2237, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5215, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7557, 'num_write_kb': 51088, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2978}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:39:44.151 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:39:44.151 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'7442", 'reported_seq': 7526, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:41.560337+0000', 'last_change': '2024-09-05T16:34:56.903436+0000', 'last_active': '2024-09-05T16:39:41.560337+0000', 'last_peered': '2024-09-05T16:39:41.560337+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:39:41.560337+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7442, 'log_dups_size': 0, 'ondisk_log_size': 7442, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42759328, 'num_objects': 5220, 'num_object_clones': 0, 'num_object_copies': 20880, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2142, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5220, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7440, 'num_write_kb': 50636, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3080}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:39:44.151 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-05T16:39:44.151 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "35'7595", 'reported_seq': 7672, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:41.890680+0000', 'last_change': '2024-09-05T16:34:56.883319+0000', 'last_active': '2024-09-05T16:39:41.890680+0000', 'last_peered': '2024-09-05T16:39:41.890680+0000', 'last_clean': '2024-09-05T16:33:53.505305+0000', 'last_became_active': '2024-09-05T16:34:56.873389+0000', 'last_became_peered': '2024-09-05T16:34:56.873389+0000', 'last_unstale': '2024-09-05T16:39:41.890680+0000', 'last_undegraded': '2024-09-05T16:34:56.602567+0000', 'last_fullsized': '2024-09-05T16:34:56.559958+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7595, 'log_dups_size': 0, 'ondisk_log_size': 7595, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42464416, 'num_objects': 5184, 'num_object_clones': 0, 'num_object_copies': 20736, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2237, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5184, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7593, 'num_write_kb': 51108, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 6, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2949}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:39:44.152 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:39:44.152 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'7598", 'reported_seq': 7669, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:41.540439+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:39:41.540439+0000', 'last_peered': '2024-09-05T16:39:41.540439+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:39:41.540439+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7598, 'log_dups_size': 0, 'ondisk_log_size': 7598, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42395968, 'num_objects': 5176, 'num_object_clones': 0, 'num_object_copies': 20704, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5176, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7595, 'num_write_kb': 51081, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 8, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2920}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:39:44.152 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:39:44.152 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'7658", 'reported_seq': 7752, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:41.931947+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:39:41.931947+0000', 'last_peered': '2024-09-05T16:39:41.931947+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:39:41.931947+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7658, 'log_dups_size': 0, 'ondisk_log_size': 7658, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42202272, 'num_objects': 5152, 'num_object_clones': 0, 'num_object_copies': 20608, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5152, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7656, 'num_write_kb': 51230, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2897}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:39:44.152 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:39:44.152 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'7829", 'reported_seq': 7909, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:41.640614+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:39:41.640614+0000', 'last_peered': '2024-09-05T16:39:41.640614+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:39:41.640614+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7829, 'log_dups_size': 0, 'ondisk_log_size': 7829, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43364080, 'num_objects': 5294, 'num_object_clones': 0, 'num_object_copies': 21176, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2304, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5294, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7826, 'num_write_kb': 52476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2993}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:39:44.152 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:39:44.152 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'7576", 'reported_seq': 7641, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:41.612575+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:39:41.612575+0000', 'last_peered': '2024-09-05T16:39:41.612575+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:39:41.612575+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7576, 'log_dups_size': 0, 'ondisk_log_size': 7576, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42457680, 'num_objects': 5183, 'num_object_clones': 0, 'num_object_copies': 20732, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2253, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5183, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7575, 'num_write_kb': 51031, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2931}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:39:44.153 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-05T16:39:44.153 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'7667", 'reported_seq': 10811, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:42.123713+0000', 'last_change': '2024-09-05T16:37:30.634570+0000', 'last_active': '2024-09-05T16:39:42.123713+0000', 'last_peered': '2024-09-05T16:39:42.123713+0000', 'last_clean': '2024-09-05T16:33:45.082039+0000', 'last_became_active': '2024-09-05T16:34:56.792386+0000', 'last_became_peered': '2024-09-05T16:34:56.792386+0000', 'last_unstale': '2024-09-05T16:39:42.123713+0000', 'last_undegraded': '2024-09-05T16:34:56.600595+0000', 'last_fullsized': '2024-09-05T16:34:56.558203+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7667, 'log_dups_size': 0, 'ondisk_log_size': 7667, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42121808, 'num_objects': 5142, 'num_object_clones': 0, 'num_object_copies': 20568, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 763, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5142, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7666, 'num_write_kb': 51231, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1533, 'num_bytes_recovered': 12541952, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 4379}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:39:44.153 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:39:44.153 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'7820", 'reported_seq': 7899, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:41.522121+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:39:41.522121+0000', 'last_peered': '2024-09-05T16:39:41.522121+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:39:41.522121+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7820, 'log_dups_size': 0, 'ondisk_log_size': 7820, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43463840, 'num_objects': 5306, 'num_object_clones': 0, 'num_object_copies': 21224, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2348, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5306, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7818, 'num_write_kb': 52500, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2960}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:39:44.153 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:39:44.154 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'7647", 'reported_seq': 7742, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:42.099928+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:39:42.099928+0000', 'last_peered': '2024-09-05T16:39:42.099928+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:39:42.099928+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7647, 'log_dups_size': 0, 'ondisk_log_size': 7647, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42480800, 'num_objects': 5186, 'num_object_clones': 0, 'num_object_copies': 20744, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5186, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7645, 'num_write_kb': 51325, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 8, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2872}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:39:44.154 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:39:44.154 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'7609", 'reported_seq': 7696, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:41.755202+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:39:41.755202+0000', 'last_peered': '2024-09-05T16:39:41.755202+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:39:41.755202+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7609, 'log_dups_size': 0, 'ondisk_log_size': 7609, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42316960, 'num_objects': 5166, 'num_object_clones': 0, 'num_object_copies': 20664, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2230, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5166, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7607, 'num_write_kb': 51092, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2938}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:39:44.154 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:39:44.878 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:39:44.878 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 32256/331704 objects degraded (9.724%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":388,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":82926,"data_bytes":679283768,"bytes_used":2936954880,"bytes_avail":383610101760,"bytes_total":386547056640,"degraded_objects":32256,"degraded_total":331704,"degraded_ratio":0.097243325374430215,"recovering_objects_per_sec":17,"recovering_bytes_per_sec":131018,"recovering_keys_per_sec":0,"num_objects_recovered":212,"num_bytes_recovered":1572864,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":303},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:39:44.902 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:39:44.903 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:39:45.574 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:39:45.575 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:39:45.593 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":220,"stamp":"2024-09-05T16:39:44.917223+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":679283768,"num_objects":82926,"num_object_clones":0,"num_object_copies":331704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32256,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82926,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":116311,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1677,"num_bytes_recovered":12541952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":122036,"ondisk_log_size":122036,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2868120,"kb_used_data":1267524,"kb_used_omap":14,"kb_used_meta":1600497,"kb_avail":374619240,"statfs":{"total":386547056640,"available":383610101760,"internally_reserved":0,"allocated":1297944576,"data_stored":1297176412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1638909495},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,5,11,16,15,3,2,0,0,0,0,3],"upper_bound":8192},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19922944,"num_objects":-2432,"num_object_clones":0,"num_object_copies":-9728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-145,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2432,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2432,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":145,"num_bytes_recovered":1032192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004567"},"pg_stats":[{"pgid":"1.f","version":"35'7720","reported_seq":7793,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:41.524179+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:39:41.524179+0000","last_peered":"2024-09-05T16:39:41.524179+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:39:41.524179+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7720,"log_dups_size":0,"ondisk_log_size":7720,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42467328,"num_objects":5184,"num_object_clones":0,"num_object_copies":20736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2249,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5184,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7720,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2935}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'7523","reported_seq":7608,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:41.759007+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:39:41.759007+0000","last_peered":"2024-09-05T16:39:41.759007+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:39:41.759007+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7523,"log_dups_size":0,"ondisk_log_size":7523,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41211040,"num_objects":5031,"num_object_clones":0,"num_object_copies":20124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2261,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5031,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7521,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2772}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'7619","reported_seq":7710,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:42.060606+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:39:42.060606+0000","last_peered":"2024-09-05T16:39:42.060606+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:39:42.060606+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7619,"log_dups_size":0,"ondisk_log_size":7619,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42048104,"num_objects":5134,"num_object_clones":0,"num_object_copies":20536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2246,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5134,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7618,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2889}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'7705","reported_seq":12429,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-05T16:39:40.913303+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:39:40.913303+0000","last_peered":"2024-09-05T16:39:40.913303+0000","last_clean":"2024-09-05T16:39:40.913303+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:39:40.913303+0000","last_undegraded":"2024-09-05T16:39:40.913303+0000","last_fullsized":"2024-09-05T16:39:40.913303+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":7705,"log_dups_size":0,"ondisk_log_size":7705,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42835968,"num_objects":5229,"num_object_clones":0,"num_object_copies":20916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5229,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2005,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'7471","reported_seq":7550,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:41.635749+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:39:41.635749+0000","last_peered":"2024-09-05T16:39:41.635749+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:39:41.635749+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7471,"log_dups_size":0,"ondisk_log_size":7471,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41972896,"num_objects":5124,"num_object_clones":0,"num_object_copies":20496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5124,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7469,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2972}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'7557","reported_seq":7630,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:41.496833+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:39:41.496833+0000","last_peered":"2024-09-05T16:39:41.496833+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:39:41.496833+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7557,"log_dups_size":0,"ondisk_log_size":7557,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42721280,"num_objects":5215,"num_object_clones":0,"num_object_copies":20860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2237,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5215,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7557,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2978}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'7442","reported_seq":7526,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:41.560337+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:39:41.560337+0000","last_peered":"2024-09-05T16:39:41.560337+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:39:41.560337+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7442,"log_dups_size":0,"ondisk_log_size":7442,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42759328,"num_objects":5220,"num_object_clones":0,"num_object_copies":20880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2142,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5220,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7440,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3080}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'7595","reported_seq":7672,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:41.890680+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:39:41.890680+0000","last_peered":"2024-09-05T16:39:41.890680+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:39:41.890680+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7595,"log_dups_size":0,"ondisk_log_size":7595,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42464416,"num_objects":5184,"num_object_clones":0,"num_object_copies":20736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2237,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5184,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7593,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2949}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'7598","reported_seq":7669,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:41.540439+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:39:41.540439+0000","last_peered":"2024-09-05T16:39:41.540439+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:39:41.540439+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7598,"log_dups_size":0,"ondisk_log_size":7598,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42395968,"num_objects":5176,"num_object_clones":0,"num_object_copies":20704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5176,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7595,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2920}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'7658","reported_seq":7752,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:41.931947+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:39:41.931947+0000","last_peered":"2024-09-05T16:39:41.931947+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:39:41.931947+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7658,"log_dups_size":0,"ondisk_log_size":7658,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42202272,"num_objects":5152,"num_object_clones":0,"num_object_copies":20608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5152,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7656,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2897}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'7829","reported_seq":7909,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:41.640614+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:39:41.640614+0000","last_peered":"2024-09-05T16:39:41.640614+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:39:41.640614+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7829,"log_dups_size":0,"ondisk_log_size":7829,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43364080,"num_objects":5294,"num_object_clones":0,"num_object_copies":21176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5294,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7826,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2993}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'7576","reported_seq":7641,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:41.612575+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:39:41.612575+0000","last_peered":"2024-09-05T16:39:41.612575+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:39:41.612575+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7576,"log_dups_size":0,"ondisk_log_size":7576,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42457680,"num_objects":5183,"num_object_clones":0,"num_object_copies":20732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2253,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5183,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7575,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2931}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'7667","reported_seq":10811,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:42.123713+0000","last_change":"2024-09-05T16:37:30.634570+0000","last_active":"2024-09-05T16:39:42.123713+0000","last_peered":"2024-09-05T16:39:42.123713+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:39:42.123713+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7667,"log_dups_size":0,"ondisk_log_size":7667,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42121808,"num_objects":5142,"num_object_clones":0,"num_object_copies":20568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":763,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5142,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7666,"num_write_kb":51231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1533,"num_bytes_recovered":12541952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4379}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'7820","reported_seq":7899,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:41.522121+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:39:41.522121+0000","last_peered":"2024-09-05T16:39:41.522121+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:39:41.522121+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7820,"log_dups_size":0,"ondisk_log_size":7820,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43463840,"num_objects":5306,"num_object_clones":0,"num_object_copies":21224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2348,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5306,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7818,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2960}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'7647","reported_seq":7742,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:42.099928+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:39:42.099928+0000","last_peered":"2024-09-05T16:39:42.099928+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:39:42.099928+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7647,"log_dups_size":0,"ondisk_log_size":7647,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42480800,"num_objects":5186,"num_object_clones":0,"num_object_copies":20744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5186,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7645,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2872}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'7609","reported_seq":7696,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:41.755202+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:39:41.755202+0000","last_peered":"2024-09-05T16:39:41.755202+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:39:41.755202+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7609,"log_dups_size":0,"ondisk_log_size":7609,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42316960,"num_objects":5166,"num_object_clones":0,"num_object_copies":20664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5166,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7607,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2938}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":679283768,"num_objects":82926,"num_object_clones":0,"num_object_copies":331704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32256,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82926,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":116311,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1677,"num_bytes_recovered":12541952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1295360000,"data_stored":1295360000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":122036,"ondisk_log_size":122036,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084357,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":757536,"kb_used_data":332944,"kb_used_omap":4,"kb_used_meta":424571,"kb_avail":93614304,"statfs":{"total":96636764160,"available":95861047296,"internally_reserved":0,"allocated":340934656,"data_stored":340742615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":434761427},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,4,6,4,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986234,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":581528,"kb_used_data":270780,"kb_used_omap":4,"kb_used_meta":310715,"kb_avail":93790312,"statfs":{"total":96636764160,"available":96041279488,"internally_reserved":0,"allocated":277278720,"data_stored":277086679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":318172882},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,3,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673040,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":764236,"kb_used_data":331836,"kb_used_omap":1,"kb_used_meta":432382,"kb_avail":93607604,"statfs":{"total":96636764160,"available":95854186496,"internally_reserved":0,"allocated":339800064,"data_stored":339608023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":442759621},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2,3,1,0,0,0,0,0,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182471,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":764820,"kb_used_data":331964,"kb_used_omap":4,"kb_used_meta":432827,"kb_avail":93607020,"statfs":{"total":96636764160,"available":95853588480,"internally_reserved":0,"allocated":339931136,"data_stored":339739095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":443215565},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,1,5,5,0,1,0,0,0,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":339476480,"data_stored":339476480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":276910080,"data_stored":276910080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":339234816,"data_stored":339234816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":339738624,"data_stored":339738624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:39:48.595 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:39:49.836 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:39:49.837 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:39:49.860 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":222,"stamp":"2024-09-05T16:39:48.918661+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":669404216,"num_objects":81720,"num_object_clones":0,"num_object_copies":326880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32172,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81720,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":117517,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1761,"num_bytes_recovered":13033472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":123242,"ondisk_log_size":123242,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2853508,"kb_used_data":1252888,"kb_used_omap":14,"kb_used_meta":1600497,"kb_avail":374633852,"statfs":{"total":386547056640,"available":383625064448,"internally_reserved":0,"allocated":1282957312,"data_stored":1282189148,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1638909495},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,1,10,13,21,11,3,0,1,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22380544,"num_objects":-2732,"num_object_clones":0,"num_object_copies":-10928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2732,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2732,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":223,"num_bytes_recovered":1523712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004282"},"pg_stats":[{"pgid":"1.f","version":"35'7788","reported_seq":7861,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:46.541706+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:39:46.541706+0000","last_peered":"2024-09-05T16:39:46.541706+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:39:46.541706+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7788,"log_dups_size":0,"ondisk_log_size":7788,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41910272,"num_objects":5116,"num_object_clones":0,"num_object_copies":20464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2249,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5116,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7788,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2867}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'7604","reported_seq":7690,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:46.921403+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:39:46.921403+0000","last_peered":"2024-09-05T16:39:46.921403+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:39:46.921403+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7604,"log_dups_size":0,"ondisk_log_size":7604,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40547488,"num_objects":4950,"num_object_clones":0,"num_object_copies":19800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4950,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7602,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2692}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'7692","reported_seq":7785,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:47.128641+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:39:47.128641+0000","last_peered":"2024-09-05T16:39:47.128641+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:39:47.128641+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7692,"log_dups_size":0,"ondisk_log_size":7692,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41450088,"num_objects":5061,"num_object_clones":0,"num_object_copies":20244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2245,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5061,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7691,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2817}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'7774","reported_seq":12498,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-05T16:39:46.038144+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:39:46.038144+0000","last_peered":"2024-09-05T16:39:46.038144+0000","last_clean":"2024-09-05T16:39:46.038144+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:39:46.038144+0000","last_undegraded":"2024-09-05T16:39:46.038144+0000","last_fullsized":"2024-09-05T16:39:46.038144+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":7774,"log_dups_size":0,"ondisk_log_size":7774,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42270720,"num_objects":5160,"num_object_clones":0,"num_object_copies":20640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5160,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2074,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'7560","reported_seq":7641,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:46.648021+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:39:46.648021+0000","last_peered":"2024-09-05T16:39:46.648021+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:39:46.648021+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7560,"log_dups_size":0,"ondisk_log_size":7560,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41243808,"num_objects":5035,"num_object_clones":0,"num_object_copies":20140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2153,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5035,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7558,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2884}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'7627","reported_seq":7706,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:46.656035+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:39:46.656035+0000","last_peered":"2024-09-05T16:39:46.656035+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:39:46.656035+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7627,"log_dups_size":0,"ondisk_log_size":7627,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42147840,"num_objects":5145,"num_object_clones":0,"num_object_copies":20580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5145,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7627,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2911}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'7531","reported_seq":7615,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:46.662980+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:39:46.662980+0000","last_peered":"2024-09-05T16:39:46.662980+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:39:46.662980+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7531,"log_dups_size":0,"ondisk_log_size":7531,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42030240,"num_objects":5131,"num_object_clones":0,"num_object_copies":20524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2142,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5131,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7529,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2991}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'7652","reported_seq":7731,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:46.867540+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:39:46.867540+0000","last_peered":"2024-09-05T16:39:46.867540+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:39:46.867540+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7652,"log_dups_size":0,"ondisk_log_size":7652,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41997472,"num_objects":5127,"num_object_clones":0,"num_object_copies":20508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5127,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7650,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2893}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'7681","reported_seq":7758,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:46.359086+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:39:46.359086+0000","last_peered":"2024-09-05T16:39:46.359086+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:39:46.359086+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7681,"log_dups_size":0,"ondisk_log_size":7681,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41716032,"num_objects":5093,"num_object_clones":0,"num_object_copies":20372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2256,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5093,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7678,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2840}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'7720","reported_seq":7820,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:47.130409+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:39:47.130409+0000","last_peered":"2024-09-05T16:39:47.130409+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:39:47.130409+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7720,"log_dups_size":0,"ondisk_log_size":7720,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41694368,"num_objects":5090,"num_object_clones":0,"num_object_copies":20360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2255,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5090,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7718,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2837}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'7924","reported_seq":8018,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:46.647764+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:39:46.647764+0000","last_peered":"2024-09-05T16:39:46.647764+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:39:46.647764+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7924,"log_dups_size":0,"ondisk_log_size":7924,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42585840,"num_objects":5199,"num_object_clones":0,"num_object_copies":20796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2297,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5199,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7921,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2905}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'7650","reported_seq":7717,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:46.702468+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:39:46.702468+0000","last_peered":"2024-09-05T16:39:46.702468+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:39:46.702468+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7650,"log_dups_size":0,"ondisk_log_size":7650,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41851472,"num_objects":5109,"num_object_clones":0,"num_object_copies":20436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5109,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7649,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2858}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'7744","reported_seq":11008,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:47.137043+0000","last_change":"2024-09-05T16:37:30.634570+0000","last_active":"2024-09-05T16:39:47.137043+0000","last_peered":"2024-09-05T16:39:47.137043+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:39:47.137043+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7744,"log_dups_size":0,"ondisk_log_size":7744,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41491024,"num_objects":5065,"num_object_clones":0,"num_object_copies":20260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":703,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5065,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7743,"num_write_kb":51231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1593,"num_bytes_recovered":13033472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4362}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'7906","reported_seq":7989,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:46.563901+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:39:46.563901+0000","last_peered":"2024-09-05T16:39:46.563901+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:39:46.563901+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7906,"log_dups_size":0,"ondisk_log_size":7906,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42759328,"num_objects":5220,"num_object_clones":0,"num_object_copies":20880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2345,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5220,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7904,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2877}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'7712","reported_seq":7807,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:47.077795+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:39:47.077795+0000","last_peered":"2024-09-05T16:39:47.077795+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:39:47.077795+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7712,"log_dups_size":0,"ondisk_log_size":7712,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41948320,"num_objects":5121,"num_object_clones":0,"num_object_copies":20484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5121,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7710,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2807}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'7677","reported_seq":7766,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:46.717897+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:39:46.717897+0000","last_peered":"2024-09-05T16:39:46.717897+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:39:46.717897+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7677,"log_dups_size":0,"ondisk_log_size":7677,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41759904,"num_objects":5098,"num_object_clones":0,"num_object_copies":20392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2229,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5098,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7675,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2871}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":669404216,"num_objects":81720,"num_object_clones":0,"num_object_copies":326880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32172,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81720,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":117517,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1761,"num_bytes_recovered":13033472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1280184320,"data_stored":1280184320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":123242,"ondisk_log_size":123242,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084358,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":757048,"kb_used_data":328288,"kb_used_omap":4,"kb_used_meta":428731,"kb_avail":93614792,"statfs":{"total":96636764160,"available":95861547008,"internally_reserved":0,"allocated":336166912,"data_stored":335974871,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":439021267},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,8,6,5,3,0,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986235,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":568600,"kb_used_data":270344,"kb_used_omap":4,"kb_used_meta":298235,"kb_avail":93803240,"statfs":{"total":96636764160,"available":96054517760,"internally_reserved":0,"allocated":276832256,"data_stored":276640215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":305393362},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,4,0,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673041,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":763592,"kb_used_data":327012,"kb_used_omap":1,"kb_used_meta":436542,"kb_avail":93608248,"statfs":{"total":96636764160,"available":95854845952,"internally_reserved":0,"allocated":334860288,"data_stored":334668247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":447019461},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,4,4,1,0,0,0,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182472,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":764268,"kb_used_data":327244,"kb_used_omap":4,"kb_used_meta":436987,"kb_avail":93607572,"statfs":{"total":96636764160,"available":95854153728,"internally_reserved":0,"allocated":335097856,"data_stored":334905815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":447475405},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,3,1,6,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":334577664,"data_stored":334577664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":276525056,"data_stored":276525056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":334315520,"data_stored":334315520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":334766080,"data_stored":334766080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:39:49.861 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:39:50.469 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:39:50.470 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:39:50.500 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":222,"stamp":"2024-09-05T16:39:48.918661+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":669404216,"num_objects":81720,"num_object_clones":0,"num_object_copies":326880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32172,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81720,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":117517,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1761,"num_bytes_recovered":13033472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":123242,"ondisk_log_size":123242,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2853508,"kb_used_data":1252888,"kb_used_omap":14,"kb_used_meta":1600497,"kb_avail":374633852,"statfs":{"total":386547056640,"available":383625064448,"internally_reserved":0,"allocated":1282957312,"data_stored":1282189148,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1638909495},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,1,10,13,21,11,3,0,1,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22380544,"num_objects":-2732,"num_object_clones":0,"num_object_copies":-10928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2732,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2732,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":223,"num_bytes_recovered":1523712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004282"},"pg_stats":[{"pgid":"1.f","version":"35'7788","reported_seq":7861,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:46.541706+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:39:46.541706+0000","last_peered":"2024-09-05T16:39:46.541706+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:39:46.541706+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7788,"log_dups_size":0,"ondisk_log_size":7788,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41910272,"num_objects":5116,"num_object_clones":0,"num_object_copies":20464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2249,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5116,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7788,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2867}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'7604","reported_seq":7690,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:46.921403+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:39:46.921403+0000","last_peered":"2024-09-05T16:39:46.921403+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:39:46.921403+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7604,"log_dups_size":0,"ondisk_log_size":7604,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40547488,"num_objects":4950,"num_object_clones":0,"num_object_copies":19800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4950,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7602,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2692}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'7692","reported_seq":7785,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:47.128641+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:39:47.128641+0000","last_peered":"2024-09-05T16:39:47.128641+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:39:47.128641+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7692,"log_dups_size":0,"ondisk_log_size":7692,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41450088,"num_objects":5061,"num_object_clones":0,"num_object_copies":20244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2245,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5061,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7691,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2817}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'7774","reported_seq":12498,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-05T16:39:46.038144+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:39:46.038144+0000","last_peered":"2024-09-05T16:39:46.038144+0000","last_clean":"2024-09-05T16:39:46.038144+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:39:46.038144+0000","last_undegraded":"2024-09-05T16:39:46.038144+0000","last_fullsized":"2024-09-05T16:39:46.038144+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":7774,"log_dups_size":0,"ondisk_log_size":7774,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42270720,"num_objects":5160,"num_object_clones":0,"num_object_copies":20640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5160,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2074,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'7560","reported_seq":7641,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:46.648021+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:39:46.648021+0000","last_peered":"2024-09-05T16:39:46.648021+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:39:46.648021+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7560,"log_dups_size":0,"ondisk_log_size":7560,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41243808,"num_objects":5035,"num_object_clones":0,"num_object_copies":20140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2153,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5035,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7558,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2884}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'7627","reported_seq":7706,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:46.656035+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:39:46.656035+0000","last_peered":"2024-09-05T16:39:46.656035+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:39:46.656035+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7627,"log_dups_size":0,"ondisk_log_size":7627,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42147840,"num_objects":5145,"num_object_clones":0,"num_object_copies":20580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5145,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7627,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2911}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'7531","reported_seq":7615,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:46.662980+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:39:46.662980+0000","last_peered":"2024-09-05T16:39:46.662980+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:39:46.662980+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7531,"log_dups_size":0,"ondisk_log_size":7531,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42030240,"num_objects":5131,"num_object_clones":0,"num_object_copies":20524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2142,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5131,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7529,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2991}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'7652","reported_seq":7731,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:46.867540+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:39:46.867540+0000","last_peered":"2024-09-05T16:39:46.867540+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:39:46.867540+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7652,"log_dups_size":0,"ondisk_log_size":7652,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41997472,"num_objects":5127,"num_object_clones":0,"num_object_copies":20508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5127,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7650,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2893}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'7681","reported_seq":7758,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:46.359086+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:39:46.359086+0000","last_peered":"2024-09-05T16:39:46.359086+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:39:46.359086+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7681,"log_dups_size":0,"ondisk_log_size":7681,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41716032,"num_objects":5093,"num_object_clones":0,"num_object_copies":20372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2256,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5093,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7678,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2840}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'7720","reported_seq":7820,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:47.130409+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:39:47.130409+0000","last_peered":"2024-09-05T16:39:47.130409+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:39:47.130409+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7720,"log_dups_size":0,"ondisk_log_size":7720,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41694368,"num_objects":5090,"num_object_clones":0,"num_object_copies":20360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2255,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5090,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7718,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2837}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'7924","reported_seq":8018,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:46.647764+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:39:46.647764+0000","last_peered":"2024-09-05T16:39:46.647764+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:39:46.647764+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7924,"log_dups_size":0,"ondisk_log_size":7924,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42585840,"num_objects":5199,"num_object_clones":0,"num_object_copies":20796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2297,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5199,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7921,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2905}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'7650","reported_seq":7717,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:46.702468+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:39:46.702468+0000","last_peered":"2024-09-05T16:39:46.702468+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:39:46.702468+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7650,"log_dups_size":0,"ondisk_log_size":7650,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41851472,"num_objects":5109,"num_object_clones":0,"num_object_copies":20436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5109,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7649,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2858}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'7744","reported_seq":11008,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:47.137043+0000","last_change":"2024-09-05T16:37:30.634570+0000","last_active":"2024-09-05T16:39:47.137043+0000","last_peered":"2024-09-05T16:39:47.137043+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:39:47.137043+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7744,"log_dups_size":0,"ondisk_log_size":7744,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41491024,"num_objects":5065,"num_object_clones":0,"num_object_copies":20260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":703,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5065,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7743,"num_write_kb":51231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1593,"num_bytes_recovered":13033472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4362}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'7906","reported_seq":7989,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:46.563901+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:39:46.563901+0000","last_peered":"2024-09-05T16:39:46.563901+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:39:46.563901+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7906,"log_dups_size":0,"ondisk_log_size":7906,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42759328,"num_objects":5220,"num_object_clones":0,"num_object_copies":20880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2345,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5220,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7904,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2877}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'7712","reported_seq":7807,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:47.077795+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:39:47.077795+0000","last_peered":"2024-09-05T16:39:47.077795+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:39:47.077795+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7712,"log_dups_size":0,"ondisk_log_size":7712,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41948320,"num_objects":5121,"num_object_clones":0,"num_object_copies":20484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5121,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7710,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2807}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'7677","reported_seq":7766,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:46.717897+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:39:46.717897+0000","last_peered":"2024-09-05T16:39:46.717897+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:39:46.717897+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7677,"log_dups_size":0,"ondisk_log_size":7677,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41759904,"num_objects":5098,"num_object_clones":0,"num_object_copies":20392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2229,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5098,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7675,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2871}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":669404216,"num_objects":81720,"num_object_clones":0,"num_object_copies":326880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32172,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81720,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":117517,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1761,"num_bytes_recovered":13033472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1280184320,"data_stored":1280184320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":123242,"ondisk_log_size":123242,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084358,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":757048,"kb_used_data":328288,"kb_used_omap":4,"kb_used_meta":428731,"kb_avail":93614792,"statfs":{"total":96636764160,"available":95861547008,"internally_reserved":0,"allocated":336166912,"data_stored":335974871,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":439021267},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,8,6,5,3,0,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986235,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":568600,"kb_used_data":270344,"kb_used_omap":4,"kb_used_meta":298235,"kb_avail":93803240,"statfs":{"total":96636764160,"available":96054517760,"internally_reserved":0,"allocated":276832256,"data_stored":276640215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":305393362},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,4,0,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673041,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":763592,"kb_used_data":327012,"kb_used_omap":1,"kb_used_meta":436542,"kb_avail":93608248,"statfs":{"total":96636764160,"available":95854845952,"internally_reserved":0,"allocated":334860288,"data_stored":334668247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":447019461},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,4,4,1,0,0,0,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182472,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":764268,"kb_used_data":327244,"kb_used_omap":4,"kb_used_meta":436987,"kb_avail":93607572,"statfs":{"total":96636764160,"available":95854153728,"internally_reserved":0,"allocated":335097856,"data_stored":334905815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":447475405},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,3,1,6,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":334577664,"data_stored":334577664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":276525056,"data_stored":276525056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":334315520,"data_stored":334315520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":334766080,"data_stored":334766080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:39:50.501 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-05T16:39:50.501 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'7788", 'reported_seq': 7861, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:46.541706+0000', 'last_change': '2024-09-05T16:34:56.833497+0000', 'last_active': '2024-09-05T16:39:46.541706+0000', 'last_peered': '2024-09-05T16:39:46.541706+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.828289+0000', 'last_became_peered': '2024-09-05T16:34:56.828289+0000', 'last_unstale': '2024-09-05T16:39:46.541706+0000', 'last_undegraded': '2024-09-05T16:34:56.612160+0000', 'last_fullsized': '2024-09-05T16:34:56.557632+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7788, 'log_dups_size': 0, 'ondisk_log_size': 7788, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41910272, 'num_objects': 5116, 'num_object_clones': 0, 'num_object_copies': 20464, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2249, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5116, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7788, 'num_write_kb': 51616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2867}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:39:50.501 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-05T16:39:50.501 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "35'7604", 'reported_seq': 7690, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:46.921403+0000', 'last_change': '2024-09-05T16:34:56.825031+0000', 'last_active': '2024-09-05T16:39:46.921403+0000', 'last_peered': '2024-09-05T16:39:46.921403+0000', 'last_clean': '2024-09-05T16:33:53.200762+0000', 'last_became_active': '2024-09-05T16:34:56.820292+0000', 'last_became_peered': '2024-09-05T16:34:56.820292+0000', 'last_unstale': '2024-09-05T16:39:46.921403+0000', 'last_undegraded': '2024-09-05T16:34:56.605776+0000', 'last_fullsized': '2024-09-05T16:34:56.559383+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7604, 'log_dups_size': 0, 'ondisk_log_size': 7604, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40547488, 'num_objects': 4950, 'num_object_clones': 0, 'num_object_copies': 19800, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2260, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4950, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7602, 'num_write_kb': 50212, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2692}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:39:50.502 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:39:50.502 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'7692", 'reported_seq': 7785, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:47.128641+0000', 'last_change': '2024-09-05T16:34:56.867900+0000', 'last_active': '2024-09-05T16:39:47.128641+0000', 'last_peered': '2024-09-05T16:39:47.128641+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:39:47.128641+0000', 'last_undegraded': '2024-09-05T16:34:56.590972+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7692, 'log_dups_size': 0, 'ondisk_log_size': 7692, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41450088, 'num_objects': 5061, 'num_object_clones': 0, 'num_object_copies': 20244, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2245, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5061, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 7691, 'num_write_kb': 50999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2817}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:39:50.502 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:39:50.502 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'7560", 'reported_seq': 7641, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:46.648021+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:39:46.648021+0000', 'last_peered': '2024-09-05T16:39:46.648021+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:39:46.648021+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7560, 'log_dups_size': 0, 'ondisk_log_size': 7560, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41243808, 'num_objects': 5035, 'num_object_clones': 0, 'num_object_copies': 20140, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2153, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5035, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7558, 'num_write_kb': 50373, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2884}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:39:50.502 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:39:50.502 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'7627", 'reported_seq': 7706, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:46.656035+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:39:46.656035+0000', 'last_peered': '2024-09-05T16:39:46.656035+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:39:46.656035+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7627, 'log_dups_size': 0, 'ondisk_log_size': 7627, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42147840, 'num_objects': 5145, 'num_object_clones': 0, 'num_object_copies': 20580, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2234, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5145, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7627, 'num_write_kb': 51088, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 16, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2911}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:39:50.502 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:39:50.502 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'7531", 'reported_seq': 7615, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:46.662980+0000', 'last_change': '2024-09-05T16:34:56.903436+0000', 'last_active': '2024-09-05T16:39:46.662980+0000', 'last_peered': '2024-09-05T16:39:46.662980+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:39:46.662980+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7531, 'log_dups_size': 0, 'ondisk_log_size': 7531, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42030240, 'num_objects': 5131, 'num_object_clones': 0, 'num_object_copies': 20524, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2142, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5131, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7529, 'num_write_kb': 50636, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2991}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:39:50.502 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-05T16:39:50.502 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "35'7652", 'reported_seq': 7731, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:46.867540+0000', 'last_change': '2024-09-05T16:34:56.883319+0000', 'last_active': '2024-09-05T16:39:46.867540+0000', 'last_peered': '2024-09-05T16:39:46.867540+0000', 'last_clean': '2024-09-05T16:33:53.505305+0000', 'last_became_active': '2024-09-05T16:34:56.873389+0000', 'last_became_peered': '2024-09-05T16:34:56.873389+0000', 'last_unstale': '2024-09-05T16:39:46.867540+0000', 'last_undegraded': '2024-09-05T16:34:56.602567+0000', 'last_fullsized': '2024-09-05T16:34:56.559958+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7652, 'log_dups_size': 0, 'ondisk_log_size': 7652, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41997472, 'num_objects': 5127, 'num_object_clones': 0, 'num_object_copies': 20508, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2236, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5127, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7650, 'num_write_kb': 51108, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2893}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:39:50.503 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:39:50.503 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'7681", 'reported_seq': 7758, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:46.359086+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:39:46.359086+0000', 'last_peered': '2024-09-05T16:39:46.359086+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:39:46.359086+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7681, 'log_dups_size': 0, 'ondisk_log_size': 7681, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41716032, 'num_objects': 5093, 'num_object_clones': 0, 'num_object_copies': 20372, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2256, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5093, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7678, 'num_write_kb': 51081, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2840}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:39:50.503 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:39:50.503 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'7720", 'reported_seq': 7820, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:47.130409+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:39:47.130409+0000', 'last_peered': '2024-09-05T16:39:47.130409+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:39:47.130409+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7720, 'log_dups_size': 0, 'ondisk_log_size': 7720, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41694368, 'num_objects': 5090, 'num_object_clones': 0, 'num_object_copies': 20360, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2255, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5090, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7718, 'num_write_kb': 51230, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2837}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:39:50.503 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:39:50.503 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'7924", 'reported_seq': 8018, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:46.647764+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:39:46.647764+0000', 'last_peered': '2024-09-05T16:39:46.647764+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:39:46.647764+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7924, 'log_dups_size': 0, 'ondisk_log_size': 7924, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42585840, 'num_objects': 5199, 'num_object_clones': 0, 'num_object_copies': 20796, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2297, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5199, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7921, 'num_write_kb': 52476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2905}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:39:50.503 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:39:50.503 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'7650", 'reported_seq': 7717, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:46.702468+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:39:46.702468+0000', 'last_peered': '2024-09-05T16:39:46.702468+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:39:46.702468+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7650, 'log_dups_size': 0, 'ondisk_log_size': 7650, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41851472, 'num_objects': 5109, 'num_object_clones': 0, 'num_object_copies': 20436, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2252, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5109, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7649, 'num_write_kb': 51031, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2858}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:39:50.503 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-05T16:39:50.503 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'7744", 'reported_seq': 11008, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:47.137043+0000', 'last_change': '2024-09-05T16:37:30.634570+0000', 'last_active': '2024-09-05T16:39:47.137043+0000', 'last_peered': '2024-09-05T16:39:47.137043+0000', 'last_clean': '2024-09-05T16:33:45.082039+0000', 'last_became_active': '2024-09-05T16:34:56.792386+0000', 'last_became_peered': '2024-09-05T16:34:56.792386+0000', 'last_unstale': '2024-09-05T16:39:47.137043+0000', 'last_undegraded': '2024-09-05T16:34:56.600595+0000', 'last_fullsized': '2024-09-05T16:34:56.558203+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7744, 'log_dups_size': 0, 'ondisk_log_size': 7744, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41491024, 'num_objects': 5065, 'num_object_clones': 0, 'num_object_copies': 20260, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 703, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5065, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7743, 'num_write_kb': 51231, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1593, 'num_bytes_recovered': 13033472, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 4362}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:39:50.504 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:39:50.504 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'7906", 'reported_seq': 7989, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:46.563901+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:39:46.563901+0000', 'last_peered': '2024-09-05T16:39:46.563901+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:39:46.563901+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7906, 'log_dups_size': 0, 'ondisk_log_size': 7906, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42759328, 'num_objects': 5220, 'num_object_clones': 0, 'num_object_copies': 20880, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2345, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5220, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7904, 'num_write_kb': 52500, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 16, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2877}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:39:50.504 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:39:50.504 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'7712", 'reported_seq': 7807, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:47.077795+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:39:47.077795+0000', 'last_peered': '2024-09-05T16:39:47.077795+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:39:47.077795+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7712, 'log_dups_size': 0, 'ondisk_log_size': 7712, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41948320, 'num_objects': 5121, 'num_object_clones': 0, 'num_object_copies': 20484, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5121, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7710, 'num_write_kb': 51325, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 8, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2807}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:39:50.504 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:39:50.504 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'7677", 'reported_seq': 7766, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:46.717897+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:39:46.717897+0000', 'last_peered': '2024-09-05T16:39:46.717897+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:39:46.717897+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7677, 'log_dups_size': 0, 'ondisk_log_size': 7677, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41759904, 'num_objects': 5098, 'num_object_clones': 0, 'num_object_copies': 20392, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2229, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5098, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7675, 'num_write_kb': 51092, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2871}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:39:50.504 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:39:51.259 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:39:51.259 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 32172/326880 objects degraded (9.842%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":394,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":81720,"data_bytes":669404216,"bytes_used":2921992192,"bytes_avail":383625064448,"bytes_total":386547056640,"degraded_objects":32172,"degraded_total":326880,"degraded_ratio":0.098421439060205576,"recovering_objects_per_sec":12,"recovering_bytes_per_sec":78478,"recovering_keys_per_sec":0,"num_objects_recovered":150,"num_bytes_recovered":942080,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":201},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:39:51.287 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:39:51.287 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:39:51.910 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:39:51.910 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:39:51.933 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":223,"stamp":"2024-09-05T16:39:50.919492+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":669404216,"num_objects":81720,"num_object_clones":0,"num_object_copies":326880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32172,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81720,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":117517,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1761,"num_bytes_recovered":13033472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":123242,"ondisk_log_size":123242,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2853508,"kb_used_data":1252888,"kb_used_omap":14,"kb_used_meta":1600497,"kb_avail":374633852,"statfs":{"total":386547056640,"available":383625064448,"internally_reserved":0,"allocated":1282957312,"data_stored":1282189148,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1638909495},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,1,10,13,21,11,3,0,1,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19824640,"num_objects":-2420,"num_object_clones":0,"num_object_copies":-9680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-150,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2420,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2420,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":150,"num_bytes_recovered":942080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004238"},"pg_stats":[{"pgid":"1.f","version":"35'7788","reported_seq":7861,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:46.541706+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:39:46.541706+0000","last_peered":"2024-09-05T16:39:46.541706+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:39:46.541706+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7788,"log_dups_size":0,"ondisk_log_size":7788,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41910272,"num_objects":5116,"num_object_clones":0,"num_object_copies":20464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2249,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5116,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7788,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2867}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'7604","reported_seq":7690,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:46.921403+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:39:46.921403+0000","last_peered":"2024-09-05T16:39:46.921403+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:39:46.921403+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7604,"log_dups_size":0,"ondisk_log_size":7604,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40547488,"num_objects":4950,"num_object_clones":0,"num_object_copies":19800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4950,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7602,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2692}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'7692","reported_seq":7785,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:47.128641+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:39:47.128641+0000","last_peered":"2024-09-05T16:39:47.128641+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:39:47.128641+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7692,"log_dups_size":0,"ondisk_log_size":7692,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41450088,"num_objects":5061,"num_object_clones":0,"num_object_copies":20244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2245,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5061,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7691,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2817}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'7774","reported_seq":12498,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-05T16:39:46.038144+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:39:46.038144+0000","last_peered":"2024-09-05T16:39:46.038144+0000","last_clean":"2024-09-05T16:39:46.038144+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:39:46.038144+0000","last_undegraded":"2024-09-05T16:39:46.038144+0000","last_fullsized":"2024-09-05T16:39:46.038144+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":7774,"log_dups_size":0,"ondisk_log_size":7774,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42270720,"num_objects":5160,"num_object_clones":0,"num_object_copies":20640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5160,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2074,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'7560","reported_seq":7641,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:46.648021+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:39:46.648021+0000","last_peered":"2024-09-05T16:39:46.648021+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:39:46.648021+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7560,"log_dups_size":0,"ondisk_log_size":7560,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41243808,"num_objects":5035,"num_object_clones":0,"num_object_copies":20140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2153,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5035,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7558,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2884}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'7627","reported_seq":7706,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:46.656035+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:39:46.656035+0000","last_peered":"2024-09-05T16:39:46.656035+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:39:46.656035+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7627,"log_dups_size":0,"ondisk_log_size":7627,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42147840,"num_objects":5145,"num_object_clones":0,"num_object_copies":20580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5145,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7627,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2911}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'7531","reported_seq":7615,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:46.662980+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:39:46.662980+0000","last_peered":"2024-09-05T16:39:46.662980+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:39:46.662980+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7531,"log_dups_size":0,"ondisk_log_size":7531,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42030240,"num_objects":5131,"num_object_clones":0,"num_object_copies":20524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2142,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5131,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7529,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2991}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'7652","reported_seq":7731,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:46.867540+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:39:46.867540+0000","last_peered":"2024-09-05T16:39:46.867540+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:39:46.867540+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7652,"log_dups_size":0,"ondisk_log_size":7652,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41997472,"num_objects":5127,"num_object_clones":0,"num_object_copies":20508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5127,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7650,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2893}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'7681","reported_seq":7758,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:46.359086+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:39:46.359086+0000","last_peered":"2024-09-05T16:39:46.359086+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:39:46.359086+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7681,"log_dups_size":0,"ondisk_log_size":7681,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41716032,"num_objects":5093,"num_object_clones":0,"num_object_copies":20372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2256,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5093,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7678,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2840}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'7720","reported_seq":7820,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:47.130409+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:39:47.130409+0000","last_peered":"2024-09-05T16:39:47.130409+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:39:47.130409+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7720,"log_dups_size":0,"ondisk_log_size":7720,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41694368,"num_objects":5090,"num_object_clones":0,"num_object_copies":20360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2255,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5090,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7718,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2837}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'7924","reported_seq":8018,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:46.647764+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:39:46.647764+0000","last_peered":"2024-09-05T16:39:46.647764+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:39:46.647764+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7924,"log_dups_size":0,"ondisk_log_size":7924,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42585840,"num_objects":5199,"num_object_clones":0,"num_object_copies":20796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2297,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5199,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7921,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2905}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'7650","reported_seq":7717,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:46.702468+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:39:46.702468+0000","last_peered":"2024-09-05T16:39:46.702468+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:39:46.702468+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7650,"log_dups_size":0,"ondisk_log_size":7650,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41851472,"num_objects":5109,"num_object_clones":0,"num_object_copies":20436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5109,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7649,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2858}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'7744","reported_seq":11008,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:47.137043+0000","last_change":"2024-09-05T16:37:30.634570+0000","last_active":"2024-09-05T16:39:47.137043+0000","last_peered":"2024-09-05T16:39:47.137043+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:39:47.137043+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7744,"log_dups_size":0,"ondisk_log_size":7744,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41491024,"num_objects":5065,"num_object_clones":0,"num_object_copies":20260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":703,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5065,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7743,"num_write_kb":51231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1593,"num_bytes_recovered":13033472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4362}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'7906","reported_seq":7989,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:46.563901+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:39:46.563901+0000","last_peered":"2024-09-05T16:39:46.563901+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:39:46.563901+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7906,"log_dups_size":0,"ondisk_log_size":7906,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42759328,"num_objects":5220,"num_object_clones":0,"num_object_copies":20880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2345,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5220,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7904,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2877}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'7712","reported_seq":7807,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:47.077795+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:39:47.077795+0000","last_peered":"2024-09-05T16:39:47.077795+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:39:47.077795+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7712,"log_dups_size":0,"ondisk_log_size":7712,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41948320,"num_objects":5121,"num_object_clones":0,"num_object_copies":20484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5121,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7710,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2807}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'7677","reported_seq":7766,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:46.717897+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:39:46.717897+0000","last_peered":"2024-09-05T16:39:46.717897+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:39:46.717897+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7677,"log_dups_size":0,"ondisk_log_size":7677,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41759904,"num_objects":5098,"num_object_clones":0,"num_object_copies":20392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2229,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5098,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7675,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2871}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":669404216,"num_objects":81720,"num_object_clones":0,"num_object_copies":326880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32172,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81720,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":117517,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1761,"num_bytes_recovered":13033472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1280184320,"data_stored":1280184320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":123242,"ondisk_log_size":123242,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084358,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":757048,"kb_used_data":328288,"kb_used_omap":4,"kb_used_meta":428731,"kb_avail":93614792,"statfs":{"total":96636764160,"available":95861547008,"internally_reserved":0,"allocated":336166912,"data_stored":335974871,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":439021267},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,8,6,5,3,0,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986235,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":568600,"kb_used_data":270344,"kb_used_omap":4,"kb_used_meta":298235,"kb_avail":93803240,"statfs":{"total":96636764160,"available":96054517760,"internally_reserved":0,"allocated":276832256,"data_stored":276640215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":305393362},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,4,0,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673041,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":763592,"kb_used_data":327012,"kb_used_omap":1,"kb_used_meta":436542,"kb_avail":93608248,"statfs":{"total":96636764160,"available":95854845952,"internally_reserved":0,"allocated":334860288,"data_stored":334668247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":447019461},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,4,4,1,0,0,0,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182472,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":764268,"kb_used_data":327244,"kb_used_omap":4,"kb_used_meta":436987,"kb_avail":93607572,"statfs":{"total":96636764160,"available":95854153728,"internally_reserved":0,"allocated":335097856,"data_stored":334905815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":447475405},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,3,1,6,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":334577664,"data_stored":334577664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":276525056,"data_stored":276525056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":334315520,"data_stored":334315520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":334766080,"data_stored":334766080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:39:54.935 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:39:55.564 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:39:55.565 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:39:55.587 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":225,"stamp":"2024-09-05T16:39:54.920738+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":659377208,"num_objects":80496,"num_object_clones":0,"num_object_copies":321984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32100,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80496,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":118741,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1833,"num_bytes_recovered":13598720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":124466,"ondisk_log_size":124466,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2824972,"kb_used_data":1236860,"kb_used_omap":14,"kb_used_meta":1588017,"kb_avail":374662388,"statfs":{"total":386547056640,"available":383654285312,"internally_reserved":0,"allocated":1266544640,"data_stored":1265776476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1626129975},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1,8,11,11,15,0,0,0,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19906560,"num_objects":-2430,"num_object_clones":0,"num_object_copies":-9720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-156,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2430,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2430,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":156,"num_bytes_recovered":1056768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004083"},"pg_stats":[{"pgid":"1.f","version":"35'7857","reported_seq":7930,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:51.744446+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:39:51.744446+0000","last_peered":"2024-09-05T16:39:51.744446+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:39:51.744446+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7857,"log_dups_size":0,"ondisk_log_size":7857,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41345024,"num_objects":5047,"num_object_clones":0,"num_object_copies":20188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2249,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5047,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7857,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2798}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'7688","reported_seq":7775,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:51.896703+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:39:51.896703+0000","last_peered":"2024-09-05T16:39:51.896703+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:39:51.896703+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7688,"log_dups_size":0,"ondisk_log_size":7688,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39859360,"num_objects":4866,"num_object_clones":0,"num_object_copies":19464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4866,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7686,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2608}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'7760","reported_seq":7853,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:52.102259+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:39:52.102259+0000","last_peered":"2024-09-05T16:39:52.102259+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:39:52.102259+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7760,"log_dups_size":0,"ondisk_log_size":7760,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40893032,"num_objects":4993,"num_object_clones":0,"num_object_copies":19972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2245,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4993,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7759,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2749}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'7863","reported_seq":12587,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-05T16:39:50.957074+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:39:50.957074+0000","last_peered":"2024-09-05T16:39:50.957074+0000","last_clean":"2024-09-05T16:39:50.957074+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:39:50.957074+0000","last_undegraded":"2024-09-05T16:39:50.957074+0000","last_fullsized":"2024-09-05T16:39:50.957074+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":7863,"log_dups_size":0,"ondisk_log_size":7863,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41541632,"num_objects":5071,"num_object_clones":0,"num_object_copies":20284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5071,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2163,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'7626","reported_seq":7707,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:51.637253+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:39:51.637253+0000","last_peered":"2024-09-05T16:39:51.637253+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:39:51.637253+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7626,"log_dups_size":0,"ondisk_log_size":7626,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40703136,"num_objects":4969,"num_object_clones":0,"num_object_copies":19876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2153,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4969,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7624,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2818}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'7715","reported_seq":7793,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:51.685790+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:39:51.685790+0000","last_peered":"2024-09-05T16:39:51.685790+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:39:51.685790+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7715,"log_dups_size":0,"ondisk_log_size":7715,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41426944,"num_objects":5057,"num_object_clones":0,"num_object_copies":20228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5057,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7715,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2823}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'7608","reported_seq":7696,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:51.845435+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:39:51.845435+0000","last_peered":"2024-09-05T16:39:51.845435+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:39:51.845435+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7608,"log_dups_size":0,"ondisk_log_size":7608,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41399456,"num_objects":5054,"num_object_clones":0,"num_object_copies":20216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2140,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5054,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7606,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2916}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'7723","reported_seq":7802,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:51.868194+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:39:51.868194+0000","last_peered":"2024-09-05T16:39:51.868194+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:39:51.868194+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7723,"log_dups_size":0,"ondisk_log_size":7723,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41415840,"num_objects":5056,"num_object_clones":0,"num_object_copies":20224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5056,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7721,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2822}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'7772","reported_seq":7849,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:51.555889+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:39:51.555889+0000","last_peered":"2024-09-05T16:39:51.555889+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:39:51.555889+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7772,"log_dups_size":0,"ondisk_log_size":7772,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40970560,"num_objects":5002,"num_object_clones":0,"num_object_copies":20008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2256,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5002,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7769,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2749}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'7799","reported_seq":7899,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:52.042243+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:39:52.042243+0000","last_peered":"2024-09-05T16:39:52.042243+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:39:52.042243+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7799,"log_dups_size":0,"ondisk_log_size":7799,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41047200,"num_objects":5011,"num_object_clones":0,"num_object_copies":20044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2255,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5011,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7797,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2758}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'7993","reported_seq":8088,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:51.730626+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:39:51.730626+0000","last_peered":"2024-09-05T16:39:51.730626+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:39:51.730626+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7993,"log_dups_size":0,"ondisk_log_size":7993,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42020592,"num_objects":5130,"num_object_clones":0,"num_object_copies":20520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2297,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5130,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7990,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2836}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'7733","reported_seq":7800,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:51.690854+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:39:51.690854+0000","last_peered":"2024-09-05T16:39:51.690854+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:39:51.690854+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7733,"log_dups_size":0,"ondisk_log_size":7733,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41171536,"num_objects":5026,"num_object_clones":0,"num_object_copies":20104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5026,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7732,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2775}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'7807","reported_seq":11208,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:52.191537+0000","last_change":"2024-09-05T16:37:30.634570+0000","last_active":"2024-09-05T16:39:52.191537+0000","last_peered":"2024-09-05T16:39:52.191537+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:39:52.191537+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7807,"log_dups_size":0,"ondisk_log_size":7807,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40974928,"num_objects":5002,"num_object_clones":0,"num_object_copies":20008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":634,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5002,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7806,"num_write_kb":51231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1662,"num_bytes_recovered":13598720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4368}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'7979","reported_seq":8064,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:51.656075+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:39:51.656075+0000","last_peered":"2024-09-05T16:39:51.656075+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:39:51.656075+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7979,"log_dups_size":0,"ondisk_log_size":7979,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42161312,"num_objects":5147,"num_object_clones":0,"num_object_copies":20588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2345,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5147,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7977,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2804}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'7783","reported_seq":7880,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:52.000660+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:39:52.000660+0000","last_peered":"2024-09-05T16:39:52.000660+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:39:52.000660+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7783,"log_dups_size":0,"ondisk_log_size":7783,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41366688,"num_objects":5050,"num_object_clones":0,"num_object_copies":20200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5050,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7781,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2737}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'7760","reported_seq":7849,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:51.710890+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:39:51.710890+0000","last_peered":"2024-09-05T16:39:51.710890+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:39:51.710890+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7760,"log_dups_size":0,"ondisk_log_size":7760,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41079968,"num_objects":5015,"num_object_clones":0,"num_object_copies":20060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2229,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5015,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7758,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2788}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":659377208,"num_objects":80496,"num_object_clones":0,"num_object_copies":321984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32100,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80496,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":118741,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1833,"num_bytes_recovered":13598720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1264807936,"data_stored":1264807936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":124466,"ondisk_log_size":124466,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084359,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":759392,"kb_used_data":322308,"kb_used_omap":4,"kb_used_meta":437051,"kb_avail":93612448,"statfs":{"total":96636764160,"available":95859146752,"internally_reserved":0,"allocated":330043392,"data_stored":329851351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":447540947},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,4,3,1,3,0,0,0,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986236,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":572448,"kb_used_data":270032,"kb_used_omap":4,"kb_used_meta":302395,"kb_avail":93799392,"statfs":{"total":96636764160,"available":96050577408,"internally_reserved":0,"allocated":276512768,"data_stored":276320727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":309653202},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,4,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673042,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":746236,"kb_used_data":322156,"kb_used_omap":1,"kb_used_meta":424062,"kb_avail":93625604,"statfs":{"total":96636764160,"available":95872618496,"internally_reserved":0,"allocated":329887744,"data_stored":329695703,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":434239941},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,7,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182473,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":746896,"kb_used_data":322364,"kb_used_omap":4,"kb_used_meta":424507,"kb_avail":93624944,"statfs":{"total":96636764160,"available":95871942656,"internally_reserved":0,"allocated":330100736,"data_stored":329908695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":434695885},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,3,1,6,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":329580544,"data_stored":329580544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":276213760,"data_stored":276213760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":329302016,"data_stored":329302016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":329711616,"data_stored":329711616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:39:55.588 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:39:56.201 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:39:56.202 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:39:56.224 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":225,"stamp":"2024-09-05T16:39:54.920738+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":659377208,"num_objects":80496,"num_object_clones":0,"num_object_copies":321984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32100,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80496,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":118741,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1833,"num_bytes_recovered":13598720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":124466,"ondisk_log_size":124466,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2824972,"kb_used_data":1236860,"kb_used_omap":14,"kb_used_meta":1588017,"kb_avail":374662388,"statfs":{"total":386547056640,"available":383654285312,"internally_reserved":0,"allocated":1266544640,"data_stored":1265776476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1626129975},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1,8,11,11,15,0,0,0,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19906560,"num_objects":-2430,"num_object_clones":0,"num_object_copies":-9720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-156,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2430,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2430,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":156,"num_bytes_recovered":1056768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004083"},"pg_stats":[{"pgid":"1.f","version":"35'7857","reported_seq":7930,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:51.744446+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:39:51.744446+0000","last_peered":"2024-09-05T16:39:51.744446+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:39:51.744446+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7857,"log_dups_size":0,"ondisk_log_size":7857,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41345024,"num_objects":5047,"num_object_clones":0,"num_object_copies":20188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2249,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5047,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7857,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2798}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'7688","reported_seq":7775,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:51.896703+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:39:51.896703+0000","last_peered":"2024-09-05T16:39:51.896703+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:39:51.896703+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7688,"log_dups_size":0,"ondisk_log_size":7688,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39859360,"num_objects":4866,"num_object_clones":0,"num_object_copies":19464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4866,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7686,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2608}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'7760","reported_seq":7853,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:52.102259+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:39:52.102259+0000","last_peered":"2024-09-05T16:39:52.102259+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:39:52.102259+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7760,"log_dups_size":0,"ondisk_log_size":7760,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40893032,"num_objects":4993,"num_object_clones":0,"num_object_copies":19972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2245,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4993,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7759,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2749}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'7863","reported_seq":12587,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-05T16:39:50.957074+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:39:50.957074+0000","last_peered":"2024-09-05T16:39:50.957074+0000","last_clean":"2024-09-05T16:39:50.957074+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:39:50.957074+0000","last_undegraded":"2024-09-05T16:39:50.957074+0000","last_fullsized":"2024-09-05T16:39:50.957074+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":7863,"log_dups_size":0,"ondisk_log_size":7863,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41541632,"num_objects":5071,"num_object_clones":0,"num_object_copies":20284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5071,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2163,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'7626","reported_seq":7707,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:51.637253+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:39:51.637253+0000","last_peered":"2024-09-05T16:39:51.637253+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:39:51.637253+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7626,"log_dups_size":0,"ondisk_log_size":7626,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40703136,"num_objects":4969,"num_object_clones":0,"num_object_copies":19876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2153,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4969,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7624,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2818}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'7715","reported_seq":7793,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:51.685790+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:39:51.685790+0000","last_peered":"2024-09-05T16:39:51.685790+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:39:51.685790+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7715,"log_dups_size":0,"ondisk_log_size":7715,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41426944,"num_objects":5057,"num_object_clones":0,"num_object_copies":20228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5057,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7715,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2823}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'7608","reported_seq":7696,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:51.845435+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:39:51.845435+0000","last_peered":"2024-09-05T16:39:51.845435+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:39:51.845435+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7608,"log_dups_size":0,"ondisk_log_size":7608,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41399456,"num_objects":5054,"num_object_clones":0,"num_object_copies":20216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2140,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5054,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7606,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2916}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'7723","reported_seq":7802,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:51.868194+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:39:51.868194+0000","last_peered":"2024-09-05T16:39:51.868194+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:39:51.868194+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7723,"log_dups_size":0,"ondisk_log_size":7723,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41415840,"num_objects":5056,"num_object_clones":0,"num_object_copies":20224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5056,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7721,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2822}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'7772","reported_seq":7849,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:51.555889+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:39:51.555889+0000","last_peered":"2024-09-05T16:39:51.555889+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:39:51.555889+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7772,"log_dups_size":0,"ondisk_log_size":7772,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40970560,"num_objects":5002,"num_object_clones":0,"num_object_copies":20008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2256,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5002,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7769,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2749}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'7799","reported_seq":7899,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:52.042243+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:39:52.042243+0000","last_peered":"2024-09-05T16:39:52.042243+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:39:52.042243+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7799,"log_dups_size":0,"ondisk_log_size":7799,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41047200,"num_objects":5011,"num_object_clones":0,"num_object_copies":20044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2255,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5011,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7797,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2758}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'7993","reported_seq":8088,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:51.730626+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:39:51.730626+0000","last_peered":"2024-09-05T16:39:51.730626+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:39:51.730626+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7993,"log_dups_size":0,"ondisk_log_size":7993,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42020592,"num_objects":5130,"num_object_clones":0,"num_object_copies":20520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2297,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5130,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7990,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2836}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'7733","reported_seq":7800,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:51.690854+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:39:51.690854+0000","last_peered":"2024-09-05T16:39:51.690854+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:39:51.690854+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7733,"log_dups_size":0,"ondisk_log_size":7733,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41171536,"num_objects":5026,"num_object_clones":0,"num_object_copies":20104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5026,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7732,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2775}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'7807","reported_seq":11208,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:52.191537+0000","last_change":"2024-09-05T16:37:30.634570+0000","last_active":"2024-09-05T16:39:52.191537+0000","last_peered":"2024-09-05T16:39:52.191537+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:39:52.191537+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7807,"log_dups_size":0,"ondisk_log_size":7807,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40974928,"num_objects":5002,"num_object_clones":0,"num_object_copies":20008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":634,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5002,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7806,"num_write_kb":51231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1662,"num_bytes_recovered":13598720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4368}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'7979","reported_seq":8064,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:51.656075+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:39:51.656075+0000","last_peered":"2024-09-05T16:39:51.656075+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:39:51.656075+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7979,"log_dups_size":0,"ondisk_log_size":7979,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42161312,"num_objects":5147,"num_object_clones":0,"num_object_copies":20588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2345,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5147,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7977,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2804}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'7783","reported_seq":7880,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:52.000660+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:39:52.000660+0000","last_peered":"2024-09-05T16:39:52.000660+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:39:52.000660+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7783,"log_dups_size":0,"ondisk_log_size":7783,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41366688,"num_objects":5050,"num_object_clones":0,"num_object_copies":20200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5050,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7781,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2737}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'7760","reported_seq":7849,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:51.710890+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:39:51.710890+0000","last_peered":"2024-09-05T16:39:51.710890+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:39:51.710890+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7760,"log_dups_size":0,"ondisk_log_size":7760,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41079968,"num_objects":5015,"num_object_clones":0,"num_object_copies":20060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2229,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5015,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7758,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2788}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":659377208,"num_objects":80496,"num_object_clones":0,"num_object_copies":321984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32100,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80496,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":118741,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1833,"num_bytes_recovered":13598720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1264807936,"data_stored":1264807936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":124466,"ondisk_log_size":124466,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084359,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":759392,"kb_used_data":322308,"kb_used_omap":4,"kb_used_meta":437051,"kb_avail":93612448,"statfs":{"total":96636764160,"available":95859146752,"internally_reserved":0,"allocated":330043392,"data_stored":329851351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":447540947},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,4,3,1,3,0,0,0,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986236,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":572448,"kb_used_data":270032,"kb_used_omap":4,"kb_used_meta":302395,"kb_avail":93799392,"statfs":{"total":96636764160,"available":96050577408,"internally_reserved":0,"allocated":276512768,"data_stored":276320727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":309653202},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,4,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673042,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":746236,"kb_used_data":322156,"kb_used_omap":1,"kb_used_meta":424062,"kb_avail":93625604,"statfs":{"total":96636764160,"available":95872618496,"internally_reserved":0,"allocated":329887744,"data_stored":329695703,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":434239941},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,7,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182473,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":746896,"kb_used_data":322364,"kb_used_omap":4,"kb_used_meta":424507,"kb_avail":93624944,"statfs":{"total":96636764160,"available":95871942656,"internally_reserved":0,"allocated":330100736,"data_stored":329908695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":434695885},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,3,1,6,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":329580544,"data_stored":329580544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":276213760,"data_stored":276213760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":329302016,"data_stored":329302016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":329711616,"data_stored":329711616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:39:56.225 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-05T16:39:56.226 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'7857", 'reported_seq': 7930, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:51.744446+0000', 'last_change': '2024-09-05T16:34:56.833497+0000', 'last_active': '2024-09-05T16:39:51.744446+0000', 'last_peered': '2024-09-05T16:39:51.744446+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.828289+0000', 'last_became_peered': '2024-09-05T16:34:56.828289+0000', 'last_unstale': '2024-09-05T16:39:51.744446+0000', 'last_undegraded': '2024-09-05T16:34:56.612160+0000', 'last_fullsized': '2024-09-05T16:34:56.557632+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7857, 'log_dups_size': 0, 'ondisk_log_size': 7857, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41345024, 'num_objects': 5047, 'num_object_clones': 0, 'num_object_copies': 20188, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2249, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5047, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7857, 'num_write_kb': 51616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2798}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:39:56.226 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-05T16:39:56.226 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "35'7688", 'reported_seq': 7775, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:51.896703+0000', 'last_change': '2024-09-05T16:34:56.825031+0000', 'last_active': '2024-09-05T16:39:51.896703+0000', 'last_peered': '2024-09-05T16:39:51.896703+0000', 'last_clean': '2024-09-05T16:33:53.200762+0000', 'last_became_active': '2024-09-05T16:34:56.820292+0000', 'last_became_peered': '2024-09-05T16:34:56.820292+0000', 'last_unstale': '2024-09-05T16:39:51.896703+0000', 'last_undegraded': '2024-09-05T16:34:56.605776+0000', 'last_fullsized': '2024-09-05T16:34:56.559383+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7688, 'log_dups_size': 0, 'ondisk_log_size': 7688, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39859360, 'num_objects': 4866, 'num_object_clones': 0, 'num_object_copies': 19464, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2260, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4866, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7686, 'num_write_kb': 50212, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2608}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:39:56.226 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:39:56.226 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'7760", 'reported_seq': 7853, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:52.102259+0000', 'last_change': '2024-09-05T16:34:56.867900+0000', 'last_active': '2024-09-05T16:39:52.102259+0000', 'last_peered': '2024-09-05T16:39:52.102259+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:39:52.102259+0000', 'last_undegraded': '2024-09-05T16:34:56.590972+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7760, 'log_dups_size': 0, 'ondisk_log_size': 7760, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40893032, 'num_objects': 4993, 'num_object_clones': 0, 'num_object_copies': 19972, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2245, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4993, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 7759, 'num_write_kb': 50999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2749}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:39:56.226 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:39:56.226 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'7626", 'reported_seq': 7707, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:51.637253+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:39:51.637253+0000', 'last_peered': '2024-09-05T16:39:51.637253+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:39:51.637253+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7626, 'log_dups_size': 0, 'ondisk_log_size': 7626, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40703136, 'num_objects': 4969, 'num_object_clones': 0, 'num_object_copies': 19876, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2153, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4969, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7624, 'num_write_kb': 50373, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2818}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:39:56.226 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:39:56.226 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'7715", 'reported_seq': 7793, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:51.685790+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:39:51.685790+0000', 'last_peered': '2024-09-05T16:39:51.685790+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:39:51.685790+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7715, 'log_dups_size': 0, 'ondisk_log_size': 7715, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41426944, 'num_objects': 5057, 'num_object_clones': 0, 'num_object_copies': 20228, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2234, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5057, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7715, 'num_write_kb': 51088, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 16, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2823}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:39:56.227 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:39:56.227 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'7608", 'reported_seq': 7696, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:51.845435+0000', 'last_change': '2024-09-05T16:34:56.903436+0000', 'last_active': '2024-09-05T16:39:51.845435+0000', 'last_peered': '2024-09-05T16:39:51.845435+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:39:51.845435+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7608, 'log_dups_size': 0, 'ondisk_log_size': 7608, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41399456, 'num_objects': 5054, 'num_object_clones': 0, 'num_object_copies': 20216, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2140, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5054, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7606, 'num_write_kb': 50636, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2916}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:39:56.227 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-05T16:39:56.227 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "35'7723", 'reported_seq': 7802, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:51.868194+0000', 'last_change': '2024-09-05T16:34:56.883319+0000', 'last_active': '2024-09-05T16:39:51.868194+0000', 'last_peered': '2024-09-05T16:39:51.868194+0000', 'last_clean': '2024-09-05T16:33:53.505305+0000', 'last_became_active': '2024-09-05T16:34:56.873389+0000', 'last_became_peered': '2024-09-05T16:34:56.873389+0000', 'last_unstale': '2024-09-05T16:39:51.868194+0000', 'last_undegraded': '2024-09-05T16:34:56.602567+0000', 'last_fullsized': '2024-09-05T16:34:56.559958+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7723, 'log_dups_size': 0, 'ondisk_log_size': 7723, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41415840, 'num_objects': 5056, 'num_object_clones': 0, 'num_object_copies': 20224, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2236, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5056, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7721, 'num_write_kb': 51108, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2822}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:39:56.227 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:39:56.227 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'7772", 'reported_seq': 7849, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:51.555889+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:39:51.555889+0000', 'last_peered': '2024-09-05T16:39:51.555889+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:39:51.555889+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7772, 'log_dups_size': 0, 'ondisk_log_size': 7772, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40970560, 'num_objects': 5002, 'num_object_clones': 0, 'num_object_copies': 20008, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2256, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5002, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7769, 'num_write_kb': 51081, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2749}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:39:56.227 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:39:56.227 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'7799", 'reported_seq': 7899, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:52.042243+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:39:52.042243+0000', 'last_peered': '2024-09-05T16:39:52.042243+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:39:52.042243+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7799, 'log_dups_size': 0, 'ondisk_log_size': 7799, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41047200, 'num_objects': 5011, 'num_object_clones': 0, 'num_object_copies': 20044, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2255, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5011, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7797, 'num_write_kb': 51230, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2758}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:39:56.227 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:39:56.227 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'7993", 'reported_seq': 8088, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:51.730626+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:39:51.730626+0000', 'last_peered': '2024-09-05T16:39:51.730626+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:39:51.730626+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7993, 'log_dups_size': 0, 'ondisk_log_size': 7993, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42020592, 'num_objects': 5130, 'num_object_clones': 0, 'num_object_copies': 20520, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2297, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5130, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7990, 'num_write_kb': 52476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2836}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:39:56.227 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:39:56.227 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'7733", 'reported_seq': 7800, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:51.690854+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:39:51.690854+0000', 'last_peered': '2024-09-05T16:39:51.690854+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:39:51.690854+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7733, 'log_dups_size': 0, 'ondisk_log_size': 7733, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41171536, 'num_objects': 5026, 'num_object_clones': 0, 'num_object_copies': 20104, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2252, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5026, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7732, 'num_write_kb': 51031, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2775}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:39:56.228 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-05T16:39:56.228 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'7807", 'reported_seq': 11208, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:52.191537+0000', 'last_change': '2024-09-05T16:37:30.634570+0000', 'last_active': '2024-09-05T16:39:52.191537+0000', 'last_peered': '2024-09-05T16:39:52.191537+0000', 'last_clean': '2024-09-05T16:33:45.082039+0000', 'last_became_active': '2024-09-05T16:34:56.792386+0000', 'last_became_peered': '2024-09-05T16:34:56.792386+0000', 'last_unstale': '2024-09-05T16:39:52.191537+0000', 'last_undegraded': '2024-09-05T16:34:56.600595+0000', 'last_fullsized': '2024-09-05T16:34:56.558203+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7807, 'log_dups_size': 0, 'ondisk_log_size': 7807, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40974928, 'num_objects': 5002, 'num_object_clones': 0, 'num_object_copies': 20008, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 634, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5002, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7806, 'num_write_kb': 51231, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1662, 'num_bytes_recovered': 13598720, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 4368}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:39:56.228 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:39:56.228 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'7979", 'reported_seq': 8064, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:51.656075+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:39:51.656075+0000', 'last_peered': '2024-09-05T16:39:51.656075+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:39:51.656075+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7979, 'log_dups_size': 0, 'ondisk_log_size': 7979, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42161312, 'num_objects': 5147, 'num_object_clones': 0, 'num_object_copies': 20588, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2345, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5147, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7977, 'num_write_kb': 52500, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 16, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2804}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:39:56.228 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:39:56.228 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'7783", 'reported_seq': 7880, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:52.000660+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:39:52.000660+0000', 'last_peered': '2024-09-05T16:39:52.000660+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:39:52.000660+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7783, 'log_dups_size': 0, 'ondisk_log_size': 7783, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41366688, 'num_objects': 5050, 'num_object_clones': 0, 'num_object_copies': 20200, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5050, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7781, 'num_write_kb': 51325, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2737}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:39:56.228 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:39:56.228 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'7760", 'reported_seq': 7849, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:51.710890+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:39:51.710890+0000', 'last_peered': '2024-09-05T16:39:51.710890+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:39:51.710890+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7760, 'log_dups_size': 0, 'ondisk_log_size': 7760, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41079968, 'num_objects': 5015, 'num_object_clones': 0, 'num_object_copies': 20060, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2229, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5015, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7758, 'num_write_kb': 51092, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2788}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:39:56.229 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:39:56.996 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:39:56.996 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 32100/321984 objects degraded (9.969%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":400,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":80496,"data_bytes":659377208,"bytes_used":2892771328,"bytes_avail":383654285312,"bytes_total":386547056640,"degraded_objects":32100,"degraded_total":321984,"degraded_ratio":0.099694394752534291,"recovering_objects_per_sec":12,"recovering_bytes_per_sec":88034,"recovering_keys_per_sec":0,"num_objects_recovered":156,"num_bytes_recovered":1056768,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":202},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:39:57.034 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:39:57.035 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:39:57.644 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:39:57.645 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:39:57.669 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":226,"stamp":"2024-09-05T16:39:56.921210+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":658820152,"num_objects":80428,"num_object_clones":0,"num_object_copies":321712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32100,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80428,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":118809,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1833,"num_bytes_recovered":13598720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":124534,"ondisk_log_size":124534,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2828864,"kb_used_data":1236592,"kb_used_omap":14,"kb_used_meta":1592177,"kb_avail":374658496,"statfs":{"total":386547056640,"available":383650299904,"internally_reserved":0,"allocated":1266270208,"data_stored":1265502044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1630389815},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,4,9,12,8,9,1,0,0,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20463616,"num_objects":-2498,"num_object_clones":0,"num_object_copies":-9992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-156,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2498,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2498,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":156,"num_bytes_recovered":1056768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003987"},"pg_stats":[{"pgid":"1.f","version":"35'7857","reported_seq":7930,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:51.744446+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:39:51.744446+0000","last_peered":"2024-09-05T16:39:51.744446+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:39:51.744446+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7857,"log_dups_size":0,"ondisk_log_size":7857,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41345024,"num_objects":5047,"num_object_clones":0,"num_object_copies":20188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2249,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5047,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7857,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2798}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'7688","reported_seq":7775,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:51.896703+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:39:51.896703+0000","last_peered":"2024-09-05T16:39:51.896703+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:39:51.896703+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7688,"log_dups_size":0,"ondisk_log_size":7688,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39859360,"num_objects":4866,"num_object_clones":0,"num_object_copies":19464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4866,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7686,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2608}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'7760","reported_seq":7853,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:52.102259+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:39:52.102259+0000","last_peered":"2024-09-05T16:39:52.102259+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:39:52.102259+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7760,"log_dups_size":0,"ondisk_log_size":7760,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40893032,"num_objects":4993,"num_object_clones":0,"num_object_copies":19972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2245,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4993,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7759,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2749}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'7931","reported_seq":12655,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-05T16:39:55.862593+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:39:55.862593+0000","last_peered":"2024-09-05T16:39:55.862593+0000","last_clean":"2024-09-05T16:39:55.862593+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:39:55.862593+0000","last_undegraded":"2024-09-05T16:39:55.862593+0000","last_fullsized":"2024-09-05T16:39:55.862593+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":7931,"log_dups_size":0,"ondisk_log_size":7931,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40984576,"num_objects":5003,"num_object_clones":0,"num_object_copies":20012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5003,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2231,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'7626","reported_seq":7707,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:51.637253+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:39:51.637253+0000","last_peered":"2024-09-05T16:39:51.637253+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:39:51.637253+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7626,"log_dups_size":0,"ondisk_log_size":7626,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40703136,"num_objects":4969,"num_object_clones":0,"num_object_copies":19876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2153,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4969,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7624,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2818}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'7715","reported_seq":7793,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:51.685790+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:39:51.685790+0000","last_peered":"2024-09-05T16:39:51.685790+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:39:51.685790+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7715,"log_dups_size":0,"ondisk_log_size":7715,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41426944,"num_objects":5057,"num_object_clones":0,"num_object_copies":20228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5057,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7715,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2823}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'7608","reported_seq":7696,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:51.845435+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:39:51.845435+0000","last_peered":"2024-09-05T16:39:51.845435+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:39:51.845435+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7608,"log_dups_size":0,"ondisk_log_size":7608,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41399456,"num_objects":5054,"num_object_clones":0,"num_object_copies":20216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2140,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5054,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7606,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2916}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'7723","reported_seq":7802,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:51.868194+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:39:51.868194+0000","last_peered":"2024-09-05T16:39:51.868194+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:39:51.868194+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7723,"log_dups_size":0,"ondisk_log_size":7723,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41415840,"num_objects":5056,"num_object_clones":0,"num_object_copies":20224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5056,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7721,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2822}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'7772","reported_seq":7849,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:51.555889+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:39:51.555889+0000","last_peered":"2024-09-05T16:39:51.555889+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:39:51.555889+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7772,"log_dups_size":0,"ondisk_log_size":7772,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40970560,"num_objects":5002,"num_object_clones":0,"num_object_copies":20008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2256,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5002,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7769,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2749}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'7799","reported_seq":7899,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:52.042243+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:39:52.042243+0000","last_peered":"2024-09-05T16:39:52.042243+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:39:52.042243+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7799,"log_dups_size":0,"ondisk_log_size":7799,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41047200,"num_objects":5011,"num_object_clones":0,"num_object_copies":20044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2255,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5011,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7797,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2758}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'7993","reported_seq":8088,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:51.730626+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:39:51.730626+0000","last_peered":"2024-09-05T16:39:51.730626+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:39:51.730626+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7993,"log_dups_size":0,"ondisk_log_size":7993,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42020592,"num_objects":5130,"num_object_clones":0,"num_object_copies":20520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2297,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5130,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7990,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2836}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'7733","reported_seq":7800,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:51.690854+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:39:51.690854+0000","last_peered":"2024-09-05T16:39:51.690854+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:39:51.690854+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7733,"log_dups_size":0,"ondisk_log_size":7733,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41171536,"num_objects":5026,"num_object_clones":0,"num_object_copies":20104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5026,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7732,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2775}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'7807","reported_seq":11208,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:52.191537+0000","last_change":"2024-09-05T16:37:30.634570+0000","last_active":"2024-09-05T16:39:52.191537+0000","last_peered":"2024-09-05T16:39:52.191537+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:39:52.191537+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7807,"log_dups_size":0,"ondisk_log_size":7807,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40974928,"num_objects":5002,"num_object_clones":0,"num_object_copies":20008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":634,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5002,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7806,"num_write_kb":51231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1662,"num_bytes_recovered":13598720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4368}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'7979","reported_seq":8064,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:51.656075+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:39:51.656075+0000","last_peered":"2024-09-05T16:39:51.656075+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:39:51.656075+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7979,"log_dups_size":0,"ondisk_log_size":7979,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42161312,"num_objects":5147,"num_object_clones":0,"num_object_copies":20588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2345,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5147,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7977,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2804}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'7783","reported_seq":7880,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:52.000660+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:39:52.000660+0000","last_peered":"2024-09-05T16:39:52.000660+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:39:52.000660+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7783,"log_dups_size":0,"ondisk_log_size":7783,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41366688,"num_objects":5050,"num_object_clones":0,"num_object_copies":20200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5050,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7781,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2737}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'7760","reported_seq":7849,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:51.710890+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:39:51.710890+0000","last_peered":"2024-09-05T16:39:51.710890+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:39:51.710890+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7760,"log_dups_size":0,"ondisk_log_size":7760,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41079968,"num_objects":5015,"num_object_clones":0,"num_object_copies":20060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2229,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5015,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7758,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2788}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":658820152,"num_objects":80428,"num_object_clones":0,"num_object_copies":321712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32100,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80428,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":118809,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1833,"num_bytes_recovered":13598720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1264533504,"data_stored":1264533504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":124534,"ondisk_log_size":124534,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084359,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":759392,"kb_used_data":322308,"kb_used_omap":4,"kb_used_meta":437051,"kb_avail":93612448,"statfs":{"total":96636764160,"available":95859146752,"internally_reserved":0,"allocated":330043392,"data_stored":329851351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":447540947},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,4,3,1,3,0,0,0,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986237,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":576340,"kb_used_data":269764,"kb_used_omap":4,"kb_used_meta":306555,"kb_avail":93795500,"statfs":{"total":96636764160,"available":96046592000,"internally_reserved":0,"allocated":276238336,"data_stored":276046295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":313913042},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,1,1,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673042,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":746236,"kb_used_data":322156,"kb_used_omap":1,"kb_used_meta":424062,"kb_avail":93625604,"statfs":{"total":96636764160,"available":95872618496,"internally_reserved":0,"allocated":329887744,"data_stored":329695703,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":434239941},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,7,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182473,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":746896,"kb_used_data":322364,"kb_used_omap":4,"kb_used_meta":424507,"kb_avail":93624944,"statfs":{"total":96636764160,"available":95871942656,"internally_reserved":0,"allocated":330100736,"data_stored":329908695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":434695885},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,3,1,6,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":329580544,"data_stored":329580544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":275939328,"data_stored":275939328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":329302016,"data_stored":329302016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":329711616,"data_stored":329711616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:40:00.672 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:40:01.256 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:40:01.257 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:40:01.278 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":228,"stamp":"2024-09-05T16:40:00.922935+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":649940024,"num_objects":79344,"num_object_clones":0,"num_object_copies":317376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32016,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":79344,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":119893,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1917,"num_bytes_recovered":14245888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":125618,"ondisk_log_size":125618,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2810884,"kb_used_data":1222772,"kb_used_omap":14,"kb_used_meta":1588017,"kb_avail":374676476,"statfs":{"total":386547056640,"available":383668711424,"internally_reserved":0,"allocated":1252118528,"data_stored":1251350364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1626129975},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,4,10,13,5,8,3,0,0,0,0,2,6],"upper_bound":16384},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19464192,"num_objects":-2376,"num_object_clones":0,"num_object_copies":-9504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-156,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2376,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2376,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":156,"num_bytes_recovered":1212416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004274"},"pg_stats":[{"pgid":"1.f","version":"35'7944","reported_seq":8017,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:56.889147+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:39:56.889147+0000","last_peered":"2024-09-05T16:39:56.889147+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:39:56.889147+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7944,"log_dups_size":0,"ondisk_log_size":7944,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40632320,"num_objects":4960,"num_object_clones":0,"num_object_copies":19840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2249,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4960,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7944,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2711}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'7758","reported_seq":7845,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:56.896124+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:39:56.896124+0000","last_peered":"2024-09-05T16:39:56.896124+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:39:56.896124+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7758,"log_dups_size":0,"ondisk_log_size":7758,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39285920,"num_objects":4796,"num_object_clones":0,"num_object_copies":19184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4796,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7756,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2538}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'7824","reported_seq":7917,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:57.121111+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:39:57.121111+0000","last_peered":"2024-09-05T16:39:57.121111+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:39:57.121111+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7824,"log_dups_size":0,"ondisk_log_size":7824,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40368744,"num_objects":4929,"num_object_clones":0,"num_object_copies":19716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2245,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4929,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7823,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2685}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'7931","reported_seq":12655,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-05T16:39:55.862593+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:39:55.862593+0000","last_peered":"2024-09-05T16:39:55.862593+0000","last_clean":"2024-09-05T16:39:55.862593+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:39:55.862593+0000","last_undegraded":"2024-09-05T16:39:55.862593+0000","last_fullsized":"2024-09-05T16:39:55.862593+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":7931,"log_dups_size":0,"ondisk_log_size":7931,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40984576,"num_objects":5003,"num_object_clones":0,"num_object_copies":20012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5003,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2231,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'7693","reported_seq":7774,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:56.545181+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:39:56.545181+0000","last_peered":"2024-09-05T16:39:56.545181+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:39:56.545181+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7693,"log_dups_size":0,"ondisk_log_size":7693,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40154272,"num_objects":4902,"num_object_clones":0,"num_object_copies":19608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2153,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4902,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7691,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2751}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'7791","reported_seq":7874,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:56.750439+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:39:56.750439+0000","last_peered":"2024-09-05T16:39:56.750439+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:39:56.750439+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7791,"log_dups_size":0,"ondisk_log_size":7791,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40804352,"num_objects":4981,"num_object_clones":0,"num_object_copies":19924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4981,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7791,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2749}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'7670","reported_seq":7758,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:56.793357+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:39:56.793357+0000","last_peered":"2024-09-05T16:39:56.793357+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:39:56.793357+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7670,"log_dups_size":0,"ondisk_log_size":7670,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40891552,"num_objects":4992,"num_object_clones":0,"num_object_copies":19968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2140,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4992,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7668,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2854}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'7799","reported_seq":7876,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:56.842123+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:39:56.842123+0000","last_peered":"2024-09-05T16:39:56.842123+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:39:56.842123+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7799,"log_dups_size":0,"ondisk_log_size":7799,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40793248,"num_objects":4980,"num_object_clones":0,"num_object_copies":19920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4980,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7797,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2746}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'7851","reported_seq":7934,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:56.726237+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:39:56.726237+0000","last_peered":"2024-09-05T16:39:56.726237+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:39:56.726237+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7851,"log_dups_size":0,"ondisk_log_size":7851,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40323392,"num_objects":4923,"num_object_clones":0,"num_object_copies":19692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2253,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7848,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2673}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'7862","reported_seq":7962,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:56.869974+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:39:56.869974+0000","last_peered":"2024-09-05T16:39:56.869974+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:39:56.869974+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7862,"log_dups_size":0,"ondisk_log_size":7862,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40531104,"num_objects":4948,"num_object_clones":0,"num_object_copies":19792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2255,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4948,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7860,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2695}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'8077","reported_seq":8172,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:56.788098+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:39:56.788098+0000","last_peered":"2024-09-05T16:39:56.788098+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:39:56.788098+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8077,"log_dups_size":0,"ondisk_log_size":8077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41332464,"num_objects":5046,"num_object_clones":0,"num_object_copies":20184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2297,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5046,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8074,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2752}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'7811","reported_seq":7878,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:56.737000+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:39:56.737000+0000","last_peered":"2024-09-05T16:39:56.737000+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:39:56.737000+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7811,"log_dups_size":0,"ondisk_log_size":7811,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40532560,"num_objects":4948,"num_object_clones":0,"num_object_copies":19792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4948,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7810,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2697}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'7875","reported_seq":11434,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:57.198630+0000","last_change":"2024-09-05T16:37:30.634570+0000","last_active":"2024-09-05T16:39:57.198630+0000","last_peered":"2024-09-05T16:39:57.198630+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:39:57.198630+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7875,"log_dups_size":0,"ondisk_log_size":7875,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40417872,"num_objects":4934,"num_object_clones":0,"num_object_copies":19736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":555,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4934,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7874,"num_write_kb":51231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1741,"num_bytes_recovered":14245888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4379}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'8049","reported_seq":8134,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:56.803501+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:39:56.803501+0000","last_peered":"2024-09-05T16:39:56.803501+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:39:56.803501+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8049,"log_dups_size":0,"ondisk_log_size":8049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41587872,"num_objects":5077,"num_object_clones":0,"num_object_copies":20308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2345,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5077,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8047,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2734}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'7864","reported_seq":7961,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:57.054094+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:39:57.054094+0000","last_peered":"2024-09-05T16:39:57.054094+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:39:57.054094+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7864,"log_dups_size":0,"ondisk_log_size":7864,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40703136,"num_objects":4969,"num_object_clones":0,"num_object_copies":19876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4969,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7862,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2656}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'7819","reported_seq":7907,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:56.781994+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:39:56.781994+0000","last_peered":"2024-09-05T16:39:56.781994+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:39:56.781994+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7819,"log_dups_size":0,"ondisk_log_size":7819,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40596640,"num_objects":4956,"num_object_clones":0,"num_object_copies":19824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2229,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7817,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2729}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":649940024,"num_objects":79344,"num_object_clones":0,"num_object_copies":317376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32016,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":79344,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":119893,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1917,"num_bytes_recovered":14245888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1250471936,"data_stored":1250471936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":125618,"ondisk_log_size":125618,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084360,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":742336,"kb_used_data":317728,"kb_used_omap":4,"kb_used_meta":424571,"kb_avail":93629504,"statfs":{"total":96636764160,"available":95876612096,"internally_reserved":0,"allocated":325353472,"data_stored":325161431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":434761427},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,1,4,4,3,4,1,0,0,0,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986237,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":576340,"kb_used_data":269764,"kb_used_omap":4,"kb_used_meta":306555,"kb_avail":93795500,"statfs":{"total":96636764160,"available":96046592000,"internally_reserved":0,"allocated":276238336,"data_stored":276046295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":313913042},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,1,1,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673043,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":745776,"kb_used_data":317536,"kb_used_omap":1,"kb_used_meta":428222,"kb_avail":93626064,"statfs":{"total":96636764160,"available":95873089536,"internally_reserved":0,"allocated":325156864,"data_stored":324964823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":438499781},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,3,0,0,0,0,0,0,0,0,6],"upper_bound":16384},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182474,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":746432,"kb_used_data":317744,"kb_used_omap":4,"kb_used_meta":428667,"kb_avail":93625408,"statfs":{"total":96636764160,"available":95872417792,"internally_reserved":0,"allocated":325369856,"data_stored":325177815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":438955725},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,5,1,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":324902912,"data_stored":324902912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":275939328,"data_stored":275939328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":324685824,"data_stored":324685824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":324943872,"data_stored":324943872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:40:01.279 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:40:01.857 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:40:01.952 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:40:01.953 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":228,"stamp":"2024-09-05T16:40:00.922935+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":649940024,"num_objects":79344,"num_object_clones":0,"num_object_copies":317376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32016,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":79344,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":119893,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1917,"num_bytes_recovered":14245888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":125618,"ondisk_log_size":125618,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2810884,"kb_used_data":1222772,"kb_used_omap":14,"kb_used_meta":1588017,"kb_avail":374676476,"statfs":{"total":386547056640,"available":383668711424,"internally_reserved":0,"allocated":1252118528,"data_stored":1251350364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1626129975},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,4,10,13,5,8,3,0,0,0,0,2,6],"upper_bound":16384},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19464192,"num_objects":-2376,"num_object_clones":0,"num_object_copies":-9504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-156,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2376,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2376,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":156,"num_bytes_recovered":1212416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004274"},"pg_stats":[{"pgid":"1.f","version":"35'7944","reported_seq":8017,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:56.889147+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:39:56.889147+0000","last_peered":"2024-09-05T16:39:56.889147+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:39:56.889147+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7944,"log_dups_size":0,"ondisk_log_size":7944,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40632320,"num_objects":4960,"num_object_clones":0,"num_object_copies":19840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2249,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4960,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7944,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2711}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'7758","reported_seq":7845,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:56.896124+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:39:56.896124+0000","last_peered":"2024-09-05T16:39:56.896124+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:39:56.896124+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7758,"log_dups_size":0,"ondisk_log_size":7758,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39285920,"num_objects":4796,"num_object_clones":0,"num_object_copies":19184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4796,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7756,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2538}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'7824","reported_seq":7917,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:57.121111+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:39:57.121111+0000","last_peered":"2024-09-05T16:39:57.121111+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:39:57.121111+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7824,"log_dups_size":0,"ondisk_log_size":7824,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40368744,"num_objects":4929,"num_object_clones":0,"num_object_copies":19716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2245,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4929,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7823,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2685}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'7931","reported_seq":12655,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-05T16:39:55.862593+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:39:55.862593+0000","last_peered":"2024-09-05T16:39:55.862593+0000","last_clean":"2024-09-05T16:39:55.862593+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:39:55.862593+0000","last_undegraded":"2024-09-05T16:39:55.862593+0000","last_fullsized":"2024-09-05T16:39:55.862593+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":7931,"log_dups_size":0,"ondisk_log_size":7931,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40984576,"num_objects":5003,"num_object_clones":0,"num_object_copies":20012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5003,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2231,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'7693","reported_seq":7774,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:56.545181+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:39:56.545181+0000","last_peered":"2024-09-05T16:39:56.545181+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:39:56.545181+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7693,"log_dups_size":0,"ondisk_log_size":7693,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40154272,"num_objects":4902,"num_object_clones":0,"num_object_copies":19608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2153,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4902,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7691,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2751}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'7791","reported_seq":7874,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:56.750439+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:39:56.750439+0000","last_peered":"2024-09-05T16:39:56.750439+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:39:56.750439+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7791,"log_dups_size":0,"ondisk_log_size":7791,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40804352,"num_objects":4981,"num_object_clones":0,"num_object_copies":19924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4981,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7791,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2749}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'7670","reported_seq":7758,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:56.793357+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:39:56.793357+0000","last_peered":"2024-09-05T16:39:56.793357+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:39:56.793357+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7670,"log_dups_size":0,"ondisk_log_size":7670,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40891552,"num_objects":4992,"num_object_clones":0,"num_object_copies":19968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2140,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4992,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7668,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2854}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'7799","reported_seq":7876,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:56.842123+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:39:56.842123+0000","last_peered":"2024-09-05T16:39:56.842123+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:39:56.842123+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7799,"log_dups_size":0,"ondisk_log_size":7799,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40793248,"num_objects":4980,"num_object_clones":0,"num_object_copies":19920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4980,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7797,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2746}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'7851","reported_seq":7934,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:56.726237+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:39:56.726237+0000","last_peered":"2024-09-05T16:39:56.726237+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:39:56.726237+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7851,"log_dups_size":0,"ondisk_log_size":7851,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40323392,"num_objects":4923,"num_object_clones":0,"num_object_copies":19692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2253,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7848,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2673}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'7862","reported_seq":7962,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:56.869974+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:39:56.869974+0000","last_peered":"2024-09-05T16:39:56.869974+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:39:56.869974+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7862,"log_dups_size":0,"ondisk_log_size":7862,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40531104,"num_objects":4948,"num_object_clones":0,"num_object_copies":19792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2255,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4948,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7860,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2695}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'8077","reported_seq":8172,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:56.788098+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:39:56.788098+0000","last_peered":"2024-09-05T16:39:56.788098+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:39:56.788098+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8077,"log_dups_size":0,"ondisk_log_size":8077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41332464,"num_objects":5046,"num_object_clones":0,"num_object_copies":20184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2297,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5046,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8074,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2752}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'7811","reported_seq":7878,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:56.737000+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:39:56.737000+0000","last_peered":"2024-09-05T16:39:56.737000+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:39:56.737000+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7811,"log_dups_size":0,"ondisk_log_size":7811,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40532560,"num_objects":4948,"num_object_clones":0,"num_object_copies":19792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4948,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7810,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2697}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'7875","reported_seq":11434,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:57.198630+0000","last_change":"2024-09-05T16:37:30.634570+0000","last_active":"2024-09-05T16:39:57.198630+0000","last_peered":"2024-09-05T16:39:57.198630+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:39:57.198630+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7875,"log_dups_size":0,"ondisk_log_size":7875,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40417872,"num_objects":4934,"num_object_clones":0,"num_object_copies":19736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":555,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4934,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7874,"num_write_kb":51231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1741,"num_bytes_recovered":14245888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4379}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'8049","reported_seq":8134,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:56.803501+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:39:56.803501+0000","last_peered":"2024-09-05T16:39:56.803501+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:39:56.803501+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8049,"log_dups_size":0,"ondisk_log_size":8049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41587872,"num_objects":5077,"num_object_clones":0,"num_object_copies":20308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2345,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5077,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8047,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2734}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'7864","reported_seq":7961,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:57.054094+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:39:57.054094+0000","last_peered":"2024-09-05T16:39:57.054094+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:39:57.054094+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7864,"log_dups_size":0,"ondisk_log_size":7864,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40703136,"num_objects":4969,"num_object_clones":0,"num_object_copies":19876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4969,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7862,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2656}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'7819","reported_seq":7907,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:39:56.781994+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:39:56.781994+0000","last_peered":"2024-09-05T16:39:56.781994+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:39:56.781994+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7819,"log_dups_size":0,"ondisk_log_size":7819,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40596640,"num_objects":4956,"num_object_clones":0,"num_object_copies":19824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2229,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7817,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2729}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":649940024,"num_objects":79344,"num_object_clones":0,"num_object_copies":317376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32016,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":79344,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":119893,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1917,"num_bytes_recovered":14245888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1250471936,"data_stored":1250471936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":125618,"ondisk_log_size":125618,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084360,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":742336,"kb_used_data":317728,"kb_used_omap":4,"kb_used_meta":424571,"kb_avail":93629504,"statfs":{"total":96636764160,"available":95876612096,"internally_reserved":0,"allocated":325353472,"data_stored":325161431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":434761427},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,1,4,4,3,4,1,0,0,0,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986237,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":576340,"kb_used_data":269764,"kb_used_omap":4,"kb_used_meta":306555,"kb_avail":93795500,"statfs":{"total":96636764160,"available":96046592000,"internally_reserved":0,"allocated":276238336,"data_stored":276046295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":313913042},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,1,1,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673043,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":745776,"kb_used_data":317536,"kb_used_omap":1,"kb_used_meta":428222,"kb_avail":93626064,"statfs":{"total":96636764160,"available":95873089536,"internally_reserved":0,"allocated":325156864,"data_stored":324964823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":438499781},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,3,0,0,0,0,0,0,0,0,6],"upper_bound":16384},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182474,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":746432,"kb_used_data":317744,"kb_used_omap":4,"kb_used_meta":428667,"kb_avail":93625408,"statfs":{"total":96636764160,"available":95872417792,"internally_reserved":0,"allocated":325369856,"data_stored":325177815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":438955725},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,5,1,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":324902912,"data_stored":324902912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":275939328,"data_stored":275939328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":324685824,"data_stored":324685824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":324943872,"data_stored":324943872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:40:01.954 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-05T16:40:01.954 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'7944", 'reported_seq': 8017, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:56.889147+0000', 'last_change': '2024-09-05T16:34:56.833497+0000', 'last_active': '2024-09-05T16:39:56.889147+0000', 'last_peered': '2024-09-05T16:39:56.889147+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.828289+0000', 'last_became_peered': '2024-09-05T16:34:56.828289+0000', 'last_unstale': '2024-09-05T16:39:56.889147+0000', 'last_undegraded': '2024-09-05T16:34:56.612160+0000', 'last_fullsized': '2024-09-05T16:34:56.557632+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7944, 'log_dups_size': 0, 'ondisk_log_size': 7944, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40632320, 'num_objects': 4960, 'num_object_clones': 0, 'num_object_copies': 19840, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2249, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4960, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7944, 'num_write_kb': 51616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2711}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:40:01.954 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-05T16:40:01.955 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "35'7758", 'reported_seq': 7845, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:56.896124+0000', 'last_change': '2024-09-05T16:34:56.825031+0000', 'last_active': '2024-09-05T16:39:56.896124+0000', 'last_peered': '2024-09-05T16:39:56.896124+0000', 'last_clean': '2024-09-05T16:33:53.200762+0000', 'last_became_active': '2024-09-05T16:34:56.820292+0000', 'last_became_peered': '2024-09-05T16:34:56.820292+0000', 'last_unstale': '2024-09-05T16:39:56.896124+0000', 'last_undegraded': '2024-09-05T16:34:56.605776+0000', 'last_fullsized': '2024-09-05T16:34:56.559383+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7758, 'log_dups_size': 0, 'ondisk_log_size': 7758, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39285920, 'num_objects': 4796, 'num_object_clones': 0, 'num_object_copies': 19184, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2260, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4796, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7756, 'num_write_kb': 50212, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2538}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:40:01.955 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:40:01.955 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'7824", 'reported_seq': 7917, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:57.121111+0000', 'last_change': '2024-09-05T16:34:56.867900+0000', 'last_active': '2024-09-05T16:39:57.121111+0000', 'last_peered': '2024-09-05T16:39:57.121111+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:39:57.121111+0000', 'last_undegraded': '2024-09-05T16:34:56.590972+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7824, 'log_dups_size': 0, 'ondisk_log_size': 7824, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40368744, 'num_objects': 4929, 'num_object_clones': 0, 'num_object_copies': 19716, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2245, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4929, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 7823, 'num_write_kb': 50999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2685}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:40:01.955 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:40:01.955 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'7693", 'reported_seq': 7774, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:56.545181+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:39:56.545181+0000', 'last_peered': '2024-09-05T16:39:56.545181+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:39:56.545181+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7693, 'log_dups_size': 0, 'ondisk_log_size': 7693, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40154272, 'num_objects': 4902, 'num_object_clones': 0, 'num_object_copies': 19608, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2153, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4902, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7691, 'num_write_kb': 50373, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2751}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:40:01.955 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:40:01.955 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'7791", 'reported_seq': 7874, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:56.750439+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:39:56.750439+0000', 'last_peered': '2024-09-05T16:39:56.750439+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:39:56.750439+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7791, 'log_dups_size': 0, 'ondisk_log_size': 7791, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40804352, 'num_objects': 4981, 'num_object_clones': 0, 'num_object_copies': 19924, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2232, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4981, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7791, 'num_write_kb': 51088, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2749}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:40:01.955 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:40:01.955 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'7670", 'reported_seq': 7758, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:56.793357+0000', 'last_change': '2024-09-05T16:34:56.903436+0000', 'last_active': '2024-09-05T16:39:56.793357+0000', 'last_peered': '2024-09-05T16:39:56.793357+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:39:56.793357+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7670, 'log_dups_size': 0, 'ondisk_log_size': 7670, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40891552, 'num_objects': 4992, 'num_object_clones': 0, 'num_object_copies': 19968, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2140, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4992, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7668, 'num_write_kb': 50636, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2854}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:40:01.955 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-05T16:40:01.955 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "35'7799", 'reported_seq': 7876, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:56.842123+0000', 'last_change': '2024-09-05T16:34:56.883319+0000', 'last_active': '2024-09-05T16:39:56.842123+0000', 'last_peered': '2024-09-05T16:39:56.842123+0000', 'last_clean': '2024-09-05T16:33:53.505305+0000', 'last_became_active': '2024-09-05T16:34:56.873389+0000', 'last_became_peered': '2024-09-05T16:34:56.873389+0000', 'last_unstale': '2024-09-05T16:39:56.842123+0000', 'last_undegraded': '2024-09-05T16:34:56.602567+0000', 'last_fullsized': '2024-09-05T16:34:56.559958+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7799, 'log_dups_size': 0, 'ondisk_log_size': 7799, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40793248, 'num_objects': 4980, 'num_object_clones': 0, 'num_object_copies': 19920, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2236, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4980, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7797, 'num_write_kb': 51108, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2746}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:40:01.956 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:40:01.956 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'7851", 'reported_seq': 7934, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:56.726237+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:39:56.726237+0000', 'last_peered': '2024-09-05T16:39:56.726237+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:39:56.726237+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7851, 'log_dups_size': 0, 'ondisk_log_size': 7851, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40323392, 'num_objects': 4923, 'num_object_clones': 0, 'num_object_copies': 19692, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2253, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4923, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7848, 'num_write_kb': 51081, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2673}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:40:01.956 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:40:01.956 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'7862", 'reported_seq': 7962, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:56.869974+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:39:56.869974+0000', 'last_peered': '2024-09-05T16:39:56.869974+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:39:56.869974+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7862, 'log_dups_size': 0, 'ondisk_log_size': 7862, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40531104, 'num_objects': 4948, 'num_object_clones': 0, 'num_object_copies': 19792, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2255, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4948, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7860, 'num_write_kb': 51230, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2695}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:40:01.956 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:40:01.956 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'8077", 'reported_seq': 8172, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:56.788098+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:39:56.788098+0000', 'last_peered': '2024-09-05T16:39:56.788098+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:39:56.788098+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8077, 'log_dups_size': 0, 'ondisk_log_size': 8077, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41332464, 'num_objects': 5046, 'num_object_clones': 0, 'num_object_copies': 20184, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2297, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5046, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8074, 'num_write_kb': 52476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2752}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:40:01.956 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:40:01.956 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'7811", 'reported_seq': 7878, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:56.737000+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:39:56.737000+0000', 'last_peered': '2024-09-05T16:39:56.737000+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:39:56.737000+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7811, 'log_dups_size': 0, 'ondisk_log_size': 7811, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40532560, 'num_objects': 4948, 'num_object_clones': 0, 'num_object_copies': 19792, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2252, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4948, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7810, 'num_write_kb': 51031, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2697}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:40:01.956 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-05T16:40:01.956 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'7875", 'reported_seq': 11434, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:57.198630+0000', 'last_change': '2024-09-05T16:37:30.634570+0000', 'last_active': '2024-09-05T16:39:57.198630+0000', 'last_peered': '2024-09-05T16:39:57.198630+0000', 'last_clean': '2024-09-05T16:33:45.082039+0000', 'last_became_active': '2024-09-05T16:34:56.792386+0000', 'last_became_peered': '2024-09-05T16:34:56.792386+0000', 'last_unstale': '2024-09-05T16:39:57.198630+0000', 'last_undegraded': '2024-09-05T16:34:56.600595+0000', 'last_fullsized': '2024-09-05T16:34:56.558203+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7875, 'log_dups_size': 0, 'ondisk_log_size': 7875, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40417872, 'num_objects': 4934, 'num_object_clones': 0, 'num_object_copies': 19736, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 555, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4934, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7874, 'num_write_kb': 51231, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1741, 'num_bytes_recovered': 14245888, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 4379}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:40:01.957 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:40:01.957 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'8049", 'reported_seq': 8134, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:56.803501+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:39:56.803501+0000', 'last_peered': '2024-09-05T16:39:56.803501+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:39:56.803501+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8049, 'log_dups_size': 0, 'ondisk_log_size': 8049, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41587872, 'num_objects': 5077, 'num_object_clones': 0, 'num_object_copies': 20308, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2345, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5077, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8047, 'num_write_kb': 52500, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 16, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2734}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:40:01.957 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:40:01.957 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'7864", 'reported_seq': 7961, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:57.054094+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:39:57.054094+0000', 'last_peered': '2024-09-05T16:39:57.054094+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:39:57.054094+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7864, 'log_dups_size': 0, 'ondisk_log_size': 7864, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40703136, 'num_objects': 4969, 'num_object_clones': 0, 'num_object_copies': 19876, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4969, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7862, 'num_write_kb': 51325, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2656}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:40:01.957 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:40:01.957 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'7819", 'reported_seq': 7907, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:39:56.781994+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:39:56.781994+0000', 'last_peered': '2024-09-05T16:39:56.781994+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:39:56.781994+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7819, 'log_dups_size': 0, 'ondisk_log_size': 7819, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40596640, 'num_objects': 4956, 'num_object_clones': 0, 'num_object_copies': 19824, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2229, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4956, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7817, 'num_write_kb': 51092, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2729}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:40:01.957 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:40:02.648 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:40:02.648 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 32016/317376 objects degraded (10.088%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":405,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":79344,"data_bytes":649940024,"bytes_used":2878345216,"bytes_avail":383668711424,"bytes_total":386547056640,"degraded_objects":32016,"degraded_total":317376,"degraded_ratio":0.10087719298245613,"recovering_objects_per_sec":12,"recovering_bytes_per_sec":100998,"recovering_keys_per_sec":0,"num_objects_recovered":156,"num_bytes_recovered":1212416,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":197},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:40:02.671 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:40:02.672 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:40:03.223 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:40:03.223 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:40:03.245 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":229,"stamp":"2024-09-05T16:40:02.923807+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":640846904,"num_objects":78234,"num_object_clones":0,"num_object_copies":312936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":78234,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":121003,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2024,"num_bytes_recovered":15040512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":126728,"ondisk_log_size":126728,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2818160,"kb_used_data":1209256,"kb_used_omap":14,"kb_used_meta":1608817,"kb_avail":374669200,"statfs":{"total":386547056640,"available":383661260800,"internally_reserved":0,"allocated":1238278144,"data_stored":1237509980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1647429175},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,11,8,5,2,5,0,0,0,0,0,8],"upper_bound":16384},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-28557312,"num_objects":-3486,"num_object_clones":0,"num_object_copies":-13944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3486,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3486,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":263,"num_bytes_recovered":2007040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004315"},"pg_stats":[{"pgid":"1.f","version":"35'8015","reported_seq":8092,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:01.805760+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:40:01.805760+0000","last_peered":"2024-09-05T16:40:01.805760+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:40:01.805760+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8015,"log_dups_size":0,"ondisk_log_size":8015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40050688,"num_objects":4889,"num_object_clones":0,"num_object_copies":19556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4889,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8015,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2642}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'7819","reported_seq":7906,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:01.891167+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:40:01.891167+0000","last_peered":"2024-09-05T16:40:01.891167+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:40:01.891167+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7819,"log_dups_size":0,"ondisk_log_size":7819,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38786208,"num_objects":4735,"num_object_clones":0,"num_object_copies":18940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4735,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7817,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2477}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'7884","reported_seq":7977,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:01.979444+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:40:01.979444+0000","last_peered":"2024-09-05T16:40:01.979444+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:40:01.979444+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7884,"log_dups_size":0,"ondisk_log_size":7884,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39877224,"num_objects":4869,"num_object_clones":0,"num_object_copies":19476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2245,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4869,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7883,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2625}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'7990","reported_seq":12714,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-05T16:40:00.935837+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:40:00.935837+0000","last_peered":"2024-09-05T16:40:00.935837+0000","last_clean":"2024-09-05T16:40:00.935837+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:40:00.935837+0000","last_undegraded":"2024-09-05T16:40:00.935837+0000","last_fullsized":"2024-09-05T16:40:00.935837+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":7990,"log_dups_size":0,"ondisk_log_size":7990,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40501248,"num_objects":4944,"num_object_clones":0,"num_object_copies":19776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4944,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2290,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'7765","reported_seq":7848,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:01.837943+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:40:01.837943+0000","last_peered":"2024-09-05T16:40:01.837943+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:40:01.837943+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7765,"log_dups_size":0,"ondisk_log_size":7765,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39564448,"num_objects":4830,"num_object_clones":0,"num_object_copies":19320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2152,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4830,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7763,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2680}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'7862","reported_seq":7947,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:01.771921+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:40:01.771921+0000","last_peered":"2024-09-05T16:40:01.771921+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:40:01.771921+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7862,"log_dups_size":0,"ondisk_log_size":7862,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40222720,"num_objects":4910,"num_object_clones":0,"num_object_copies":19640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7862,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2679}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'7728","reported_seq":7816,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:01.914805+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:40:01.914805+0000","last_peered":"2024-09-05T16:40:01.914805+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:40:01.914805+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7728,"log_dups_size":0,"ondisk_log_size":7728,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40416416,"num_objects":4934,"num_object_clones":0,"num_object_copies":19736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2140,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4934,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7726,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2796}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'7873","reported_seq":7952,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:01.864138+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:40:01.864138+0000","last_peered":"2024-09-05T16:40:01.864138+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:40:01.864138+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7873,"log_dups_size":0,"ondisk_log_size":7873,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40187040,"num_objects":4906,"num_object_clones":0,"num_object_copies":19624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4906,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7871,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2672}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'7917","reported_seq":8001,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:01.888105+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:40:01.888105+0000","last_peered":"2024-09-05T16:40:01.888105+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:40:01.888105+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7917,"log_dups_size":0,"ondisk_log_size":7917,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39782720,"num_objects":4857,"num_object_clones":0,"num_object_copies":19428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4857,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7914,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2608}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'7937","reported_seq":8038,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:02.089634+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:40:02.089634+0000","last_peered":"2024-09-05T16:40:02.089634+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:40:02.089634+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7937,"log_dups_size":0,"ondisk_log_size":7937,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39916704,"num_objects":4873,"num_object_clones":0,"num_object_copies":19492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2255,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4873,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7935,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2620}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'8148","reported_seq":8243,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:01.793545+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:40:01.793545+0000","last_peered":"2024-09-05T16:40:01.793545+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:40:01.793545+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8148,"log_dups_size":0,"ondisk_log_size":8148,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40750832,"num_objects":4975,"num_object_clones":0,"num_object_copies":19900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2297,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4975,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8145,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2681}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'7888","reported_seq":7963,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:01.754974+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:40:01.754974+0000","last_peered":"2024-09-05T16:40:01.754974+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:40:01.754974+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7888,"log_dups_size":0,"ondisk_log_size":7888,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39901776,"num_objects":4871,"num_object_clones":0,"num_object_copies":19484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2248,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4871,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7887,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2624}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'7957","reported_seq":11711,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:02.267808+0000","last_change":"2024-09-05T16:37:30.634570+0000","last_active":"2024-09-05T16:40:02.267808+0000","last_peered":"2024-09-05T16:40:02.267808+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:40:02.267808+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7957,"log_dups_size":0,"ondisk_log_size":7957,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39746128,"num_objects":4852,"num_object_clones":0,"num_object_copies":19408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":458,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4852,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7956,"num_write_kb":51231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1838,"num_bytes_recovered":15040512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4394}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'8109","reported_seq":8196,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:01.928534+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:40:01.928534+0000","last_peered":"2024-09-05T16:40:01.928534+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:40:01.928534+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8109,"log_dups_size":0,"ondisk_log_size":8109,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41096352,"num_objects":5017,"num_object_clones":0,"num_object_copies":20068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2344,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5017,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8107,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2675}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'7941","reported_seq":8038,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:02.178003+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:40:02.178003+0000","last_peered":"2024-09-05T16:40:02.178003+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:40:02.178003+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7941,"log_dups_size":0,"ondisk_log_size":7941,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40072352,"num_objects":4892,"num_object_clones":0,"num_object_copies":19568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4892,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7939,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2579}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'7895","reported_seq":7984,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:01.720872+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:40:01.720872+0000","last_peered":"2024-09-05T16:40:01.720872+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:40:01.720872+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7895,"log_dups_size":0,"ondisk_log_size":7895,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39974048,"num_objects":4880,"num_object_clones":0,"num_object_copies":19520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2229,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4880,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7893,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2653}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":640846904,"num_objects":78234,"num_object_clones":0,"num_object_copies":312936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":78234,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":121003,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2024,"num_bytes_recovered":15040512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1236672512,"data_stored":1236672512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":126728,"ondisk_log_size":126728,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084361,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":742068,"kb_used_data":313308,"kb_used_omap":4,"kb_used_meta":428731,"kb_avail":93629772,"statfs":{"total":96636764160,"available":95876886528,"internally_reserved":0,"allocated":320827392,"data_stored":320635351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":439021267},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,0,1,2,5,0,0,0,0,0,2],"upper_bound":16384},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986238,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":584476,"kb_used_data":269580,"kb_used_omap":4,"kb_used_meta":314875,"kb_avail":93787364,"statfs":{"total":96636764160,"available":96038260736,"internally_reserved":0,"allocated":276049920,"data_stored":275857879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":322432722},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673044,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":745508,"kb_used_data":313108,"kb_used_omap":1,"kb_used_meta":432382,"kb_avail":93626332,"statfs":{"total":96636764160,"available":95873363968,"internally_reserved":0,"allocated":320622592,"data_stored":320430551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":442759621},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,3,0,0,0,0,0,0,0,0,6],"upper_bound":16384},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182475,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":746108,"kb_used_data":313260,"kb_used_omap":4,"kb_used_meta":432827,"kb_avail":93625732,"statfs":{"total":96636764160,"available":95872749568,"internally_reserved":0,"allocated":320778240,"data_stored":320586199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":443215565},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":320385024,"data_stored":320385024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":275763200,"data_stored":275763200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":320131072,"data_stored":320131072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":320393216,"data_stored":320393216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:40:06.247 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:40:06.843 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:40:06.844 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:40:06.872 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":230,"stamp":"2024-09-05T16:40:04.924734+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":640846904,"num_objects":78234,"num_object_clones":0,"num_object_copies":312936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":78234,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":121003,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2024,"num_bytes_recovered":15040512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":126728,"ondisk_log_size":126728,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2818160,"kb_used_data":1209256,"kb_used_omap":14,"kb_used_meta":1608817,"kb_avail":374669200,"statfs":{"total":386547056640,"available":383661260800,"internally_reserved":0,"allocated":1238278144,"data_stored":1237509980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1647429175},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,11,8,5,2,5,0,0,0,0,0,8],"upper_bound":16384},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18530304,"num_objects":-2262,"num_object_clones":0,"num_object_copies":-9048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2262,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2262,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":191,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004535"},"pg_stats":[{"pgid":"1.f","version":"35'8015","reported_seq":8092,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:01.805760+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:40:01.805760+0000","last_peered":"2024-09-05T16:40:01.805760+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:40:01.805760+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8015,"log_dups_size":0,"ondisk_log_size":8015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40050688,"num_objects":4889,"num_object_clones":0,"num_object_copies":19556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4889,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8015,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2642}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'7819","reported_seq":7906,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:01.891167+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:40:01.891167+0000","last_peered":"2024-09-05T16:40:01.891167+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:40:01.891167+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7819,"log_dups_size":0,"ondisk_log_size":7819,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38786208,"num_objects":4735,"num_object_clones":0,"num_object_copies":18940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4735,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7817,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2477}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'7884","reported_seq":7977,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:01.979444+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:40:01.979444+0000","last_peered":"2024-09-05T16:40:01.979444+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:40:01.979444+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7884,"log_dups_size":0,"ondisk_log_size":7884,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39877224,"num_objects":4869,"num_object_clones":0,"num_object_copies":19476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2245,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4869,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7883,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2625}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'7990","reported_seq":12714,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-05T16:40:00.935837+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:40:00.935837+0000","last_peered":"2024-09-05T16:40:00.935837+0000","last_clean":"2024-09-05T16:40:00.935837+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:40:00.935837+0000","last_undegraded":"2024-09-05T16:40:00.935837+0000","last_fullsized":"2024-09-05T16:40:00.935837+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":7990,"log_dups_size":0,"ondisk_log_size":7990,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40501248,"num_objects":4944,"num_object_clones":0,"num_object_copies":19776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4944,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2290,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'7765","reported_seq":7848,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:01.837943+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:40:01.837943+0000","last_peered":"2024-09-05T16:40:01.837943+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:40:01.837943+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7765,"log_dups_size":0,"ondisk_log_size":7765,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39564448,"num_objects":4830,"num_object_clones":0,"num_object_copies":19320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2152,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4830,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7763,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2680}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'7862","reported_seq":7947,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:01.771921+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:40:01.771921+0000","last_peered":"2024-09-05T16:40:01.771921+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:40:01.771921+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7862,"log_dups_size":0,"ondisk_log_size":7862,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40222720,"num_objects":4910,"num_object_clones":0,"num_object_copies":19640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7862,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2679}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'7728","reported_seq":7816,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:01.914805+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:40:01.914805+0000","last_peered":"2024-09-05T16:40:01.914805+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:40:01.914805+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7728,"log_dups_size":0,"ondisk_log_size":7728,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40416416,"num_objects":4934,"num_object_clones":0,"num_object_copies":19736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2140,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4934,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7726,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2796}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'7873","reported_seq":7952,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:01.864138+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:40:01.864138+0000","last_peered":"2024-09-05T16:40:01.864138+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:40:01.864138+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7873,"log_dups_size":0,"ondisk_log_size":7873,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40187040,"num_objects":4906,"num_object_clones":0,"num_object_copies":19624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4906,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7871,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2672}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'7917","reported_seq":8001,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:01.888105+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:40:01.888105+0000","last_peered":"2024-09-05T16:40:01.888105+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:40:01.888105+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7917,"log_dups_size":0,"ondisk_log_size":7917,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39782720,"num_objects":4857,"num_object_clones":0,"num_object_copies":19428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4857,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7914,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2608}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'7937","reported_seq":8038,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:02.089634+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:40:02.089634+0000","last_peered":"2024-09-05T16:40:02.089634+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:40:02.089634+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7937,"log_dups_size":0,"ondisk_log_size":7937,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39916704,"num_objects":4873,"num_object_clones":0,"num_object_copies":19492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2255,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4873,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7935,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2620}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'8148","reported_seq":8243,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:01.793545+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:40:01.793545+0000","last_peered":"2024-09-05T16:40:01.793545+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:40:01.793545+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8148,"log_dups_size":0,"ondisk_log_size":8148,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40750832,"num_objects":4975,"num_object_clones":0,"num_object_copies":19900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2297,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4975,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8145,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2681}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'7888","reported_seq":7963,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:01.754974+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:40:01.754974+0000","last_peered":"2024-09-05T16:40:01.754974+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:40:01.754974+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7888,"log_dups_size":0,"ondisk_log_size":7888,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39901776,"num_objects":4871,"num_object_clones":0,"num_object_copies":19484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2248,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4871,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7887,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2624}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'7957","reported_seq":11711,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:02.267808+0000","last_change":"2024-09-05T16:37:30.634570+0000","last_active":"2024-09-05T16:40:02.267808+0000","last_peered":"2024-09-05T16:40:02.267808+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:40:02.267808+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7957,"log_dups_size":0,"ondisk_log_size":7957,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39746128,"num_objects":4852,"num_object_clones":0,"num_object_copies":19408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":458,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4852,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7956,"num_write_kb":51231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1838,"num_bytes_recovered":15040512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4394}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'8109","reported_seq":8196,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:01.928534+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:40:01.928534+0000","last_peered":"2024-09-05T16:40:01.928534+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:40:01.928534+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8109,"log_dups_size":0,"ondisk_log_size":8109,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41096352,"num_objects":5017,"num_object_clones":0,"num_object_copies":20068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2344,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5017,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8107,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2675}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'7941","reported_seq":8038,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:02.178003+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:40:02.178003+0000","last_peered":"2024-09-05T16:40:02.178003+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:40:02.178003+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7941,"log_dups_size":0,"ondisk_log_size":7941,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40072352,"num_objects":4892,"num_object_clones":0,"num_object_copies":19568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4892,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7939,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2579}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'7895","reported_seq":7984,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:01.720872+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:40:01.720872+0000","last_peered":"2024-09-05T16:40:01.720872+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:40:01.720872+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7895,"log_dups_size":0,"ondisk_log_size":7895,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39974048,"num_objects":4880,"num_object_clones":0,"num_object_copies":19520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2229,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4880,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7893,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2653}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":640846904,"num_objects":78234,"num_object_clones":0,"num_object_copies":312936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":78234,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":121003,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2024,"num_bytes_recovered":15040512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1236672512,"data_stored":1236672512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":126728,"ondisk_log_size":126728,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084361,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":742068,"kb_used_data":313308,"kb_used_omap":4,"kb_used_meta":428731,"kb_avail":93629772,"statfs":{"total":96636764160,"available":95876886528,"internally_reserved":0,"allocated":320827392,"data_stored":320635351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":439021267},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,0,1,2,5,0,0,0,0,0,2],"upper_bound":16384},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986238,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":584476,"kb_used_data":269580,"kb_used_omap":4,"kb_used_meta":314875,"kb_avail":93787364,"statfs":{"total":96636764160,"available":96038260736,"internally_reserved":0,"allocated":276049920,"data_stored":275857879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":322432722},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673044,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":745508,"kb_used_data":313108,"kb_used_omap":1,"kb_used_meta":432382,"kb_avail":93626332,"statfs":{"total":96636764160,"available":95873363968,"internally_reserved":0,"allocated":320622592,"data_stored":320430551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":442759621},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,3,0,0,0,0,0,0,0,0,6],"upper_bound":16384},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182475,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":746108,"kb_used_data":313260,"kb_used_omap":4,"kb_used_meta":432827,"kb_avail":93625732,"statfs":{"total":96636764160,"available":95872749568,"internally_reserved":0,"allocated":320778240,"data_stored":320586199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":443215565},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":320385024,"data_stored":320385024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":275763200,"data_stored":275763200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":320131072,"data_stored":320131072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":320393216,"data_stored":320393216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:40:06.874 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:40:07.457 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:40:07.458 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:40:07.477 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":231,"stamp":"2024-09-05T16:40:06.925498+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":640314424,"num_objects":78169,"num_object_clones":0,"num_object_copies":312676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":78169,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":121068,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2024,"num_bytes_recovered":15040512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":126793,"ondisk_log_size":126793,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2805644,"kb_used_data":1209220,"kb_used_omap":14,"kb_used_meta":1596337,"kb_avail":374681716,"statfs":{"total":386547056640,"available":383674077184,"internally_reserved":0,"allocated":1238241280,"data_stored":1237473116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1634649655},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,3,10,7,2,2,5,0,0,0,0,0,8],"upper_bound":16384},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19062784,"num_objects":-2327,"num_object_clones":0,"num_object_copies":-9308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2327,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2327,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":191,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004760"},"pg_stats":[{"pgid":"1.f","version":"35'8015","reported_seq":8092,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:01.805760+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:40:01.805760+0000","last_peered":"2024-09-05T16:40:01.805760+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:40:01.805760+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8015,"log_dups_size":0,"ondisk_log_size":8015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40050688,"num_objects":4889,"num_object_clones":0,"num_object_copies":19556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4889,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8015,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2642}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'7819","reported_seq":7906,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:01.891167+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:40:01.891167+0000","last_peered":"2024-09-05T16:40:01.891167+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:40:01.891167+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7819,"log_dups_size":0,"ondisk_log_size":7819,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38786208,"num_objects":4735,"num_object_clones":0,"num_object_copies":18940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4735,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7817,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2477}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'7884","reported_seq":7977,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:01.979444+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:40:01.979444+0000","last_peered":"2024-09-05T16:40:01.979444+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:40:01.979444+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7884,"log_dups_size":0,"ondisk_log_size":7884,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39877224,"num_objects":4869,"num_object_clones":0,"num_object_copies":19476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2245,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4869,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7883,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2625}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'8055","reported_seq":12779,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-05T16:40:06.163819+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:40:06.163819+0000","last_peered":"2024-09-05T16:40:06.163819+0000","last_clean":"2024-09-05T16:40:06.163819+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:40:06.163819+0000","last_undegraded":"2024-09-05T16:40:06.163819+0000","last_fullsized":"2024-09-05T16:40:06.163819+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":8055,"log_dups_size":0,"ondisk_log_size":8055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39968768,"num_objects":4879,"num_object_clones":0,"num_object_copies":19516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4879,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2355,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'7765","reported_seq":7848,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:01.837943+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:40:01.837943+0000","last_peered":"2024-09-05T16:40:01.837943+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:40:01.837943+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7765,"log_dups_size":0,"ondisk_log_size":7765,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39564448,"num_objects":4830,"num_object_clones":0,"num_object_copies":19320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2152,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4830,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7763,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2680}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'7862","reported_seq":7947,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:01.771921+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:40:01.771921+0000","last_peered":"2024-09-05T16:40:01.771921+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:40:01.771921+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7862,"log_dups_size":0,"ondisk_log_size":7862,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40222720,"num_objects":4910,"num_object_clones":0,"num_object_copies":19640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7862,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2679}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'7728","reported_seq":7816,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:01.914805+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:40:01.914805+0000","last_peered":"2024-09-05T16:40:01.914805+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:40:01.914805+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7728,"log_dups_size":0,"ondisk_log_size":7728,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40416416,"num_objects":4934,"num_object_clones":0,"num_object_copies":19736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2140,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4934,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7726,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2796}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'7873","reported_seq":7952,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:01.864138+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:40:01.864138+0000","last_peered":"2024-09-05T16:40:01.864138+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:40:01.864138+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7873,"log_dups_size":0,"ondisk_log_size":7873,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40187040,"num_objects":4906,"num_object_clones":0,"num_object_copies":19624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4906,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7871,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2672}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'7917","reported_seq":8001,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:01.888105+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:40:01.888105+0000","last_peered":"2024-09-05T16:40:01.888105+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:40:01.888105+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7917,"log_dups_size":0,"ondisk_log_size":7917,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39782720,"num_objects":4857,"num_object_clones":0,"num_object_copies":19428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4857,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7914,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2608}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'7937","reported_seq":8038,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:02.089634+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:40:02.089634+0000","last_peered":"2024-09-05T16:40:02.089634+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:40:02.089634+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7937,"log_dups_size":0,"ondisk_log_size":7937,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39916704,"num_objects":4873,"num_object_clones":0,"num_object_copies":19492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2255,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4873,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7935,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2620}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'8148","reported_seq":8243,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:01.793545+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:40:01.793545+0000","last_peered":"2024-09-05T16:40:01.793545+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:40:01.793545+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8148,"log_dups_size":0,"ondisk_log_size":8148,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40750832,"num_objects":4975,"num_object_clones":0,"num_object_copies":19900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2297,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4975,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8145,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2681}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'7888","reported_seq":7963,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:01.754974+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:40:01.754974+0000","last_peered":"2024-09-05T16:40:01.754974+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:40:01.754974+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7888,"log_dups_size":0,"ondisk_log_size":7888,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39901776,"num_objects":4871,"num_object_clones":0,"num_object_copies":19484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2248,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4871,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7887,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2624}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'7957","reported_seq":11711,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:02.267808+0000","last_change":"2024-09-05T16:37:30.634570+0000","last_active":"2024-09-05T16:40:02.267808+0000","last_peered":"2024-09-05T16:40:02.267808+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:40:02.267808+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7957,"log_dups_size":0,"ondisk_log_size":7957,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39746128,"num_objects":4852,"num_object_clones":0,"num_object_copies":19408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":458,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4852,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7956,"num_write_kb":51231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1838,"num_bytes_recovered":15040512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4394}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'8109","reported_seq":8196,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:01.928534+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:40:01.928534+0000","last_peered":"2024-09-05T16:40:01.928534+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:40:01.928534+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8109,"log_dups_size":0,"ondisk_log_size":8109,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41096352,"num_objects":5017,"num_object_clones":0,"num_object_copies":20068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2344,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5017,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8107,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2675}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'7941","reported_seq":8038,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:02.178003+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:40:02.178003+0000","last_peered":"2024-09-05T16:40:02.178003+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:40:02.178003+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7941,"log_dups_size":0,"ondisk_log_size":7941,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40072352,"num_objects":4892,"num_object_clones":0,"num_object_copies":19568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4892,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7939,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2579}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'7895","reported_seq":7984,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:01.720872+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:40:01.720872+0000","last_peered":"2024-09-05T16:40:01.720872+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:40:01.720872+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7895,"log_dups_size":0,"ondisk_log_size":7895,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39974048,"num_objects":4880,"num_object_clones":0,"num_object_copies":19520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2229,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4880,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7893,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2653}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":640314424,"num_objects":78169,"num_object_clones":0,"num_object_copies":312676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":78169,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":121068,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2024,"num_bytes_recovered":15040512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1236561920,"data_stored":1236561920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":126793,"ondisk_log_size":126793,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084361,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":742068,"kb_used_data":313308,"kb_used_omap":4,"kb_used_meta":428731,"kb_avail":93629772,"statfs":{"total":96636764160,"available":95876886528,"internally_reserved":0,"allocated":320827392,"data_stored":320635351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":439021267},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,0,1,2,5,0,0,0,0,0,2],"upper_bound":16384},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986239,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":571960,"kb_used_data":269544,"kb_used_omap":4,"kb_used_meta":302395,"kb_avail":93799880,"statfs":{"total":96636764160,"available":96051077120,"internally_reserved":0,"allocated":276013056,"data_stored":275821015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":309653202},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,0,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673044,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":745508,"kb_used_data":313108,"kb_used_omap":1,"kb_used_meta":432382,"kb_avail":93626332,"statfs":{"total":96636764160,"available":95873363968,"internally_reserved":0,"allocated":320622592,"data_stored":320430551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":442759621},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,3,0,0,0,0,0,0,0,0,6],"upper_bound":16384},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182475,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":746108,"kb_used_data":313260,"kb_used_omap":4,"kb_used_meta":432827,"kb_avail":93625732,"statfs":{"total":96636764160,"available":95872749568,"internally_reserved":0,"allocated":320778240,"data_stored":320586199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":443215565},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":320385024,"data_stored":320385024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":275652608,"data_stored":275652608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":320131072,"data_stored":320131072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":320393216,"data_stored":320393216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:40:07.478 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-05T16:40:07.479 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'8015", 'reported_seq': 8092, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:01.805760+0000', 'last_change': '2024-09-05T16:34:56.833497+0000', 'last_active': '2024-09-05T16:40:01.805760+0000', 'last_peered': '2024-09-05T16:40:01.805760+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.828289+0000', 'last_became_peered': '2024-09-05T16:34:56.828289+0000', 'last_unstale': '2024-09-05T16:40:01.805760+0000', 'last_undegraded': '2024-09-05T16:34:56.612160+0000', 'last_fullsized': '2024-09-05T16:34:56.557632+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8015, 'log_dups_size': 0, 'ondisk_log_size': 8015, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40050688, 'num_objects': 4889, 'num_object_clones': 0, 'num_object_copies': 19556, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4889, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8015, 'num_write_kb': 51616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2642}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:40:07.479 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-05T16:40:07.479 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "35'7819", 'reported_seq': 7906, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:01.891167+0000', 'last_change': '2024-09-05T16:34:56.825031+0000', 'last_active': '2024-09-05T16:40:01.891167+0000', 'last_peered': '2024-09-05T16:40:01.891167+0000', 'last_clean': '2024-09-05T16:33:53.200762+0000', 'last_became_active': '2024-09-05T16:34:56.820292+0000', 'last_became_peered': '2024-09-05T16:34:56.820292+0000', 'last_unstale': '2024-09-05T16:40:01.891167+0000', 'last_undegraded': '2024-09-05T16:34:56.605776+0000', 'last_fullsized': '2024-09-05T16:34:56.559383+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7819, 'log_dups_size': 0, 'ondisk_log_size': 7819, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38786208, 'num_objects': 4735, 'num_object_clones': 0, 'num_object_copies': 18940, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2260, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4735, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7817, 'num_write_kb': 50212, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2477}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:40:07.479 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:40:07.479 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'7884", 'reported_seq': 7977, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:01.979444+0000', 'last_change': '2024-09-05T16:34:56.867900+0000', 'last_active': '2024-09-05T16:40:01.979444+0000', 'last_peered': '2024-09-05T16:40:01.979444+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:40:01.979444+0000', 'last_undegraded': '2024-09-05T16:34:56.590972+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7884, 'log_dups_size': 0, 'ondisk_log_size': 7884, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39877224, 'num_objects': 4869, 'num_object_clones': 0, 'num_object_copies': 19476, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2245, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4869, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 7883, 'num_write_kb': 50999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2625}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:40:07.479 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:40:07.479 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'7765", 'reported_seq': 7848, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:01.837943+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:40:01.837943+0000', 'last_peered': '2024-09-05T16:40:01.837943+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:40:01.837943+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7765, 'log_dups_size': 0, 'ondisk_log_size': 7765, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39564448, 'num_objects': 4830, 'num_object_clones': 0, 'num_object_copies': 19320, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2152, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4830, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7763, 'num_write_kb': 50373, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2680}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:40:07.479 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:40:07.479 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'7862", 'reported_seq': 7947, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:01.771921+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:40:01.771921+0000', 'last_peered': '2024-09-05T16:40:01.771921+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:40:01.771921+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7862, 'log_dups_size': 0, 'ondisk_log_size': 7862, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40222720, 'num_objects': 4910, 'num_object_clones': 0, 'num_object_copies': 19640, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2231, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4910, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7862, 'num_write_kb': 51088, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2679}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:40:07.480 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:40:07.480 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'7728", 'reported_seq': 7816, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:01.914805+0000', 'last_change': '2024-09-05T16:34:56.903436+0000', 'last_active': '2024-09-05T16:40:01.914805+0000', 'last_peered': '2024-09-05T16:40:01.914805+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:40:01.914805+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7728, 'log_dups_size': 0, 'ondisk_log_size': 7728, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40416416, 'num_objects': 4934, 'num_object_clones': 0, 'num_object_copies': 19736, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2140, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4934, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7726, 'num_write_kb': 50636, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2796}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:40:07.480 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-05T16:40:07.480 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "35'7873", 'reported_seq': 7952, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:01.864138+0000', 'last_change': '2024-09-05T16:34:56.883319+0000', 'last_active': '2024-09-05T16:40:01.864138+0000', 'last_peered': '2024-09-05T16:40:01.864138+0000', 'last_clean': '2024-09-05T16:33:53.505305+0000', 'last_became_active': '2024-09-05T16:34:56.873389+0000', 'last_became_peered': '2024-09-05T16:34:56.873389+0000', 'last_unstale': '2024-09-05T16:40:01.864138+0000', 'last_undegraded': '2024-09-05T16:34:56.602567+0000', 'last_fullsized': '2024-09-05T16:34:56.559958+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7873, 'log_dups_size': 0, 'ondisk_log_size': 7873, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40187040, 'num_objects': 4906, 'num_object_clones': 0, 'num_object_copies': 19624, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2236, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4906, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7871, 'num_write_kb': 51108, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2672}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:40:07.480 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:40:07.480 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'7917", 'reported_seq': 8001, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:01.888105+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:40:01.888105+0000', 'last_peered': '2024-09-05T16:40:01.888105+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:40:01.888105+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7917, 'log_dups_size': 0, 'ondisk_log_size': 7917, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39782720, 'num_objects': 4857, 'num_object_clones': 0, 'num_object_copies': 19428, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2252, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4857, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7914, 'num_write_kb': 51081, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2608}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:40:07.480 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:40:07.480 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'7937", 'reported_seq': 8038, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:02.089634+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:40:02.089634+0000', 'last_peered': '2024-09-05T16:40:02.089634+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:40:02.089634+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7937, 'log_dups_size': 0, 'ondisk_log_size': 7937, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39916704, 'num_objects': 4873, 'num_object_clones': 0, 'num_object_copies': 19492, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2255, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4873, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7935, 'num_write_kb': 51230, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2620}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:40:07.480 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:40:07.480 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'8148", 'reported_seq': 8243, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:01.793545+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:40:01.793545+0000', 'last_peered': '2024-09-05T16:40:01.793545+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:40:01.793545+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8148, 'log_dups_size': 0, 'ondisk_log_size': 8148, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40750832, 'num_objects': 4975, 'num_object_clones': 0, 'num_object_copies': 19900, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2297, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4975, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8145, 'num_write_kb': 52476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2681}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:40:07.481 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:40:07.481 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'7888", 'reported_seq': 7963, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:01.754974+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:40:01.754974+0000', 'last_peered': '2024-09-05T16:40:01.754974+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:40:01.754974+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7888, 'log_dups_size': 0, 'ondisk_log_size': 7888, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39901776, 'num_objects': 4871, 'num_object_clones': 0, 'num_object_copies': 19484, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2248, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4871, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7887, 'num_write_kb': 51031, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2624}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:40:07.481 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-05T16:40:07.481 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'7957", 'reported_seq': 11711, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:02.267808+0000', 'last_change': '2024-09-05T16:37:30.634570+0000', 'last_active': '2024-09-05T16:40:02.267808+0000', 'last_peered': '2024-09-05T16:40:02.267808+0000', 'last_clean': '2024-09-05T16:33:45.082039+0000', 'last_became_active': '2024-09-05T16:34:56.792386+0000', 'last_became_peered': '2024-09-05T16:34:56.792386+0000', 'last_unstale': '2024-09-05T16:40:02.267808+0000', 'last_undegraded': '2024-09-05T16:34:56.600595+0000', 'last_fullsized': '2024-09-05T16:34:56.558203+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7957, 'log_dups_size': 0, 'ondisk_log_size': 7957, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39746128, 'num_objects': 4852, 'num_object_clones': 0, 'num_object_copies': 19408, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 458, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4852, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7956, 'num_write_kb': 51231, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1838, 'num_bytes_recovered': 15040512, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 4394}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:40:07.481 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:40:07.481 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'8109", 'reported_seq': 8196, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:01.928534+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:40:01.928534+0000', 'last_peered': '2024-09-05T16:40:01.928534+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:40:01.928534+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8109, 'log_dups_size': 0, 'ondisk_log_size': 8109, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41096352, 'num_objects': 5017, 'num_object_clones': 0, 'num_object_copies': 20068, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2344, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5017, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8107, 'num_write_kb': 52500, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2675}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:40:07.481 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:40:07.481 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'7941", 'reported_seq': 8038, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:02.178003+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:40:02.178003+0000', 'last_peered': '2024-09-05T16:40:02.178003+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:40:02.178003+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7941, 'log_dups_size': 0, 'ondisk_log_size': 7941, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40072352, 'num_objects': 4892, 'num_object_clones': 0, 'num_object_copies': 19568, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4892, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7939, 'num_write_kb': 51325, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2579}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:40:07.481 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:40:07.481 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'7895", 'reported_seq': 7984, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:01.720872+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:40:01.720872+0000', 'last_peered': '2024-09-05T16:40:01.720872+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:40:01.720872+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7895, 'log_dups_size': 0, 'ondisk_log_size': 7895, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39974048, 'num_objects': 4880, 'num_object_clones': 0, 'num_object_copies': 19520, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2229, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4880, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7893, 'num_write_kb': 51092, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2653}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:40:07.482 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:40:08.224 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:40:08.225 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 31909/312676 objects degraded (10.205%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":411,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+clean","count":1}],"num_pgs":16,"num_pools":1,"num_objects":78169,"data_bytes":640314424,"bytes_used":2872979456,"bytes_avail":383674077184,"bytes_total":386547056640,"degraded_objects":31909,"degraded_total":312676,"degraded_ratio":0.10205132469393238,"recovering_objects_per_sec":15,"recovering_bytes_per_sec":120101,"recovering_keys_per_sec":0,"num_objects_recovered":191,"num_bytes_recovered":1441792,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":193},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:40:08.255 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:40:08.255 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:40:08.884 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:40:08.884 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:40:08.904 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":231,"stamp":"2024-09-05T16:40:06.925498+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":640314424,"num_objects":78169,"num_object_clones":0,"num_object_copies":312676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":78169,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":121068,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2024,"num_bytes_recovered":15040512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":126793,"ondisk_log_size":126793,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2805644,"kb_used_data":1209220,"kb_used_omap":14,"kb_used_meta":1596337,"kb_avail":374681716,"statfs":{"total":386547056640,"available":383674077184,"internally_reserved":0,"allocated":1238241280,"data_stored":1237473116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1634649655},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,3,10,7,2,2,5,0,0,0,0,0,8],"upper_bound":16384},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19062784,"num_objects":-2327,"num_object_clones":0,"num_object_copies":-9308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2327,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2327,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":191,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004760"},"pg_stats":[{"pgid":"1.f","version":"35'8015","reported_seq":8092,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:01.805760+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:40:01.805760+0000","last_peered":"2024-09-05T16:40:01.805760+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:40:01.805760+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8015,"log_dups_size":0,"ondisk_log_size":8015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40050688,"num_objects":4889,"num_object_clones":0,"num_object_copies":19556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4889,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8015,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2642}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'7819","reported_seq":7906,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:01.891167+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:40:01.891167+0000","last_peered":"2024-09-05T16:40:01.891167+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:40:01.891167+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7819,"log_dups_size":0,"ondisk_log_size":7819,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38786208,"num_objects":4735,"num_object_clones":0,"num_object_copies":18940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4735,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7817,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2477}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'7884","reported_seq":7977,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:01.979444+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:40:01.979444+0000","last_peered":"2024-09-05T16:40:01.979444+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:40:01.979444+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7884,"log_dups_size":0,"ondisk_log_size":7884,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39877224,"num_objects":4869,"num_object_clones":0,"num_object_copies":19476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2245,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4869,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7883,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2625}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'8055","reported_seq":12779,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-05T16:40:06.163819+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:40:06.163819+0000","last_peered":"2024-09-05T16:40:06.163819+0000","last_clean":"2024-09-05T16:40:06.163819+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:40:06.163819+0000","last_undegraded":"2024-09-05T16:40:06.163819+0000","last_fullsized":"2024-09-05T16:40:06.163819+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":8055,"log_dups_size":0,"ondisk_log_size":8055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39968768,"num_objects":4879,"num_object_clones":0,"num_object_copies":19516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4879,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2355,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'7765","reported_seq":7848,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:01.837943+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:40:01.837943+0000","last_peered":"2024-09-05T16:40:01.837943+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:40:01.837943+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7765,"log_dups_size":0,"ondisk_log_size":7765,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39564448,"num_objects":4830,"num_object_clones":0,"num_object_copies":19320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2152,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4830,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7763,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2680}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'7862","reported_seq":7947,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:01.771921+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:40:01.771921+0000","last_peered":"2024-09-05T16:40:01.771921+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:40:01.771921+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7862,"log_dups_size":0,"ondisk_log_size":7862,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40222720,"num_objects":4910,"num_object_clones":0,"num_object_copies":19640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7862,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2679}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'7728","reported_seq":7816,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:01.914805+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:40:01.914805+0000","last_peered":"2024-09-05T16:40:01.914805+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:40:01.914805+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7728,"log_dups_size":0,"ondisk_log_size":7728,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40416416,"num_objects":4934,"num_object_clones":0,"num_object_copies":19736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2140,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4934,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7726,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2796}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'7873","reported_seq":7952,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:01.864138+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:40:01.864138+0000","last_peered":"2024-09-05T16:40:01.864138+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:40:01.864138+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7873,"log_dups_size":0,"ondisk_log_size":7873,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40187040,"num_objects":4906,"num_object_clones":0,"num_object_copies":19624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4906,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7871,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2672}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'7917","reported_seq":8001,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:01.888105+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:40:01.888105+0000","last_peered":"2024-09-05T16:40:01.888105+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:40:01.888105+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7917,"log_dups_size":0,"ondisk_log_size":7917,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39782720,"num_objects":4857,"num_object_clones":0,"num_object_copies":19428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4857,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7914,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2608}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'7937","reported_seq":8038,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:02.089634+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:40:02.089634+0000","last_peered":"2024-09-05T16:40:02.089634+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:40:02.089634+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7937,"log_dups_size":0,"ondisk_log_size":7937,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39916704,"num_objects":4873,"num_object_clones":0,"num_object_copies":19492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2255,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4873,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7935,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2620}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'8148","reported_seq":8243,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:01.793545+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:40:01.793545+0000","last_peered":"2024-09-05T16:40:01.793545+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:40:01.793545+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8148,"log_dups_size":0,"ondisk_log_size":8148,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40750832,"num_objects":4975,"num_object_clones":0,"num_object_copies":19900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2297,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4975,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8145,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2681}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'7888","reported_seq":7963,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:01.754974+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:40:01.754974+0000","last_peered":"2024-09-05T16:40:01.754974+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:40:01.754974+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7888,"log_dups_size":0,"ondisk_log_size":7888,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39901776,"num_objects":4871,"num_object_clones":0,"num_object_copies":19484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2248,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4871,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7887,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2624}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'7957","reported_seq":11711,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:02.267808+0000","last_change":"2024-09-05T16:37:30.634570+0000","last_active":"2024-09-05T16:40:02.267808+0000","last_peered":"2024-09-05T16:40:02.267808+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:40:02.267808+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7957,"log_dups_size":0,"ondisk_log_size":7957,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39746128,"num_objects":4852,"num_object_clones":0,"num_object_copies":19408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":458,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4852,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7956,"num_write_kb":51231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1838,"num_bytes_recovered":15040512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4394}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'8109","reported_seq":8196,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:01.928534+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:40:01.928534+0000","last_peered":"2024-09-05T16:40:01.928534+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:40:01.928534+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8109,"log_dups_size":0,"ondisk_log_size":8109,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41096352,"num_objects":5017,"num_object_clones":0,"num_object_copies":20068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2344,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5017,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8107,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2675}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'7941","reported_seq":8038,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:02.178003+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:40:02.178003+0000","last_peered":"2024-09-05T16:40:02.178003+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:40:02.178003+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7941,"log_dups_size":0,"ondisk_log_size":7941,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40072352,"num_objects":4892,"num_object_clones":0,"num_object_copies":19568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4892,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7939,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2579}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'7895","reported_seq":7984,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:01.720872+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:40:01.720872+0000","last_peered":"2024-09-05T16:40:01.720872+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:40:01.720872+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7895,"log_dups_size":0,"ondisk_log_size":7895,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39974048,"num_objects":4880,"num_object_clones":0,"num_object_copies":19520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2229,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4880,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7893,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2653}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":640314424,"num_objects":78169,"num_object_clones":0,"num_object_copies":312676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":78169,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":121068,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2024,"num_bytes_recovered":15040512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1236561920,"data_stored":1236561920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":126793,"ondisk_log_size":126793,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084361,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":742068,"kb_used_data":313308,"kb_used_omap":4,"kb_used_meta":428731,"kb_avail":93629772,"statfs":{"total":96636764160,"available":95876886528,"internally_reserved":0,"allocated":320827392,"data_stored":320635351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":439021267},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,0,1,2,5,0,0,0,0,0,2],"upper_bound":16384},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986239,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":571960,"kb_used_data":269544,"kb_used_omap":4,"kb_used_meta":302395,"kb_avail":93799880,"statfs":{"total":96636764160,"available":96051077120,"internally_reserved":0,"allocated":276013056,"data_stored":275821015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":309653202},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,0,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673044,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":745508,"kb_used_data":313108,"kb_used_omap":1,"kb_used_meta":432382,"kb_avail":93626332,"statfs":{"total":96636764160,"available":95873363968,"internally_reserved":0,"allocated":320622592,"data_stored":320430551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":442759621},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,3,0,0,0,0,0,0,0,0,6],"upper_bound":16384},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182475,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":746108,"kb_used_data":313260,"kb_used_omap":4,"kb_used_meta":432827,"kb_avail":93625732,"statfs":{"total":96636764160,"available":95872749568,"internally_reserved":0,"allocated":320778240,"data_stored":320586199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":443215565},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":320385024,"data_stored":320385024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":275652608,"data_stored":275652608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":320131072,"data_stored":320131072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":320393216,"data_stored":320393216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:40:11.911 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:40:12.531 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:40:12.570 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:40:12.570 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":233,"stamp":"2024-09-05T16:40:10.926746+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":631991352,"num_objects":77153,"num_object_clones":0,"num_object_copies":308612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31799,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":77153,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":122084,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2134,"num_bytes_recovered":15802368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":127809,"ondisk_log_size":127809,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2805364,"kb_used_data":1196456,"kb_used_omap":14,"kb_used_meta":1608817,"kb_avail":374681996,"statfs":{"total":386547056640,"available":383674363904,"internally_reserved":0,"allocated":1225170944,"data_stored":1224402780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1647429175},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[7,0,0,4,11,6,3,1,0,0,0,0,0,0,6],"upper_bound":32768},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17948672,"num_objects":-2191,"num_object_clones":0,"num_object_copies":-8764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-217,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2191,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2191,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":217,"num_bytes_recovered":1556480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004912"},"pg_stats":[{"pgid":"1.f","version":"35'8073","reported_seq":8158,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:06.826124+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:40:06.826124+0000","last_peered":"2024-09-05T16:40:06.826124+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:40:06.826124+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8073,"log_dups_size":0,"ondisk_log_size":8073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39575552,"num_objects":4831,"num_object_clones":0,"num_object_copies":19324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2243,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4831,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8073,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2588}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'7872","reported_seq":7959,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:06.777047+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:40:06.777047+0000","last_peered":"2024-09-05T16:40:06.777047+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:40:06.777047+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7872,"log_dups_size":0,"ondisk_log_size":7872,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38352032,"num_objects":4682,"num_object_clones":0,"num_object_copies":18728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4682,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7870,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2424}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'7958","reported_seq":8051,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:07.264844+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:40:07.264844+0000","last_peered":"2024-09-05T16:40:07.264844+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:40:07.264844+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7958,"log_dups_size":0,"ondisk_log_size":7958,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39271016,"num_objects":4795,"num_object_clones":0,"num_object_copies":19180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2245,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4795,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7957,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2551}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'8055","reported_seq":12779,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-05T16:40:06.163819+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:40:06.163819+0000","last_peered":"2024-09-05T16:40:06.163819+0000","last_clean":"2024-09-05T16:40:06.163819+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:40:06.163819+0000","last_undegraded":"2024-09-05T16:40:06.163819+0000","last_fullsized":"2024-09-05T16:40:06.163819+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":8055,"log_dups_size":0,"ondisk_log_size":8055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39968768,"num_objects":4879,"num_object_clones":0,"num_object_copies":19516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4879,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2355,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'7822","reported_seq":7905,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:06.947184+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:40:06.947184+0000","last_peered":"2024-09-05T16:40:06.947184+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:40:06.947184+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7822,"log_dups_size":0,"ondisk_log_size":7822,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39097504,"num_objects":4773,"num_object_clones":0,"num_object_copies":19092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2152,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4773,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7820,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2623}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'7932","reported_seq":8021,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:06.993321+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:40:06.993321+0000","last_peered":"2024-09-05T16:40:06.993321+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:40:06.993321+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7932,"log_dups_size":0,"ondisk_log_size":7932,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39649280,"num_objects":4840,"num_object_clones":0,"num_object_copies":19360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2229,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4840,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7932,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2611}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'7783","reported_seq":7871,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:06.898471+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:40:06.898471+0000","last_peered":"2024-09-05T16:40:06.898471+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:40:06.898471+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7783,"log_dups_size":0,"ondisk_log_size":7783,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39965856,"num_objects":4879,"num_object_clones":0,"num_object_copies":19516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2140,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4879,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7781,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2741}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'7944","reported_seq":8025,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:06.880722+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:40:06.880722+0000","last_peered":"2024-09-05T16:40:06.880722+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:40:06.880722+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7944,"log_dups_size":0,"ondisk_log_size":7944,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39605408,"num_objects":4835,"num_object_clones":0,"num_object_copies":19340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4835,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7942,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2602}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'8003","reported_seq":8092,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:07.004478+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:40:07.004478+0000","last_peered":"2024-09-05T16:40:07.004478+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:40:07.004478+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8003,"log_dups_size":0,"ondisk_log_size":8003,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39078208,"num_objects":4771,"num_object_clones":0,"num_object_copies":19084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4771,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8000,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2524}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'8004","reported_seq":8105,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:07.177324+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:40:07.177324+0000","last_peered":"2024-09-05T16:40:07.177324+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:40:07.177324+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8004,"log_dups_size":0,"ondisk_log_size":8004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39367840,"num_objects":4806,"num_object_clones":0,"num_object_copies":19224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2255,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4806,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8002,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2553}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'8218","reported_seq":8321,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:06.785632+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:40:06.785632+0000","last_peered":"2024-09-05T16:40:06.785632+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:40:06.785632+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8218,"log_dups_size":0,"ondisk_log_size":8218,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40177392,"num_objects":4905,"num_object_clones":0,"num_object_copies":19620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2293,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4905,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8215,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2615}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'7946","reported_seq":8020,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:06.897348+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:40:06.897348+0000","last_peered":"2024-09-05T16:40:06.897348+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:40:06.897348+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7946,"log_dups_size":0,"ondisk_log_size":7946,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39426640,"num_objects":4813,"num_object_clones":0,"num_object_copies":19252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2248,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4813,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7945,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2566}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'8021","reported_seq":11961,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:07.268802+0000","last_change":"2024-09-05T16:37:30.634570+0000","last_active":"2024-09-05T16:40:07.268802+0000","last_peered":"2024-09-05T16:40:07.268802+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:40:07.268802+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8021,"log_dups_size":0,"ondisk_log_size":8021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39221840,"num_objects":4788,"num_object_clones":0,"num_object_copies":19152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":365,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8020,"num_write_kb":51231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1931,"num_bytes_recovered":15802368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4423}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'8188","reported_seq":8279,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:06.791228+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:40:06.791228+0000","last_peered":"2024-09-05T16:40:06.791228+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:40:06.791228+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8188,"log_dups_size":0,"ondisk_log_size":8188,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40449184,"num_objects":4938,"num_object_clones":0,"num_object_copies":19752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2342,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4938,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8186,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2598}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'8020","reported_seq":8117,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:07.107767+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:40:07.107767+0000","last_peered":"2024-09-05T16:40:07.107767+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:40:07.107767+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8020,"log_dups_size":0,"ondisk_log_size":8020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39425184,"num_objects":4813,"num_object_clones":0,"num_object_copies":19252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4813,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8018,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2500}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'7970","reported_seq":8063,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:06.882402+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:40:06.882402+0000","last_peered":"2024-09-05T16:40:06.882402+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:40:06.882402+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7970,"log_dups_size":0,"ondisk_log_size":7970,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39359648,"num_objects":4805,"num_object_clones":0,"num_object_copies":19220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2227,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4805,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7968,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2580}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":631991352,"num_objects":77153,"num_object_clones":0,"num_object_copies":308612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31799,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":77153,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":122084,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2134,"num_bytes_recovered":15802368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1223352320,"data_stored":1223352320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":127809,"ondisk_log_size":127809,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084362,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":741956,"kb_used_data":309040,"kb_used_omap":4,"kb_used_meta":432891,"kb_avail":93629884,"statfs":{"total":96636764160,"available":95877001216,"internally_reserved":0,"allocated":316456960,"data_stored":316264919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":443281107},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986239,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":571960,"kb_used_data":269544,"kb_used_omap":4,"kb_used_meta":302395,"kb_avail":93799880,"statfs":{"total":96636764160,"available":96051077120,"internally_reserved":0,"allocated":276013056,"data_stored":275821015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":309653202},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,0,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673045,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":745428,"kb_used_data":308860,"kb_used_omap":1,"kb_used_meta":436542,"kb_avail":93626412,"statfs":{"total":96636764160,"available":95873445888,"internally_reserved":0,"allocated":316272640,"data_stored":316080599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":447019461},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,5,2,1,0,0,0,0,0,0,0,6],"upper_bound":32768},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182476,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":746020,"kb_used_data":309012,"kb_used_omap":4,"kb_used_meta":436987,"kb_avail":93625820,"statfs":{"total":96636764160,"available":95872839680,"internally_reserved":0,"allocated":316428288,"data_stored":316236247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":447475405},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,1,1,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":316010496,"data_stored":316010496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":275652608,"data_stored":275652608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":315723776,"data_stored":315723776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":315965440,"data_stored":315965440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:40:12.571 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:40:13.157 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:40:13.158 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:40:13.186 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":234,"stamp":"2024-09-05T16:40:12.927310+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":622636088,"num_objects":76011,"num_object_clones":0,"num_object_copies":304044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31695,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76011,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":123226,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2238,"num_bytes_recovered":16580608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128951,"ondisk_log_size":128951,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2829056,"kb_used_data":1182620,"kb_used_omap":14,"kb_used_meta":1646321,"kb_avail":374658304,"statfs":{"total":386547056640,"available":383650103296,"internally_reserved":0,"allocated":1211002880,"data_stored":1210234716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1685833271},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,2,8,11,14,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-27303936,"num_objects":-3333,"num_object_clones":0,"num_object_copies":-13332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3333,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3333,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":321,"num_bytes_recovered":2334720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004374"},"pg_stats":[{"pgid":"1.f","version":"35'8147","reported_seq":8234,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:11.834701+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:40:11.834701+0000","last_peered":"2024-09-05T16:40:11.834701+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:40:11.834701+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8147,"log_dups_size":0,"ondisk_log_size":8147,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38969344,"num_objects":4757,"num_object_clones":0,"num_object_copies":19028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2242,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4757,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8147,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2515}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'7943","reported_seq":8034,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:11.820685+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:40:11.820685+0000","last_peered":"2024-09-05T16:40:11.820685+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:40:11.820685+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7943,"log_dups_size":0,"ondisk_log_size":7943,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37770400,"num_objects":4611,"num_object_clones":0,"num_object_copies":18444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2258,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4611,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7941,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2355}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'8028","reported_seq":8125,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:12.313104+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:40:12.313104+0000","last_peered":"2024-09-05T16:40:12.313104+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:40:12.313104+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8028,"log_dups_size":0,"ondisk_log_size":8028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38697576,"num_objects":4725,"num_object_clones":0,"num_object_copies":18900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2243,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4725,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8027,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2483}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'8135","reported_seq":12859,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-05T16:40:11.259465+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:40:11.259465+0000","last_peered":"2024-09-05T16:40:11.259465+0000","last_clean":"2024-09-05T16:40:11.259465+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:40:11.259465+0000","last_undegraded":"2024-09-05T16:40:11.259465+0000","last_fullsized":"2024-09-05T16:40:11.259465+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":8135,"log_dups_size":0,"ondisk_log_size":8135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39313408,"num_objects":4799,"num_object_clones":0,"num_object_copies":19196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4799,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2435,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'7883","reported_seq":7966,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:11.966408+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:40:11.966408+0000","last_peered":"2024-09-05T16:40:11.966408+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:40:11.966408+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7883,"log_dups_size":0,"ondisk_log_size":7883,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38597792,"num_objects":4712,"num_object_clones":0,"num_object_copies":18848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2152,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4712,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7881,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2562}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'8011","reported_seq":8100,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:12.037210+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:40:12.037210+0000","last_peered":"2024-09-05T16:40:12.037210+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:40:12.037210+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8011,"log_dups_size":0,"ondisk_log_size":8011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39002112,"num_objects":4761,"num_object_clones":0,"num_object_copies":19044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2229,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8011,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2532}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'7843","reported_seq":7931,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:11.932187+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:40:11.932187+0000","last_peered":"2024-09-05T16:40:11.932187+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:40:11.932187+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7843,"log_dups_size":0,"ondisk_log_size":7843,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39474336,"num_objects":4819,"num_object_clones":0,"num_object_copies":19276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2140,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4819,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7841,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2681}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'8016","reported_seq":8097,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:11.775134+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:40:11.775134+0000","last_peered":"2024-09-05T16:40:11.775134+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:40:11.775134+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8016,"log_dups_size":0,"ondisk_log_size":8016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39015584,"num_objects":4763,"num_object_clones":0,"num_object_copies":19052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4763,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8014,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2530}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'8072","reported_seq":8161,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:12.066586+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:40:12.066586+0000","last_peered":"2024-09-05T16:40:12.066586+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:40:12.066586+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8072,"log_dups_size":0,"ondisk_log_size":8072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38512960,"num_objects":4702,"num_object_clones":0,"num_object_copies":18808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4702,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8069,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2455}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'8072","reported_seq":8179,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:12.188319+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:40:12.188319+0000","last_peered":"2024-09-05T16:40:12.188319+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:40:12.188319+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8072,"log_dups_size":0,"ondisk_log_size":8072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38810784,"num_objects":4738,"num_object_clones":0,"num_object_copies":18952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4738,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8070,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2488}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'8280","reported_seq":8382,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:12.063370+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:40:12.063370+0000","last_peered":"2024-09-05T16:40:12.063370+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:40:12.063370+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8280,"log_dups_size":0,"ondisk_log_size":8280,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39669488,"num_objects":4843,"num_object_clones":0,"num_object_copies":19372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2293,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4843,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8277,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2553}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'8023","reported_seq":8098,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:11.939773+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:40:11.939773+0000","last_peered":"2024-09-05T16:40:11.939773+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:40:11.939773+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8023,"log_dups_size":0,"ondisk_log_size":8023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38795856,"num_objects":4736,"num_object_clones":0,"num_object_copies":18944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2248,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4736,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8022,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2489}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'8086","reported_seq":12216,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:12.284030+0000","last_change":"2024-09-05T16:37:30.634570+0000","last_active":"2024-09-05T16:40:12.284030+0000","last_peered":"2024-09-05T16:40:12.284030+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:40:12.284030+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8086,"log_dups_size":0,"ondisk_log_size":8086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38689360,"num_objects":4723,"num_object_clones":0,"num_object_copies":18892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4723,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8085,"num_write_kb":51231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":16580608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4453}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'8276","reported_seq":8367,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:12.003553+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:40:12.003553+0000","last_peered":"2024-09-05T16:40:12.003553+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:40:12.003553+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8276,"log_dups_size":0,"ondisk_log_size":8276,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39728288,"num_objects":4850,"num_object_clones":0,"num_object_copies":19400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2342,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4850,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8274,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2510}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'8093","reported_seq":8192,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:12.334530+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:40:12.334530+0000","last_peered":"2024-09-05T16:40:12.334530+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:40:12.334530+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8093,"log_dups_size":0,"ondisk_log_size":8093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38827168,"num_objects":4740,"num_object_clones":0,"num_object_copies":18960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4740,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8091,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2428}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'8043","reported_seq":8136,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:11.673626+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:40:11.673626+0000","last_peered":"2024-09-05T16:40:11.673626+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:40:11.673626+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8043,"log_dups_size":0,"ondisk_log_size":8043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38761632,"num_objects":4732,"num_object_clones":0,"num_object_copies":18928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2227,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4732,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8041,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2507}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":622636088,"num_objects":76011,"num_object_clones":0,"num_object_copies":304044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31695,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76011,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":123226,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2238,"num_bytes_recovered":16580608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1208979456,"data_stored":1208979456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128951,"ondisk_log_size":128951,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084363,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":759516,"kb_used_data":304388,"kb_used_omap":4,"kb_used_meta":455099,"kb_avail":93612324,"statfs":{"total":96636764160,"available":95859019776,"internally_reserved":0,"allocated":311693312,"data_stored":311501271,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":466022099},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,5,4,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986240,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":575868,"kb_used_data":269292,"kb_used_omap":4,"kb_used_meta":306555,"kb_avail":93795972,"statfs":{"total":96636764160,"available":96047075328,"internally_reserved":0,"allocated":275755008,"data_stored":275562967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":313913042},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673046,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":746444,"kb_used_data":304316,"kb_used_omap":1,"kb_used_meta":442110,"kb_avail":93625396,"statfs":{"total":96636764160,"available":95872405504,"internally_reserved":0,"allocated":311619584,"data_stored":311427543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":452721093},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,1,7,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182477,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":747228,"kb_used_data":304624,"kb_used_omap":4,"kb_used_meta":442555,"kb_avail":93624612,"statfs":{"total":96636764160,"available":95871602688,"internally_reserved":0,"allocated":311934976,"data_stored":311742935,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":453177037},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,1,3,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":311316480,"data_stored":311316480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":275443712,"data_stored":275443712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":310980608,"data_stored":310980608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":311238656,"data_stored":311238656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:40:13.188 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-05T16:40:13.188 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'8147", 'reported_seq': 8234, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:11.834701+0000', 'last_change': '2024-09-05T16:34:56.833497+0000', 'last_active': '2024-09-05T16:40:11.834701+0000', 'last_peered': '2024-09-05T16:40:11.834701+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.828289+0000', 'last_became_peered': '2024-09-05T16:34:56.828289+0000', 'last_unstale': '2024-09-05T16:40:11.834701+0000', 'last_undegraded': '2024-09-05T16:34:56.612160+0000', 'last_fullsized': '2024-09-05T16:34:56.557632+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8147, 'log_dups_size': 0, 'ondisk_log_size': 8147, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38969344, 'num_objects': 4757, 'num_object_clones': 0, 'num_object_copies': 19028, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2242, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4757, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8147, 'num_write_kb': 51616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2515}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:40:13.188 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-05T16:40:13.188 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "35'7943", 'reported_seq': 8034, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:11.820685+0000', 'last_change': '2024-09-05T16:34:56.825031+0000', 'last_active': '2024-09-05T16:40:11.820685+0000', 'last_peered': '2024-09-05T16:40:11.820685+0000', 'last_clean': '2024-09-05T16:33:53.200762+0000', 'last_became_active': '2024-09-05T16:34:56.820292+0000', 'last_became_peered': '2024-09-05T16:34:56.820292+0000', 'last_unstale': '2024-09-05T16:40:11.820685+0000', 'last_undegraded': '2024-09-05T16:34:56.605776+0000', 'last_fullsized': '2024-09-05T16:34:56.559383+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7943, 'log_dups_size': 0, 'ondisk_log_size': 7943, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37770400, 'num_objects': 4611, 'num_object_clones': 0, 'num_object_copies': 18444, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2258, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4611, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7941, 'num_write_kb': 50212, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2355}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:40:13.188 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:40:13.188 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'8028", 'reported_seq': 8125, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:12.313104+0000', 'last_change': '2024-09-05T16:34:56.867900+0000', 'last_active': '2024-09-05T16:40:12.313104+0000', 'last_peered': '2024-09-05T16:40:12.313104+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:40:12.313104+0000', 'last_undegraded': '2024-09-05T16:34:56.590972+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8028, 'log_dups_size': 0, 'ondisk_log_size': 8028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38697576, 'num_objects': 4725, 'num_object_clones': 0, 'num_object_copies': 18900, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2243, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4725, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 8027, 'num_write_kb': 50999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2483}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:40:13.188 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:40:13.188 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'7883", 'reported_seq': 7966, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:11.966408+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:40:11.966408+0000', 'last_peered': '2024-09-05T16:40:11.966408+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:40:11.966408+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7883, 'log_dups_size': 0, 'ondisk_log_size': 7883, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38597792, 'num_objects': 4712, 'num_object_clones': 0, 'num_object_copies': 18848, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2152, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4712, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7881, 'num_write_kb': 50373, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2562}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:40:13.189 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:40:13.189 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'8011", 'reported_seq': 8100, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:12.037210+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:40:12.037210+0000', 'last_peered': '2024-09-05T16:40:12.037210+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:40:12.037210+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8011, 'log_dups_size': 0, 'ondisk_log_size': 8011, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39002112, 'num_objects': 4761, 'num_object_clones': 0, 'num_object_copies': 19044, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2229, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4761, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8011, 'num_write_kb': 51088, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2532}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:40:13.189 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:40:13.189 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'7843", 'reported_seq': 7931, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:11.932187+0000', 'last_change': '2024-09-05T16:34:56.903436+0000', 'last_active': '2024-09-05T16:40:11.932187+0000', 'last_peered': '2024-09-05T16:40:11.932187+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:40:11.932187+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7843, 'log_dups_size': 0, 'ondisk_log_size': 7843, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39474336, 'num_objects': 4819, 'num_object_clones': 0, 'num_object_copies': 19276, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2140, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4819, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7841, 'num_write_kb': 50636, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2681}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:40:13.189 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-05T16:40:13.189 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "35'8016", 'reported_seq': 8097, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:11.775134+0000', 'last_change': '2024-09-05T16:34:56.883319+0000', 'last_active': '2024-09-05T16:40:11.775134+0000', 'last_peered': '2024-09-05T16:40:11.775134+0000', 'last_clean': '2024-09-05T16:33:53.505305+0000', 'last_became_active': '2024-09-05T16:34:56.873389+0000', 'last_became_peered': '2024-09-05T16:34:56.873389+0000', 'last_unstale': '2024-09-05T16:40:11.775134+0000', 'last_undegraded': '2024-09-05T16:34:56.602567+0000', 'last_fullsized': '2024-09-05T16:34:56.559958+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8016, 'log_dups_size': 0, 'ondisk_log_size': 8016, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39015584, 'num_objects': 4763, 'num_object_clones': 0, 'num_object_copies': 19052, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2235, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4763, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8014, 'num_write_kb': 51108, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 8, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2530}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:40:13.189 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:40:13.189 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'8072", 'reported_seq': 8161, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:12.066586+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:40:12.066586+0000', 'last_peered': '2024-09-05T16:40:12.066586+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:40:12.066586+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8072, 'log_dups_size': 0, 'ondisk_log_size': 8072, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38512960, 'num_objects': 4702, 'num_object_clones': 0, 'num_object_copies': 18808, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4702, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8069, 'num_write_kb': 51081, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2455}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:40:13.189 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:40:13.189 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'8072", 'reported_seq': 8179, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:12.188319+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:40:12.188319+0000', 'last_peered': '2024-09-05T16:40:12.188319+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:40:12.188319+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8072, 'log_dups_size': 0, 'ondisk_log_size': 8072, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38810784, 'num_objects': 4738, 'num_object_clones': 0, 'num_object_copies': 18952, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2252, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4738, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8070, 'num_write_kb': 51230, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2488}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:40:13.190 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:40:13.190 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'8280", 'reported_seq': 8382, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:12.063370+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:40:12.063370+0000', 'last_peered': '2024-09-05T16:40:12.063370+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:40:12.063370+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8280, 'log_dups_size': 0, 'ondisk_log_size': 8280, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39669488, 'num_objects': 4843, 'num_object_clones': 0, 'num_object_copies': 19372, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2293, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4843, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8277, 'num_write_kb': 52476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 24, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2553}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:40:13.190 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:40:13.190 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'8023", 'reported_seq': 8098, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:11.939773+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:40:11.939773+0000', 'last_peered': '2024-09-05T16:40:11.939773+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:40:11.939773+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8023, 'log_dups_size': 0, 'ondisk_log_size': 8023, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38795856, 'num_objects': 4736, 'num_object_clones': 0, 'num_object_copies': 18944, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2248, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4736, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8022, 'num_write_kb': 51031, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2489}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:40:13.190 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-05T16:40:13.190 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'8086", 'reported_seq': 12216, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:12.284030+0000', 'last_change': '2024-09-05T16:37:30.634570+0000', 'last_active': '2024-09-05T16:40:12.284030+0000', 'last_peered': '2024-09-05T16:40:12.284030+0000', 'last_clean': '2024-09-05T16:33:45.082039+0000', 'last_became_active': '2024-09-05T16:34:56.792386+0000', 'last_became_peered': '2024-09-05T16:34:56.792386+0000', 'last_unstale': '2024-09-05T16:40:12.284030+0000', 'last_undegraded': '2024-09-05T16:34:56.600595+0000', 'last_fullsized': '2024-09-05T16:34:56.558203+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8086, 'log_dups_size': 0, 'ondisk_log_size': 8086, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38689360, 'num_objects': 4723, 'num_object_clones': 0, 'num_object_copies': 18892, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 270, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4723, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8085, 'num_write_kb': 51231, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2026, 'num_bytes_recovered': 16580608, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 4453}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:40:13.190 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:40:13.190 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'8276", 'reported_seq': 8367, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:12.003553+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:40:12.003553+0000', 'last_peered': '2024-09-05T16:40:12.003553+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:40:12.003553+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8276, 'log_dups_size': 0, 'ondisk_log_size': 8276, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39728288, 'num_objects': 4850, 'num_object_clones': 0, 'num_object_copies': 19400, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2342, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4850, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8274, 'num_write_kb': 52500, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2510}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:40:13.190 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:40:13.190 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'8093", 'reported_seq': 8192, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:12.334530+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:40:12.334530+0000', 'last_peered': '2024-09-05T16:40:12.334530+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:40:12.334530+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8093, 'log_dups_size': 0, 'ondisk_log_size': 8093, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38827168, 'num_objects': 4740, 'num_object_clones': 0, 'num_object_copies': 18960, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2314, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4740, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8091, 'num_write_kb': 51325, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2428}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:40:13.190 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:40:13.191 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'8043", 'reported_seq': 8136, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:11.673626+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:40:11.673626+0000', 'last_peered': '2024-09-05T16:40:11.673626+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:40:11.673626+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8043, 'log_dups_size': 0, 'ondisk_log_size': 8043, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38761632, 'num_objects': 4732, 'num_object_clones': 0, 'num_object_copies': 18928, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2227, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4732, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8041, 'num_write_kb': 51092, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2507}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:40:13.191 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:40:13.931 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:40:13.931 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 31695/304044 objects degraded (10.424%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":417,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":76011,"data_bytes":622636088,"bytes_used":2896953344,"bytes_avail":383650103296,"bytes_total":386547056640,"degraded_objects":31695,"degraded_total":304044,"degraded_ratio":0.10424478036073728,"recovering_objects_per_sec":26,"recovering_bytes_per_sec":194489,"recovering_keys_per_sec":0,"num_objects_recovered":321,"num_bytes_recovered":2334720,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":277},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:40:13.957 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:40:13.958 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:40:14.529 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:40:14.530 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:40:14.550 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":234,"stamp":"2024-09-05T16:40:12.927310+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":622636088,"num_objects":76011,"num_object_clones":0,"num_object_copies":304044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31695,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76011,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":123226,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2238,"num_bytes_recovered":16580608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128951,"ondisk_log_size":128951,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2829056,"kb_used_data":1182620,"kb_used_omap":14,"kb_used_meta":1646321,"kb_avail":374658304,"statfs":{"total":386547056640,"available":383650103296,"internally_reserved":0,"allocated":1211002880,"data_stored":1210234716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1685833271},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,2,8,11,14,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-27303936,"num_objects":-3333,"num_object_clones":0,"num_object_copies":-13332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3333,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3333,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":321,"num_bytes_recovered":2334720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004374"},"pg_stats":[{"pgid":"1.f","version":"35'8147","reported_seq":8234,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:11.834701+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:40:11.834701+0000","last_peered":"2024-09-05T16:40:11.834701+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:40:11.834701+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8147,"log_dups_size":0,"ondisk_log_size":8147,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38969344,"num_objects":4757,"num_object_clones":0,"num_object_copies":19028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2242,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4757,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8147,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2515}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'7943","reported_seq":8034,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:11.820685+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:40:11.820685+0000","last_peered":"2024-09-05T16:40:11.820685+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:40:11.820685+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7943,"log_dups_size":0,"ondisk_log_size":7943,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37770400,"num_objects":4611,"num_object_clones":0,"num_object_copies":18444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2258,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4611,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7941,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2355}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'8028","reported_seq":8125,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:12.313104+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:40:12.313104+0000","last_peered":"2024-09-05T16:40:12.313104+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:40:12.313104+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8028,"log_dups_size":0,"ondisk_log_size":8028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38697576,"num_objects":4725,"num_object_clones":0,"num_object_copies":18900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2243,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4725,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8027,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2483}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'8135","reported_seq":12859,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-05T16:40:11.259465+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:40:11.259465+0000","last_peered":"2024-09-05T16:40:11.259465+0000","last_clean":"2024-09-05T16:40:11.259465+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:40:11.259465+0000","last_undegraded":"2024-09-05T16:40:11.259465+0000","last_fullsized":"2024-09-05T16:40:11.259465+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":8135,"log_dups_size":0,"ondisk_log_size":8135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39313408,"num_objects":4799,"num_object_clones":0,"num_object_copies":19196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4799,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2435,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'7883","reported_seq":7966,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:11.966408+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:40:11.966408+0000","last_peered":"2024-09-05T16:40:11.966408+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:40:11.966408+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7883,"log_dups_size":0,"ondisk_log_size":7883,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38597792,"num_objects":4712,"num_object_clones":0,"num_object_copies":18848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2152,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4712,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7881,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2562}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'8011","reported_seq":8100,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:12.037210+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:40:12.037210+0000","last_peered":"2024-09-05T16:40:12.037210+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:40:12.037210+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8011,"log_dups_size":0,"ondisk_log_size":8011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39002112,"num_objects":4761,"num_object_clones":0,"num_object_copies":19044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2229,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8011,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2532}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'7843","reported_seq":7931,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:11.932187+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:40:11.932187+0000","last_peered":"2024-09-05T16:40:11.932187+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:40:11.932187+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7843,"log_dups_size":0,"ondisk_log_size":7843,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39474336,"num_objects":4819,"num_object_clones":0,"num_object_copies":19276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2140,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4819,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7841,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2681}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'8016","reported_seq":8097,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:11.775134+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:40:11.775134+0000","last_peered":"2024-09-05T16:40:11.775134+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:40:11.775134+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8016,"log_dups_size":0,"ondisk_log_size":8016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39015584,"num_objects":4763,"num_object_clones":0,"num_object_copies":19052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4763,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8014,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2530}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'8072","reported_seq":8161,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:12.066586+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:40:12.066586+0000","last_peered":"2024-09-05T16:40:12.066586+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:40:12.066586+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8072,"log_dups_size":0,"ondisk_log_size":8072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38512960,"num_objects":4702,"num_object_clones":0,"num_object_copies":18808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4702,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8069,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2455}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'8072","reported_seq":8179,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:12.188319+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:40:12.188319+0000","last_peered":"2024-09-05T16:40:12.188319+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:40:12.188319+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8072,"log_dups_size":0,"ondisk_log_size":8072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38810784,"num_objects":4738,"num_object_clones":0,"num_object_copies":18952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4738,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8070,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2488}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'8280","reported_seq":8382,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:12.063370+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:40:12.063370+0000","last_peered":"2024-09-05T16:40:12.063370+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:40:12.063370+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8280,"log_dups_size":0,"ondisk_log_size":8280,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39669488,"num_objects":4843,"num_object_clones":0,"num_object_copies":19372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2293,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4843,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8277,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2553}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'8023","reported_seq":8098,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:11.939773+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:40:11.939773+0000","last_peered":"2024-09-05T16:40:11.939773+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:40:11.939773+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8023,"log_dups_size":0,"ondisk_log_size":8023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38795856,"num_objects":4736,"num_object_clones":0,"num_object_copies":18944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2248,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4736,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8022,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2489}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'8086","reported_seq":12216,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:12.284030+0000","last_change":"2024-09-05T16:37:30.634570+0000","last_active":"2024-09-05T16:40:12.284030+0000","last_peered":"2024-09-05T16:40:12.284030+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:40:12.284030+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8086,"log_dups_size":0,"ondisk_log_size":8086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38689360,"num_objects":4723,"num_object_clones":0,"num_object_copies":18892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4723,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8085,"num_write_kb":51231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":16580608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4453}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'8276","reported_seq":8367,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:12.003553+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:40:12.003553+0000","last_peered":"2024-09-05T16:40:12.003553+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:40:12.003553+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8276,"log_dups_size":0,"ondisk_log_size":8276,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39728288,"num_objects":4850,"num_object_clones":0,"num_object_copies":19400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2342,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4850,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8274,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2510}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'8093","reported_seq":8192,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:12.334530+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:40:12.334530+0000","last_peered":"2024-09-05T16:40:12.334530+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:40:12.334530+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8093,"log_dups_size":0,"ondisk_log_size":8093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38827168,"num_objects":4740,"num_object_clones":0,"num_object_copies":18960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4740,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8091,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2428}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'8043","reported_seq":8136,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:11.673626+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:40:11.673626+0000","last_peered":"2024-09-05T16:40:11.673626+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:40:11.673626+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8043,"log_dups_size":0,"ondisk_log_size":8043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38761632,"num_objects":4732,"num_object_clones":0,"num_object_copies":18928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2227,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4732,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8041,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2507}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":622636088,"num_objects":76011,"num_object_clones":0,"num_object_copies":304044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31695,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76011,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":123226,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2238,"num_bytes_recovered":16580608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1208979456,"data_stored":1208979456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128951,"ondisk_log_size":128951,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084363,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":759516,"kb_used_data":304388,"kb_used_omap":4,"kb_used_meta":455099,"kb_avail":93612324,"statfs":{"total":96636764160,"available":95859019776,"internally_reserved":0,"allocated":311693312,"data_stored":311501271,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":466022099},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,5,4,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986240,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":575868,"kb_used_data":269292,"kb_used_omap":4,"kb_used_meta":306555,"kb_avail":93795972,"statfs":{"total":96636764160,"available":96047075328,"internally_reserved":0,"allocated":275755008,"data_stored":275562967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":313913042},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673046,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":746444,"kb_used_data":304316,"kb_used_omap":1,"kb_used_meta":442110,"kb_avail":93625396,"statfs":{"total":96636764160,"available":95872405504,"internally_reserved":0,"allocated":311619584,"data_stored":311427543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":452721093},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,1,7,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182477,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":747228,"kb_used_data":304624,"kb_used_omap":4,"kb_used_meta":442555,"kb_avail":93624612,"statfs":{"total":96636764160,"available":95871602688,"internally_reserved":0,"allocated":311934976,"data_stored":311742935,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":453177037},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,1,3,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":311316480,"data_stored":311316480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":275443712,"data_stored":275443712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":310980608,"data_stored":310980608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":311238656,"data_stored":311238656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:40:17.552 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:40:18.184 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:40:18.185 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:40:18.214 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":236,"stamp":"2024-09-05T16:40:16.928977+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":621964344,"num_objects":75929,"num_object_clones":0,"num_object_copies":303716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31695,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":75929,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":123308,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2238,"num_bytes_recovered":16580608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":129033,"ondisk_log_size":129033,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2832884,"kb_used_data":1182288,"kb_used_omap":14,"kb_used_meta":1650481,"kb_avail":374654476,"statfs":{"total":386547056640,"available":383646183424,"internally_reserved":0,"allocated":1210662912,"data_stored":1209894748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1690093111},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,2,6,9,11,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18882560,"num_objects":-2305,"num_object_clones":0,"num_object_copies":-9220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-214,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2305,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2305,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":214,"num_bytes_recovered":1540096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004242"},"pg_stats":[{"pgid":"1.f","version":"35'8147","reported_seq":8234,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:11.834701+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:40:11.834701+0000","last_peered":"2024-09-05T16:40:11.834701+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:40:11.834701+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8147,"log_dups_size":0,"ondisk_log_size":8147,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38969344,"num_objects":4757,"num_object_clones":0,"num_object_copies":19028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2242,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4757,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8147,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2515}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'7943","reported_seq":8034,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:11.820685+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:40:11.820685+0000","last_peered":"2024-09-05T16:40:11.820685+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:40:11.820685+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7943,"log_dups_size":0,"ondisk_log_size":7943,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37770400,"num_objects":4611,"num_object_clones":0,"num_object_copies":18444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2258,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4611,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7941,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2355}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'8028","reported_seq":8125,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:12.313104+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:40:12.313104+0000","last_peered":"2024-09-05T16:40:12.313104+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:40:12.313104+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8028,"log_dups_size":0,"ondisk_log_size":8028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38697576,"num_objects":4725,"num_object_clones":0,"num_object_copies":18900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2243,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4725,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8027,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2483}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'8217","reported_seq":12941,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-05T16:40:16.219672+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:40:16.219672+0000","last_peered":"2024-09-05T16:40:16.219672+0000","last_clean":"2024-09-05T16:40:16.219672+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:40:16.219672+0000","last_undegraded":"2024-09-05T16:40:16.219672+0000","last_fullsized":"2024-09-05T16:40:16.219672+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":8217,"log_dups_size":0,"ondisk_log_size":8217,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38641664,"num_objects":4717,"num_object_clones":0,"num_object_copies":18868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4717,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2517,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'7883","reported_seq":7966,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:11.966408+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:40:11.966408+0000","last_peered":"2024-09-05T16:40:11.966408+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:40:11.966408+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7883,"log_dups_size":0,"ondisk_log_size":7883,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38597792,"num_objects":4712,"num_object_clones":0,"num_object_copies":18848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2152,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4712,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7881,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2562}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'8011","reported_seq":8100,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:12.037210+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:40:12.037210+0000","last_peered":"2024-09-05T16:40:12.037210+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:40:12.037210+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8011,"log_dups_size":0,"ondisk_log_size":8011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39002112,"num_objects":4761,"num_object_clones":0,"num_object_copies":19044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2229,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8011,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2532}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'7843","reported_seq":7931,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:11.932187+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:40:11.932187+0000","last_peered":"2024-09-05T16:40:11.932187+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:40:11.932187+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7843,"log_dups_size":0,"ondisk_log_size":7843,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39474336,"num_objects":4819,"num_object_clones":0,"num_object_copies":19276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2140,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4819,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7841,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2681}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'8016","reported_seq":8097,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:11.775134+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:40:11.775134+0000","last_peered":"2024-09-05T16:40:11.775134+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:40:11.775134+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8016,"log_dups_size":0,"ondisk_log_size":8016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39015584,"num_objects":4763,"num_object_clones":0,"num_object_copies":19052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4763,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8014,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2530}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'8072","reported_seq":8161,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:12.066586+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:40:12.066586+0000","last_peered":"2024-09-05T16:40:12.066586+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:40:12.066586+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8072,"log_dups_size":0,"ondisk_log_size":8072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38512960,"num_objects":4702,"num_object_clones":0,"num_object_copies":18808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4702,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8069,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2455}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'8072","reported_seq":8179,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:12.188319+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:40:12.188319+0000","last_peered":"2024-09-05T16:40:12.188319+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:40:12.188319+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8072,"log_dups_size":0,"ondisk_log_size":8072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38810784,"num_objects":4738,"num_object_clones":0,"num_object_copies":18952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4738,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8070,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2488}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'8280","reported_seq":8382,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:12.063370+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:40:12.063370+0000","last_peered":"2024-09-05T16:40:12.063370+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:40:12.063370+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8280,"log_dups_size":0,"ondisk_log_size":8280,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39669488,"num_objects":4843,"num_object_clones":0,"num_object_copies":19372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2293,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4843,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8277,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2553}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'8023","reported_seq":8098,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:11.939773+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:40:11.939773+0000","last_peered":"2024-09-05T16:40:11.939773+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:40:11.939773+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8023,"log_dups_size":0,"ondisk_log_size":8023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38795856,"num_objects":4736,"num_object_clones":0,"num_object_copies":18944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2248,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4736,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8022,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2489}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'8086","reported_seq":12216,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:12.284030+0000","last_change":"2024-09-05T16:37:30.634570+0000","last_active":"2024-09-05T16:40:12.284030+0000","last_peered":"2024-09-05T16:40:12.284030+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:40:12.284030+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8086,"log_dups_size":0,"ondisk_log_size":8086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38689360,"num_objects":4723,"num_object_clones":0,"num_object_copies":18892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4723,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8085,"num_write_kb":51231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":16580608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4453}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'8276","reported_seq":8367,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:12.003553+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:40:12.003553+0000","last_peered":"2024-09-05T16:40:12.003553+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:40:12.003553+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8276,"log_dups_size":0,"ondisk_log_size":8276,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39728288,"num_objects":4850,"num_object_clones":0,"num_object_copies":19400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2342,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4850,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8274,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2510}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'8093","reported_seq":8192,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:12.334530+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:40:12.334530+0000","last_peered":"2024-09-05T16:40:12.334530+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:40:12.334530+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8093,"log_dups_size":0,"ondisk_log_size":8093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38827168,"num_objects":4740,"num_object_clones":0,"num_object_copies":18960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4740,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8091,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2428}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'8043","reported_seq":8136,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:11.673626+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:40:11.673626+0000","last_peered":"2024-09-05T16:40:11.673626+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:40:11.673626+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8043,"log_dups_size":0,"ondisk_log_size":8043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38761632,"num_objects":4732,"num_object_clones":0,"num_object_copies":18928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2227,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4732,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8041,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2507}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":621964344,"num_objects":75929,"num_object_clones":0,"num_object_copies":303716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31695,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":75929,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":123308,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2238,"num_bytes_recovered":16580608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1208619008,"data_stored":1208619008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":129033,"ondisk_log_size":129033,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084363,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":759516,"kb_used_data":304388,"kb_used_omap":4,"kb_used_meta":455099,"kb_avail":93612324,"statfs":{"total":96636764160,"available":95859019776,"internally_reserved":0,"allocated":311693312,"data_stored":311501271,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":466022099},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,5,4,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986241,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":579696,"kb_used_data":268960,"kb_used_omap":4,"kb_used_meta":310715,"kb_avail":93792144,"statfs":{"total":96636764160,"available":96043155456,"internally_reserved":0,"allocated":275415040,"data_stored":275222999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":318172882},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673046,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":746444,"kb_used_data":304316,"kb_used_omap":1,"kb_used_meta":442110,"kb_avail":93625396,"statfs":{"total":96636764160,"available":95872405504,"internally_reserved":0,"allocated":311619584,"data_stored":311427543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":452721093},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,1,7,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182477,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":747228,"kb_used_data":304624,"kb_used_omap":4,"kb_used_meta":442555,"kb_avail":93624612,"statfs":{"total":96636764160,"available":95871602688,"internally_reserved":0,"allocated":311934976,"data_stored":311742935,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":453177037},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,1,3,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":311316480,"data_stored":311316480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":275083264,"data_stored":275083264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":310980608,"data_stored":310980608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":311238656,"data_stored":311238656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:40:18.215 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:40:18.837 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:40:18.837 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:40:18.857 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":236,"stamp":"2024-09-05T16:40:16.928977+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":621964344,"num_objects":75929,"num_object_clones":0,"num_object_copies":303716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31695,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":75929,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":123308,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2238,"num_bytes_recovered":16580608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":129033,"ondisk_log_size":129033,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2832884,"kb_used_data":1182288,"kb_used_omap":14,"kb_used_meta":1650481,"kb_avail":374654476,"statfs":{"total":386547056640,"available":383646183424,"internally_reserved":0,"allocated":1210662912,"data_stored":1209894748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1690093111},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,2,6,9,11,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18882560,"num_objects":-2305,"num_object_clones":0,"num_object_copies":-9220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-214,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2305,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2305,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":214,"num_bytes_recovered":1540096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004242"},"pg_stats":[{"pgid":"1.f","version":"35'8147","reported_seq":8234,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:11.834701+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:40:11.834701+0000","last_peered":"2024-09-05T16:40:11.834701+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:40:11.834701+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8147,"log_dups_size":0,"ondisk_log_size":8147,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38969344,"num_objects":4757,"num_object_clones":0,"num_object_copies":19028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2242,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4757,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8147,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2515}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'7943","reported_seq":8034,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:11.820685+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:40:11.820685+0000","last_peered":"2024-09-05T16:40:11.820685+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:40:11.820685+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7943,"log_dups_size":0,"ondisk_log_size":7943,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37770400,"num_objects":4611,"num_object_clones":0,"num_object_copies":18444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2258,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4611,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7941,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2355}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'8028","reported_seq":8125,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:12.313104+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:40:12.313104+0000","last_peered":"2024-09-05T16:40:12.313104+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:40:12.313104+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8028,"log_dups_size":0,"ondisk_log_size":8028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38697576,"num_objects":4725,"num_object_clones":0,"num_object_copies":18900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2243,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4725,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8027,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2483}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'8217","reported_seq":12941,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-05T16:40:16.219672+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:40:16.219672+0000","last_peered":"2024-09-05T16:40:16.219672+0000","last_clean":"2024-09-05T16:40:16.219672+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:40:16.219672+0000","last_undegraded":"2024-09-05T16:40:16.219672+0000","last_fullsized":"2024-09-05T16:40:16.219672+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":8217,"log_dups_size":0,"ondisk_log_size":8217,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38641664,"num_objects":4717,"num_object_clones":0,"num_object_copies":18868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4717,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2517,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'7883","reported_seq":7966,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:11.966408+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:40:11.966408+0000","last_peered":"2024-09-05T16:40:11.966408+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:40:11.966408+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7883,"log_dups_size":0,"ondisk_log_size":7883,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38597792,"num_objects":4712,"num_object_clones":0,"num_object_copies":18848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2152,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4712,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7881,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2562}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'8011","reported_seq":8100,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:12.037210+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:40:12.037210+0000","last_peered":"2024-09-05T16:40:12.037210+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:40:12.037210+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8011,"log_dups_size":0,"ondisk_log_size":8011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39002112,"num_objects":4761,"num_object_clones":0,"num_object_copies":19044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2229,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8011,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2532}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'7843","reported_seq":7931,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:11.932187+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:40:11.932187+0000","last_peered":"2024-09-05T16:40:11.932187+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:40:11.932187+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7843,"log_dups_size":0,"ondisk_log_size":7843,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39474336,"num_objects":4819,"num_object_clones":0,"num_object_copies":19276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2140,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4819,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7841,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2681}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'8016","reported_seq":8097,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:11.775134+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:40:11.775134+0000","last_peered":"2024-09-05T16:40:11.775134+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:40:11.775134+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8016,"log_dups_size":0,"ondisk_log_size":8016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39015584,"num_objects":4763,"num_object_clones":0,"num_object_copies":19052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4763,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8014,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2530}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'8072","reported_seq":8161,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:12.066586+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:40:12.066586+0000","last_peered":"2024-09-05T16:40:12.066586+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:40:12.066586+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8072,"log_dups_size":0,"ondisk_log_size":8072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38512960,"num_objects":4702,"num_object_clones":0,"num_object_copies":18808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4702,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8069,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2455}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'8072","reported_seq":8179,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:12.188319+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:40:12.188319+0000","last_peered":"2024-09-05T16:40:12.188319+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:40:12.188319+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8072,"log_dups_size":0,"ondisk_log_size":8072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38810784,"num_objects":4738,"num_object_clones":0,"num_object_copies":18952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4738,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8070,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2488}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'8280","reported_seq":8382,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:12.063370+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:40:12.063370+0000","last_peered":"2024-09-05T16:40:12.063370+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:40:12.063370+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8280,"log_dups_size":0,"ondisk_log_size":8280,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39669488,"num_objects":4843,"num_object_clones":0,"num_object_copies":19372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2293,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4843,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8277,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2553}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'8023","reported_seq":8098,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:11.939773+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:40:11.939773+0000","last_peered":"2024-09-05T16:40:11.939773+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:40:11.939773+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8023,"log_dups_size":0,"ondisk_log_size":8023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38795856,"num_objects":4736,"num_object_clones":0,"num_object_copies":18944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2248,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4736,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8022,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2489}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'8086","reported_seq":12216,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:12.284030+0000","last_change":"2024-09-05T16:37:30.634570+0000","last_active":"2024-09-05T16:40:12.284030+0000","last_peered":"2024-09-05T16:40:12.284030+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:40:12.284030+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8086,"log_dups_size":0,"ondisk_log_size":8086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38689360,"num_objects":4723,"num_object_clones":0,"num_object_copies":18892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4723,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8085,"num_write_kb":51231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":16580608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4453}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'8276","reported_seq":8367,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:12.003553+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:40:12.003553+0000","last_peered":"2024-09-05T16:40:12.003553+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:40:12.003553+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8276,"log_dups_size":0,"ondisk_log_size":8276,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39728288,"num_objects":4850,"num_object_clones":0,"num_object_copies":19400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2342,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4850,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8274,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2510}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'8093","reported_seq":8192,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:12.334530+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:40:12.334530+0000","last_peered":"2024-09-05T16:40:12.334530+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:40:12.334530+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8093,"log_dups_size":0,"ondisk_log_size":8093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38827168,"num_objects":4740,"num_object_clones":0,"num_object_copies":18960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4740,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8091,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2428}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'8043","reported_seq":8136,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:11.673626+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:40:11.673626+0000","last_peered":"2024-09-05T16:40:11.673626+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:40:11.673626+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8043,"log_dups_size":0,"ondisk_log_size":8043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38761632,"num_objects":4732,"num_object_clones":0,"num_object_copies":18928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2227,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4732,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8041,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2507}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":621964344,"num_objects":75929,"num_object_clones":0,"num_object_copies":303716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31695,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":75929,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":123308,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2238,"num_bytes_recovered":16580608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1208619008,"data_stored":1208619008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":129033,"ondisk_log_size":129033,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084363,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":759516,"kb_used_data":304388,"kb_used_omap":4,"kb_used_meta":455099,"kb_avail":93612324,"statfs":{"total":96636764160,"available":95859019776,"internally_reserved":0,"allocated":311693312,"data_stored":311501271,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":466022099},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,5,4,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986241,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":579696,"kb_used_data":268960,"kb_used_omap":4,"kb_used_meta":310715,"kb_avail":93792144,"statfs":{"total":96636764160,"available":96043155456,"internally_reserved":0,"allocated":275415040,"data_stored":275222999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":318172882},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673046,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":746444,"kb_used_data":304316,"kb_used_omap":1,"kb_used_meta":442110,"kb_avail":93625396,"statfs":{"total":96636764160,"available":95872405504,"internally_reserved":0,"allocated":311619584,"data_stored":311427543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":452721093},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,1,7,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182477,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":747228,"kb_used_data":304624,"kb_used_omap":4,"kb_used_meta":442555,"kb_avail":93624612,"statfs":{"total":96636764160,"available":95871602688,"internally_reserved":0,"allocated":311934976,"data_stored":311742935,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":453177037},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,1,3,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":311316480,"data_stored":311316480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":275083264,"data_stored":275083264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":310980608,"data_stored":310980608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":311238656,"data_stored":311238656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:40:18.859 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-05T16:40:18.859 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'8147", 'reported_seq': 8234, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:11.834701+0000', 'last_change': '2024-09-05T16:34:56.833497+0000', 'last_active': '2024-09-05T16:40:11.834701+0000', 'last_peered': '2024-09-05T16:40:11.834701+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.828289+0000', 'last_became_peered': '2024-09-05T16:34:56.828289+0000', 'last_unstale': '2024-09-05T16:40:11.834701+0000', 'last_undegraded': '2024-09-05T16:34:56.612160+0000', 'last_fullsized': '2024-09-05T16:34:56.557632+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8147, 'log_dups_size': 0, 'ondisk_log_size': 8147, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38969344, 'num_objects': 4757, 'num_object_clones': 0, 'num_object_copies': 19028, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2242, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4757, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8147, 'num_write_kb': 51616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2515}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:40:18.860 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-05T16:40:18.860 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "35'7943", 'reported_seq': 8034, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:11.820685+0000', 'last_change': '2024-09-05T16:34:56.825031+0000', 'last_active': '2024-09-05T16:40:11.820685+0000', 'last_peered': '2024-09-05T16:40:11.820685+0000', 'last_clean': '2024-09-05T16:33:53.200762+0000', 'last_became_active': '2024-09-05T16:34:56.820292+0000', 'last_became_peered': '2024-09-05T16:34:56.820292+0000', 'last_unstale': '2024-09-05T16:40:11.820685+0000', 'last_undegraded': '2024-09-05T16:34:56.605776+0000', 'last_fullsized': '2024-09-05T16:34:56.559383+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7943, 'log_dups_size': 0, 'ondisk_log_size': 7943, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37770400, 'num_objects': 4611, 'num_object_clones': 0, 'num_object_copies': 18444, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2258, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4611, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7941, 'num_write_kb': 50212, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2355}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:40:18.860 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:40:18.860 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'8028", 'reported_seq': 8125, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:12.313104+0000', 'last_change': '2024-09-05T16:34:56.867900+0000', 'last_active': '2024-09-05T16:40:12.313104+0000', 'last_peered': '2024-09-05T16:40:12.313104+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:40:12.313104+0000', 'last_undegraded': '2024-09-05T16:34:56.590972+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8028, 'log_dups_size': 0, 'ondisk_log_size': 8028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38697576, 'num_objects': 4725, 'num_object_clones': 0, 'num_object_copies': 18900, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2243, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4725, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 8027, 'num_write_kb': 50999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2483}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:40:18.860 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:40:18.860 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'7883", 'reported_seq': 7966, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:11.966408+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:40:11.966408+0000', 'last_peered': '2024-09-05T16:40:11.966408+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:40:11.966408+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7883, 'log_dups_size': 0, 'ondisk_log_size': 7883, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38597792, 'num_objects': 4712, 'num_object_clones': 0, 'num_object_copies': 18848, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2152, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4712, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7881, 'num_write_kb': 50373, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2562}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:40:18.861 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:40:18.861 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'8011", 'reported_seq': 8100, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:12.037210+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:40:12.037210+0000', 'last_peered': '2024-09-05T16:40:12.037210+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:40:12.037210+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8011, 'log_dups_size': 0, 'ondisk_log_size': 8011, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39002112, 'num_objects': 4761, 'num_object_clones': 0, 'num_object_copies': 19044, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2229, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4761, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8011, 'num_write_kb': 51088, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2532}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:40:18.861 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:40:18.861 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'7843", 'reported_seq': 7931, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:11.932187+0000', 'last_change': '2024-09-05T16:34:56.903436+0000', 'last_active': '2024-09-05T16:40:11.932187+0000', 'last_peered': '2024-09-05T16:40:11.932187+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:40:11.932187+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7843, 'log_dups_size': 0, 'ondisk_log_size': 7843, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39474336, 'num_objects': 4819, 'num_object_clones': 0, 'num_object_copies': 19276, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2140, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4819, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7841, 'num_write_kb': 50636, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2681}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:40:18.861 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-05T16:40:18.861 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "35'8016", 'reported_seq': 8097, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:11.775134+0000', 'last_change': '2024-09-05T16:34:56.883319+0000', 'last_active': '2024-09-05T16:40:11.775134+0000', 'last_peered': '2024-09-05T16:40:11.775134+0000', 'last_clean': '2024-09-05T16:33:53.505305+0000', 'last_became_active': '2024-09-05T16:34:56.873389+0000', 'last_became_peered': '2024-09-05T16:34:56.873389+0000', 'last_unstale': '2024-09-05T16:40:11.775134+0000', 'last_undegraded': '2024-09-05T16:34:56.602567+0000', 'last_fullsized': '2024-09-05T16:34:56.559958+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8016, 'log_dups_size': 0, 'ondisk_log_size': 8016, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39015584, 'num_objects': 4763, 'num_object_clones': 0, 'num_object_copies': 19052, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2235, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4763, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8014, 'num_write_kb': 51108, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 8, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2530}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:40:18.862 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:40:18.862 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'8072", 'reported_seq': 8161, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:12.066586+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:40:12.066586+0000', 'last_peered': '2024-09-05T16:40:12.066586+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:40:12.066586+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8072, 'log_dups_size': 0, 'ondisk_log_size': 8072, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38512960, 'num_objects': 4702, 'num_object_clones': 0, 'num_object_copies': 18808, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4702, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8069, 'num_write_kb': 51081, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2455}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:40:18.862 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:40:18.862 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'8072", 'reported_seq': 8179, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:12.188319+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:40:12.188319+0000', 'last_peered': '2024-09-05T16:40:12.188319+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:40:12.188319+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8072, 'log_dups_size': 0, 'ondisk_log_size': 8072, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38810784, 'num_objects': 4738, 'num_object_clones': 0, 'num_object_copies': 18952, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2252, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4738, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8070, 'num_write_kb': 51230, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2488}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:40:18.862 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:40:18.862 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'8280", 'reported_seq': 8382, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:12.063370+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:40:12.063370+0000', 'last_peered': '2024-09-05T16:40:12.063370+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:40:12.063370+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8280, 'log_dups_size': 0, 'ondisk_log_size': 8280, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39669488, 'num_objects': 4843, 'num_object_clones': 0, 'num_object_copies': 19372, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2293, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4843, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8277, 'num_write_kb': 52476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 24, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2553}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:40:18.862 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:40:18.862 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'8023", 'reported_seq': 8098, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:11.939773+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:40:11.939773+0000', 'last_peered': '2024-09-05T16:40:11.939773+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:40:11.939773+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8023, 'log_dups_size': 0, 'ondisk_log_size': 8023, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38795856, 'num_objects': 4736, 'num_object_clones': 0, 'num_object_copies': 18944, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2248, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4736, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8022, 'num_write_kb': 51031, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2489}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:40:18.862 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-05T16:40:18.863 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'8086", 'reported_seq': 12216, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:12.284030+0000', 'last_change': '2024-09-05T16:37:30.634570+0000', 'last_active': '2024-09-05T16:40:12.284030+0000', 'last_peered': '2024-09-05T16:40:12.284030+0000', 'last_clean': '2024-09-05T16:33:45.082039+0000', 'last_became_active': '2024-09-05T16:34:56.792386+0000', 'last_became_peered': '2024-09-05T16:34:56.792386+0000', 'last_unstale': '2024-09-05T16:40:12.284030+0000', 'last_undegraded': '2024-09-05T16:34:56.600595+0000', 'last_fullsized': '2024-09-05T16:34:56.558203+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8086, 'log_dups_size': 0, 'ondisk_log_size': 8086, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38689360, 'num_objects': 4723, 'num_object_clones': 0, 'num_object_copies': 18892, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 270, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4723, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8085, 'num_write_kb': 51231, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2026, 'num_bytes_recovered': 16580608, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 4453}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:40:18.863 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:40:18.863 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'8276", 'reported_seq': 8367, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:12.003553+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:40:12.003553+0000', 'last_peered': '2024-09-05T16:40:12.003553+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:40:12.003553+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8276, 'log_dups_size': 0, 'ondisk_log_size': 8276, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39728288, 'num_objects': 4850, 'num_object_clones': 0, 'num_object_copies': 19400, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2342, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4850, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8274, 'num_write_kb': 52500, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2510}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:40:18.863 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:40:18.863 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'8093", 'reported_seq': 8192, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:12.334530+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:40:12.334530+0000', 'last_peered': '2024-09-05T16:40:12.334530+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:40:12.334530+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8093, 'log_dups_size': 0, 'ondisk_log_size': 8093, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38827168, 'num_objects': 4740, 'num_object_clones': 0, 'num_object_copies': 18960, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2314, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4740, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8091, 'num_write_kb': 51325, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2428}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:40:18.863 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:40:18.863 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'8043", 'reported_seq': 8136, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:11.673626+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:40:11.673626+0000', 'last_peered': '2024-09-05T16:40:11.673626+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:40:11.673626+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8043, 'log_dups_size': 0, 'ondisk_log_size': 8043, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38761632, 'num_objects': 4732, 'num_object_clones': 0, 'num_object_copies': 18928, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2227, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4732, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8041, 'num_write_kb': 51092, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2507}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:40:18.863 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:40:19.611 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:40:19.611 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 31621/299388 objects degraded (10.562%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":422,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":74847,"data_bytes":613100600,"bytes_used":2886643712,"bytes_avail":383660412928,"bytes_total":386547056640,"degraded_objects":31621,"degraded_total":299388,"degraded_ratio":0.10561879567651342,"recovering_objects_per_sec":23,"recovering_bytes_per_sec":175384,"recovering_keys_per_sec":0,"num_objects_recovered":288,"num_bytes_recovered":2105344,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":276},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:40:19.632 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:40:19.633 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:40:20.240 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:40:20.240 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:40:20.260 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":237,"stamp":"2024-09-05T16:40:18.929664+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":613100600,"num_objects":74847,"num_object_clones":0,"num_object_copies":299388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31621,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74847,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":124390,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2312,"num_bytes_recovered":17145856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130115,"ondisk_log_size":130115,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2818988,"kb_used_data":1168412,"kb_used_omap":14,"kb_used_meta":1650481,"kb_avail":374668372,"statfs":{"total":386547056640,"available":383660412928,"internally_reserved":0,"allocated":1196453888,"data_stored":1195685724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1690093111},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,1,10,8,8,12,6,0,0,0,4],"upper_bound":4096},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-27213824,"num_objects":-3322,"num_object_clones":0,"num_object_copies":-13288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-288,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3322,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3322,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":2105344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004165"},"pg_stats":[{"pgid":"1.f","version":"35'8212","reported_seq":8301,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:16.937662+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:40:16.937662+0000","last_peered":"2024-09-05T16:40:16.937662+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:40:16.937662+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8212,"log_dups_size":0,"ondisk_log_size":8212,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38436864,"num_objects":4692,"num_object_clones":0,"num_object_copies":18768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4692,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8212,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2451}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'8023","reported_seq":8115,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:17.041385+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:40:17.041385+0000","last_peered":"2024-09-05T16:40:17.041385+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:40:17.041385+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8023,"log_dups_size":0,"ondisk_log_size":8023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37115040,"num_objects":4531,"num_object_clones":0,"num_object_copies":18124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8021,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2276}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'8093","reported_seq":8190,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:17.405339+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:40:17.405339+0000","last_peered":"2024-09-05T16:40:17.405339+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:40:17.405339+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8093,"log_dups_size":0,"ondisk_log_size":8093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38165096,"num_objects":4660,"num_object_clones":0,"num_object_copies":18640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2243,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4660,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8092,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2418}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'8217","reported_seq":12941,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-05T16:40:16.219672+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:40:16.219672+0000","last_peered":"2024-09-05T16:40:16.219672+0000","last_clean":"2024-09-05T16:40:16.219672+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:40:16.219672+0000","last_undegraded":"2024-09-05T16:40:16.219672+0000","last_fullsized":"2024-09-05T16:40:16.219672+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":8217,"log_dups_size":0,"ondisk_log_size":8217,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38641664,"num_objects":4717,"num_object_clones":0,"num_object_copies":18868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4717,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2517,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'7942","reported_seq":8027,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:16.867802+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:40:16.867802+0000","last_peered":"2024-09-05T16:40:16.867802+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:40:16.867802+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7942,"log_dups_size":0,"ondisk_log_size":7942,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38114464,"num_objects":4653,"num_object_clones":0,"num_object_copies":18612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2151,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4653,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7940,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2504}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'8074","reported_seq":8163,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:16.885817+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:40:16.885817+0000","last_peered":"2024-09-05T16:40:16.885817+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:40:16.885817+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8074,"log_dups_size":0,"ondisk_log_size":8074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38486016,"num_objects":4698,"num_object_clones":0,"num_object_copies":18792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2229,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4698,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8074,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2469}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'7913","reported_seq":8001,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:16.827062+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:40:16.827062+0000","last_peered":"2024-09-05T16:40:16.827062+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:40:16.827062+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7913,"log_dups_size":0,"ondisk_log_size":7913,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38900896,"num_objects":4749,"num_object_clones":0,"num_object_copies":18996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2140,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4749,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7911,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2611}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'8084","reported_seq":8165,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:16.721660+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:40:16.721660+0000","last_peered":"2024-09-05T16:40:16.721660+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:40:16.721660+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8084,"log_dups_size":0,"ondisk_log_size":8084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38458528,"num_objects":4695,"num_object_clones":0,"num_object_copies":18780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4695,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8082,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2462}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'8147","reported_seq":8236,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:17.053778+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:40:17.053778+0000","last_peered":"2024-09-05T16:40:17.053778+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:40:17.053778+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8147,"log_dups_size":0,"ondisk_log_size":8147,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37898560,"num_objects":4627,"num_object_clones":0,"num_object_copies":18508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4627,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8144,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2380}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'8147","reported_seq":8254,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:17.263456+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:40:17.263456+0000","last_peered":"2024-09-05T16:40:17.263456+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:40:17.263456+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8147,"log_dups_size":0,"ondisk_log_size":8147,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38196384,"num_objects":4663,"num_object_clones":0,"num_object_copies":18652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8145,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2413}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'8355","reported_seq":8458,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:17.026339+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:40:17.026339+0000","last_peered":"2024-09-05T16:40:17.026339+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:40:17.026339+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8355,"log_dups_size":0,"ondisk_log_size":8355,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39055088,"num_objects":4768,"num_object_clones":0,"num_object_copies":19072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2293,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4768,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8352,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2478}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'8101","reported_seq":8178,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:17.000872+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:40:17.000872+0000","last_peered":"2024-09-05T16:40:17.000872+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:40:17.000872+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8101,"log_dups_size":0,"ondisk_log_size":8101,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38156880,"num_objects":4658,"num_object_clones":0,"num_object_copies":18632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4658,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8100,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2412}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'8164","reported_seq":12432,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:17.356437+0000","last_change":"2024-09-05T16:37:30.634570+0000","last_active":"2024-09-05T16:40:17.356437+0000","last_peered":"2024-09-05T16:40:17.356437+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:40:17.356437+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8164,"log_dups_size":0,"ondisk_log_size":8164,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38050384,"num_objects":4645,"num_object_clones":0,"num_object_copies":18580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":201,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4645,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8163,"num_write_kb":51231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":17145856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4444}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'8352","reported_seq":8445,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:17.029397+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:40:17.029397+0000","last_peered":"2024-09-05T16:40:17.029397+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:40:17.029397+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8352,"log_dups_size":0,"ondisk_log_size":8352,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39105696,"num_objects":4774,"num_object_clones":0,"num_object_copies":19096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2341,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4774,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8350,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2435}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'8180","reported_seq":8279,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:17.316975+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:40:17.316975+0000","last_peered":"2024-09-05T16:40:17.316975+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:40:17.316975+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8180,"log_dups_size":0,"ondisk_log_size":8180,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38114464,"num_objects":4653,"num_object_clones":0,"num_object_copies":18612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4653,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8178,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2341}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'8111","reported_seq":8204,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:16.931981+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:40:16.931981+0000","last_peered":"2024-09-05T16:40:16.931981+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:40:16.931981+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8111,"log_dups_size":0,"ondisk_log_size":8111,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38204576,"num_objects":4664,"num_object_clones":0,"num_object_copies":18656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2227,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4664,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8109,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2439}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":613100600,"num_objects":74847,"num_object_clones":0,"num_object_copies":299388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31621,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74847,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":124390,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2312,"num_bytes_recovered":17145856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1194283008,"data_stored":1194283008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130115,"ondisk_log_size":130115,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084364,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":742368,"kb_used_data":299724,"kb_used_omap":4,"kb_used_meta":442619,"kb_avail":93629472,"statfs":{"total":96636764160,"available":95876579328,"internally_reserved":0,"allocated":306917376,"data_stored":306725335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":453242579},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,1,7,4,1,0,0,0,4],"upper_bound":4096},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986241,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":579696,"kb_used_data":268960,"kb_used_omap":4,"kb_used_meta":310715,"kb_avail":93792144,"statfs":{"total":96636764160,"available":96043155456,"internally_reserved":0,"allocated":275415040,"data_stored":275222999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":318172882},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673047,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":746072,"kb_used_data":299772,"kb_used_omap":1,"kb_used_meta":446270,"kb_avail":93625768,"statfs":{"total":96636764160,"available":95872786432,"internally_reserved":0,"allocated":306966528,"data_stored":306774487,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":456980933},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2,5,1,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182478,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":750852,"kb_used_data":299956,"kb_used_omap":4,"kb_used_meta":450875,"kb_avail":93620988,"statfs":{"total":96636764160,"available":95867891712,"internally_reserved":0,"allocated":307154944,"data_stored":306962903,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":461696717},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,8,1,0,4,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":306536448,"data_stored":306536448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":275083264,"data_stored":275083264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":306180096,"data_stored":306180096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":306483200,"data_stored":306483200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:40:23.263 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:40:23.891 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:40:23.891 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:40:23.926 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":239,"stamp":"2024-09-05T16:40:22.931443+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":603966520,"num_objects":73732,"num_object_clones":0,"num_object_copies":294928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31517,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73732,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":125505,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2416,"num_bytes_recovered":17932288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":131230,"ondisk_log_size":131230,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2877764,"kb_used_data":1154980,"kb_used_omap":14,"kb_used_meta":1722673,"kb_avail":374609596,"statfs":{"total":386547056640,"available":383600226304,"internally_reserved":0,"allocated":1182699520,"data_stored":1181931356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1764017719},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,5,4,5,13,4,0,5,0,0,0,1,4],"upper_bound":16384},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-28024832,"num_objects":-3421,"num_object_clones":0,"num_object_copies":-13684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-282,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3421,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3421,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":282,"num_bytes_recovered":2129920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004697"},"pg_stats":[{"pgid":"1.f","version":"35'8269","reported_seq":8362,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:21.713420+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:40:21.713420+0000","last_peered":"2024-09-05T16:40:21.713420+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:40:21.713420+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8269,"log_dups_size":0,"ondisk_log_size":8269,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37969920,"num_objects":4635,"num_object_clones":0,"num_object_copies":18540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4635,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8269,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2396}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'8105","reported_seq":8198,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:22.030172+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:40:22.030172+0000","last_peered":"2024-09-05T16:40:22.030172+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:40:22.030172+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8105,"log_dups_size":0,"ondisk_log_size":8105,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36443296,"num_objects":4449,"num_object_clones":0,"num_object_copies":17796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4449,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8103,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2194}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'8175","reported_seq":8272,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:22.309441+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:40:22.309441+0000","last_peered":"2024-09-05T16:40:22.309441+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:40:22.309441+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8175,"log_dups_size":0,"ondisk_log_size":8175,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37493352,"num_objects":4578,"num_object_clones":0,"num_object_copies":18312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2243,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4578,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8174,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2336}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'8293","reported_seq":13017,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-05T16:40:21.296712+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:40:21.296712+0000","last_peered":"2024-09-05T16:40:21.296712+0000","last_clean":"2024-09-05T16:40:21.296712+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:40:21.296712+0000","last_undegraded":"2024-09-05T16:40:21.296712+0000","last_fullsized":"2024-09-05T16:40:21.296712+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":8293,"log_dups_size":0,"ondisk_log_size":8293,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38019072,"num_objects":4641,"num_object_clones":0,"num_object_copies":18564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4641,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2593,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'8008","reported_seq":8095,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:22.102765+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:40:22.102765+0000","last_peered":"2024-09-05T16:40:22.102765+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:40:22.102765+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8008,"log_dups_size":0,"ondisk_log_size":8008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37573792,"num_objects":4587,"num_object_clones":0,"num_object_copies":18348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2150,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4587,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8006,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2439}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'8150","reported_seq":8241,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:22.079179+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:40:22.079179+0000","last_peered":"2024-09-05T16:40:22.079179+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:40:22.079179+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8150,"log_dups_size":0,"ondisk_log_size":8150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37863424,"num_objects":4622,"num_object_clones":0,"num_object_copies":18488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4622,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8150,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2394}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'7975","reported_seq":8064,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:21.910365+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:40:21.910365+0000","last_peered":"2024-09-05T16:40:21.910365+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:40:21.910365+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7975,"log_dups_size":0,"ondisk_log_size":7975,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38392992,"num_objects":4687,"num_object_clones":0,"num_object_copies":18748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2140,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7973,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2549}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'8153","reported_seq":8234,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:21.808447+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:40:21.808447+0000","last_peered":"2024-09-05T16:40:21.808447+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:40:21.808447+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8153,"log_dups_size":0,"ondisk_log_size":8153,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37893280,"num_objects":4626,"num_object_clones":0,"num_object_copies":18504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4626,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8151,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2393}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'8212","reported_seq":8303,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:22.086149+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:40:22.086149+0000","last_peered":"2024-09-05T16:40:22.086149+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:40:22.086149+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8212,"log_dups_size":0,"ondisk_log_size":8212,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37366080,"num_objects":4562,"num_object_clones":0,"num_object_copies":18248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2249,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4562,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8209,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2316}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'8199","reported_seq":8306,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:22.355489+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:40:22.355489+0000","last_peered":"2024-09-05T16:40:22.355489+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:40:22.355489+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8199,"log_dups_size":0,"ondisk_log_size":8199,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37770400,"num_objects":4611,"num_object_clones":0,"num_object_copies":18444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4611,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8197,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2361}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'8424","reported_seq":8531,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:22.133596+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:40:22.133596+0000","last_peered":"2024-09-05T16:40:22.133596+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:40:22.133596+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8424,"log_dups_size":0,"ondisk_log_size":8424,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38489840,"num_objects":4699,"num_object_clones":0,"num_object_copies":18796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4699,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8421,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2411}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'8178","reported_seq":8256,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:22.141319+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:40:22.141319+0000","last_peered":"2024-09-05T16:40:22.141319+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:40:22.141319+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8178,"log_dups_size":0,"ondisk_log_size":8178,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37526096,"num_objects":4581,"num_object_clones":0,"num_object_copies":18324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2246,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4581,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8177,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2336}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'8241","reported_seq":12701,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:22.465407+0000","last_change":"2024-09-05T16:37:30.634570+0000","last_active":"2024-09-05T16:40:22.465407+0000","last_peered":"2024-09-05T16:40:22.465407+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:40:22.465407+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8241,"log_dups_size":0,"ondisk_log_size":8241,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37419600,"num_objects":4568,"num_object_clones":0,"num_object_copies":18272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":105,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4568,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8240,"num_write_kb":51231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2191,"num_bytes_recovered":17932288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4463}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'8442","reported_seq":8535,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:22.106378+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:40:22.106378+0000","last_peered":"2024-09-05T16:40:22.106378+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:40:22.106378+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8442,"log_dups_size":0,"ondisk_log_size":8442,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38368416,"num_objects":4684,"num_object_clones":0,"num_object_copies":18736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2341,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4684,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8440,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2345}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'8248","reported_seq":8347,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:22.422720+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:40:22.422720+0000","last_peered":"2024-09-05T16:40:22.422720+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:40:22.422720+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8248,"log_dups_size":0,"ondisk_log_size":8248,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37557408,"num_objects":4585,"num_object_clones":0,"num_object_copies":18340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4585,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8246,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2273}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'8158","reported_seq":8251,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:21.863407+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:40:21.863407+0000","last_peered":"2024-09-05T16:40:21.863407+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:40:21.863407+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8158,"log_dups_size":0,"ondisk_log_size":8158,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37819552,"num_objects":4617,"num_object_clones":0,"num_object_copies":18468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2227,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4617,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8156,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2392}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":603966520,"num_objects":73732,"num_object_clones":0,"num_object_copies":294928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31517,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73732,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":125505,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2416,"num_bytes_recovered":17932288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1180426240,"data_stored":1180426240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":131230,"ondisk_log_size":131230,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084365,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":760144,"kb_used_data":295296,"kb_used_omap":4,"kb_used_meta":464827,"kb_avail":93611696,"statfs":{"total":96636764160,"available":95858376704,"internally_reserved":0,"allocated":302383104,"data_stored":302191063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":475983571},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,2,0,2,3,0,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986242,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":584952,"kb_used_data":268648,"kb_used_omap":4,"kb_used_meta":316283,"kb_avail":93786888,"statfs":{"total":96636764160,"available":96037773312,"internally_reserved":0,"allocated":275095552,"data_stored":274903511,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":323874514},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673048,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":763964,"kb_used_data":295456,"kb_used_omap":1,"kb_used_meta":468478,"kb_avail":93607876,"statfs":{"total":96636764160,"available":95854465024,"internally_reserved":0,"allocated":302546944,"data_stored":302354903,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":479721925},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,3,8,0,0,0,0,0,0,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182479,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":768704,"kb_used_data":295580,"kb_used_omap":4,"kb_used_meta":473083,"kb_avail":93603136,"statfs":{"total":96636764160,"available":95849611264,"internally_reserved":0,"allocated":302673920,"data_stored":302481879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":484437709},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2,0,3,0,0,0,0,0,0,0,4],"upper_bound":16384},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":302030848,"data_stored":302030848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":274792448,"data_stored":274792448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":301658112,"data_stored":301658112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":301944832,"data_stored":301944832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:40:23.927 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:40:24.530 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:40:24.530 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:40:24.556 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":239,"stamp":"2024-09-05T16:40:22.931443+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":603966520,"num_objects":73732,"num_object_clones":0,"num_object_copies":294928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31517,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73732,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":125505,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2416,"num_bytes_recovered":17932288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":131230,"ondisk_log_size":131230,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2877764,"kb_used_data":1154980,"kb_used_omap":14,"kb_used_meta":1722673,"kb_avail":374609596,"statfs":{"total":386547056640,"available":383600226304,"internally_reserved":0,"allocated":1182699520,"data_stored":1181931356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1764017719},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,5,4,5,13,4,0,5,0,0,0,1,4],"upper_bound":16384},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-28024832,"num_objects":-3421,"num_object_clones":0,"num_object_copies":-13684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-282,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3421,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3421,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":282,"num_bytes_recovered":2129920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004697"},"pg_stats":[{"pgid":"1.f","version":"35'8269","reported_seq":8362,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:21.713420+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:40:21.713420+0000","last_peered":"2024-09-05T16:40:21.713420+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:40:21.713420+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8269,"log_dups_size":0,"ondisk_log_size":8269,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37969920,"num_objects":4635,"num_object_clones":0,"num_object_copies":18540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4635,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8269,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2396}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'8105","reported_seq":8198,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:22.030172+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:40:22.030172+0000","last_peered":"2024-09-05T16:40:22.030172+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:40:22.030172+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8105,"log_dups_size":0,"ondisk_log_size":8105,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36443296,"num_objects":4449,"num_object_clones":0,"num_object_copies":17796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4449,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8103,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2194}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'8175","reported_seq":8272,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:22.309441+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:40:22.309441+0000","last_peered":"2024-09-05T16:40:22.309441+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:40:22.309441+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8175,"log_dups_size":0,"ondisk_log_size":8175,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37493352,"num_objects":4578,"num_object_clones":0,"num_object_copies":18312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2243,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4578,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8174,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2336}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'8293","reported_seq":13017,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-05T16:40:21.296712+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:40:21.296712+0000","last_peered":"2024-09-05T16:40:21.296712+0000","last_clean":"2024-09-05T16:40:21.296712+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:40:21.296712+0000","last_undegraded":"2024-09-05T16:40:21.296712+0000","last_fullsized":"2024-09-05T16:40:21.296712+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":8293,"log_dups_size":0,"ondisk_log_size":8293,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38019072,"num_objects":4641,"num_object_clones":0,"num_object_copies":18564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4641,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2593,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'8008","reported_seq":8095,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:22.102765+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:40:22.102765+0000","last_peered":"2024-09-05T16:40:22.102765+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:40:22.102765+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8008,"log_dups_size":0,"ondisk_log_size":8008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37573792,"num_objects":4587,"num_object_clones":0,"num_object_copies":18348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2150,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4587,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8006,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2439}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'8150","reported_seq":8241,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:22.079179+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:40:22.079179+0000","last_peered":"2024-09-05T16:40:22.079179+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:40:22.079179+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8150,"log_dups_size":0,"ondisk_log_size":8150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37863424,"num_objects":4622,"num_object_clones":0,"num_object_copies":18488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4622,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8150,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2394}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'7975","reported_seq":8064,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:21.910365+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:40:21.910365+0000","last_peered":"2024-09-05T16:40:21.910365+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:40:21.910365+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7975,"log_dups_size":0,"ondisk_log_size":7975,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38392992,"num_objects":4687,"num_object_clones":0,"num_object_copies":18748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2140,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7973,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2549}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'8153","reported_seq":8234,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:21.808447+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:40:21.808447+0000","last_peered":"2024-09-05T16:40:21.808447+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:40:21.808447+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8153,"log_dups_size":0,"ondisk_log_size":8153,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37893280,"num_objects":4626,"num_object_clones":0,"num_object_copies":18504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4626,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8151,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2393}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'8212","reported_seq":8303,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:22.086149+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:40:22.086149+0000","last_peered":"2024-09-05T16:40:22.086149+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:40:22.086149+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8212,"log_dups_size":0,"ondisk_log_size":8212,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37366080,"num_objects":4562,"num_object_clones":0,"num_object_copies":18248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2249,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4562,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8209,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2316}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'8199","reported_seq":8306,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:22.355489+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:40:22.355489+0000","last_peered":"2024-09-05T16:40:22.355489+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:40:22.355489+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8199,"log_dups_size":0,"ondisk_log_size":8199,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37770400,"num_objects":4611,"num_object_clones":0,"num_object_copies":18444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4611,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8197,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2361}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'8424","reported_seq":8531,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:22.133596+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:40:22.133596+0000","last_peered":"2024-09-05T16:40:22.133596+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:40:22.133596+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8424,"log_dups_size":0,"ondisk_log_size":8424,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38489840,"num_objects":4699,"num_object_clones":0,"num_object_copies":18796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4699,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8421,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2411}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'8178","reported_seq":8256,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:22.141319+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:40:22.141319+0000","last_peered":"2024-09-05T16:40:22.141319+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:40:22.141319+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8178,"log_dups_size":0,"ondisk_log_size":8178,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37526096,"num_objects":4581,"num_object_clones":0,"num_object_copies":18324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2246,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4581,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8177,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2336}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'8241","reported_seq":12701,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:22.465407+0000","last_change":"2024-09-05T16:37:30.634570+0000","last_active":"2024-09-05T16:40:22.465407+0000","last_peered":"2024-09-05T16:40:22.465407+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:40:22.465407+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8241,"log_dups_size":0,"ondisk_log_size":8241,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37419600,"num_objects":4568,"num_object_clones":0,"num_object_copies":18272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":105,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4568,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8240,"num_write_kb":51231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2191,"num_bytes_recovered":17932288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4463}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'8442","reported_seq":8535,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:22.106378+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:40:22.106378+0000","last_peered":"2024-09-05T16:40:22.106378+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:40:22.106378+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8442,"log_dups_size":0,"ondisk_log_size":8442,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38368416,"num_objects":4684,"num_object_clones":0,"num_object_copies":18736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2341,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4684,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8440,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2345}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'8248","reported_seq":8347,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:22.422720+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:40:22.422720+0000","last_peered":"2024-09-05T16:40:22.422720+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:40:22.422720+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8248,"log_dups_size":0,"ondisk_log_size":8248,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37557408,"num_objects":4585,"num_object_clones":0,"num_object_copies":18340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4585,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8246,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2273}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'8158","reported_seq":8251,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:21.863407+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:40:21.863407+0000","last_peered":"2024-09-05T16:40:21.863407+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:40:21.863407+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8158,"log_dups_size":0,"ondisk_log_size":8158,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37819552,"num_objects":4617,"num_object_clones":0,"num_object_copies":18468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2227,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4617,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8156,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2392}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":603966520,"num_objects":73732,"num_object_clones":0,"num_object_copies":294928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31517,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73732,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":125505,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2416,"num_bytes_recovered":17932288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1180426240,"data_stored":1180426240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":131230,"ondisk_log_size":131230,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084365,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":760144,"kb_used_data":295296,"kb_used_omap":4,"kb_used_meta":464827,"kb_avail":93611696,"statfs":{"total":96636764160,"available":95858376704,"internally_reserved":0,"allocated":302383104,"data_stored":302191063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":475983571},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,2,0,2,3,0,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986242,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":584952,"kb_used_data":268648,"kb_used_omap":4,"kb_used_meta":316283,"kb_avail":93786888,"statfs":{"total":96636764160,"available":96037773312,"internally_reserved":0,"allocated":275095552,"data_stored":274903511,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":323874514},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673048,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":763964,"kb_used_data":295456,"kb_used_omap":1,"kb_used_meta":468478,"kb_avail":93607876,"statfs":{"total":96636764160,"available":95854465024,"internally_reserved":0,"allocated":302546944,"data_stored":302354903,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":479721925},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,3,8,0,0,0,0,0,0,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182479,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":768704,"kb_used_data":295580,"kb_used_omap":4,"kb_used_meta":473083,"kb_avail":93603136,"statfs":{"total":96636764160,"available":95849611264,"internally_reserved":0,"allocated":302673920,"data_stored":302481879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":484437709},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2,0,3,0,0,0,0,0,0,0,4],"upper_bound":16384},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":302030848,"data_stored":302030848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":274792448,"data_stored":274792448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":301658112,"data_stored":301658112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":301944832,"data_stored":301944832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:40:24.557 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-05T16:40:24.557 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'8269", 'reported_seq': 8362, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:21.713420+0000', 'last_change': '2024-09-05T16:34:56.833497+0000', 'last_active': '2024-09-05T16:40:21.713420+0000', 'last_peered': '2024-09-05T16:40:21.713420+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.828289+0000', 'last_became_peered': '2024-09-05T16:34:56.828289+0000', 'last_unstale': '2024-09-05T16:40:21.713420+0000', 'last_undegraded': '2024-09-05T16:34:56.612160+0000', 'last_fullsized': '2024-09-05T16:34:56.557632+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8269, 'log_dups_size': 0, 'ondisk_log_size': 8269, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37969920, 'num_objects': 4635, 'num_object_clones': 0, 'num_object_copies': 18540, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4635, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8269, 'num_write_kb': 51616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 23, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2396}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:40:24.557 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-05T16:40:24.557 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "35'8105", 'reported_seq': 8198, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:22.030172+0000', 'last_change': '2024-09-05T16:34:56.825031+0000', 'last_active': '2024-09-05T16:40:22.030172+0000', 'last_peered': '2024-09-05T16:40:22.030172+0000', 'last_clean': '2024-09-05T16:33:53.200762+0000', 'last_became_active': '2024-09-05T16:34:56.820292+0000', 'last_became_peered': '2024-09-05T16:34:56.820292+0000', 'last_unstale': '2024-09-05T16:40:22.030172+0000', 'last_undegraded': '2024-09-05T16:34:56.605776+0000', 'last_fullsized': '2024-09-05T16:34:56.559383+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8105, 'log_dups_size': 0, 'ondisk_log_size': 8105, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36443296, 'num_objects': 4449, 'num_object_clones': 0, 'num_object_copies': 17796, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4449, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8103, 'num_write_kb': 50212, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2194}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:40:24.558 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:40:24.558 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'8175", 'reported_seq': 8272, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:22.309441+0000', 'last_change': '2024-09-05T16:34:56.867900+0000', 'last_active': '2024-09-05T16:40:22.309441+0000', 'last_peered': '2024-09-05T16:40:22.309441+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:40:22.309441+0000', 'last_undegraded': '2024-09-05T16:34:56.590972+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8175, 'log_dups_size': 0, 'ondisk_log_size': 8175, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37493352, 'num_objects': 4578, 'num_object_clones': 0, 'num_object_copies': 18312, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2243, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4578, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 8174, 'num_write_kb': 50999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2336}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:40:24.558 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:40:24.558 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'8008", 'reported_seq': 8095, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:22.102765+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:40:22.102765+0000', 'last_peered': '2024-09-05T16:40:22.102765+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:40:22.102765+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8008, 'log_dups_size': 0, 'ondisk_log_size': 8008, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37573792, 'num_objects': 4587, 'num_object_clones': 0, 'num_object_copies': 18348, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2150, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4587, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8006, 'num_write_kb': 50373, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 16, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2439}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:40:24.558 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:40:24.558 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'8150", 'reported_seq': 8241, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:22.079179+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:40:22.079179+0000', 'last_peered': '2024-09-05T16:40:22.079179+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:40:22.079179+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8150, 'log_dups_size': 0, 'ondisk_log_size': 8150, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37863424, 'num_objects': 4622, 'num_object_clones': 0, 'num_object_copies': 18488, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2228, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4622, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8150, 'num_write_kb': 51088, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 22, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2394}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:40:24.558 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:40:24.558 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'7975", 'reported_seq': 8064, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:21.910365+0000', 'last_change': '2024-09-05T16:34:56.903436+0000', 'last_active': '2024-09-05T16:40:21.910365+0000', 'last_peered': '2024-09-05T16:40:21.910365+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:40:21.910365+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 7975, 'log_dups_size': 0, 'ondisk_log_size': 7975, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38392992, 'num_objects': 4687, 'num_object_clones': 0, 'num_object_copies': 18748, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2140, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4687, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7973, 'num_write_kb': 50636, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2549}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:40:24.558 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-05T16:40:24.558 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "35'8153", 'reported_seq': 8234, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:21.808447+0000', 'last_change': '2024-09-05T16:34:56.883319+0000', 'last_active': '2024-09-05T16:40:21.808447+0000', 'last_peered': '2024-09-05T16:40:21.808447+0000', 'last_clean': '2024-09-05T16:33:53.505305+0000', 'last_became_active': '2024-09-05T16:34:56.873389+0000', 'last_became_peered': '2024-09-05T16:34:56.873389+0000', 'last_unstale': '2024-09-05T16:40:21.808447+0000', 'last_undegraded': '2024-09-05T16:34:56.602567+0000', 'last_fullsized': '2024-09-05T16:34:56.559958+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8153, 'log_dups_size': 0, 'ondisk_log_size': 8153, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37893280, 'num_objects': 4626, 'num_object_clones': 0, 'num_object_copies': 18504, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2235, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4626, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8151, 'num_write_kb': 51108, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 8, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2393}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:40:24.559 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:40:24.559 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'8212", 'reported_seq': 8303, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:22.086149+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:40:22.086149+0000', 'last_peered': '2024-09-05T16:40:22.086149+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:40:22.086149+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8212, 'log_dups_size': 0, 'ondisk_log_size': 8212, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37366080, 'num_objects': 4562, 'num_object_clones': 0, 'num_object_copies': 18248, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2249, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4562, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8209, 'num_write_kb': 51081, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2316}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:40:24.559 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:40:24.559 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'8199", 'reported_seq': 8306, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:22.355489+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:40:22.355489+0000', 'last_peered': '2024-09-05T16:40:22.355489+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:40:22.355489+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8199, 'log_dups_size': 0, 'ondisk_log_size': 8199, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37770400, 'num_objects': 4611, 'num_object_clones': 0, 'num_object_copies': 18444, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2252, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4611, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8197, 'num_write_kb': 51230, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2361}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:40:24.559 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:40:24.559 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'8424", 'reported_seq': 8531, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:22.133596+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:40:22.133596+0000', 'last_peered': '2024-09-05T16:40:22.133596+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:40:22.133596+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8424, 'log_dups_size': 0, 'ondisk_log_size': 8424, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38489840, 'num_objects': 4699, 'num_object_clones': 0, 'num_object_copies': 18796, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2291, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4699, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8421, 'num_write_kb': 52476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2411}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:40:24.559 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:40:24.559 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'8178", 'reported_seq': 8256, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:22.141319+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:40:22.141319+0000', 'last_peered': '2024-09-05T16:40:22.141319+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:40:22.141319+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8178, 'log_dups_size': 0, 'ondisk_log_size': 8178, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37526096, 'num_objects': 4581, 'num_object_clones': 0, 'num_object_copies': 18324, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2246, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4581, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8177, 'num_write_kb': 51031, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2336}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:40:24.559 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-05T16:40:24.559 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'8241", 'reported_seq': 12701, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:22.465407+0000', 'last_change': '2024-09-05T16:37:30.634570+0000', 'last_active': '2024-09-05T16:40:22.465407+0000', 'last_peered': '2024-09-05T16:40:22.465407+0000', 'last_clean': '2024-09-05T16:33:45.082039+0000', 'last_became_active': '2024-09-05T16:34:56.792386+0000', 'last_became_peered': '2024-09-05T16:34:56.792386+0000', 'last_unstale': '2024-09-05T16:40:22.465407+0000', 'last_undegraded': '2024-09-05T16:34:56.600595+0000', 'last_fullsized': '2024-09-05T16:34:56.558203+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8241, 'log_dups_size': 0, 'ondisk_log_size': 8241, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37419600, 'num_objects': 4568, 'num_object_clones': 0, 'num_object_copies': 18272, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 105, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4568, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8240, 'num_write_kb': 51231, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2191, 'num_bytes_recovered': 17932288, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 4463}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:40:24.560 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:40:24.560 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'8442", 'reported_seq': 8535, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:22.106378+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:40:22.106378+0000', 'last_peered': '2024-09-05T16:40:22.106378+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:40:22.106378+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8442, 'log_dups_size': 0, 'ondisk_log_size': 8442, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38368416, 'num_objects': 4684, 'num_object_clones': 0, 'num_object_copies': 18736, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2341, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4684, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8440, 'num_write_kb': 52500, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2345}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:40:24.560 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:40:24.560 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'8248", 'reported_seq': 8347, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:22.422720+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:40:22.422720+0000', 'last_peered': '2024-09-05T16:40:22.422720+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:40:22.422720+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8248, 'log_dups_size': 0, 'ondisk_log_size': 8248, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37557408, 'num_objects': 4585, 'num_object_clones': 0, 'num_object_copies': 18340, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2314, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4585, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8246, 'num_write_kb': 51325, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2273}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:40:24.560 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:40:24.560 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'8158", 'reported_seq': 8251, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:21.863407+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:40:21.863407+0000', 'last_peered': '2024-09-05T16:40:21.863407+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:40:21.863407+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8158, 'log_dups_size': 0, 'ondisk_log_size': 8158, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37819552, 'num_objects': 4617, 'num_object_clones': 0, 'num_object_copies': 18468, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2227, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4617, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8156, 'num_write_kb': 51092, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2392}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:40:24.560 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:40:25.317 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:40:25.317 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 31517/294928 objects degraded (10.686%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":428,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":73732,"data_bytes":603966520,"bytes_used":2946830336,"bytes_avail":383600226304,"bytes_total":386547056640,"degraded_objects":31517,"degraded_total":294928,"degraded_ratio":0.10686337004285792,"recovering_objects_per_sec":14,"recovering_bytes_per_sec":112592,"recovering_keys_per_sec":0,"num_objects_recovered":178,"num_bytes_recovered":1351680,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":189},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:40:25.349 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:40:25.349 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:40:25.986 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:40:25.986 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:40:26.013 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":240,"stamp":"2024-09-05T16:40:24.932382+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":603966520,"num_objects":73732,"num_object_clones":0,"num_object_copies":294928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31517,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73732,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":125505,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2416,"num_bytes_recovered":17932288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":131230,"ondisk_log_size":131230,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2877764,"kb_used_data":1154980,"kb_used_omap":14,"kb_used_meta":1722673,"kb_avail":374609596,"statfs":{"total":386547056640,"available":383600226304,"internally_reserved":0,"allocated":1182699520,"data_stored":1181931356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1764017719},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,5,4,5,13,4,0,5,0,0,0,1,4],"upper_bound":16384},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18669568,"num_objects":-2279,"num_object_clones":0,"num_object_copies":-9116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2279,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2279,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":178,"num_bytes_recovered":1351680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005072"},"pg_stats":[{"pgid":"1.f","version":"35'8269","reported_seq":8362,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:21.713420+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:40:21.713420+0000","last_peered":"2024-09-05T16:40:21.713420+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:40:21.713420+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8269,"log_dups_size":0,"ondisk_log_size":8269,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37969920,"num_objects":4635,"num_object_clones":0,"num_object_copies":18540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4635,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8269,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2396}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'8105","reported_seq":8198,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:22.030172+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:40:22.030172+0000","last_peered":"2024-09-05T16:40:22.030172+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:40:22.030172+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8105,"log_dups_size":0,"ondisk_log_size":8105,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36443296,"num_objects":4449,"num_object_clones":0,"num_object_copies":17796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4449,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8103,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2194}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'8175","reported_seq":8272,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:22.309441+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:40:22.309441+0000","last_peered":"2024-09-05T16:40:22.309441+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:40:22.309441+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8175,"log_dups_size":0,"ondisk_log_size":8175,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37493352,"num_objects":4578,"num_object_clones":0,"num_object_copies":18312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2243,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4578,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8174,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2336}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'8293","reported_seq":13017,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-05T16:40:21.296712+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:40:21.296712+0000","last_peered":"2024-09-05T16:40:21.296712+0000","last_clean":"2024-09-05T16:40:21.296712+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:40:21.296712+0000","last_undegraded":"2024-09-05T16:40:21.296712+0000","last_fullsized":"2024-09-05T16:40:21.296712+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":8293,"log_dups_size":0,"ondisk_log_size":8293,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38019072,"num_objects":4641,"num_object_clones":0,"num_object_copies":18564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4641,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2593,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'8008","reported_seq":8095,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:22.102765+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:40:22.102765+0000","last_peered":"2024-09-05T16:40:22.102765+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:40:22.102765+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8008,"log_dups_size":0,"ondisk_log_size":8008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37573792,"num_objects":4587,"num_object_clones":0,"num_object_copies":18348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2150,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4587,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8006,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2439}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'8150","reported_seq":8241,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:22.079179+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:40:22.079179+0000","last_peered":"2024-09-05T16:40:22.079179+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:40:22.079179+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8150,"log_dups_size":0,"ondisk_log_size":8150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37863424,"num_objects":4622,"num_object_clones":0,"num_object_copies":18488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4622,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8150,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2394}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'7975","reported_seq":8064,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:21.910365+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:40:21.910365+0000","last_peered":"2024-09-05T16:40:21.910365+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:40:21.910365+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":7975,"log_dups_size":0,"ondisk_log_size":7975,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38392992,"num_objects":4687,"num_object_clones":0,"num_object_copies":18748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2140,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7973,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2549}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'8153","reported_seq":8234,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:21.808447+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:40:21.808447+0000","last_peered":"2024-09-05T16:40:21.808447+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:40:21.808447+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8153,"log_dups_size":0,"ondisk_log_size":8153,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37893280,"num_objects":4626,"num_object_clones":0,"num_object_copies":18504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4626,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8151,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2393}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'8212","reported_seq":8303,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:22.086149+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:40:22.086149+0000","last_peered":"2024-09-05T16:40:22.086149+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:40:22.086149+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8212,"log_dups_size":0,"ondisk_log_size":8212,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37366080,"num_objects":4562,"num_object_clones":0,"num_object_copies":18248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2249,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4562,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8209,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2316}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'8199","reported_seq":8306,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:22.355489+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:40:22.355489+0000","last_peered":"2024-09-05T16:40:22.355489+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:40:22.355489+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8199,"log_dups_size":0,"ondisk_log_size":8199,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37770400,"num_objects":4611,"num_object_clones":0,"num_object_copies":18444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4611,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8197,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2361}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'8424","reported_seq":8531,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:22.133596+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:40:22.133596+0000","last_peered":"2024-09-05T16:40:22.133596+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:40:22.133596+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8424,"log_dups_size":0,"ondisk_log_size":8424,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38489840,"num_objects":4699,"num_object_clones":0,"num_object_copies":18796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4699,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8421,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2411}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'8178","reported_seq":8256,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:22.141319+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:40:22.141319+0000","last_peered":"2024-09-05T16:40:22.141319+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:40:22.141319+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8178,"log_dups_size":0,"ondisk_log_size":8178,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37526096,"num_objects":4581,"num_object_clones":0,"num_object_copies":18324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2246,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4581,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8177,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2336}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'8241","reported_seq":12701,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:22.465407+0000","last_change":"2024-09-05T16:37:30.634570+0000","last_active":"2024-09-05T16:40:22.465407+0000","last_peered":"2024-09-05T16:40:22.465407+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:40:22.465407+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8241,"log_dups_size":0,"ondisk_log_size":8241,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37419600,"num_objects":4568,"num_object_clones":0,"num_object_copies":18272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":105,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4568,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8240,"num_write_kb":51231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2191,"num_bytes_recovered":17932288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4463}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'8442","reported_seq":8535,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:22.106378+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:40:22.106378+0000","last_peered":"2024-09-05T16:40:22.106378+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:40:22.106378+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8442,"log_dups_size":0,"ondisk_log_size":8442,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38368416,"num_objects":4684,"num_object_clones":0,"num_object_copies":18736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2341,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4684,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8440,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2345}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'8248","reported_seq":8347,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:22.422720+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:40:22.422720+0000","last_peered":"2024-09-05T16:40:22.422720+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:40:22.422720+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8248,"log_dups_size":0,"ondisk_log_size":8248,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37557408,"num_objects":4585,"num_object_clones":0,"num_object_copies":18340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4585,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8246,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2273}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'8158","reported_seq":8251,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:21.863407+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:40:21.863407+0000","last_peered":"2024-09-05T16:40:21.863407+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:40:21.863407+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8158,"log_dups_size":0,"ondisk_log_size":8158,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37819552,"num_objects":4617,"num_object_clones":0,"num_object_copies":18468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2227,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4617,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8156,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2392}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":603966520,"num_objects":73732,"num_object_clones":0,"num_object_copies":294928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31517,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73732,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":125505,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2416,"num_bytes_recovered":17932288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1180426240,"data_stored":1180426240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":131230,"ondisk_log_size":131230,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084365,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":760144,"kb_used_data":295296,"kb_used_omap":4,"kb_used_meta":464827,"kb_avail":93611696,"statfs":{"total":96636764160,"available":95858376704,"internally_reserved":0,"allocated":302383104,"data_stored":302191063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":475983571},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,2,0,2,3,0,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986242,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":584952,"kb_used_data":268648,"kb_used_omap":4,"kb_used_meta":316283,"kb_avail":93786888,"statfs":{"total":96636764160,"available":96037773312,"internally_reserved":0,"allocated":275095552,"data_stored":274903511,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":323874514},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673048,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":763964,"kb_used_data":295456,"kb_used_omap":1,"kb_used_meta":468478,"kb_avail":93607876,"statfs":{"total":96636764160,"available":95854465024,"internally_reserved":0,"allocated":302546944,"data_stored":302354903,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":479721925},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,3,8,0,0,0,0,0,0,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182479,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":768704,"kb_used_data":295580,"kb_used_omap":4,"kb_used_meta":473083,"kb_avail":93603136,"statfs":{"total":96636764160,"available":95849611264,"internally_reserved":0,"allocated":302673920,"data_stored":302481879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":484437709},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2,0,3,0,0,0,0,0,0,0,4],"upper_bound":16384},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":302030848,"data_stored":302030848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":274792448,"data_stored":274792448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":301658112,"data_stored":301658112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":301944832,"data_stored":301944832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:40:29.015 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:40:29.634 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:40:29.634 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:40:29.659 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":243,"stamp":"2024-09-05T16:40:28.934092+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594603064,"num_objects":72589,"num_object_clones":0,"num_object_copies":290356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31409,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72589,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":126648,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2524,"num_bytes_recovered":18636800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":132373,"ondisk_log_size":132373,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2880572,"kb_used_data":1141164,"kb_used_omap":14,"kb_used_meta":1739313,"kb_avail":374606788,"statfs":{"total":386547056640,"available":383597350912,"internally_reserved":0,"allocated":1168551936,"data_stored":1167783772,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1781057079},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,5,7,9,7,4,5,0,0,0,0,0,0,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18497536,"num_objects":-2258,"num_object_clones":0,"num_object_copies":-9032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2258,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2258,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":212,"num_bytes_recovered":1490944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.004428"},"pg_stats":[{"pgid":"1.f","version":"35'8342","reported_seq":8439,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:26.997192+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:40:26.997192+0000","last_peered":"2024-09-05T16:40:26.997192+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:40:26.997192+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8342,"log_dups_size":0,"ondisk_log_size":8342,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37371904,"num_objects":4562,"num_object_clones":0,"num_object_copies":18248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2237,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4562,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8342,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2325}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'8171","reported_seq":8264,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:27.016899+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:40:27.016899+0000","last_peered":"2024-09-05T16:40:27.016899+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:40:27.016899+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8171,"log_dups_size":0,"ondisk_log_size":8171,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35902624,"num_objects":4383,"num_object_clones":0,"num_object_copies":17532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4383,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8169,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2128}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'8241","reported_seq":8338,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:27.409217+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:40:27.409217+0000","last_peered":"2024-09-05T16:40:27.409217+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:40:27.409217+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8241,"log_dups_size":0,"ondisk_log_size":8241,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36952680,"num_objects":4512,"num_object_clones":0,"num_object_copies":18048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2243,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4512,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8240,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2270}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'8357","reported_seq":13081,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-05T16:40:26.154350+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:40:26.154350+0000","last_peered":"2024-09-05T16:40:26.154350+0000","last_clean":"2024-09-05T16:40:26.154350+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:40:26.154350+0000","last_undegraded":"2024-09-05T16:40:26.154350+0000","last_fullsized":"2024-09-05T16:40:26.154350+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":8357,"log_dups_size":0,"ondisk_log_size":8357,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37494784,"num_objects":4577,"num_object_clones":0,"num_object_copies":18308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4577,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2657,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'8076","reported_seq":8166,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:27.046146+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:40:27.046146+0000","last_peered":"2024-09-05T16:40:27.046146+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:40:27.046146+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8076,"log_dups_size":0,"ondisk_log_size":8076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37016736,"num_objects":4519,"num_object_clones":0,"num_object_copies":18076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4519,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8074,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2372}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'8235","reported_seq":8336,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:27.189203+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:40:27.189203+0000","last_peered":"2024-09-05T16:40:27.189203+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:40:27.189203+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8235,"log_dups_size":0,"ondisk_log_size":8235,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37167104,"num_objects":4537,"num_object_clones":0,"num_object_copies":18148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4537,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8235,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2314}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'8062","reported_seq":8158,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:27.018964+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:40:27.018964+0000","last_peered":"2024-09-05T16:40:27.018964+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:40:27.018964+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8062,"log_dups_size":0,"ondisk_log_size":8062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37680288,"num_objects":4600,"num_object_clones":0,"num_object_copies":18400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4600,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8060,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2466}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'8209","reported_seq":8290,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:27.001372+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:40:27.001372+0000","last_peered":"2024-09-05T16:40:27.001372+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:40:27.001372+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8209,"log_dups_size":0,"ondisk_log_size":8209,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37434528,"num_objects":4570,"num_object_clones":0,"num_object_copies":18280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8207,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2337}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'8278","reported_seq":8371,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:27.084946+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:40:27.084946+0000","last_peered":"2024-09-05T16:40:27.084946+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:40:27.084946+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8278,"log_dups_size":0,"ondisk_log_size":8278,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36825408,"num_objects":4496,"num_object_clones":0,"num_object_copies":17984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2248,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4496,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8275,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2251}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'8278","reported_seq":8385,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:27.488720+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:40:27.488720+0000","last_peered":"2024-09-05T16:40:27.488720+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:40:27.488720+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8278,"log_dups_size":0,"ondisk_log_size":8278,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37123232,"num_objects":4532,"num_object_clones":0,"num_object_copies":18128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4532,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8276,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2282}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'8496","reported_seq":8605,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:27.198472+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:40:27.198472+0000","last_peered":"2024-09-05T16:40:27.198472+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:40:27.198472+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8496,"log_dups_size":0,"ondisk_log_size":8496,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37900016,"num_objects":4627,"num_object_clones":0,"num_object_copies":18508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4627,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8493,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2340}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'8264","reported_seq":8355,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:27.183390+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:40:27.183390+0000","last_peered":"2024-09-05T16:40:27.183390+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:40:27.183390+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8264,"log_dups_size":0,"ondisk_log_size":8264,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36821584,"num_objects":4495,"num_object_clones":0,"num_object_copies":17980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4495,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8263,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2256}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'8312","reported_seq":12944,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:27.497570+0000","last_change":"2024-09-05T16:37:30.634570+0000","last_active":"2024-09-05T16:40:27.497570+0000","last_peered":"2024-09-05T16:40:27.497570+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:40:27.497570+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8312,"log_dups_size":0,"ondisk_log_size":8312,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36837968,"num_objects":4497,"num_object_clones":0,"num_object_copies":17988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4497,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8311,"num_write_kb":51231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2277,"num_bytes_recovered":18636800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4478}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'8513","reported_seq":8608,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:27.031438+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:40:27.031438+0000","last_peered":"2024-09-05T16:40:27.031438+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:40:27.031438+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8513,"log_dups_size":0,"ondisk_log_size":8513,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37786784,"num_objects":4613,"num_object_clones":0,"num_object_copies":18452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4613,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8511,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2275}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'8320","reported_seq":8419,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:27.421123+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:40:27.421123+0000","last_peered":"2024-09-05T16:40:27.421123+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:40:27.421123+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8320,"log_dups_size":0,"ondisk_log_size":8320,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36967584,"num_objects":4513,"num_object_clones":0,"num_object_copies":18052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4513,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8318,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2201}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'8219","reported_seq":8314,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:26.881044+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:40:26.881044+0000","last_peered":"2024-09-05T16:40:26.881044+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:40:26.881044+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8219,"log_dups_size":0,"ondisk_log_size":8219,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37319840,"num_objects":4556,"num_object_clones":0,"num_object_copies":18224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4556,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8217,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2332}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":594603064,"num_objects":72589,"num_object_clones":0,"num_object_copies":290356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31409,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72589,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":126648,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2524,"num_bytes_recovered":18636800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1166237696,"data_stored":1166237696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":132373,"ondisk_log_size":132373,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084366,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":759640,"kb_used_data":290624,"kb_used_omap":4,"kb_used_meta":468987,"kb_avail":93612200,"statfs":{"total":96636764160,"available":95858892800,"internally_reserved":0,"allocated":297598976,"data_stored":297406935,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":480243411},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,2,0,2,3,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986243,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":588968,"kb_used_data":268504,"kb_used_omap":4,"kb_used_meta":320443,"kb_avail":93782872,"statfs":{"total":96636764160,"available":96033660928,"internally_reserved":0,"allocated":274948096,"data_stored":274756055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":328134354},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,7,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673049,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":763668,"kb_used_data":291008,"kb_used_omap":1,"kb_used_meta":472638,"kb_avail":93608172,"statfs":{"total":96636764160,"available":95854768128,"internally_reserved":0,"allocated":297992192,"data_stored":297800151,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":483981765},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,0,0,0,0,0,0,0,0,0,0,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182480,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":768296,"kb_used_data":291028,"kb_used_omap":4,"kb_used_meta":477243,"kb_avail":93603544,"statfs":{"total":96636764160,"available":95850029056,"internally_reserved":0,"allocated":298012672,"data_stored":297820631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":488697549},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,2,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":297394176,"data_stored":297394176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":274657280,"data_stored":274657280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":296931328,"data_stored":296931328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":297254912,"data_stored":297254912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:40:29.660 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:40:30.244 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:40:30.245 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:40:30.273 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":244,"stamp":"2024-09-05T16:40:29.926960+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594603064,"num_objects":72589,"num_object_clones":0,"num_object_copies":290356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31409,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72589,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":126648,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2524,"num_bytes_recovered":18636800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":132373,"ondisk_log_size":132373,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2880572,"kb_used_data":1141164,"kb_used_omap":14,"kb_used_meta":1739313,"kb_avail":374606788,"statfs":{"total":386547056640,"available":383597350912,"internally_reserved":0,"allocated":1168551936,"data_stored":1167783772,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1781057079},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,5,7,9,7,4,5,0,0,0,0,0,0,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18497536,"num_objects":-2258,"num_object_clones":0,"num_object_copies":-9032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2258,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2258,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":212,"num_bytes_recovered":1490944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.996352"},"pg_stats":[{"pgid":"1.f","version":"35'8342","reported_seq":8439,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:26.997192+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:40:26.997192+0000","last_peered":"2024-09-05T16:40:26.997192+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:40:26.997192+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8342,"log_dups_size":0,"ondisk_log_size":8342,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37371904,"num_objects":4562,"num_object_clones":0,"num_object_copies":18248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2237,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4562,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8342,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2325}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'8171","reported_seq":8264,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:27.016899+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:40:27.016899+0000","last_peered":"2024-09-05T16:40:27.016899+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:40:27.016899+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8171,"log_dups_size":0,"ondisk_log_size":8171,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35902624,"num_objects":4383,"num_object_clones":0,"num_object_copies":17532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4383,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8169,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2128}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'8241","reported_seq":8338,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:27.409217+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:40:27.409217+0000","last_peered":"2024-09-05T16:40:27.409217+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:40:27.409217+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8241,"log_dups_size":0,"ondisk_log_size":8241,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36952680,"num_objects":4512,"num_object_clones":0,"num_object_copies":18048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2243,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4512,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8240,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2270}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'8357","reported_seq":13081,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-05T16:40:26.154350+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:40:26.154350+0000","last_peered":"2024-09-05T16:40:26.154350+0000","last_clean":"2024-09-05T16:40:26.154350+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:40:26.154350+0000","last_undegraded":"2024-09-05T16:40:26.154350+0000","last_fullsized":"2024-09-05T16:40:26.154350+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":8357,"log_dups_size":0,"ondisk_log_size":8357,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37494784,"num_objects":4577,"num_object_clones":0,"num_object_copies":18308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4577,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2657,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'8076","reported_seq":8166,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:27.046146+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:40:27.046146+0000","last_peered":"2024-09-05T16:40:27.046146+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:40:27.046146+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8076,"log_dups_size":0,"ondisk_log_size":8076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37016736,"num_objects":4519,"num_object_clones":0,"num_object_copies":18076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4519,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8074,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2372}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'8235","reported_seq":8336,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:27.189203+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:40:27.189203+0000","last_peered":"2024-09-05T16:40:27.189203+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:40:27.189203+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8235,"log_dups_size":0,"ondisk_log_size":8235,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37167104,"num_objects":4537,"num_object_clones":0,"num_object_copies":18148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4537,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8235,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2314}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'8062","reported_seq":8158,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:27.018964+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:40:27.018964+0000","last_peered":"2024-09-05T16:40:27.018964+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:40:27.018964+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8062,"log_dups_size":0,"ondisk_log_size":8062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37680288,"num_objects":4600,"num_object_clones":0,"num_object_copies":18400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4600,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8060,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2466}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'8209","reported_seq":8290,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:27.001372+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:40:27.001372+0000","last_peered":"2024-09-05T16:40:27.001372+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:40:27.001372+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8209,"log_dups_size":0,"ondisk_log_size":8209,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37434528,"num_objects":4570,"num_object_clones":0,"num_object_copies":18280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8207,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2337}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'8278","reported_seq":8371,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:27.084946+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:40:27.084946+0000","last_peered":"2024-09-05T16:40:27.084946+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:40:27.084946+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8278,"log_dups_size":0,"ondisk_log_size":8278,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36825408,"num_objects":4496,"num_object_clones":0,"num_object_copies":17984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2248,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4496,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8275,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2251}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'8278","reported_seq":8385,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:27.488720+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:40:27.488720+0000","last_peered":"2024-09-05T16:40:27.488720+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:40:27.488720+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8278,"log_dups_size":0,"ondisk_log_size":8278,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37123232,"num_objects":4532,"num_object_clones":0,"num_object_copies":18128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4532,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8276,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2282}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'8496","reported_seq":8605,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:27.198472+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:40:27.198472+0000","last_peered":"2024-09-05T16:40:27.198472+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:40:27.198472+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8496,"log_dups_size":0,"ondisk_log_size":8496,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37900016,"num_objects":4627,"num_object_clones":0,"num_object_copies":18508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4627,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8493,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2340}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'8264","reported_seq":8355,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:27.183390+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:40:27.183390+0000","last_peered":"2024-09-05T16:40:27.183390+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:40:27.183390+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8264,"log_dups_size":0,"ondisk_log_size":8264,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36821584,"num_objects":4495,"num_object_clones":0,"num_object_copies":17980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4495,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8263,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2256}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'8312","reported_seq":12944,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:27.497570+0000","last_change":"2024-09-05T16:37:30.634570+0000","last_active":"2024-09-05T16:40:27.497570+0000","last_peered":"2024-09-05T16:40:27.497570+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:40:27.497570+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8312,"log_dups_size":0,"ondisk_log_size":8312,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36837968,"num_objects":4497,"num_object_clones":0,"num_object_copies":17988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4497,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8311,"num_write_kb":51231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2277,"num_bytes_recovered":18636800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4478}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'8513","reported_seq":8608,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:27.031438+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:40:27.031438+0000","last_peered":"2024-09-05T16:40:27.031438+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:40:27.031438+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8513,"log_dups_size":0,"ondisk_log_size":8513,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37786784,"num_objects":4613,"num_object_clones":0,"num_object_copies":18452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4613,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8511,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2275}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'8320","reported_seq":8419,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:27.421123+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:40:27.421123+0000","last_peered":"2024-09-05T16:40:27.421123+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:40:27.421123+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8320,"log_dups_size":0,"ondisk_log_size":8320,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36967584,"num_objects":4513,"num_object_clones":0,"num_object_copies":18052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4513,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8318,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2201}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'8219","reported_seq":8314,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:26.881044+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:40:26.881044+0000","last_peered":"2024-09-05T16:40:26.881044+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:40:26.881044+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8219,"log_dups_size":0,"ondisk_log_size":8219,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37319840,"num_objects":4556,"num_object_clones":0,"num_object_copies":18224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4556,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8217,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2332}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":594603064,"num_objects":72589,"num_object_clones":0,"num_object_copies":290356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31409,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72589,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":126648,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2524,"num_bytes_recovered":18636800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1166237696,"data_stored":1166237696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":132373,"ondisk_log_size":132373,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084366,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":759640,"kb_used_data":290624,"kb_used_omap":4,"kb_used_meta":468987,"kb_avail":93612200,"statfs":{"total":96636764160,"available":95858892800,"internally_reserved":0,"allocated":297598976,"data_stored":297406935,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":480243411},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,2,0,2,3,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986243,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":588968,"kb_used_data":268504,"kb_used_omap":4,"kb_used_meta":320443,"kb_avail":93782872,"statfs":{"total":96636764160,"available":96033660928,"internally_reserved":0,"allocated":274948096,"data_stored":274756055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":328134354},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,7,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673049,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":763668,"kb_used_data":291008,"kb_used_omap":1,"kb_used_meta":472638,"kb_avail":93608172,"statfs":{"total":96636764160,"available":95854768128,"internally_reserved":0,"allocated":297992192,"data_stored":297800151,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":483981765},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,0,0,0,0,0,0,0,0,0,0,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182480,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":768296,"kb_used_data":291028,"kb_used_omap":4,"kb_used_meta":477243,"kb_avail":93603544,"statfs":{"total":96636764160,"available":95850029056,"internally_reserved":0,"allocated":298012672,"data_stored":297820631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":488697549},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,2,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":297394176,"data_stored":297394176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":274657280,"data_stored":274657280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":296931328,"data_stored":296931328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":297254912,"data_stored":297254912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:40:30.274 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-05T16:40:30.274 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'8342", 'reported_seq': 8439, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:26.997192+0000', 'last_change': '2024-09-05T16:34:56.833497+0000', 'last_active': '2024-09-05T16:40:26.997192+0000', 'last_peered': '2024-09-05T16:40:26.997192+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.828289+0000', 'last_became_peered': '2024-09-05T16:34:56.828289+0000', 'last_unstale': '2024-09-05T16:40:26.997192+0000', 'last_undegraded': '2024-09-05T16:34:56.612160+0000', 'last_fullsized': '2024-09-05T16:34:56.557632+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8342, 'log_dups_size': 0, 'ondisk_log_size': 8342, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37371904, 'num_objects': 4562, 'num_object_clones': 0, 'num_object_copies': 18248, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2237, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4562, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8342, 'num_write_kb': 51616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2325}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:40:30.275 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-05T16:40:30.275 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "35'8171", 'reported_seq': 8264, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:27.016899+0000', 'last_change': '2024-09-05T16:34:56.825031+0000', 'last_active': '2024-09-05T16:40:27.016899+0000', 'last_peered': '2024-09-05T16:40:27.016899+0000', 'last_clean': '2024-09-05T16:33:53.200762+0000', 'last_became_active': '2024-09-05T16:34:56.820292+0000', 'last_became_peered': '2024-09-05T16:34:56.820292+0000', 'last_unstale': '2024-09-05T16:40:27.016899+0000', 'last_undegraded': '2024-09-05T16:34:56.605776+0000', 'last_fullsized': '2024-09-05T16:34:56.559383+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8171, 'log_dups_size': 0, 'ondisk_log_size': 8171, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35902624, 'num_objects': 4383, 'num_object_clones': 0, 'num_object_copies': 17532, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4383, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8169, 'num_write_kb': 50212, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2128}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:40:30.275 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:40:30.275 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'8241", 'reported_seq': 8338, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:27.409217+0000', 'last_change': '2024-09-05T16:34:56.867900+0000', 'last_active': '2024-09-05T16:40:27.409217+0000', 'last_peered': '2024-09-05T16:40:27.409217+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:40:27.409217+0000', 'last_undegraded': '2024-09-05T16:34:56.590972+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8241, 'log_dups_size': 0, 'ondisk_log_size': 8241, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36952680, 'num_objects': 4512, 'num_object_clones': 0, 'num_object_copies': 18048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2243, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4512, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 8240, 'num_write_kb': 50999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2270}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:40:30.275 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:40:30.275 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'8076", 'reported_seq': 8166, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:27.046146+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:40:27.046146+0000', 'last_peered': '2024-09-05T16:40:27.046146+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:40:27.046146+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8076, 'log_dups_size': 0, 'ondisk_log_size': 8076, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37016736, 'num_objects': 4519, 'num_object_clones': 0, 'num_object_copies': 18076, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2149, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4519, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8074, 'num_write_kb': 50373, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2372}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:40:30.275 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:40:30.275 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'8235", 'reported_seq': 8336, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:27.189203+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:40:27.189203+0000', 'last_peered': '2024-09-05T16:40:27.189203+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:40:27.189203+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8235, 'log_dups_size': 0, 'ondisk_log_size': 8235, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37167104, 'num_objects': 4537, 'num_object_clones': 0, 'num_object_copies': 18148, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2223, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4537, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8235, 'num_write_kb': 51088, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2314}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:40:30.275 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:40:30.276 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'8062", 'reported_seq': 8158, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:27.018964+0000', 'last_change': '2024-09-05T16:34:56.903436+0000', 'last_active': '2024-09-05T16:40:27.018964+0000', 'last_peered': '2024-09-05T16:40:27.018964+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:40:27.018964+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8062, 'log_dups_size': 0, 'ondisk_log_size': 8062, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37680288, 'num_objects': 4600, 'num_object_clones': 0, 'num_object_copies': 18400, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2136, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4600, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8060, 'num_write_kb': 50636, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2466}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:40:30.276 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-05T16:40:30.276 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "35'8209", 'reported_seq': 8290, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:27.001372+0000', 'last_change': '2024-09-05T16:34:56.883319+0000', 'last_active': '2024-09-05T16:40:27.001372+0000', 'last_peered': '2024-09-05T16:40:27.001372+0000', 'last_clean': '2024-09-05T16:33:53.505305+0000', 'last_became_active': '2024-09-05T16:34:56.873389+0000', 'last_became_peered': '2024-09-05T16:34:56.873389+0000', 'last_unstale': '2024-09-05T16:40:27.001372+0000', 'last_undegraded': '2024-09-05T16:34:56.602567+0000', 'last_fullsized': '2024-09-05T16:34:56.559958+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8209, 'log_dups_size': 0, 'ondisk_log_size': 8209, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37434528, 'num_objects': 4570, 'num_object_clones': 0, 'num_object_copies': 18280, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2235, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4570, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8207, 'num_write_kb': 51108, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 8, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2337}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:40:30.276 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:40:30.276 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'8278", 'reported_seq': 8371, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:27.084946+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:40:27.084946+0000', 'last_peered': '2024-09-05T16:40:27.084946+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:40:27.084946+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8278, 'log_dups_size': 0, 'ondisk_log_size': 8278, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36825408, 'num_objects': 4496, 'num_object_clones': 0, 'num_object_copies': 17984, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2248, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4496, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8275, 'num_write_kb': 51081, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2251}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:40:30.276 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:40:30.276 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'8278", 'reported_seq': 8385, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:27.488720+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:40:27.488720+0000', 'last_peered': '2024-09-05T16:40:27.488720+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:40:27.488720+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8278, 'log_dups_size': 0, 'ondisk_log_size': 8278, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37123232, 'num_objects': 4532, 'num_object_clones': 0, 'num_object_copies': 18128, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2252, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4532, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8276, 'num_write_kb': 51230, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2282}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:40:30.276 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:40:30.276 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'8496", 'reported_seq': 8605, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:27.198472+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:40:27.198472+0000', 'last_peered': '2024-09-05T16:40:27.198472+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:40:27.198472+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8496, 'log_dups_size': 0, 'ondisk_log_size': 8496, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37900016, 'num_objects': 4627, 'num_object_clones': 0, 'num_object_copies': 18508, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2290, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4627, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8493, 'num_write_kb': 52476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2340}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:40:30.277 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:40:30.277 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'8264", 'reported_seq': 8355, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:27.183390+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:40:27.183390+0000', 'last_peered': '2024-09-05T16:40:27.183390+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:40:27.183390+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8264, 'log_dups_size': 0, 'ondisk_log_size': 8264, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36821584, 'num_objects': 4495, 'num_object_clones': 0, 'num_object_copies': 17980, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2240, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4495, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8263, 'num_write_kb': 51031, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2256}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:40:30.277 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-05T16:40:30.277 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'8312", 'reported_seq': 12944, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:27.497570+0000', 'last_change': '2024-09-05T16:37:30.634570+0000', 'last_active': '2024-09-05T16:40:27.497570+0000', 'last_peered': '2024-09-05T16:40:27.497570+0000', 'last_clean': '2024-09-05T16:33:45.082039+0000', 'last_became_active': '2024-09-05T16:34:56.792386+0000', 'last_became_peered': '2024-09-05T16:34:56.792386+0000', 'last_unstale': '2024-09-05T16:40:27.497570+0000', 'last_undegraded': '2024-09-05T16:34:56.600595+0000', 'last_fullsized': '2024-09-05T16:34:56.558203+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8312, 'log_dups_size': 0, 'ondisk_log_size': 8312, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36837968, 'num_objects': 4497, 'num_object_clones': 0, 'num_object_copies': 17988, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4497, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8311, 'num_write_kb': 51231, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2277, 'num_bytes_recovered': 18636800, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 4478}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:40:30.277 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:40:30.277 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'8513", 'reported_seq': 8608, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:27.031438+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:40:27.031438+0000', 'last_peered': '2024-09-05T16:40:27.031438+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:40:27.031438+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8513, 'log_dups_size': 0, 'ondisk_log_size': 8513, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37786784, 'num_objects': 4613, 'num_object_clones': 0, 'num_object_copies': 18452, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2340, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4613, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8511, 'num_write_kb': 52500, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2275}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:40:30.277 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:40:30.277 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'8320", 'reported_seq': 8419, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:27.421123+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:40:27.421123+0000', 'last_peered': '2024-09-05T16:40:27.421123+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:40:27.421123+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8320, 'log_dups_size': 0, 'ondisk_log_size': 8320, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36967584, 'num_objects': 4513, 'num_object_clones': 0, 'num_object_copies': 18052, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2314, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4513, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8318, 'num_write_kb': 51325, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2201}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:40:30.277 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:40:30.278 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'8219", 'reported_seq': 8314, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:26.881044+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:40:26.881044+0000', 'last_peered': '2024-09-05T16:40:26.881044+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:40:26.881044+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8219, 'log_dups_size': 0, 'ondisk_log_size': 8219, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37319840, 'num_objects': 4556, 'num_object_clones': 0, 'num_object_copies': 18224, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2226, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4556, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8217, 'num_write_kb': 51092, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2332}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:40:30.278 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:40:31.020 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:40:31.020 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 31409/290356 objects degraded (10.817%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":434,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":72589,"data_bytes":594603064,"bytes_used":2949705728,"bytes_avail":383597350912,"bytes_total":386547056640,"degraded_objects":31409,"degraded_total":290356,"degraded_ratio":0.10817410351430658,"recovering_objects_per_sec":21,"recovering_bytes_per_sec":149028,"recovering_keys_per_sec":0,"num_objects_recovered":212,"num_bytes_recovered":1490944,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":225},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:40:31.042 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:40:31.042 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:40:31.618 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:40:31.618 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:40:31.642 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":245,"stamp":"2024-09-05T16:40:30.934871+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594603064,"num_objects":72589,"num_object_clones":0,"num_object_copies":290356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31409,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72589,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":126648,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2524,"num_bytes_recovered":18636800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":132373,"ondisk_log_size":132373,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2880572,"kb_used_data":1141164,"kb_used_omap":14,"kb_used_meta":1739313,"kb_avail":374606788,"statfs":{"total":386547056640,"available":383597350912,"internally_reserved":0,"allocated":1168551936,"data_stored":1167783772,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1781057079},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,5,7,9,7,4,5,0,0,0,0,0,0,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-9363456,"num_objects":-1143,"num_object_clones":0,"num_object_copies":-4572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-108,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1143,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1143,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":108,"num_bytes_recovered":704512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003427"},"pg_stats":[{"pgid":"1.f","version":"35'8342","reported_seq":8439,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:26.997192+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:40:26.997192+0000","last_peered":"2024-09-05T16:40:26.997192+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:40:26.997192+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8342,"log_dups_size":0,"ondisk_log_size":8342,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37371904,"num_objects":4562,"num_object_clones":0,"num_object_copies":18248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2237,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4562,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8342,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2325}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'8171","reported_seq":8264,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:27.016899+0000","last_change":"2024-09-05T16:34:56.825031+0000","last_active":"2024-09-05T16:40:27.016899+0000","last_peered":"2024-09-05T16:40:27.016899+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:40:27.016899+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8171,"log_dups_size":0,"ondisk_log_size":8171,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35902624,"num_objects":4383,"num_object_clones":0,"num_object_copies":17532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4383,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8169,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2128}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'8241","reported_seq":8338,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:27.409217+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:40:27.409217+0000","last_peered":"2024-09-05T16:40:27.409217+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:40:27.409217+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8241,"log_dups_size":0,"ondisk_log_size":8241,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36952680,"num_objects":4512,"num_object_clones":0,"num_object_copies":18048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2243,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4512,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8240,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2270}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'8357","reported_seq":13081,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-05T16:40:26.154350+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:40:26.154350+0000","last_peered":"2024-09-05T16:40:26.154350+0000","last_clean":"2024-09-05T16:40:26.154350+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:40:26.154350+0000","last_undegraded":"2024-09-05T16:40:26.154350+0000","last_fullsized":"2024-09-05T16:40:26.154350+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":8357,"log_dups_size":0,"ondisk_log_size":8357,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37494784,"num_objects":4577,"num_object_clones":0,"num_object_copies":18308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4577,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2657,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'8076","reported_seq":8166,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:27.046146+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:40:27.046146+0000","last_peered":"2024-09-05T16:40:27.046146+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:40:27.046146+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8076,"log_dups_size":0,"ondisk_log_size":8076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37016736,"num_objects":4519,"num_object_clones":0,"num_object_copies":18076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4519,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8074,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2372}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'8235","reported_seq":8336,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:27.189203+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:40:27.189203+0000","last_peered":"2024-09-05T16:40:27.189203+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:40:27.189203+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8235,"log_dups_size":0,"ondisk_log_size":8235,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37167104,"num_objects":4537,"num_object_clones":0,"num_object_copies":18148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4537,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8235,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2314}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'8062","reported_seq":8158,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:27.018964+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:40:27.018964+0000","last_peered":"2024-09-05T16:40:27.018964+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:40:27.018964+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8062,"log_dups_size":0,"ondisk_log_size":8062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37680288,"num_objects":4600,"num_object_clones":0,"num_object_copies":18400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4600,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8060,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2466}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'8209","reported_seq":8290,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:27.001372+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:40:27.001372+0000","last_peered":"2024-09-05T16:40:27.001372+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:40:27.001372+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8209,"log_dups_size":0,"ondisk_log_size":8209,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37434528,"num_objects":4570,"num_object_clones":0,"num_object_copies":18280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8207,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2337}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'8278","reported_seq":8371,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:27.084946+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:40:27.084946+0000","last_peered":"2024-09-05T16:40:27.084946+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:40:27.084946+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8278,"log_dups_size":0,"ondisk_log_size":8278,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36825408,"num_objects":4496,"num_object_clones":0,"num_object_copies":17984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2248,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4496,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8275,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2251}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'8278","reported_seq":8385,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:27.488720+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:40:27.488720+0000","last_peered":"2024-09-05T16:40:27.488720+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:40:27.488720+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8278,"log_dups_size":0,"ondisk_log_size":8278,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37123232,"num_objects":4532,"num_object_clones":0,"num_object_copies":18128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4532,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8276,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2282}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'8496","reported_seq":8605,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:27.198472+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:40:27.198472+0000","last_peered":"2024-09-05T16:40:27.198472+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:40:27.198472+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8496,"log_dups_size":0,"ondisk_log_size":8496,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37900016,"num_objects":4627,"num_object_clones":0,"num_object_copies":18508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4627,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8493,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2340}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'8264","reported_seq":8355,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:27.183390+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:40:27.183390+0000","last_peered":"2024-09-05T16:40:27.183390+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:40:27.183390+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8264,"log_dups_size":0,"ondisk_log_size":8264,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36821584,"num_objects":4495,"num_object_clones":0,"num_object_copies":17980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4495,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8263,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2256}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'8312","reported_seq":12944,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:27.497570+0000","last_change":"2024-09-05T16:37:30.634570+0000","last_active":"2024-09-05T16:40:27.497570+0000","last_peered":"2024-09-05T16:40:27.497570+0000","last_clean":"2024-09-05T16:33:45.082039+0000","last_became_active":"2024-09-05T16:34:56.792386+0000","last_became_peered":"2024-09-05T16:34:56.792386+0000","last_unstale":"2024-09-05T16:40:27.497570+0000","last_undegraded":"2024-09-05T16:34:56.600595+0000","last_fullsized":"2024-09-05T16:34:56.558203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8312,"log_dups_size":0,"ondisk_log_size":8312,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36837968,"num_objects":4497,"num_object_clones":0,"num_object_copies":17988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4497,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8311,"num_write_kb":51231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2277,"num_bytes_recovered":18636800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4478}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'8513","reported_seq":8608,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:27.031438+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:40:27.031438+0000","last_peered":"2024-09-05T16:40:27.031438+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:40:27.031438+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8513,"log_dups_size":0,"ondisk_log_size":8513,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37786784,"num_objects":4613,"num_object_clones":0,"num_object_copies":18452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4613,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8511,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2275}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'8320","reported_seq":8419,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:27.421123+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:40:27.421123+0000","last_peered":"2024-09-05T16:40:27.421123+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:40:27.421123+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8320,"log_dups_size":0,"ondisk_log_size":8320,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36967584,"num_objects":4513,"num_object_clones":0,"num_object_copies":18052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4513,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8318,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2201}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'8219","reported_seq":8314,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:26.881044+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:40:26.881044+0000","last_peered":"2024-09-05T16:40:26.881044+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:40:26.881044+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8219,"log_dups_size":0,"ondisk_log_size":8219,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37319840,"num_objects":4556,"num_object_clones":0,"num_object_copies":18224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4556,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8217,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2332}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":594603064,"num_objects":72589,"num_object_clones":0,"num_object_copies":290356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31409,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72589,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":126648,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2524,"num_bytes_recovered":18636800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1166237696,"data_stored":1166237696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":132373,"ondisk_log_size":132373,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084366,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":759640,"kb_used_data":290624,"kb_used_omap":4,"kb_used_meta":468987,"kb_avail":93612200,"statfs":{"total":96636764160,"available":95858892800,"internally_reserved":0,"allocated":297598976,"data_stored":297406935,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":480243411},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,2,0,2,3,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986243,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":588968,"kb_used_data":268504,"kb_used_omap":4,"kb_used_meta":320443,"kb_avail":93782872,"statfs":{"total":96636764160,"available":96033660928,"internally_reserved":0,"allocated":274948096,"data_stored":274756055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":328134354},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,7,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673049,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":763668,"kb_used_data":291008,"kb_used_omap":1,"kb_used_meta":472638,"kb_avail":93608172,"statfs":{"total":96636764160,"available":95854768128,"internally_reserved":0,"allocated":297992192,"data_stored":297800151,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":483981765},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,0,0,0,0,0,0,0,0,0,0,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182480,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":768296,"kb_used_data":291028,"kb_used_omap":4,"kb_used_meta":477243,"kb_avail":93603544,"statfs":{"total":96636764160,"available":95850029056,"internally_reserved":0,"allocated":298012672,"data_stored":297820631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":488697549},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,2,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":297394176,"data_stored":297394176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":274657280,"data_stored":274657280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":296931328,"data_stored":296931328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":297254912,"data_stored":297254912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:40:34.645 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:40:35.232 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:40:35.233 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:40:35.269 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":247,"stamp":"2024-09-05T16:40:34.936407+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":585419832,"num_objects":71468,"num_object_clones":0,"num_object_copies":285872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31345,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71468,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":127769,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2587,"num_bytes_recovered":19111936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":133494,"ondisk_log_size":133494,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2854348,"kb_used_data":1127420,"kb_used_omap":14,"kb_used_meta":1726833,"kb_avail":374633012,"statfs":{"total":386547056640,"available":383624204288,"internally_reserved":0,"allocated":1154478080,"data_stored":1153672984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1768277559},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,10,4,15,3,4,0,0,0,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18022400,"num_objects":-2200,"num_object_clones":0,"num_object_copies":-8800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-172,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2200,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2200,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":171,"num_bytes_recovered":1179648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003187"},"pg_stats":[{"pgid":"1.f","version":"37'8407","reported_seq":8508,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:32.030540+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:40:32.030540+0000","last_peered":"2024-09-05T16:40:32.030540+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:40:32.030540+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8407,"log_dups_size":0,"ondisk_log_size":8407,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36839424,"num_objects":4497,"num_object_clones":0,"num_object_copies":17988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2237,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4497,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8407,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2260}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'8230","reported_seq":8411,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:32.070887+0000","last_change":"2024-09-05T16:40:28.919189+0000","last_active":"2024-09-05T16:40:32.070887+0000","last_peered":"2024-09-05T16:40:32.070887+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:40:32.070887+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8230,"log_dups_size":0,"ondisk_log_size":8230,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35419296,"num_objects":4324,"num_object_clones":0,"num_object_copies":17296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4324,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8228,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2109}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'8316","reported_seq":8420,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:32.538186+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:40:32.538186+0000","last_peered":"2024-09-05T16:40:32.538186+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:40:32.538186+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8316,"log_dups_size":0,"ondisk_log_size":8316,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36338280,"num_objects":4437,"num_object_clones":0,"num_object_copies":17748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2242,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4437,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8315,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2196}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'8425","reported_seq":13153,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:40:31.378267+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:40:31.378267+0000","last_peered":"2024-09-05T16:40:31.378267+0000","last_clean":"2024-09-05T16:40:31.378267+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:40:31.378267+0000","last_undegraded":"2024-09-05T16:40:31.378267+0000","last_fullsized":"2024-09-05T16:40:31.378267+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":8425,"log_dups_size":0,"ondisk_log_size":8425,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36937728,"num_objects":4509,"num_object_clones":0,"num_object_copies":18036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4509,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2725,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'8153","reported_seq":8247,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:32.076948+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:40:32.076948+0000","last_peered":"2024-09-05T16:40:32.076948+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:40:32.076948+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8153,"log_dups_size":0,"ondisk_log_size":8153,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36385952,"num_objects":4442,"num_object_clones":0,"num_object_copies":17768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2148,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4442,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8151,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2296}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'8302","reported_seq":8407,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:31.998036+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:40:31.998036+0000","last_peered":"2024-09-05T16:40:31.998036+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:40:31.998036+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8302,"log_dups_size":0,"ondisk_log_size":8302,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36618240,"num_objects":4470,"num_object_clones":0,"num_object_copies":17880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4470,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8302,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2247}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'8133","reported_seq":8234,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:31.855824+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:40:31.855824+0000","last_peered":"2024-09-05T16:40:31.855824+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:40:31.855824+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8133,"log_dups_size":0,"ondisk_log_size":8133,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37098656,"num_objects":4529,"num_object_clones":0,"num_object_copies":18116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8131,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2395}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'8296","reported_seq":8382,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:32.024937+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:40:32.024937+0000","last_peered":"2024-09-05T16:40:32.024937+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:40:32.024937+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8296,"log_dups_size":0,"ondisk_log_size":8296,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36721824,"num_objects":4483,"num_object_clones":0,"num_object_copies":17932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4483,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8294,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2250}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'8343","reported_seq":8440,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:32.213058+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:40:32.213058+0000","last_peered":"2024-09-05T16:40:32.213058+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:40:32.213058+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8343,"log_dups_size":0,"ondisk_log_size":8343,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36292928,"num_objects":4431,"num_object_clones":0,"num_object_copies":17724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2248,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4431,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8340,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2186}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'8349","reported_seq":8459,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:32.359631+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:40:32.359631+0000","last_peered":"2024-09-05T16:40:32.359631+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:40:32.359631+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8349,"log_dups_size":0,"ondisk_log_size":8349,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36541600,"num_objects":4461,"num_object_clones":0,"num_object_copies":17844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8347,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2211}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'8567","reported_seq":8680,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:31.949568+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:40:31.949568+0000","last_peered":"2024-09-05T16:40:31.949568+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:40:31.949568+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8567,"log_dups_size":0,"ondisk_log_size":8567,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37318384,"num_objects":4556,"num_object_clones":0,"num_object_copies":18224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4556,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8564,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2269}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'8324","reported_seq":8419,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:32.196003+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:40:32.196003+0000","last_peered":"2024-09-05T16:40:32.196003+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:40:32.196003+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8324,"log_dups_size":0,"ondisk_log_size":8324,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36330064,"num_objects":4435,"num_object_clones":0,"num_object_copies":17740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4435,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8323,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2196}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'8367","reported_seq":13056,"reported_epoch":37,"state":"active+clean+scrubbing","last_fresh":"2024-09-05T16:40:31.252230+0000","last_change":"2024-09-05T16:40:30.197924+0000","last_active":"2024-09-05T16:40:31.252230+0000","last_peered":"2024-09-05T16:40:31.252230+0000","last_clean":"2024-09-05T16:40:31.252230+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:40:31.252230+0000","last_undegraded":"2024-09-05T16:40:31.252230+0000","last_fullsized":"2024-09-05T16:40:31.252230+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":448,"log_size":8367,"log_dups_size":0,"ondisk_log_size":8367,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 1s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36387408,"num_objects":4442,"num_object_clones":0,"num_object_copies":17768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4442,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8366,"num_write_kb":51231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2295,"num_bytes_recovered":18784256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'8572","reported_seq":8671,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:32.191870+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:40:32.191870+0000","last_peered":"2024-09-05T16:40:32.191870+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:40:32.191870+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8572,"log_dups_size":0,"ondisk_log_size":8572,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37303456,"num_objects":4554,"num_object_clones":0,"num_object_copies":18216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4554,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8570,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2217}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'8409","reported_seq":8512,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:32.505251+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:40:32.505251+0000","last_peered":"2024-09-05T16:40:32.505251+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:40:32.505251+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8409,"log_dups_size":0,"ondisk_log_size":8409,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36238496,"num_objects":4424,"num_object_clones":0,"num_object_copies":17696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4424,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8407,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2112}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'8301","reported_seq":8400,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:32.035112+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:40:32.035112+0000","last_peered":"2024-09-05T16:40:32.035112+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:40:32.035112+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8301,"log_dups_size":0,"ondisk_log_size":8301,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36648096,"num_objects":4474,"num_object_clones":0,"num_object_copies":17896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4474,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8299,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2250}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":585419832,"num_objects":71468,"num_object_clones":0,"num_object_copies":285872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31345,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71468,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":127769,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2587,"num_bytes_recovered":19111936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1151942656,"data_stored":1151942656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":133494,"ondisk_log_size":133494,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084367,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":759344,"kb_used_data":286164,"kb_used_omap":4,"kb_used_meta":473147,"kb_avail":93612496,"statfs":{"total":96636764160,"available":95859195904,"internally_reserved":0,"allocated":293031936,"data_stored":292830662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":484503251},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,3,0,5,2,1,0,0,0,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986244,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":592804,"kb_used_data":268180,"kb_used_omap":4,"kb_used_meta":324603,"kb_avail":93779036,"statfs":{"total":96636764160,"available":96029732864,"internally_reserved":0,"allocated":274616320,"data_stored":274415046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":332394194},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,1,3,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673050,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":750912,"kb_used_data":286576,"kb_used_omap":1,"kb_used_meta":464318,"kb_avail":93620928,"statfs":{"total":96636764160,"available":95867830272,"internally_reserved":0,"allocated":293453824,"data_stored":293252550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":475462085},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,3,6],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182481,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":751288,"kb_used_data":286500,"kb_used_omap":4,"kb_used_meta":464763,"kb_avail":93620552,"statfs":{"total":96636764160,"available":95867445248,"internally_reserved":0,"allocated":293376000,"data_stored":293174726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":475918029},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":292753408,"data_stored":292753408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":274251776,"data_stored":274251776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":292302848,"data_stored":292302848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":292634624,"data_stored":292634624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:40:35.271 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:40:35.868 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:40:35.869 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:40:35.894 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":247,"stamp":"2024-09-05T16:40:34.936407+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":585419832,"num_objects":71468,"num_object_clones":0,"num_object_copies":285872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31345,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71468,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":127769,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2587,"num_bytes_recovered":19111936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":133494,"ondisk_log_size":133494,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2854348,"kb_used_data":1127420,"kb_used_omap":14,"kb_used_meta":1726833,"kb_avail":374633012,"statfs":{"total":386547056640,"available":383624204288,"internally_reserved":0,"allocated":1154478080,"data_stored":1153672984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1768277559},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,10,4,15,3,4,0,0,0,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18022400,"num_objects":-2200,"num_object_clones":0,"num_object_copies":-8800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-172,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2200,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2200,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":171,"num_bytes_recovered":1179648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003187"},"pg_stats":[{"pgid":"1.f","version":"37'8407","reported_seq":8508,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:32.030540+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:40:32.030540+0000","last_peered":"2024-09-05T16:40:32.030540+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:40:32.030540+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8407,"log_dups_size":0,"ondisk_log_size":8407,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36839424,"num_objects":4497,"num_object_clones":0,"num_object_copies":17988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2237,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4497,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8407,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2260}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'8230","reported_seq":8411,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:32.070887+0000","last_change":"2024-09-05T16:40:28.919189+0000","last_active":"2024-09-05T16:40:32.070887+0000","last_peered":"2024-09-05T16:40:32.070887+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:40:32.070887+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8230,"log_dups_size":0,"ondisk_log_size":8230,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35419296,"num_objects":4324,"num_object_clones":0,"num_object_copies":17296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4324,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8228,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2109}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'8316","reported_seq":8420,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:32.538186+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:40:32.538186+0000","last_peered":"2024-09-05T16:40:32.538186+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:40:32.538186+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8316,"log_dups_size":0,"ondisk_log_size":8316,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36338280,"num_objects":4437,"num_object_clones":0,"num_object_copies":17748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2242,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4437,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8315,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2196}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'8425","reported_seq":13153,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:40:31.378267+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:40:31.378267+0000","last_peered":"2024-09-05T16:40:31.378267+0000","last_clean":"2024-09-05T16:40:31.378267+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:40:31.378267+0000","last_undegraded":"2024-09-05T16:40:31.378267+0000","last_fullsized":"2024-09-05T16:40:31.378267+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":8425,"log_dups_size":0,"ondisk_log_size":8425,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36937728,"num_objects":4509,"num_object_clones":0,"num_object_copies":18036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4509,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2725,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'8153","reported_seq":8247,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:32.076948+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:40:32.076948+0000","last_peered":"2024-09-05T16:40:32.076948+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:40:32.076948+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8153,"log_dups_size":0,"ondisk_log_size":8153,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36385952,"num_objects":4442,"num_object_clones":0,"num_object_copies":17768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2148,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4442,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8151,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2296}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'8302","reported_seq":8407,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:31.998036+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:40:31.998036+0000","last_peered":"2024-09-05T16:40:31.998036+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:40:31.998036+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8302,"log_dups_size":0,"ondisk_log_size":8302,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36618240,"num_objects":4470,"num_object_clones":0,"num_object_copies":17880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4470,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8302,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2247}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'8133","reported_seq":8234,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:31.855824+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:40:31.855824+0000","last_peered":"2024-09-05T16:40:31.855824+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:40:31.855824+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8133,"log_dups_size":0,"ondisk_log_size":8133,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37098656,"num_objects":4529,"num_object_clones":0,"num_object_copies":18116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8131,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2395}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'8296","reported_seq":8382,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:32.024937+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:40:32.024937+0000","last_peered":"2024-09-05T16:40:32.024937+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:40:32.024937+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8296,"log_dups_size":0,"ondisk_log_size":8296,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36721824,"num_objects":4483,"num_object_clones":0,"num_object_copies":17932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4483,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8294,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2250}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'8343","reported_seq":8440,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:32.213058+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:40:32.213058+0000","last_peered":"2024-09-05T16:40:32.213058+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:40:32.213058+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8343,"log_dups_size":0,"ondisk_log_size":8343,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36292928,"num_objects":4431,"num_object_clones":0,"num_object_copies":17724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2248,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4431,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8340,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2186}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'8349","reported_seq":8459,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:32.359631+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:40:32.359631+0000","last_peered":"2024-09-05T16:40:32.359631+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:40:32.359631+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8349,"log_dups_size":0,"ondisk_log_size":8349,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36541600,"num_objects":4461,"num_object_clones":0,"num_object_copies":17844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8347,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2211}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'8567","reported_seq":8680,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:31.949568+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:40:31.949568+0000","last_peered":"2024-09-05T16:40:31.949568+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:40:31.949568+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8567,"log_dups_size":0,"ondisk_log_size":8567,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37318384,"num_objects":4556,"num_object_clones":0,"num_object_copies":18224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4556,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8564,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2269}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'8324","reported_seq":8419,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:32.196003+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:40:32.196003+0000","last_peered":"2024-09-05T16:40:32.196003+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:40:32.196003+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8324,"log_dups_size":0,"ondisk_log_size":8324,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36330064,"num_objects":4435,"num_object_clones":0,"num_object_copies":17740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4435,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8323,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2196}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'8367","reported_seq":13056,"reported_epoch":37,"state":"active+clean+scrubbing","last_fresh":"2024-09-05T16:40:31.252230+0000","last_change":"2024-09-05T16:40:30.197924+0000","last_active":"2024-09-05T16:40:31.252230+0000","last_peered":"2024-09-05T16:40:31.252230+0000","last_clean":"2024-09-05T16:40:31.252230+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:40:31.252230+0000","last_undegraded":"2024-09-05T16:40:31.252230+0000","last_fullsized":"2024-09-05T16:40:31.252230+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":448,"log_size":8367,"log_dups_size":0,"ondisk_log_size":8367,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 1s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36387408,"num_objects":4442,"num_object_clones":0,"num_object_copies":17768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4442,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8366,"num_write_kb":51231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2295,"num_bytes_recovered":18784256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'8572","reported_seq":8671,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:32.191870+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:40:32.191870+0000","last_peered":"2024-09-05T16:40:32.191870+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:40:32.191870+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8572,"log_dups_size":0,"ondisk_log_size":8572,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37303456,"num_objects":4554,"num_object_clones":0,"num_object_copies":18216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4554,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8570,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2217}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'8409","reported_seq":8512,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:32.505251+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:40:32.505251+0000","last_peered":"2024-09-05T16:40:32.505251+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:40:32.505251+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8409,"log_dups_size":0,"ondisk_log_size":8409,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36238496,"num_objects":4424,"num_object_clones":0,"num_object_copies":17696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4424,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8407,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2112}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'8301","reported_seq":8400,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:32.035112+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:40:32.035112+0000","last_peered":"2024-09-05T16:40:32.035112+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:40:32.035112+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8301,"log_dups_size":0,"ondisk_log_size":8301,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36648096,"num_objects":4474,"num_object_clones":0,"num_object_copies":17896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4474,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8299,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2250}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":585419832,"num_objects":71468,"num_object_clones":0,"num_object_copies":285872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31345,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71468,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":127769,"num_write_kb":774158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2587,"num_bytes_recovered":19111936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1151942656,"data_stored":1151942656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":133494,"ondisk_log_size":133494,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084367,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":759344,"kb_used_data":286164,"kb_used_omap":4,"kb_used_meta":473147,"kb_avail":93612496,"statfs":{"total":96636764160,"available":95859195904,"internally_reserved":0,"allocated":293031936,"data_stored":292830662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":484503251},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,3,0,5,2,1,0,0,0,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986244,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":592804,"kb_used_data":268180,"kb_used_omap":4,"kb_used_meta":324603,"kb_avail":93779036,"statfs":{"total":96636764160,"available":96029732864,"internally_reserved":0,"allocated":274616320,"data_stored":274415046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":332394194},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,1,3,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673050,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":750912,"kb_used_data":286576,"kb_used_omap":1,"kb_used_meta":464318,"kb_avail":93620928,"statfs":{"total":96636764160,"available":95867830272,"internally_reserved":0,"allocated":293453824,"data_stored":293252550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":475462085},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,3,6],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182481,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":751288,"kb_used_data":286500,"kb_used_omap":4,"kb_used_meta":464763,"kb_avail":93620552,"statfs":{"total":96636764160,"available":95867445248,"internally_reserved":0,"allocated":293376000,"data_stored":293174726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":475918029},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":292753408,"data_stored":292753408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":274251776,"data_stored":274251776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":292302848,"data_stored":292302848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":292634624,"data_stored":292634624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:40:35.895 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-05T16:40:35.895 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'8407", 'reported_seq': 8508, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:32.030540+0000', 'last_change': '2024-09-05T16:34:56.833497+0000', 'last_active': '2024-09-05T16:40:32.030540+0000', 'last_peered': '2024-09-05T16:40:32.030540+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.828289+0000', 'last_became_peered': '2024-09-05T16:34:56.828289+0000', 'last_unstale': '2024-09-05T16:40:32.030540+0000', 'last_undegraded': '2024-09-05T16:34:56.612160+0000', 'last_fullsized': '2024-09-05T16:34:56.557632+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8407, 'log_dups_size': 0, 'ondisk_log_size': 8407, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36839424, 'num_objects': 4497, 'num_object_clones': 0, 'num_object_copies': 17988, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2237, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4497, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8407, 'num_write_kb': 51616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2260}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:40:35.895 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-05T16:40:35.895 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "37'8230", 'reported_seq': 8411, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:32.070887+0000', 'last_change': '2024-09-05T16:40:28.919189+0000', 'last_active': '2024-09-05T16:40:32.070887+0000', 'last_peered': '2024-09-05T16:40:32.070887+0000', 'last_clean': '2024-09-05T16:33:53.200762+0000', 'last_became_active': '2024-09-05T16:34:56.820292+0000', 'last_became_peered': '2024-09-05T16:34:56.820292+0000', 'last_unstale': '2024-09-05T16:40:32.070887+0000', 'last_undegraded': '2024-09-05T16:34:56.605776+0000', 'last_fullsized': '2024-09-05T16:34:56.559383+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8230, 'log_dups_size': 0, 'ondisk_log_size': 8230, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35419296, 'num_objects': 4324, 'num_object_clones': 0, 'num_object_copies': 17296, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2215, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4324, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8228, 'num_write_kb': 50212, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 56, 'num_bytes_recovered': 327680, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2109}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:40:35.896 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:40:35.896 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'8316", 'reported_seq': 8420, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:32.538186+0000', 'last_change': '2024-09-05T16:34:56.867900+0000', 'last_active': '2024-09-05T16:40:32.538186+0000', 'last_peered': '2024-09-05T16:40:32.538186+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:40:32.538186+0000', 'last_undegraded': '2024-09-05T16:34:56.590972+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8316, 'log_dups_size': 0, 'ondisk_log_size': 8316, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36338280, 'num_objects': 4437, 'num_object_clones': 0, 'num_object_copies': 17748, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2242, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4437, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 8315, 'num_write_kb': 50999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2196}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:40:35.896 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:40:35.896 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'8153", 'reported_seq': 8247, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:32.076948+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:40:32.076948+0000', 'last_peered': '2024-09-05T16:40:32.076948+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:40:32.076948+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8153, 'log_dups_size': 0, 'ondisk_log_size': 8153, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36385952, 'num_objects': 4442, 'num_object_clones': 0, 'num_object_copies': 17768, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2148, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4442, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8151, 'num_write_kb': 50373, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2296}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:40:35.896 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:40:35.896 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'8302", 'reported_seq': 8407, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:31.998036+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:40:31.998036+0000', 'last_peered': '2024-09-05T16:40:31.998036+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:40:31.998036+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8302, 'log_dups_size': 0, 'ondisk_log_size': 8302, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36618240, 'num_objects': 4470, 'num_object_clones': 0, 'num_object_copies': 17880, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2223, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4470, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8302, 'num_write_kb': 51088, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2247}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:40:35.896 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:40:35.896 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'8133", 'reported_seq': 8234, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:31.855824+0000', 'last_change': '2024-09-05T16:34:56.903436+0000', 'last_active': '2024-09-05T16:40:31.855824+0000', 'last_peered': '2024-09-05T16:40:31.855824+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:40:31.855824+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8133, 'log_dups_size': 0, 'ondisk_log_size': 8133, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37098656, 'num_objects': 4529, 'num_object_clones': 0, 'num_object_copies': 18116, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2136, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4529, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8131, 'num_write_kb': 50636, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2395}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:40:35.896 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-05T16:40:35.896 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "37'8296", 'reported_seq': 8382, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:32.024937+0000', 'last_change': '2024-09-05T16:34:56.883319+0000', 'last_active': '2024-09-05T16:40:32.024937+0000', 'last_peered': '2024-09-05T16:40:32.024937+0000', 'last_clean': '2024-09-05T16:33:53.505305+0000', 'last_became_active': '2024-09-05T16:34:56.873389+0000', 'last_became_peered': '2024-09-05T16:34:56.873389+0000', 'last_unstale': '2024-09-05T16:40:32.024937+0000', 'last_undegraded': '2024-09-05T16:34:56.602567+0000', 'last_fullsized': '2024-09-05T16:34:56.559958+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8296, 'log_dups_size': 0, 'ondisk_log_size': 8296, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36721824, 'num_objects': 4483, 'num_object_clones': 0, 'num_object_copies': 17932, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2235, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4483, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8294, 'num_write_kb': 51108, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 8, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2250}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:40:35.897 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:40:35.897 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'8343", 'reported_seq': 8440, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:32.213058+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:40:32.213058+0000', 'last_peered': '2024-09-05T16:40:32.213058+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:40:32.213058+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8343, 'log_dups_size': 0, 'ondisk_log_size': 8343, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36292928, 'num_objects': 4431, 'num_object_clones': 0, 'num_object_copies': 17724, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2248, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4431, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8340, 'num_write_kb': 51081, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2186}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:40:35.897 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:40:35.897 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'8349", 'reported_seq': 8459, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:32.359631+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:40:32.359631+0000', 'last_peered': '2024-09-05T16:40:32.359631+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:40:32.359631+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8349, 'log_dups_size': 0, 'ondisk_log_size': 8349, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36541600, 'num_objects': 4461, 'num_object_clones': 0, 'num_object_copies': 17844, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2252, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4461, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8347, 'num_write_kb': 51230, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2211}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:40:35.897 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:40:35.897 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'8567", 'reported_seq': 8680, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:31.949568+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:40:31.949568+0000', 'last_peered': '2024-09-05T16:40:31.949568+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:40:31.949568+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8567, 'log_dups_size': 0, 'ondisk_log_size': 8567, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37318384, 'num_objects': 4556, 'num_object_clones': 0, 'num_object_copies': 18224, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2290, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4556, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8564, 'num_write_kb': 52476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2269}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:40:35.897 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:40:35.897 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'8324", 'reported_seq': 8419, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:32.196003+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:40:32.196003+0000', 'last_peered': '2024-09-05T16:40:32.196003+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:40:32.196003+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8324, 'log_dups_size': 0, 'ondisk_log_size': 8324, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36330064, 'num_objects': 4435, 'num_object_clones': 0, 'num_object_copies': 17740, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2240, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4435, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8323, 'num_write_kb': 51031, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2196}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:40:35.897 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-05T16:40:35.897 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "37'8367", 'reported_seq': 13056, 'reported_epoch': 37, 'state': 'active+clean+scrubbing', 'last_fresh': '2024-09-05T16:40:31.252230+0000', 'last_change': '2024-09-05T16:40:30.197924+0000', 'last_active': '2024-09-05T16:40:31.252230+0000', 'last_peered': '2024-09-05T16:40:31.252230+0000', 'last_clean': '2024-09-05T16:40:31.252230+0000', 'last_became_active': '2024-09-05T16:40:29.954942+0000', 'last_became_peered': '2024-09-05T16:40:29.954942+0000', 'last_unstale': '2024-09-05T16:40:31.252230+0000', 'last_undegraded': '2024-09-05T16:40:31.252230+0000', 'last_fullsized': '2024-09-05T16:40:31.252230+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 37, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 448, 'log_size': 8367, 'log_dups_size': 0, 'ondisk_log_size': 8367, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'scrubbing for 1s', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36387408, 'num_objects': 4442, 'num_object_clones': 0, 'num_object_copies': 17768, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4442, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8366, 'num_write_kb': 51231, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2295, 'num_bytes_recovered': 18784256, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [1, 2, 3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-05T16:40:35.898 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:40:35.898 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'8572", 'reported_seq': 8671, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:32.191870+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:40:32.191870+0000', 'last_peered': '2024-09-05T16:40:32.191870+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:40:32.191870+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8572, 'log_dups_size': 0, 'ondisk_log_size': 8572, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37303456, 'num_objects': 4554, 'num_object_clones': 0, 'num_object_copies': 18216, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2339, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4554, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8570, 'num_write_kb': 52500, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 22, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2217}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:40:35.898 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:40:35.898 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'8409", 'reported_seq': 8512, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:32.505251+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:40:32.505251+0000', 'last_peered': '2024-09-05T16:40:32.505251+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:40:32.505251+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8409, 'log_dups_size': 0, 'ondisk_log_size': 8409, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36238496, 'num_objects': 4424, 'num_object_clones': 0, 'num_object_copies': 17696, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2314, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4424, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8407, 'num_write_kb': 51325, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2112}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:40:35.898 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:40:35.898 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'8301", 'reported_seq': 8400, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:32.035112+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:40:32.035112+0000', 'last_peered': '2024-09-05T16:40:32.035112+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:40:32.035112+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8301, 'log_dups_size': 0, 'ondisk_log_size': 8301, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36648096, 'num_objects': 4474, 'num_object_clones': 0, 'num_object_copies': 17896, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2226, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4474, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8299, 'num_write_kb': 51092, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2250}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:40:35.898 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:40:36.771 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:40:36.772 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 31345/285872 objects degraded (10.965%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":440,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean+scrubbing","count":1},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":71468,"data_bytes":585419832,"bytes_used":2922852352,"bytes_avail":383624204288,"bytes_total":386547056640,"degraded_objects":31345,"degraded_total":285872,"degraded_ratio":0.10964697486987184,"recovering_objects_per_sec":21,"recovering_bytes_per_sec":147397,"recovering_keys_per_sec":0,"num_objects_recovered":171,"num_bytes_recovered":1179648,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":274},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:40:36.796 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:40:36.796 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:40:37.393 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:40:37.393 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:40:37.417 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":248,"stamp":"2024-09-05T16:40:36.937069+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":584282600,"num_objects":71329,"num_object_clones":0,"num_object_copies":285316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31345,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71329,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":119508,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":292,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":133632,"ondisk_log_size":133632,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2857900,"kb_used_data":1126808,"kb_used_omap":14,"kb_used_meta":1730993,"kb_avail":374629460,"statfs":{"total":386547056640,"available":383620567040,"internally_reserved":0,"allocated":1153851392,"data_stored":1153046296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1772537399},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,8,6,16,8,2,0,0,0,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-10320464,"num_objects":-1260,"num_object_clones":0,"num_object_copies":-5040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-64,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1260,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-7140,"num_write_kb":-51231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-2232,"num_bytes_recovered":-18309120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.013695"},"pg_stats":[{"pgid":"1.f","version":"37'8407","reported_seq":8508,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:32.030540+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:40:32.030540+0000","last_peered":"2024-09-05T16:40:32.030540+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:40:32.030540+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8407,"log_dups_size":0,"ondisk_log_size":8407,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36839424,"num_objects":4497,"num_object_clones":0,"num_object_copies":17988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2237,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4497,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8407,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2260}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'8230","reported_seq":8411,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:32.070887+0000","last_change":"2024-09-05T16:40:28.919189+0000","last_active":"2024-09-05T16:40:32.070887+0000","last_peered":"2024-09-05T16:40:32.070887+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:40:32.070887+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8230,"log_dups_size":0,"ondisk_log_size":8230,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35419296,"num_objects":4324,"num_object_clones":0,"num_object_copies":17296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4324,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8228,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2109}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'8316","reported_seq":8420,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:32.538186+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:40:32.538186+0000","last_peered":"2024-09-05T16:40:32.538186+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:40:32.538186+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8316,"log_dups_size":0,"ondisk_log_size":8316,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36338280,"num_objects":4437,"num_object_clones":0,"num_object_copies":17748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2242,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4437,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8315,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2196}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'8486","reported_seq":13213,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:40:36.441960+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:40:36.441960+0000","last_peered":"2024-09-05T16:40:36.441960+0000","last_clean":"2024-09-05T16:40:36.441960+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:40:36.441960+0000","last_undegraded":"2024-09-05T16:40:36.441960+0000","last_fullsized":"2024-09-05T16:40:36.441960+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":8486,"log_dups_size":0,"ondisk_log_size":8486,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36438016,"num_objects":4448,"num_object_clones":0,"num_object_copies":17792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4448,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2786,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'8153","reported_seq":8247,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:32.076948+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:40:32.076948+0000","last_peered":"2024-09-05T16:40:32.076948+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:40:32.076948+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8153,"log_dups_size":0,"ondisk_log_size":8153,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36385952,"num_objects":4442,"num_object_clones":0,"num_object_copies":17768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2148,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4442,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8151,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2296}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'8302","reported_seq":8407,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:31.998036+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:40:31.998036+0000","last_peered":"2024-09-05T16:40:31.998036+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:40:31.998036+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8302,"log_dups_size":0,"ondisk_log_size":8302,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36618240,"num_objects":4470,"num_object_clones":0,"num_object_copies":17880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4470,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8302,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2247}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'8133","reported_seq":8234,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:31.855824+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:40:31.855824+0000","last_peered":"2024-09-05T16:40:31.855824+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:40:31.855824+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8133,"log_dups_size":0,"ondisk_log_size":8133,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37098656,"num_objects":4529,"num_object_clones":0,"num_object_copies":18116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8131,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2395}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'8296","reported_seq":8382,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:32.024937+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:40:32.024937+0000","last_peered":"2024-09-05T16:40:32.024937+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:40:32.024937+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8296,"log_dups_size":0,"ondisk_log_size":8296,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36721824,"num_objects":4483,"num_object_clones":0,"num_object_copies":17932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4483,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8294,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2250}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'8343","reported_seq":8440,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:32.213058+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:40:32.213058+0000","last_peered":"2024-09-05T16:40:32.213058+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:40:32.213058+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8343,"log_dups_size":0,"ondisk_log_size":8343,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36292928,"num_objects":4431,"num_object_clones":0,"num_object_copies":17724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2248,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4431,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8340,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2186}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'8349","reported_seq":8459,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:32.359631+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:40:32.359631+0000","last_peered":"2024-09-05T16:40:32.359631+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:40:32.359631+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8349,"log_dups_size":0,"ondisk_log_size":8349,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36541600,"num_objects":4461,"num_object_clones":0,"num_object_copies":17844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8347,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2211}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'8567","reported_seq":8680,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:31.949568+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:40:31.949568+0000","last_peered":"2024-09-05T16:40:31.949568+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:40:31.949568+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8567,"log_dups_size":0,"ondisk_log_size":8567,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37318384,"num_objects":4556,"num_object_clones":0,"num_object_copies":18224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4556,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8564,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2269}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'8324","reported_seq":8419,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:32.196003+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:40:32.196003+0000","last_peered":"2024-09-05T16:40:32.196003+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:40:32.196003+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8324,"log_dups_size":0,"ondisk_log_size":8324,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36330064,"num_objects":4435,"num_object_clones":0,"num_object_copies":17740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4435,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8323,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2196}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'8444","reported_seq":13179,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:40:36.222099+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:40:36.222099+0000","last_peered":"2024-09-05T16:40:36.222099+0000","last_clean":"2024-09-05T16:40:36.222099+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:40:36.222099+0000","last_undegraded":"2024-09-05T16:40:36.222099+0000","last_fullsized":"2024-09-05T16:40:36.222099+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":8444,"log_dups_size":0,"ondisk_log_size":8444,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35749888,"num_objects":4364,"num_object_clones":0,"num_object_copies":17456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4364,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'8572","reported_seq":8671,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:32.191870+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:40:32.191870+0000","last_peered":"2024-09-05T16:40:32.191870+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:40:32.191870+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8572,"log_dups_size":0,"ondisk_log_size":8572,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37303456,"num_objects":4554,"num_object_clones":0,"num_object_copies":18216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4554,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8570,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2217}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'8409","reported_seq":8512,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:32.505251+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:40:32.505251+0000","last_peered":"2024-09-05T16:40:32.505251+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:40:32.505251+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8409,"log_dups_size":0,"ondisk_log_size":8409,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36238496,"num_objects":4424,"num_object_clones":0,"num_object_copies":17696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4424,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8407,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2112}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'8301","reported_seq":8400,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:32.035112+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:40:32.035112+0000","last_peered":"2024-09-05T16:40:32.035112+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:40:32.035112+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8301,"log_dups_size":0,"ondisk_log_size":8301,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36648096,"num_objects":4474,"num_object_clones":0,"num_object_copies":17896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4474,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8299,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2250}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":584282600,"num_objects":71329,"num_object_clones":0,"num_object_copies":285316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31345,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71329,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":119508,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":292,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1151369216,"data_stored":1151369216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":133632,"ondisk_log_size":133632,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084367,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":759344,"kb_used_data":286164,"kb_used_omap":4,"kb_used_meta":473147,"kb_avail":93612496,"statfs":{"total":96636764160,"available":95859195904,"internally_reserved":0,"allocated":293031936,"data_stored":292830662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":484503251},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,3,0,5,2,1,0,0,0,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986245,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":596356,"kb_used_data":267568,"kb_used_omap":4,"kb_used_meta":328763,"kb_avail":93775484,"statfs":{"total":96636764160,"available":96026095616,"internally_reserved":0,"allocated":273989632,"data_stored":273788358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":336654034},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,4,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673050,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":750912,"kb_used_data":286576,"kb_used_omap":1,"kb_used_meta":464318,"kb_avail":93620928,"statfs":{"total":96636764160,"available":95867830272,"internally_reserved":0,"allocated":293453824,"data_stored":293252550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":475462085},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,3,6],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182481,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":751288,"kb_used_data":286500,"kb_used_omap":4,"kb_used_meta":464763,"kb_avail":93620552,"statfs":{"total":96636764160,"available":95867445248,"internally_reserved":0,"allocated":293376000,"data_stored":293174726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":475918029},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":292753408,"data_stored":292753408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":273678336,"data_stored":273678336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":292302848,"data_stored":292302848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":292634624,"data_stored":292634624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:40:40.420 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:40:41.067 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:40:41.067 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:40:41.088 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":250,"stamp":"2024-09-05T16:40:40.945739+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":576565736,"num_objects":70387,"num_object_clones":0,"num_object_copies":281548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70387,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":120450,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":346,"num_bytes_recovered":745472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":134574,"ondisk_log_size":134574,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2844768,"kb_used_data":1113676,"kb_used_omap":14,"kb_used_meta":1730993,"kb_avail":374642592,"statfs":{"total":386547056640,"available":383634014208,"internally_reserved":0,"allocated":1140404224,"data_stored":1139599128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1772537399},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,11,12,7,17,11,4,0,0,0,0,0,4],"upper_bound":16384},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18037328,"num_objects":-2202,"num_object_clones":0,"num_object_copies":-8808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2202,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-6198,"num_write_kb":-51231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-2178,"num_bytes_recovered":-17891328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.018779"},"pg_stats":[{"pgid":"1.f","version":"37'8479","reported_seq":8582,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:37.083107+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:40:37.083107+0000","last_peered":"2024-09-05T16:40:37.083107+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:40:37.083107+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8479,"log_dups_size":0,"ondisk_log_size":8479,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36249600,"num_objects":4425,"num_object_clones":0,"num_object_copies":17700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4425,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8479,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2189}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'8297","reported_seq":8581,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:37.131916+0000","last_change":"2024-09-05T16:40:28.919189+0000","last_active":"2024-09-05T16:40:37.131916+0000","last_peered":"2024-09-05T16:40:37.131916+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:40:37.131916+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8297,"log_dups_size":0,"ondisk_log_size":8297,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34870432,"num_objects":4257,"num_object_clones":0,"num_object_copies":17028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4257,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8295,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":107,"num_bytes_recovered":745472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2093}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'8366","reported_seq":8468,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:37.522138+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:40:37.522138+0000","last_peered":"2024-09-05T16:40:37.522138+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:40:37.522138+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8366,"log_dups_size":0,"ondisk_log_size":8366,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35928680,"num_objects":4387,"num_object_clones":0,"num_object_copies":17548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2242,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4387,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8365,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2146}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'8486","reported_seq":13213,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:40:36.441960+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:40:36.441960+0000","last_peered":"2024-09-05T16:40:36.441960+0000","last_clean":"2024-09-05T16:40:36.441960+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:40:36.441960+0000","last_undegraded":"2024-09-05T16:40:36.441960+0000","last_fullsized":"2024-09-05T16:40:36.441960+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":8486,"log_dups_size":0,"ondisk_log_size":8486,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36438016,"num_objects":4448,"num_object_clones":0,"num_object_copies":17792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4448,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2786,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'8217","reported_seq":8313,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:37.062371+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:40:37.062371+0000","last_peered":"2024-09-05T16:40:37.062371+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:40:37.062371+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8217,"log_dups_size":0,"ondisk_log_size":8217,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35861664,"num_objects":4378,"num_object_clones":0,"num_object_copies":17512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2148,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4378,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8215,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2232}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'8377","reported_seq":8482,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:37.173486+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:40:37.173486+0000","last_peered":"2024-09-05T16:40:37.173486+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:40:37.173486+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8377,"log_dups_size":0,"ondisk_log_size":8377,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36003840,"num_objects":4395,"num_object_clones":0,"num_object_copies":17580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4395,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8377,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2172}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'8202","reported_seq":8303,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:37.078142+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:40:37.078142+0000","last_peered":"2024-09-05T16:40:37.078142+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:40:37.078142+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8202,"log_dups_size":0,"ondisk_log_size":8202,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36533408,"num_objects":4460,"num_object_clones":0,"num_object_copies":17840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4460,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8200,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2326}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'8348","reported_seq":8434,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:37.002336+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:40:37.002336+0000","last_peered":"2024-09-05T16:40:37.002336+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:40:37.002336+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8348,"log_dups_size":0,"ondisk_log_size":8348,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36295840,"num_objects":4431,"num_object_clones":0,"num_object_copies":17724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4431,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8346,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2198}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'8411","reported_seq":8509,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:37.084183+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:40:37.084183+0000","last_peered":"2024-09-05T16:40:37.084183+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:40:37.084183+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8411,"log_dups_size":0,"ondisk_log_size":8411,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35735872,"num_objects":4363,"num_object_clones":0,"num_object_copies":17452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2248,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4363,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8408,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2118}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'8412","reported_seq":8523,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:37.480269+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:40:37.480269+0000","last_peered":"2024-09-05T16:40:37.480269+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:40:37.480269+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8412,"log_dups_size":0,"ondisk_log_size":8412,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36025504,"num_objects":4398,"num_object_clones":0,"num_object_copies":17592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4398,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8410,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2148}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'8635","reported_seq":8749,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:37.213180+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:40:37.213180+0000","last_peered":"2024-09-05T16:40:37.213180+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:40:37.213180+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8635,"log_dups_size":0,"ondisk_log_size":8635,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36761328,"num_objects":4488,"num_object_clones":0,"num_object_copies":17952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4488,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8632,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2202}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'8390","reported_seq":8487,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:37.223569+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:40:37.223569+0000","last_peered":"2024-09-05T16:40:37.223569+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:40:37.223569+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8390,"log_dups_size":0,"ondisk_log_size":8390,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35789392,"num_objects":4369,"num_object_clones":0,"num_object_copies":17476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4369,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8389,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2131}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'8444","reported_seq":13179,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:40:36.222099+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:40:36.222099+0000","last_peered":"2024-09-05T16:40:36.222099+0000","last_clean":"2024-09-05T16:40:36.222099+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:40:36.222099+0000","last_undegraded":"2024-09-05T16:40:36.222099+0000","last_fullsized":"2024-09-05T16:40:36.222099+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":8444,"log_dups_size":0,"ondisk_log_size":8444,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35749888,"num_objects":4364,"num_object_clones":0,"num_object_copies":17456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4364,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'8649","reported_seq":8750,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:37.193965+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:40:37.193965+0000","last_peered":"2024-09-05T16:40:37.193965+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:40:37.193965+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8649,"log_dups_size":0,"ondisk_log_size":8649,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36672672,"num_objects":4477,"num_object_clones":0,"num_object_copies":17908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4477,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8647,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2140}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'8494","reported_seq":8598,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:37.522698+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:40:37.522698+0000","last_peered":"2024-09-05T16:40:37.522698+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:40:37.522698+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8494,"log_dups_size":0,"ondisk_log_size":8494,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35542176,"num_objects":4339,"num_object_clones":0,"num_object_copies":17356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4339,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8492,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2027}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'8367","reported_seq":8466,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:37.076644+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:40:37.076644+0000","last_peered":"2024-09-05T16:40:37.076644+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:40:37.076644+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8367,"log_dups_size":0,"ondisk_log_size":8367,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36107424,"num_objects":4408,"num_object_clones":0,"num_object_copies":17632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4408,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8365,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2184}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":576565736,"num_objects":70387,"num_object_clones":0,"num_object_copies":281548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70387,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":120450,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":346,"num_bytes_recovered":745472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1138061312,"data_stored":1138061312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":134574,"ondisk_log_size":134574,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084368,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":746600,"kb_used_data":281752,"kb_used_omap":4,"kb_used_meta":464827,"kb_avail":93625240,"statfs":{"total":96636764160,"available":95872245760,"internally_reserved":0,"allocated":288514048,"data_stored":288312774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":475983571},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,4,4,5,2,1,0,0,0,0,0,2],"upper_bound":16384},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986245,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":596356,"kb_used_data":267568,"kb_used_omap":4,"kb_used_meta":328763,"kb_avail":93775484,"statfs":{"total":96636764160,"available":96026095616,"internally_reserved":0,"allocated":273989632,"data_stored":273788358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":336654034},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,4,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673051,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":750708,"kb_used_data":282200,"kb_used_omap":1,"kb_used_meta":468478,"kb_avail":93621132,"statfs":{"total":96636764160,"available":95868039168,"internally_reserved":0,"allocated":288972800,"data_stored":288771526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":479721925},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,6,0,3,2,0,0,0,0,0,0,2],"upper_bound":16384},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182482,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":751104,"kb_used_data":282156,"kb_used_omap":4,"kb_used_meta":468923,"kb_avail":93620736,"statfs":{"total":96636764160,"available":95867633664,"internally_reserved":0,"allocated":288927744,"data_stored":288726470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":480177869},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,10,0,0,5,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":288284672,"data_stored":288284672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":273678336,"data_stored":273678336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":287911936,"data_stored":287911936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":288186368,"data_stored":288186368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:40:41.089 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:40:41.706 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:40:41.706 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:40:41.726 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":250,"stamp":"2024-09-05T16:40:40.945739+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":576565736,"num_objects":70387,"num_object_clones":0,"num_object_copies":281548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70387,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":120450,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":346,"num_bytes_recovered":745472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":134574,"ondisk_log_size":134574,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2844768,"kb_used_data":1113676,"kb_used_omap":14,"kb_used_meta":1730993,"kb_avail":374642592,"statfs":{"total":386547056640,"available":383634014208,"internally_reserved":0,"allocated":1140404224,"data_stored":1139599128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1772537399},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,11,12,7,17,11,4,0,0,0,0,0,4],"upper_bound":16384},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18037328,"num_objects":-2202,"num_object_clones":0,"num_object_copies":-8808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2202,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-6198,"num_write_kb":-51231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-2178,"num_bytes_recovered":-17891328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.018779"},"pg_stats":[{"pgid":"1.f","version":"37'8479","reported_seq":8582,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:37.083107+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:40:37.083107+0000","last_peered":"2024-09-05T16:40:37.083107+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:40:37.083107+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8479,"log_dups_size":0,"ondisk_log_size":8479,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36249600,"num_objects":4425,"num_object_clones":0,"num_object_copies":17700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4425,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8479,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2189}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'8297","reported_seq":8581,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:37.131916+0000","last_change":"2024-09-05T16:40:28.919189+0000","last_active":"2024-09-05T16:40:37.131916+0000","last_peered":"2024-09-05T16:40:37.131916+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:40:37.131916+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8297,"log_dups_size":0,"ondisk_log_size":8297,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34870432,"num_objects":4257,"num_object_clones":0,"num_object_copies":17028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4257,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8295,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":107,"num_bytes_recovered":745472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2093}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'8366","reported_seq":8468,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:37.522138+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:40:37.522138+0000","last_peered":"2024-09-05T16:40:37.522138+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:40:37.522138+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8366,"log_dups_size":0,"ondisk_log_size":8366,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35928680,"num_objects":4387,"num_object_clones":0,"num_object_copies":17548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2242,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4387,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8365,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2146}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'8486","reported_seq":13213,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:40:36.441960+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:40:36.441960+0000","last_peered":"2024-09-05T16:40:36.441960+0000","last_clean":"2024-09-05T16:40:36.441960+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:40:36.441960+0000","last_undegraded":"2024-09-05T16:40:36.441960+0000","last_fullsized":"2024-09-05T16:40:36.441960+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":8486,"log_dups_size":0,"ondisk_log_size":8486,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36438016,"num_objects":4448,"num_object_clones":0,"num_object_copies":17792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4448,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2786,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'8217","reported_seq":8313,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:37.062371+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:40:37.062371+0000","last_peered":"2024-09-05T16:40:37.062371+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:40:37.062371+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8217,"log_dups_size":0,"ondisk_log_size":8217,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35861664,"num_objects":4378,"num_object_clones":0,"num_object_copies":17512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2148,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4378,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8215,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2232}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'8377","reported_seq":8482,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:37.173486+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:40:37.173486+0000","last_peered":"2024-09-05T16:40:37.173486+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:40:37.173486+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8377,"log_dups_size":0,"ondisk_log_size":8377,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36003840,"num_objects":4395,"num_object_clones":0,"num_object_copies":17580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4395,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8377,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2172}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'8202","reported_seq":8303,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:37.078142+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:40:37.078142+0000","last_peered":"2024-09-05T16:40:37.078142+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:40:37.078142+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8202,"log_dups_size":0,"ondisk_log_size":8202,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36533408,"num_objects":4460,"num_object_clones":0,"num_object_copies":17840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4460,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8200,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2326}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'8348","reported_seq":8434,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:37.002336+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:40:37.002336+0000","last_peered":"2024-09-05T16:40:37.002336+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:40:37.002336+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8348,"log_dups_size":0,"ondisk_log_size":8348,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36295840,"num_objects":4431,"num_object_clones":0,"num_object_copies":17724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4431,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8346,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2198}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'8411","reported_seq":8509,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:37.084183+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:40:37.084183+0000","last_peered":"2024-09-05T16:40:37.084183+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:40:37.084183+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8411,"log_dups_size":0,"ondisk_log_size":8411,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35735872,"num_objects":4363,"num_object_clones":0,"num_object_copies":17452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2248,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4363,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8408,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2118}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'8412","reported_seq":8523,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:37.480269+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:40:37.480269+0000","last_peered":"2024-09-05T16:40:37.480269+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:40:37.480269+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8412,"log_dups_size":0,"ondisk_log_size":8412,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36025504,"num_objects":4398,"num_object_clones":0,"num_object_copies":17592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4398,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8410,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2148}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'8635","reported_seq":8749,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:37.213180+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:40:37.213180+0000","last_peered":"2024-09-05T16:40:37.213180+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:40:37.213180+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8635,"log_dups_size":0,"ondisk_log_size":8635,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36761328,"num_objects":4488,"num_object_clones":0,"num_object_copies":17952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4488,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8632,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2202}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'8390","reported_seq":8487,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:37.223569+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:40:37.223569+0000","last_peered":"2024-09-05T16:40:37.223569+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:40:37.223569+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8390,"log_dups_size":0,"ondisk_log_size":8390,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35789392,"num_objects":4369,"num_object_clones":0,"num_object_copies":17476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4369,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8389,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2131}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'8444","reported_seq":13179,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:40:36.222099+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:40:36.222099+0000","last_peered":"2024-09-05T16:40:36.222099+0000","last_clean":"2024-09-05T16:40:36.222099+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:40:36.222099+0000","last_undegraded":"2024-09-05T16:40:36.222099+0000","last_fullsized":"2024-09-05T16:40:36.222099+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":8444,"log_dups_size":0,"ondisk_log_size":8444,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35749888,"num_objects":4364,"num_object_clones":0,"num_object_copies":17456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4364,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'8649","reported_seq":8750,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:37.193965+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:40:37.193965+0000","last_peered":"2024-09-05T16:40:37.193965+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:40:37.193965+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8649,"log_dups_size":0,"ondisk_log_size":8649,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36672672,"num_objects":4477,"num_object_clones":0,"num_object_copies":17908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4477,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8647,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2140}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'8494","reported_seq":8598,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:37.522698+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:40:37.522698+0000","last_peered":"2024-09-05T16:40:37.522698+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:40:37.522698+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8494,"log_dups_size":0,"ondisk_log_size":8494,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35542176,"num_objects":4339,"num_object_clones":0,"num_object_copies":17356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4339,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8492,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2027}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'8367","reported_seq":8466,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:37.076644+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:40:37.076644+0000","last_peered":"2024-09-05T16:40:37.076644+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:40:37.076644+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8367,"log_dups_size":0,"ondisk_log_size":8367,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36107424,"num_objects":4408,"num_object_clones":0,"num_object_copies":17632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4408,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8365,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2184}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":576565736,"num_objects":70387,"num_object_clones":0,"num_object_copies":281548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70387,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":120450,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":346,"num_bytes_recovered":745472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1138061312,"data_stored":1138061312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":134574,"ondisk_log_size":134574,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084368,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":746600,"kb_used_data":281752,"kb_used_omap":4,"kb_used_meta":464827,"kb_avail":93625240,"statfs":{"total":96636764160,"available":95872245760,"internally_reserved":0,"allocated":288514048,"data_stored":288312774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":475983571},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,4,4,5,2,1,0,0,0,0,0,2],"upper_bound":16384},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986245,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":596356,"kb_used_data":267568,"kb_used_omap":4,"kb_used_meta":328763,"kb_avail":93775484,"statfs":{"total":96636764160,"available":96026095616,"internally_reserved":0,"allocated":273989632,"data_stored":273788358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":336654034},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,4,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673051,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":750708,"kb_used_data":282200,"kb_used_omap":1,"kb_used_meta":468478,"kb_avail":93621132,"statfs":{"total":96636764160,"available":95868039168,"internally_reserved":0,"allocated":288972800,"data_stored":288771526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":479721925},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,6,0,3,2,0,0,0,0,0,0,2],"upper_bound":16384},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182482,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":751104,"kb_used_data":282156,"kb_used_omap":4,"kb_used_meta":468923,"kb_avail":93620736,"statfs":{"total":96636764160,"available":95867633664,"internally_reserved":0,"allocated":288927744,"data_stored":288726470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":480177869},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,10,0,0,5,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":288284672,"data_stored":288284672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":273678336,"data_stored":273678336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":287911936,"data_stored":287911936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":288186368,"data_stored":288186368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:40:41.727 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-05T16:40:41.727 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'8479", 'reported_seq': 8582, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:37.083107+0000', 'last_change': '2024-09-05T16:34:56.833497+0000', 'last_active': '2024-09-05T16:40:37.083107+0000', 'last_peered': '2024-09-05T16:40:37.083107+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.828289+0000', 'last_became_peered': '2024-09-05T16:34:56.828289+0000', 'last_unstale': '2024-09-05T16:40:37.083107+0000', 'last_undegraded': '2024-09-05T16:34:56.612160+0000', 'last_fullsized': '2024-09-05T16:34:56.557632+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8479, 'log_dups_size': 0, 'ondisk_log_size': 8479, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36249600, 'num_objects': 4425, 'num_object_clones': 0, 'num_object_copies': 17700, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2236, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4425, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8479, 'num_write_kb': 51616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2189}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:40:41.728 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-05T16:40:41.728 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "37'8297", 'reported_seq': 8581, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:37.131916+0000', 'last_change': '2024-09-05T16:40:28.919189+0000', 'last_active': '2024-09-05T16:40:37.131916+0000', 'last_peered': '2024-09-05T16:40:37.131916+0000', 'last_clean': '2024-09-05T16:33:53.200762+0000', 'last_became_active': '2024-09-05T16:34:56.820292+0000', 'last_became_peered': '2024-09-05T16:34:56.820292+0000', 'last_unstale': '2024-09-05T16:40:37.131916+0000', 'last_undegraded': '2024-09-05T16:34:56.605776+0000', 'last_fullsized': '2024-09-05T16:34:56.559383+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8297, 'log_dups_size': 0, 'ondisk_log_size': 8297, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34870432, 'num_objects': 4257, 'num_object_clones': 0, 'num_object_copies': 17028, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2164, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4257, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8295, 'num_write_kb': 50212, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 107, 'num_bytes_recovered': 745472, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2093}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:40:41.728 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:40:41.728 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'8366", 'reported_seq': 8468, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:37.522138+0000', 'last_change': '2024-09-05T16:34:56.867900+0000', 'last_active': '2024-09-05T16:40:37.522138+0000', 'last_peered': '2024-09-05T16:40:37.522138+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:40:37.522138+0000', 'last_undegraded': '2024-09-05T16:34:56.590972+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8366, 'log_dups_size': 0, 'ondisk_log_size': 8366, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35928680, 'num_objects': 4387, 'num_object_clones': 0, 'num_object_copies': 17548, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2242, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4387, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 8365, 'num_write_kb': 50999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2146}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:40:41.728 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:40:41.728 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'8217", 'reported_seq': 8313, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:37.062371+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:40:37.062371+0000', 'last_peered': '2024-09-05T16:40:37.062371+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:40:37.062371+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8217, 'log_dups_size': 0, 'ondisk_log_size': 8217, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35861664, 'num_objects': 4378, 'num_object_clones': 0, 'num_object_copies': 17512, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2148, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4378, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8215, 'num_write_kb': 50373, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2232}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:40:41.728 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:40:41.728 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'8377", 'reported_seq': 8482, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:37.173486+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:40:37.173486+0000', 'last_peered': '2024-09-05T16:40:37.173486+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:40:37.173486+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8377, 'log_dups_size': 0, 'ondisk_log_size': 8377, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36003840, 'num_objects': 4395, 'num_object_clones': 0, 'num_object_copies': 17580, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2223, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4395, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8377, 'num_write_kb': 51088, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2172}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:40:41.728 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:40:41.729 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'8202", 'reported_seq': 8303, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:37.078142+0000', 'last_change': '2024-09-05T16:34:56.903436+0000', 'last_active': '2024-09-05T16:40:37.078142+0000', 'last_peered': '2024-09-05T16:40:37.078142+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:40:37.078142+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8202, 'log_dups_size': 0, 'ondisk_log_size': 8202, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36533408, 'num_objects': 4460, 'num_object_clones': 0, 'num_object_copies': 17840, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2136, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4460, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8200, 'num_write_kb': 50636, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2326}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:40:41.729 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-05T16:40:41.729 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "37'8348", 'reported_seq': 8434, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:37.002336+0000', 'last_change': '2024-09-05T16:34:56.883319+0000', 'last_active': '2024-09-05T16:40:37.002336+0000', 'last_peered': '2024-09-05T16:40:37.002336+0000', 'last_clean': '2024-09-05T16:33:53.505305+0000', 'last_became_active': '2024-09-05T16:34:56.873389+0000', 'last_became_peered': '2024-09-05T16:34:56.873389+0000', 'last_unstale': '2024-09-05T16:40:37.002336+0000', 'last_undegraded': '2024-09-05T16:34:56.602567+0000', 'last_fullsized': '2024-09-05T16:34:56.559958+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8348, 'log_dups_size': 0, 'ondisk_log_size': 8348, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36295840, 'num_objects': 4431, 'num_object_clones': 0, 'num_object_copies': 17724, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2235, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4431, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8346, 'num_write_kb': 51108, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 8, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2198}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:40:41.729 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:40:41.729 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'8411", 'reported_seq': 8509, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:37.084183+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:40:37.084183+0000', 'last_peered': '2024-09-05T16:40:37.084183+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:40:37.084183+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8411, 'log_dups_size': 0, 'ondisk_log_size': 8411, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35735872, 'num_objects': 4363, 'num_object_clones': 0, 'num_object_copies': 17452, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2248, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4363, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8408, 'num_write_kb': 51081, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2118}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:40:41.729 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:40:41.729 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'8412", 'reported_seq': 8523, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:37.480269+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:40:37.480269+0000', 'last_peered': '2024-09-05T16:40:37.480269+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:40:37.480269+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8412, 'log_dups_size': 0, 'ondisk_log_size': 8412, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36025504, 'num_objects': 4398, 'num_object_clones': 0, 'num_object_copies': 17592, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2252, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4398, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8410, 'num_write_kb': 51230, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2148}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:40:41.729 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:40:41.729 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'8635", 'reported_seq': 8749, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:37.213180+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:40:37.213180+0000', 'last_peered': '2024-09-05T16:40:37.213180+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:40:37.213180+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8635, 'log_dups_size': 0, 'ondisk_log_size': 8635, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36761328, 'num_objects': 4488, 'num_object_clones': 0, 'num_object_copies': 17952, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2289, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4488, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8632, 'num_write_kb': 52476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2202}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:40:41.729 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:40:41.730 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'8390", 'reported_seq': 8487, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:37.223569+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:40:37.223569+0000', 'last_peered': '2024-09-05T16:40:37.223569+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:40:37.223569+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8390, 'log_dups_size': 0, 'ondisk_log_size': 8390, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35789392, 'num_objects': 4369, 'num_object_clones': 0, 'num_object_copies': 17476, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4369, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8389, 'num_write_kb': 51031, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2131}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:40:41.730 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:40:41.730 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'8649", 'reported_seq': 8750, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:37.193965+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:40:37.193965+0000', 'last_peered': '2024-09-05T16:40:37.193965+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:40:37.193965+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8649, 'log_dups_size': 0, 'ondisk_log_size': 8649, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36672672, 'num_objects': 4477, 'num_object_clones': 0, 'num_object_copies': 17908, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2339, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4477, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8647, 'num_write_kb': 52500, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 22, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2140}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:40:41.730 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:40:41.730 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'8494", 'reported_seq': 8598, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:37.522698+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:40:37.522698+0000', 'last_peered': '2024-09-05T16:40:37.522698+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:40:37.522698+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8494, 'log_dups_size': 0, 'ondisk_log_size': 8494, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35542176, 'num_objects': 4339, 'num_object_clones': 0, 'num_object_copies': 17356, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2314, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4339, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8492, 'num_write_kb': 51325, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2027}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:40:41.730 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:40:41.730 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'8367", 'reported_seq': 8466, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:37.076644+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:40:37.076644+0000', 'last_peered': '2024-09-05T16:40:37.076644+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:40:37.076644+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8367, 'log_dups_size': 0, 'ondisk_log_size': 8367, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36107424, 'num_objects': 4408, 'num_object_clones': 0, 'num_object_copies': 17632, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2226, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4408, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8365, 'num_write_kb': 51092, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2184}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:40:41.730 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:40:42.492 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:40:42.492 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 31291/281548 objects degraded (11.114%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":445,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":70387,"data_bytes":576565736,"bytes_used":2913042432,"bytes_avail":383634014208,"bytes_total":386547056640,"degraded_objects":31291,"degraded_total":281548,"degraded_ratio":0.11113913080540443},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:40:42.519 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-09-05T16:40:42.519 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:40:43.125 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:40:43.125 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:40:43.160 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":251,"stamp":"2024-09-05T16:40:42.946509+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":567161320,"num_objects":69239,"num_object_clones":0,"num_object_copies":276956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69239,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":121598,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":1204224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":135722,"ondisk_log_size":135722,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2830440,"kb_used_data":1099364,"kb_used_omap":14,"kb_used_meta":1730993,"kb_avail":374656920,"statfs":{"total":386547056640,"available":383648686080,"internally_reserved":0,"allocated":1125748736,"data_stored":1124943640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1772537399},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,4,10,4,11,7,2,0,0,0,0,0,4],"upper_bound":16384},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-27441744,"num_objects":-3350,"num_object_clones":0,"num_object_copies":-13400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-175,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3350,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-5050,"num_write_kb":-51231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-2121,"num_bytes_recovered":-17432576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.011638"},"pg_stats":[{"pgid":"1.f","version":"37'8551","reported_seq":8654,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:42.114379+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:40:42.114379+0000","last_peered":"2024-09-05T16:40:42.114379+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:40:42.114379+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8551,"log_dups_size":0,"ondisk_log_size":8551,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35659776,"num_objects":4353,"num_object_clones":0,"num_object_copies":17412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4353,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8551,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2117}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'8365","reported_seq":8761,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:42.119267+0000","last_change":"2024-09-05T16:40:28.919189+0000","last_active":"2024-09-05T16:40:42.119267+0000","last_peered":"2024-09-05T16:40:42.119267+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:40:42.119267+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8365,"log_dups_size":0,"ondisk_log_size":8365,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34313376,"num_objects":4189,"num_object_clones":0,"num_object_copies":16756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2108,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4189,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8363,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":163,"num_bytes_recovered":1204224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2081}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'8445","reported_seq":8549,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:42.367735+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:40:42.367735+0000","last_peered":"2024-09-05T16:40:42.367735+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:40:42.367735+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8445,"log_dups_size":0,"ondisk_log_size":8445,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35281512,"num_objects":4308,"num_object_clones":0,"num_object_copies":17232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2242,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4308,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8444,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2067}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'8553","reported_seq":13280,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:40:41.362728+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:40:41.362728+0000","last_peered":"2024-09-05T16:40:41.362728+0000","last_clean":"2024-09-05T16:40:41.362728+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:40:41.362728+0000","last_undegraded":"2024-09-05T16:40:41.362728+0000","last_fullsized":"2024-09-05T16:40:41.362728+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":8553,"log_dups_size":0,"ondisk_log_size":8553,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35889152,"num_objects":4381,"num_object_clones":0,"num_object_copies":17524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4381,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2853,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'8276","reported_seq":8372,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:41.767828+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:40:41.767828+0000","last_peered":"2024-09-05T16:40:41.767828+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:40:41.767828+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8276,"log_dups_size":0,"ondisk_log_size":8276,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35378336,"num_objects":4319,"num_object_clones":0,"num_object_copies":17276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2148,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4319,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8274,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2173}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'8450","reported_seq":8555,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:42.015931+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:40:42.015931+0000","last_peered":"2024-09-05T16:40:42.015931+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:40:42.015931+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8450,"log_dups_size":0,"ondisk_log_size":8450,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35405824,"num_objects":4322,"num_object_clones":0,"num_object_copies":17288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4322,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8450,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2099}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'8284","reported_seq":8384,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:42.125634+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:40:42.125634+0000","last_peered":"2024-09-05T16:40:42.125634+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:40:42.125634+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8284,"log_dups_size":0,"ondisk_log_size":8284,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35861664,"num_objects":4378,"num_object_clones":0,"num_object_copies":17512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4378,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8282,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2244}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'8439","reported_seq":8527,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:41.966937+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:40:41.966937+0000","last_peered":"2024-09-05T16:40:41.966937+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:40:41.966937+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8439,"log_dups_size":0,"ondisk_log_size":8439,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35550368,"num_objects":4340,"num_object_clones":0,"num_object_copies":17360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4340,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8437,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2108}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'8474","reported_seq":8572,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:42.004168+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:40:42.004168+0000","last_peered":"2024-09-05T16:40:42.004168+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:40:42.004168+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8474,"log_dups_size":0,"ondisk_log_size":8474,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35219776,"num_objects":4300,"num_object_clones":0,"num_object_copies":17200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2248,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4300,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8471,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2055}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'8485","reported_seq":8595,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:42.481083+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:40:42.481083+0000","last_peered":"2024-09-05T16:40:42.481083+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:40:42.481083+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8485,"log_dups_size":0,"ondisk_log_size":8485,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35427488,"num_objects":4325,"num_object_clones":0,"num_object_copies":17300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4325,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8483,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2075}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'8699","reported_seq":8815,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:42.114356+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:40:42.114356+0000","last_peered":"2024-09-05T16:40:42.114356+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:40:42.114356+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8699,"log_dups_size":0,"ondisk_log_size":8699,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36237040,"num_objects":4424,"num_object_clones":0,"num_object_copies":17696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4424,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8696,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2138}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'8456","reported_seq":8553,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:42.165012+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:40:42.165012+0000","last_peered":"2024-09-05T16:40:42.165012+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:40:42.165012+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8456,"log_dups_size":0,"ondisk_log_size":8456,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35248720,"num_objects":4303,"num_object_clones":0,"num_object_copies":17212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4303,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8455,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2065}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'8518","reported_seq":13253,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:40:41.158021+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:40:41.158021+0000","last_peered":"2024-09-05T16:40:41.158021+0000","last_clean":"2024-09-05T16:40:41.158021+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:40:41.158021+0000","last_undegraded":"2024-09-05T16:40:41.158021+0000","last_fullsized":"2024-09-05T16:40:41.158021+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":8518,"log_dups_size":0,"ondisk_log_size":8518,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35143680,"num_objects":4290,"num_object_clones":0,"num_object_copies":17160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4290,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":118,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'8717","reported_seq":8818,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:41.677561+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:40:41.677561+0000","last_peered":"2024-09-05T16:40:41.677561+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:40:41.677561+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8717,"log_dups_size":0,"ondisk_log_size":8717,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36115616,"num_objects":4409,"num_object_clones":0,"num_object_copies":17636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4409,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8715,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2072}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'8567","reported_seq":8671,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:42.534175+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:40:42.534175+0000","last_peered":"2024-09-05T16:40:42.534175+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:40:42.534175+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8567,"log_dups_size":0,"ondisk_log_size":8567,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34944160,"num_objects":4266,"num_object_clones":0,"num_object_copies":17064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4266,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8565,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1954}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'8443","reported_seq":8542,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:42.125340+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:40:42.125340+0000","last_peered":"2024-09-05T16:40:42.125340+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:40:42.125340+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8443,"log_dups_size":0,"ondisk_log_size":8443,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35484832,"num_objects":4332,"num_object_clones":0,"num_object_copies":17328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4332,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8441,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2108}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":567161320,"num_objects":69239,"num_object_clones":0,"num_object_copies":276956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69239,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":121598,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":1204224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1123323904,"data_stored":1123323904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":135722,"ondisk_log_size":135722,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084369,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":746248,"kb_used_data":277240,"kb_used_omap":4,"kb_used_meta":468987,"kb_avail":93625592,"statfs":{"total":96636764160,"available":95872606208,"internally_reserved":0,"allocated":283893760,"data_stored":283692486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":480243411},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,1,1,1,1,0,0,0,0,0,2],"upper_bound":16384},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986246,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":583328,"kb_used_data":267020,"kb_used_omap":4,"kb_used_meta":316283,"kb_avail":93788512,"statfs":{"total":96636764160,"available":96039436288,"internally_reserved":0,"allocated":273428480,"data_stored":273227206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":323874514},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,4,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673052,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":750200,"kb_used_data":277544,"kb_used_omap":1,"kb_used_meta":472638,"kb_avail":93621640,"statfs":{"total":96636764160,"available":95868559360,"internally_reserved":0,"allocated":284205056,"data_stored":284003782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":483981765},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,5,0,0,0,0,0,0,0,2],"upper_bound":16384},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182483,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":750664,"kb_used_data":277560,"kb_used_omap":4,"kb_used_meta":473083,"kb_avail":93621176,"statfs":{"total":96636764160,"available":95868084224,"internally_reserved":0,"allocated":284221440,"data_stored":284020166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":484437709},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,0,5,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":283545600,"data_stored":283545600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":273100800,"data_stored":273100800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":283197440,"data_stored":283197440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":283480064,"data_stored":283480064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:40:46.162 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:40:46.760 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:40:46.760 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:40:46.787 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":252,"stamp":"2024-09-05T16:40:44.947265+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":567161320,"num_objects":69239,"num_object_clones":0,"num_object_copies":276956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69239,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":121598,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":1204224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":135722,"ondisk_log_size":135722,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2830440,"kb_used_data":1099364,"kb_used_omap":14,"kb_used_meta":1730993,"kb_avail":374656920,"statfs":{"total":386547056640,"available":383648686080,"internally_reserved":0,"allocated":1125748736,"data_stored":1124943640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1772537399},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,4,10,4,11,7,2,0,0,0,0,0,4],"upper_bound":16384},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18258512,"num_objects":-2229,"num_object_clones":0,"num_object_copies":-8916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-111,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2229,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-6171,"num_write_kb":-51231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-2184,"num_bytes_recovered":-17907712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.011756"},"pg_stats":[{"pgid":"1.f","version":"37'8551","reported_seq":8654,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:42.114379+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:40:42.114379+0000","last_peered":"2024-09-05T16:40:42.114379+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:40:42.114379+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8551,"log_dups_size":0,"ondisk_log_size":8551,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35659776,"num_objects":4353,"num_object_clones":0,"num_object_copies":17412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4353,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8551,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2117}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'8365","reported_seq":8761,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:42.119267+0000","last_change":"2024-09-05T16:40:28.919189+0000","last_active":"2024-09-05T16:40:42.119267+0000","last_peered":"2024-09-05T16:40:42.119267+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:40:42.119267+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8365,"log_dups_size":0,"ondisk_log_size":8365,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34313376,"num_objects":4189,"num_object_clones":0,"num_object_copies":16756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2108,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4189,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8363,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":163,"num_bytes_recovered":1204224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2081}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'8445","reported_seq":8549,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:42.367735+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:40:42.367735+0000","last_peered":"2024-09-05T16:40:42.367735+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:40:42.367735+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8445,"log_dups_size":0,"ondisk_log_size":8445,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35281512,"num_objects":4308,"num_object_clones":0,"num_object_copies":17232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2242,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4308,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8444,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2067}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'8553","reported_seq":13280,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:40:41.362728+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:40:41.362728+0000","last_peered":"2024-09-05T16:40:41.362728+0000","last_clean":"2024-09-05T16:40:41.362728+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:40:41.362728+0000","last_undegraded":"2024-09-05T16:40:41.362728+0000","last_fullsized":"2024-09-05T16:40:41.362728+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":8553,"log_dups_size":0,"ondisk_log_size":8553,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35889152,"num_objects":4381,"num_object_clones":0,"num_object_copies":17524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4381,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2853,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'8276","reported_seq":8372,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:41.767828+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:40:41.767828+0000","last_peered":"2024-09-05T16:40:41.767828+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:40:41.767828+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8276,"log_dups_size":0,"ondisk_log_size":8276,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35378336,"num_objects":4319,"num_object_clones":0,"num_object_copies":17276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2148,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4319,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8274,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2173}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'8450","reported_seq":8555,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:42.015931+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:40:42.015931+0000","last_peered":"2024-09-05T16:40:42.015931+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:40:42.015931+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8450,"log_dups_size":0,"ondisk_log_size":8450,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35405824,"num_objects":4322,"num_object_clones":0,"num_object_copies":17288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4322,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8450,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2099}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'8284","reported_seq":8384,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:42.125634+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:40:42.125634+0000","last_peered":"2024-09-05T16:40:42.125634+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:40:42.125634+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8284,"log_dups_size":0,"ondisk_log_size":8284,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35861664,"num_objects":4378,"num_object_clones":0,"num_object_copies":17512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4378,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8282,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2244}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'8439","reported_seq":8527,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:41.966937+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:40:41.966937+0000","last_peered":"2024-09-05T16:40:41.966937+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:40:41.966937+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8439,"log_dups_size":0,"ondisk_log_size":8439,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35550368,"num_objects":4340,"num_object_clones":0,"num_object_copies":17360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4340,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8437,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2108}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'8474","reported_seq":8572,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:42.004168+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:40:42.004168+0000","last_peered":"2024-09-05T16:40:42.004168+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:40:42.004168+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8474,"log_dups_size":0,"ondisk_log_size":8474,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35219776,"num_objects":4300,"num_object_clones":0,"num_object_copies":17200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2248,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4300,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8471,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2055}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'8485","reported_seq":8595,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:42.481083+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:40:42.481083+0000","last_peered":"2024-09-05T16:40:42.481083+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:40:42.481083+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8485,"log_dups_size":0,"ondisk_log_size":8485,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35427488,"num_objects":4325,"num_object_clones":0,"num_object_copies":17300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4325,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8483,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2075}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'8699","reported_seq":8815,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:42.114356+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:40:42.114356+0000","last_peered":"2024-09-05T16:40:42.114356+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:40:42.114356+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8699,"log_dups_size":0,"ondisk_log_size":8699,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36237040,"num_objects":4424,"num_object_clones":0,"num_object_copies":17696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4424,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8696,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2138}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'8456","reported_seq":8553,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:42.165012+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:40:42.165012+0000","last_peered":"2024-09-05T16:40:42.165012+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:40:42.165012+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8456,"log_dups_size":0,"ondisk_log_size":8456,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35248720,"num_objects":4303,"num_object_clones":0,"num_object_copies":17212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4303,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8455,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2065}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'8518","reported_seq":13253,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:40:41.158021+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:40:41.158021+0000","last_peered":"2024-09-05T16:40:41.158021+0000","last_clean":"2024-09-05T16:40:41.158021+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:40:41.158021+0000","last_undegraded":"2024-09-05T16:40:41.158021+0000","last_fullsized":"2024-09-05T16:40:41.158021+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":8518,"log_dups_size":0,"ondisk_log_size":8518,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35143680,"num_objects":4290,"num_object_clones":0,"num_object_copies":17160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4290,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":118,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'8717","reported_seq":8818,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:41.677561+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:40:41.677561+0000","last_peered":"2024-09-05T16:40:41.677561+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:40:41.677561+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8717,"log_dups_size":0,"ondisk_log_size":8717,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36115616,"num_objects":4409,"num_object_clones":0,"num_object_copies":17636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4409,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8715,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2072}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'8567","reported_seq":8671,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:42.534175+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:40:42.534175+0000","last_peered":"2024-09-05T16:40:42.534175+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:40:42.534175+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8567,"log_dups_size":0,"ondisk_log_size":8567,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34944160,"num_objects":4266,"num_object_clones":0,"num_object_copies":17064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4266,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8565,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1954}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'8443","reported_seq":8542,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:42.125340+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:40:42.125340+0000","last_peered":"2024-09-05T16:40:42.125340+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:40:42.125340+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8443,"log_dups_size":0,"ondisk_log_size":8443,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35484832,"num_objects":4332,"num_object_clones":0,"num_object_copies":17328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4332,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8441,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2108}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":567161320,"num_objects":69239,"num_object_clones":0,"num_object_copies":276956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69239,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":121598,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":1204224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1123323904,"data_stored":1123323904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":135722,"ondisk_log_size":135722,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084369,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":746248,"kb_used_data":277240,"kb_used_omap":4,"kb_used_meta":468987,"kb_avail":93625592,"statfs":{"total":96636764160,"available":95872606208,"internally_reserved":0,"allocated":283893760,"data_stored":283692486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":480243411},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,1,1,1,1,0,0,0,0,0,2],"upper_bound":16384},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986246,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":583328,"kb_used_data":267020,"kb_used_omap":4,"kb_used_meta":316283,"kb_avail":93788512,"statfs":{"total":96636764160,"available":96039436288,"internally_reserved":0,"allocated":273428480,"data_stored":273227206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":323874514},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,4,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673052,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":750200,"kb_used_data":277544,"kb_used_omap":1,"kb_used_meta":472638,"kb_avail":93621640,"statfs":{"total":96636764160,"available":95868559360,"internally_reserved":0,"allocated":284205056,"data_stored":284003782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":483981765},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,5,0,0,0,0,0,0,0,2],"upper_bound":16384},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182483,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":750664,"kb_used_data":277560,"kb_used_omap":4,"kb_used_meta":473083,"kb_avail":93621176,"statfs":{"total":96636764160,"available":95868084224,"internally_reserved":0,"allocated":284221440,"data_stored":284020166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":484437709},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,0,5,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":283545600,"data_stored":283545600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":273100800,"data_stored":273100800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":283197440,"data_stored":283197440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":283480064,"data_stored":283480064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:40:46.788 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:40:47.393 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:40:47.394 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:40:47.418 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":253,"stamp":"2024-09-05T16:40:46.947899+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":566071784,"num_objects":69106,"num_object_clones":0,"num_object_copies":276424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69106,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":121731,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":1204224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":135855,"ondisk_log_size":135855,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2838296,"kb_used_data":1098904,"kb_used_omap":14,"kb_used_meta":1739313,"kb_avail":374649064,"statfs":{"total":386547056640,"available":383640641536,"internally_reserved":0,"allocated":1125277696,"data_stored":1124472600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1781057079},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,9,5,14,9,3,0,0,0,0,0,4],"upper_bound":16384},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19348048,"num_objects":-2362,"num_object_clones":0,"num_object_copies":-9448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-111,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2362,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-6038,"num_write_kb":-51231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-2184,"num_bytes_recovered":-17907712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.011491"},"pg_stats":[{"pgid":"1.f","version":"37'8551","reported_seq":8654,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:42.114379+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:40:42.114379+0000","last_peered":"2024-09-05T16:40:42.114379+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:40:42.114379+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8551,"log_dups_size":0,"ondisk_log_size":8551,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35659776,"num_objects":4353,"num_object_clones":0,"num_object_copies":17412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4353,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8551,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2117}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'8365","reported_seq":8761,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:42.119267+0000","last_change":"2024-09-05T16:40:28.919189+0000","last_active":"2024-09-05T16:40:42.119267+0000","last_peered":"2024-09-05T16:40:42.119267+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:40:42.119267+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8365,"log_dups_size":0,"ondisk_log_size":8365,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34313376,"num_objects":4189,"num_object_clones":0,"num_object_copies":16756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2108,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4189,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8363,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":163,"num_bytes_recovered":1204224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2081}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'8445","reported_seq":8549,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:42.367735+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:40:42.367735+0000","last_peered":"2024-09-05T16:40:42.367735+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:40:42.367735+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8445,"log_dups_size":0,"ondisk_log_size":8445,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35281512,"num_objects":4308,"num_object_clones":0,"num_object_copies":17232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2242,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4308,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8444,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2067}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'8623","reported_seq":13351,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:40:46.371671+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:40:46.371671+0000","last_peered":"2024-09-05T16:40:46.371671+0000","last_clean":"2024-09-05T16:40:46.371671+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:40:46.371671+0000","last_undegraded":"2024-09-05T16:40:46.371671+0000","last_fullsized":"2024-09-05T16:40:46.371671+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":8623,"log_dups_size":0,"ondisk_log_size":8623,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35315712,"num_objects":4311,"num_object_clones":0,"num_object_copies":17244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4311,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2923,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'8276","reported_seq":8372,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:41.767828+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:40:41.767828+0000","last_peered":"2024-09-05T16:40:41.767828+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:40:41.767828+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8276,"log_dups_size":0,"ondisk_log_size":8276,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35378336,"num_objects":4319,"num_object_clones":0,"num_object_copies":17276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2148,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4319,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8274,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2173}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'8450","reported_seq":8555,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:42.015931+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:40:42.015931+0000","last_peered":"2024-09-05T16:40:42.015931+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:40:42.015931+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8450,"log_dups_size":0,"ondisk_log_size":8450,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35405824,"num_objects":4322,"num_object_clones":0,"num_object_copies":17288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4322,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8450,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2099}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'8284","reported_seq":8384,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:42.125634+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:40:42.125634+0000","last_peered":"2024-09-05T16:40:42.125634+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:40:42.125634+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8284,"log_dups_size":0,"ondisk_log_size":8284,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35861664,"num_objects":4378,"num_object_clones":0,"num_object_copies":17512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4378,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8282,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2244}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'8439","reported_seq":8527,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:41.966937+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:40:41.966937+0000","last_peered":"2024-09-05T16:40:41.966937+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:40:41.966937+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8439,"log_dups_size":0,"ondisk_log_size":8439,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35550368,"num_objects":4340,"num_object_clones":0,"num_object_copies":17360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4340,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8437,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2108}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'8474","reported_seq":8572,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:42.004168+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:40:42.004168+0000","last_peered":"2024-09-05T16:40:42.004168+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:40:42.004168+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8474,"log_dups_size":0,"ondisk_log_size":8474,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35219776,"num_objects":4300,"num_object_clones":0,"num_object_copies":17200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2248,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4300,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8471,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2055}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'8485","reported_seq":8595,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:42.481083+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:40:42.481083+0000","last_peered":"2024-09-05T16:40:42.481083+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:40:42.481083+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8485,"log_dups_size":0,"ondisk_log_size":8485,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35427488,"num_objects":4325,"num_object_clones":0,"num_object_copies":17300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4325,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8483,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2075}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'8699","reported_seq":8815,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:42.114356+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:40:42.114356+0000","last_peered":"2024-09-05T16:40:42.114356+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:40:42.114356+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8699,"log_dups_size":0,"ondisk_log_size":8699,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36237040,"num_objects":4424,"num_object_clones":0,"num_object_copies":17696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4424,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8696,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2138}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'8456","reported_seq":8553,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:42.165012+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:40:42.165012+0000","last_peered":"2024-09-05T16:40:42.165012+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:40:42.165012+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8456,"log_dups_size":0,"ondisk_log_size":8456,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35248720,"num_objects":4303,"num_object_clones":0,"num_object_copies":17212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4303,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8455,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2065}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'8581","reported_seq":13316,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:40:46.435753+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:40:46.435753+0000","last_peered":"2024-09-05T16:40:46.435753+0000","last_clean":"2024-09-05T16:40:46.435753+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:40:46.435753+0000","last_undegraded":"2024-09-05T16:40:46.435753+0000","last_fullsized":"2024-09-05T16:40:46.435753+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":8581,"log_dups_size":0,"ondisk_log_size":8581,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34627584,"num_objects":4227,"num_object_clones":0,"num_object_copies":16908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4227,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":181,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'8717","reported_seq":8818,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:41.677561+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:40:41.677561+0000","last_peered":"2024-09-05T16:40:41.677561+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:40:41.677561+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8717,"log_dups_size":0,"ondisk_log_size":8717,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36115616,"num_objects":4409,"num_object_clones":0,"num_object_copies":17636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4409,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8715,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2072}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'8567","reported_seq":8671,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:42.534175+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:40:42.534175+0000","last_peered":"2024-09-05T16:40:42.534175+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:40:42.534175+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8567,"log_dups_size":0,"ondisk_log_size":8567,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34944160,"num_objects":4266,"num_object_clones":0,"num_object_copies":17064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4266,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8565,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1954}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'8443","reported_seq":8542,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:42.125340+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:40:42.125340+0000","last_peered":"2024-09-05T16:40:42.125340+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:40:42.125340+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8443,"log_dups_size":0,"ondisk_log_size":8443,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35484832,"num_objects":4332,"num_object_clones":0,"num_object_copies":17328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4332,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8441,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2108}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":566071784,"num_objects":69106,"num_object_clones":0,"num_object_copies":276424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69106,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":121731,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":1204224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1122848768,"data_stored":1122848768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":135855,"ondisk_log_size":135855,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084369,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":746248,"kb_used_data":277240,"kb_used_omap":4,"kb_used_meta":468987,"kb_avail":93625592,"statfs":{"total":96636764160,"available":95872606208,"internally_reserved":0,"allocated":283893760,"data_stored":283692486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":480243411},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,1,1,1,1,0,0,0,0,0,2],"upper_bound":16384},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986247,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":591184,"kb_used_data":266560,"kb_used_omap":4,"kb_used_meta":324603,"kb_avail":93780656,"statfs":{"total":96636764160,"available":96031391744,"internally_reserved":0,"allocated":272957440,"data_stored":272756166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":332394194},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,3,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673052,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":750200,"kb_used_data":277544,"kb_used_omap":1,"kb_used_meta":472638,"kb_avail":93621640,"statfs":{"total":96636764160,"available":95868559360,"internally_reserved":0,"allocated":284205056,"data_stored":284003782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":483981765},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,5,0,0,0,0,0,0,0,2],"upper_bound":16384},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182483,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":750664,"kb_used_data":277560,"kb_used_omap":4,"kb_used_meta":473083,"kb_avail":93621176,"statfs":{"total":96636764160,"available":95868084224,"internally_reserved":0,"allocated":284221440,"data_stored":284020166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":484437709},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,0,5,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":283545600,"data_stored":283545600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":272625664,"data_stored":272625664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":283197440,"data_stored":283197440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":283480064,"data_stored":283480064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:40:47.419 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-05T16:40:47.420 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'8551", 'reported_seq': 8654, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:42.114379+0000', 'last_change': '2024-09-05T16:34:56.833497+0000', 'last_active': '2024-09-05T16:40:42.114379+0000', 'last_peered': '2024-09-05T16:40:42.114379+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.828289+0000', 'last_became_peered': '2024-09-05T16:34:56.828289+0000', 'last_unstale': '2024-09-05T16:40:42.114379+0000', 'last_undegraded': '2024-09-05T16:34:56.612160+0000', 'last_fullsized': '2024-09-05T16:34:56.557632+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8551, 'log_dups_size': 0, 'ondisk_log_size': 8551, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35659776, 'num_objects': 4353, 'num_object_clones': 0, 'num_object_copies': 17412, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2236, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4353, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8551, 'num_write_kb': 51616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2117}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:40:47.420 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-05T16:40:47.420 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "37'8365", 'reported_seq': 8761, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:42.119267+0000', 'last_change': '2024-09-05T16:40:28.919189+0000', 'last_active': '2024-09-05T16:40:42.119267+0000', 'last_peered': '2024-09-05T16:40:42.119267+0000', 'last_clean': '2024-09-05T16:33:53.200762+0000', 'last_became_active': '2024-09-05T16:34:56.820292+0000', 'last_became_peered': '2024-09-05T16:34:56.820292+0000', 'last_unstale': '2024-09-05T16:40:42.119267+0000', 'last_undegraded': '2024-09-05T16:34:56.605776+0000', 'last_fullsized': '2024-09-05T16:34:56.559383+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8365, 'log_dups_size': 0, 'ondisk_log_size': 8365, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34313376, 'num_objects': 4189, 'num_object_clones': 0, 'num_object_copies': 16756, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2108, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4189, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8363, 'num_write_kb': 50212, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 163, 'num_bytes_recovered': 1204224, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2081}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:40:47.420 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:40:47.420 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'8445", 'reported_seq': 8549, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:42.367735+0000', 'last_change': '2024-09-05T16:34:56.867900+0000', 'last_active': '2024-09-05T16:40:42.367735+0000', 'last_peered': '2024-09-05T16:40:42.367735+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:40:42.367735+0000', 'last_undegraded': '2024-09-05T16:34:56.590972+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8445, 'log_dups_size': 0, 'ondisk_log_size': 8445, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35281512, 'num_objects': 4308, 'num_object_clones': 0, 'num_object_copies': 17232, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2242, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4308, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 8444, 'num_write_kb': 50999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2067}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:40:47.420 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:40:47.420 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'8276", 'reported_seq': 8372, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:41.767828+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:40:41.767828+0000', 'last_peered': '2024-09-05T16:40:41.767828+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:40:41.767828+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8276, 'log_dups_size': 0, 'ondisk_log_size': 8276, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35378336, 'num_objects': 4319, 'num_object_clones': 0, 'num_object_copies': 17276, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2148, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4319, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8274, 'num_write_kb': 50373, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2173}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:40:47.420 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:40:47.420 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'8450", 'reported_seq': 8555, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:42.015931+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:40:42.015931+0000', 'last_peered': '2024-09-05T16:40:42.015931+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:40:42.015931+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8450, 'log_dups_size': 0, 'ondisk_log_size': 8450, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35405824, 'num_objects': 4322, 'num_object_clones': 0, 'num_object_copies': 17288, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2223, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4322, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8450, 'num_write_kb': 51088, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2099}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:40:47.421 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:40:47.421 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'8284", 'reported_seq': 8384, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:42.125634+0000', 'last_change': '2024-09-05T16:34:56.903436+0000', 'last_active': '2024-09-05T16:40:42.125634+0000', 'last_peered': '2024-09-05T16:40:42.125634+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:40:42.125634+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8284, 'log_dups_size': 0, 'ondisk_log_size': 8284, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35861664, 'num_objects': 4378, 'num_object_clones': 0, 'num_object_copies': 17512, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2136, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4378, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8282, 'num_write_kb': 50636, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2244}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:40:47.421 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-05T16:40:47.421 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "37'8439", 'reported_seq': 8527, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:41.966937+0000', 'last_change': '2024-09-05T16:34:56.883319+0000', 'last_active': '2024-09-05T16:40:41.966937+0000', 'last_peered': '2024-09-05T16:40:41.966937+0000', 'last_clean': '2024-09-05T16:33:53.505305+0000', 'last_became_active': '2024-09-05T16:34:56.873389+0000', 'last_became_peered': '2024-09-05T16:34:56.873389+0000', 'last_unstale': '2024-09-05T16:40:41.966937+0000', 'last_undegraded': '2024-09-05T16:34:56.602567+0000', 'last_fullsized': '2024-09-05T16:34:56.559958+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8439, 'log_dups_size': 0, 'ondisk_log_size': 8439, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35550368, 'num_objects': 4340, 'num_object_clones': 0, 'num_object_copies': 17360, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2234, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4340, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8437, 'num_write_kb': 51108, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2108}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:40:47.421 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:40:47.421 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'8474", 'reported_seq': 8572, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:42.004168+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:40:42.004168+0000', 'last_peered': '2024-09-05T16:40:42.004168+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:40:42.004168+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8474, 'log_dups_size': 0, 'ondisk_log_size': 8474, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35219776, 'num_objects': 4300, 'num_object_clones': 0, 'num_object_copies': 17200, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2248, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4300, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8471, 'num_write_kb': 51081, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2055}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:40:47.421 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:40:47.421 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'8485", 'reported_seq': 8595, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:42.481083+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:40:42.481083+0000', 'last_peered': '2024-09-05T16:40:42.481083+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:40:42.481083+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8485, 'log_dups_size': 0, 'ondisk_log_size': 8485, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35427488, 'num_objects': 4325, 'num_object_clones': 0, 'num_object_copies': 17300, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2252, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4325, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8483, 'num_write_kb': 51230, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2075}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:40:47.421 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:40:47.421 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'8699", 'reported_seq': 8815, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:42.114356+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:40:42.114356+0000', 'last_peered': '2024-09-05T16:40:42.114356+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:40:42.114356+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8699, 'log_dups_size': 0, 'ondisk_log_size': 8699, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36237040, 'num_objects': 4424, 'num_object_clones': 0, 'num_object_copies': 17696, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2289, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4424, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8696, 'num_write_kb': 52476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2138}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:40:47.422 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:40:47.422 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'8456", 'reported_seq': 8553, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:42.165012+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:40:42.165012+0000', 'last_peered': '2024-09-05T16:40:42.165012+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:40:42.165012+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8456, 'log_dups_size': 0, 'ondisk_log_size': 8456, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35248720, 'num_objects': 4303, 'num_object_clones': 0, 'num_object_copies': 17212, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4303, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8455, 'num_write_kb': 51031, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2065}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:40:47.422 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:40:47.422 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'8717", 'reported_seq': 8818, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:41.677561+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:40:41.677561+0000', 'last_peered': '2024-09-05T16:40:41.677561+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:40:41.677561+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8717, 'log_dups_size': 0, 'ondisk_log_size': 8717, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36115616, 'num_objects': 4409, 'num_object_clones': 0, 'num_object_copies': 17636, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2339, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4409, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8715, 'num_write_kb': 52500, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 22, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2072}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:40:47.422 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:40:47.422 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'8567", 'reported_seq': 8671, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:42.534175+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:40:42.534175+0000', 'last_peered': '2024-09-05T16:40:42.534175+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:40:42.534175+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8567, 'log_dups_size': 0, 'ondisk_log_size': 8567, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34944160, 'num_objects': 4266, 'num_object_clones': 0, 'num_object_copies': 17064, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2314, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4266, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8565, 'num_write_kb': 51325, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1954}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:40:47.422 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:40:47.422 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'8443", 'reported_seq': 8542, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:42.125340+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:40:42.125340+0000', 'last_peered': '2024-09-05T16:40:42.125340+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:40:42.125340+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8443, 'log_dups_size': 0, 'ondisk_log_size': 8443, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35484832, 'num_objects': 4332, 'num_object_clones': 0, 'num_object_copies': 17328, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2226, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4332, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8441, 'num_write_kb': 51092, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2108}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:40:47.422 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:40:48.152 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:40:48.152 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 31234/276424 objects degraded (11.299%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":451,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":69106,"data_bytes":566071784,"bytes_used":2906415104,"bytes_avail":383640641536,"bytes_total":386547056640,"degraded_objects":31234,"degraded_total":276424,"degraded_ratio":0.11299308308974618},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:40:48.173 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-09-05T16:40:48.173 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:40:48.816 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:40:48.817 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:40:48.848 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":253,"stamp":"2024-09-05T16:40:46.947899+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":566071784,"num_objects":69106,"num_object_clones":0,"num_object_copies":276424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69106,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":121731,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":1204224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":135855,"ondisk_log_size":135855,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2838296,"kb_used_data":1098904,"kb_used_omap":14,"kb_used_meta":1739313,"kb_avail":374649064,"statfs":{"total":386547056640,"available":383640641536,"internally_reserved":0,"allocated":1125277696,"data_stored":1124472600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1781057079},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,9,5,14,9,3,0,0,0,0,0,4],"upper_bound":16384},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19348048,"num_objects":-2362,"num_object_clones":0,"num_object_copies":-9448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-111,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2362,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-6038,"num_write_kb":-51231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-2184,"num_bytes_recovered":-17907712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.011491"},"pg_stats":[{"pgid":"1.f","version":"37'8551","reported_seq":8654,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:42.114379+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:40:42.114379+0000","last_peered":"2024-09-05T16:40:42.114379+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:40:42.114379+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8551,"log_dups_size":0,"ondisk_log_size":8551,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35659776,"num_objects":4353,"num_object_clones":0,"num_object_copies":17412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4353,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8551,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2117}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'8365","reported_seq":8761,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:42.119267+0000","last_change":"2024-09-05T16:40:28.919189+0000","last_active":"2024-09-05T16:40:42.119267+0000","last_peered":"2024-09-05T16:40:42.119267+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:40:42.119267+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8365,"log_dups_size":0,"ondisk_log_size":8365,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34313376,"num_objects":4189,"num_object_clones":0,"num_object_copies":16756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2108,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4189,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8363,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":163,"num_bytes_recovered":1204224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2081}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'8445","reported_seq":8549,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:42.367735+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:40:42.367735+0000","last_peered":"2024-09-05T16:40:42.367735+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:40:42.367735+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8445,"log_dups_size":0,"ondisk_log_size":8445,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35281512,"num_objects":4308,"num_object_clones":0,"num_object_copies":17232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2242,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4308,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8444,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2067}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'8623","reported_seq":13351,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:40:46.371671+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:40:46.371671+0000","last_peered":"2024-09-05T16:40:46.371671+0000","last_clean":"2024-09-05T16:40:46.371671+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:40:46.371671+0000","last_undegraded":"2024-09-05T16:40:46.371671+0000","last_fullsized":"2024-09-05T16:40:46.371671+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":8623,"log_dups_size":0,"ondisk_log_size":8623,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35315712,"num_objects":4311,"num_object_clones":0,"num_object_copies":17244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4311,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2923,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'8276","reported_seq":8372,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:41.767828+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:40:41.767828+0000","last_peered":"2024-09-05T16:40:41.767828+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:40:41.767828+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8276,"log_dups_size":0,"ondisk_log_size":8276,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35378336,"num_objects":4319,"num_object_clones":0,"num_object_copies":17276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2148,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4319,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8274,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2173}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'8450","reported_seq":8555,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:42.015931+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:40:42.015931+0000","last_peered":"2024-09-05T16:40:42.015931+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:40:42.015931+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8450,"log_dups_size":0,"ondisk_log_size":8450,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35405824,"num_objects":4322,"num_object_clones":0,"num_object_copies":17288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4322,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8450,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2099}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'8284","reported_seq":8384,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:42.125634+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:40:42.125634+0000","last_peered":"2024-09-05T16:40:42.125634+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:40:42.125634+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8284,"log_dups_size":0,"ondisk_log_size":8284,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35861664,"num_objects":4378,"num_object_clones":0,"num_object_copies":17512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4378,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8282,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2244}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'8439","reported_seq":8527,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:41.966937+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:40:41.966937+0000","last_peered":"2024-09-05T16:40:41.966937+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:40:41.966937+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8439,"log_dups_size":0,"ondisk_log_size":8439,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35550368,"num_objects":4340,"num_object_clones":0,"num_object_copies":17360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4340,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8437,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2108}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'8474","reported_seq":8572,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:42.004168+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:40:42.004168+0000","last_peered":"2024-09-05T16:40:42.004168+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:40:42.004168+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8474,"log_dups_size":0,"ondisk_log_size":8474,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35219776,"num_objects":4300,"num_object_clones":0,"num_object_copies":17200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2248,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4300,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8471,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2055}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'8485","reported_seq":8595,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:42.481083+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:40:42.481083+0000","last_peered":"2024-09-05T16:40:42.481083+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:40:42.481083+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8485,"log_dups_size":0,"ondisk_log_size":8485,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35427488,"num_objects":4325,"num_object_clones":0,"num_object_copies":17300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4325,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8483,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2075}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'8699","reported_seq":8815,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:42.114356+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:40:42.114356+0000","last_peered":"2024-09-05T16:40:42.114356+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:40:42.114356+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8699,"log_dups_size":0,"ondisk_log_size":8699,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36237040,"num_objects":4424,"num_object_clones":0,"num_object_copies":17696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4424,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8696,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2138}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'8456","reported_seq":8553,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:42.165012+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:40:42.165012+0000","last_peered":"2024-09-05T16:40:42.165012+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:40:42.165012+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8456,"log_dups_size":0,"ondisk_log_size":8456,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35248720,"num_objects":4303,"num_object_clones":0,"num_object_copies":17212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4303,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8455,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2065}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'8581","reported_seq":13316,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:40:46.435753+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:40:46.435753+0000","last_peered":"2024-09-05T16:40:46.435753+0000","last_clean":"2024-09-05T16:40:46.435753+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:40:46.435753+0000","last_undegraded":"2024-09-05T16:40:46.435753+0000","last_fullsized":"2024-09-05T16:40:46.435753+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":8581,"log_dups_size":0,"ondisk_log_size":8581,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34627584,"num_objects":4227,"num_object_clones":0,"num_object_copies":16908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4227,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":181,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'8717","reported_seq":8818,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:41.677561+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:40:41.677561+0000","last_peered":"2024-09-05T16:40:41.677561+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:40:41.677561+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8717,"log_dups_size":0,"ondisk_log_size":8717,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36115616,"num_objects":4409,"num_object_clones":0,"num_object_copies":17636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4409,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8715,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2072}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'8567","reported_seq":8671,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:42.534175+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:40:42.534175+0000","last_peered":"2024-09-05T16:40:42.534175+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:40:42.534175+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8567,"log_dups_size":0,"ondisk_log_size":8567,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34944160,"num_objects":4266,"num_object_clones":0,"num_object_copies":17064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4266,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8565,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1954}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'8443","reported_seq":8542,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:42.125340+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:40:42.125340+0000","last_peered":"2024-09-05T16:40:42.125340+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:40:42.125340+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8443,"log_dups_size":0,"ondisk_log_size":8443,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35484832,"num_objects":4332,"num_object_clones":0,"num_object_copies":17328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4332,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8441,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2108}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":566071784,"num_objects":69106,"num_object_clones":0,"num_object_copies":276424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69106,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":121731,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":1204224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1122848768,"data_stored":1122848768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":135855,"ondisk_log_size":135855,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084369,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":746248,"kb_used_data":277240,"kb_used_omap":4,"kb_used_meta":468987,"kb_avail":93625592,"statfs":{"total":96636764160,"available":95872606208,"internally_reserved":0,"allocated":283893760,"data_stored":283692486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":480243411},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,1,1,1,1,0,0,0,0,0,2],"upper_bound":16384},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986247,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":591184,"kb_used_data":266560,"kb_used_omap":4,"kb_used_meta":324603,"kb_avail":93780656,"statfs":{"total":96636764160,"available":96031391744,"internally_reserved":0,"allocated":272957440,"data_stored":272756166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":332394194},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,3,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673052,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":750200,"kb_used_data":277544,"kb_used_omap":1,"kb_used_meta":472638,"kb_avail":93621640,"statfs":{"total":96636764160,"available":95868559360,"internally_reserved":0,"allocated":284205056,"data_stored":284003782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":483981765},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,5,0,0,0,0,0,0,0,2],"upper_bound":16384},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182483,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":750664,"kb_used_data":277560,"kb_used_omap":4,"kb_used_meta":473083,"kb_avail":93621176,"statfs":{"total":96636764160,"available":95868084224,"internally_reserved":0,"allocated":284221440,"data_stored":284020166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":484437709},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,0,5,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":283545600,"data_stored":283545600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":272625664,"data_stored":272625664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":283197440,"data_stored":283197440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":283480064,"data_stored":283480064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:40:51.850 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:40:52.433 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:40:52.434 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:40:52.465 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":255,"stamp":"2024-09-05T16:40:50.949380+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":558305768,"num_objects":68158,"num_object_clones":0,"num_object_copies":272632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31151,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68158,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":122679,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":486,"num_bytes_recovered":1802240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":136803,"ondisk_log_size":136803,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2821120,"kb_used_data":1085864,"kb_used_omap":14,"kb_used_meta":1735153,"kb_avail":374666240,"statfs":{"total":386547056640,"available":383658229760,"internally_reserved":0,"allocated":1111924736,"data_stored":1111119640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1776797239},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,3,13,10,5,1,0,0,0,7,0,3,2],"upper_bound":32768},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18259968,"num_objects":-2229,"num_object_clones":0,"num_object_copies":-8916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-140,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2229,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2229,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":140,"num_bytes_recovered":1056768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004434"},"pg_stats":[{"pgid":"1.f","version":"37'8604","reported_seq":8707,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:47.216170+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:40:47.216170+0000","last_peered":"2024-09-05T16:40:47.216170+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:40:47.216170+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8604,"log_dups_size":0,"ondisk_log_size":8604,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35225600,"num_objects":4300,"num_object_clones":0,"num_object_copies":17200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4300,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8604,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2064}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'8420","reported_seq":8961,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:47.206535+0000","last_change":"2024-09-05T16:40:28.919189+0000","last_active":"2024-09-05T16:40:47.206535+0000","last_peered":"2024-09-05T16:40:47.206535+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:40:47.206535+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8420,"log_dups_size":0,"ondisk_log_size":8420,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33862816,"num_objects":4134,"num_object_clones":0,"num_object_copies":16536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2035,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4134,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8418,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":236,"num_bytes_recovered":1802240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2099}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'8524","reported_seq":8632,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:47.498267+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:40:47.498267+0000","last_peered":"2024-09-05T16:40:47.498267+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:40:47.498267+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8524,"log_dups_size":0,"ondisk_log_size":8524,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34634344,"num_objects":4229,"num_object_clones":0,"num_object_copies":16916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4229,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8523,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1990}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'8623","reported_seq":13351,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:40:46.371671+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:40:46.371671+0000","last_peered":"2024-09-05T16:40:46.371671+0000","last_clean":"2024-09-05T16:40:46.371671+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:40:46.371671+0000","last_undegraded":"2024-09-05T16:40:46.371671+0000","last_fullsized":"2024-09-05T16:40:46.371671+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":8623,"log_dups_size":0,"ondisk_log_size":8623,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35315712,"num_objects":4311,"num_object_clones":0,"num_object_copies":17244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4311,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2923,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'8338","reported_seq":8436,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:47.174161+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:40:47.174161+0000","last_peered":"2024-09-05T16:40:47.174161+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:40:47.174161+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8338,"log_dups_size":0,"ondisk_log_size":8338,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34870432,"num_objects":4257,"num_object_clones":0,"num_object_copies":17028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4257,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8336,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2112}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'8516","reported_seq":8621,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:47.015030+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:40:47.015030+0000","last_peered":"2024-09-05T16:40:47.015030+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:40:47.015030+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8516,"log_dups_size":0,"ondisk_log_size":8516,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34865152,"num_objects":4256,"num_object_clones":0,"num_object_copies":17024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4256,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8516,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2033}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'8362","reported_seq":8467,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:47.123575+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:40:47.123575+0000","last_peered":"2024-09-05T16:40:47.123575+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:40:47.123575+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8362,"log_dups_size":0,"ondisk_log_size":8362,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35222688,"num_objects":4300,"num_object_clones":0,"num_object_copies":17200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4300,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8360,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2168}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'8500","reported_seq":8588,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:47.075663+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:40:47.075663+0000","last_peered":"2024-09-05T16:40:47.075663+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:40:47.075663+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8500,"log_dups_size":0,"ondisk_log_size":8500,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35050656,"num_objects":4279,"num_object_clones":0,"num_object_copies":17116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4279,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8498,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2047}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'8546","reported_seq":8644,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:47.132315+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:40:47.132315+0000","last_peered":"2024-09-05T16:40:47.132315+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:40:47.132315+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8546,"log_dups_size":0,"ondisk_log_size":8546,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34629952,"num_objects":4228,"num_object_clones":0,"num_object_copies":16912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2248,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4228,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8543,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1983}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'8550","reported_seq":8663,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:47.305388+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:40:47.305388+0000","last_peered":"2024-09-05T16:40:47.305388+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:40:47.305388+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8550,"log_dups_size":0,"ondisk_log_size":8550,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34895008,"num_objects":4260,"num_object_clones":0,"num_object_copies":17040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4260,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8548,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2011}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'8765","reported_seq":8881,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:47.104898+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:40:47.104898+0000","last_peered":"2024-09-05T16:40:47.104898+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:40:47.104898+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8765,"log_dups_size":0,"ondisk_log_size":8765,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35696368,"num_objects":4358,"num_object_clones":0,"num_object_copies":17432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4358,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8762,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2072}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'8527","reported_seq":8624,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:47.187687+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:40:47.187687+0000","last_peered":"2024-09-05T16:40:47.187687+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:40:47.187687+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8527,"log_dups_size":0,"ondisk_log_size":8527,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34667088,"num_objects":4232,"num_object_clones":0,"num_object_copies":16928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4232,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8526,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1994}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'8581","reported_seq":13316,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:40:46.435753+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:40:46.435753+0000","last_peered":"2024-09-05T16:40:46.435753+0000","last_clean":"2024-09-05T16:40:46.435753+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:40:46.435753+0000","last_undegraded":"2024-09-05T16:40:46.435753+0000","last_fullsized":"2024-09-05T16:40:46.435753+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":8581,"log_dups_size":0,"ondisk_log_size":8581,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34627584,"num_objects":4227,"num_object_clones":0,"num_object_copies":16908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4227,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":181,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'8791","reported_seq":8894,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:46.971095+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:40:46.971095+0000","last_peered":"2024-09-05T16:40:46.971095+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:40:46.971095+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8791,"log_dups_size":0,"ondisk_log_size":8791,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35509408,"num_objects":4335,"num_object_clones":0,"num_object_copies":17340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2338,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4335,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8789,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1999}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'8638","reported_seq":8744,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:47.458990+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:40:47.458990+0000","last_peered":"2024-09-05T16:40:47.458990+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:40:47.458990+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8638,"log_dups_size":0,"ondisk_log_size":8638,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34362528,"num_objects":4195,"num_object_clones":0,"num_object_copies":16780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4195,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8636,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1884}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'8518","reported_seq":8621,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:47.113849+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:40:47.113849+0000","last_peered":"2024-09-05T16:40:47.113849+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:40:47.113849+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8518,"log_dups_size":0,"ondisk_log_size":8518,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34870432,"num_objects":4257,"num_object_clones":0,"num_object_copies":17028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2224,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4257,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8516,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2035}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":558305768,"num_objects":68158,"num_object_clones":0,"num_object_copies":272632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31151,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68158,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":122679,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":486,"num_bytes_recovered":1802240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1109565440,"data_stored":1109565440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":136803,"ondisk_log_size":136803,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084370,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":746084,"kb_used_data":272912,"kb_used_omap":4,"kb_used_meta":473147,"kb_avail":93625756,"statfs":{"total":96636764160,"available":95872774144,"internally_reserved":0,"allocated":279461888,"data_stored":279260614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":484503251},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,8,2,3,0,0,0,0,7],"upper_bound":4096},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986247,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":591184,"kb_used_data":266560,"kb_used_omap":4,"kb_used_meta":324603,"kb_avail":93780656,"statfs":{"total":96636764160,"available":96031391744,"internally_reserved":0,"allocated":272957440,"data_stored":272756166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":332394194},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,3,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673053,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":749980,"kb_used_data":273164,"kb_used_omap":1,"kb_used_meta":476798,"kb_avail":93621860,"statfs":{"total":96636764160,"available":95868784640,"internally_reserved":0,"allocated":279719936,"data_stored":279518662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":488241605},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,5,0,0,0,0,0,0,0,0,2],"upper_bound":32768},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182484,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":733872,"kb_used_data":273228,"kb_used_omap":4,"kb_used_meta":460603,"kb_avail":93637968,"statfs":{"total":96636764160,"available":95885279232,"internally_reserved":0,"allocated":279785472,"data_stored":279584198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":471658189},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,1,0,0,0,0,0,0,0,0,3],"upper_bound":16384},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":279117824,"data_stored":279117824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":272625664,"data_stored":272625664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":278765568,"data_stored":278765568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":279056384,"data_stored":279056384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:40:52.467 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:40:53.096 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:40:53.097 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:40:53.117 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":256,"stamp":"2024-09-05T16:40:52.950148+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":549220840,"num_objects":67049,"num_object_clones":0,"num_object_copies":268196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31069,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67049,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":123788,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":568,"num_bytes_recovered":2293760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137912,"ondisk_log_size":137912,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2807168,"kb_used_data":1071904,"kb_used_omap":14,"kb_used_meta":1735153,"kb_avail":374680192,"statfs":{"total":386547056640,"available":383672516608,"internally_reserved":0,"allocated":1097629696,"data_stored":1096824600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1776797239},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,1,5,15,6,0,0,0,0,5,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-27344896,"num_objects":-3338,"num_object_clones":0,"num_object_copies":-13352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3338,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3338,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":222,"num_bytes_recovered":1548288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004408"},"pg_stats":[{"pgid":"1.f","version":"37'8670","reported_seq":8773,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:51.921696+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:40:51.921696+0000","last_peered":"2024-09-05T16:40:51.921696+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:40:51.921696+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8670,"log_dups_size":0,"ondisk_log_size":8670,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34684928,"num_objects":4234,"num_object_clones":0,"num_object_copies":16936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4234,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8670,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1998}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'8490","reported_seq":9152,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:52.204450+0000","last_change":"2024-09-05T16:40:28.919189+0000","last_active":"2024-09-05T16:40:52.204450+0000","last_peered":"2024-09-05T16:40:52.204450+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:40:52.204450+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8490,"log_dups_size":0,"ondisk_log_size":8490,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33289376,"num_objects":4064,"num_object_clones":0,"num_object_copies":16256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1975,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4064,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8488,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":296,"num_bytes_recovered":2293760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2089}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'8587","reported_seq":8709,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:52.557168+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:40:52.557168+0000","last_peered":"2024-09-05T16:40:52.557168+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:40:52.557168+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8587,"log_dups_size":0,"ondisk_log_size":8587,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34118248,"num_objects":4166,"num_object_clones":0,"num_object_copies":16664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4166,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8586,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1934}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'8686","reported_seq":13414,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:40:51.359636+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:40:51.359636+0000","last_peered":"2024-09-05T16:40:51.359636+0000","last_clean":"2024-09-05T16:40:51.359636+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:40:51.359636+0000","last_undegraded":"2024-09-05T16:40:51.359636+0000","last_fullsized":"2024-09-05T16:40:51.359636+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":8686,"log_dups_size":0,"ondisk_log_size":8686,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34799616,"num_objects":4248,"num_object_clones":0,"num_object_copies":16992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4248,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2986,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'8405","reported_seq":8503,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:52.186911+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:40:52.186911+0000","last_peered":"2024-09-05T16:40:52.186911+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:40:52.186911+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8405,"log_dups_size":0,"ondisk_log_size":8405,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34321568,"num_objects":4190,"num_object_clones":0,"num_object_copies":16760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4190,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8403,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2045}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'8591","reported_seq":8702,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:52.245086+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:40:52.245086+0000","last_peered":"2024-09-05T16:40:52.245086+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:40:52.245086+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8591,"log_dups_size":0,"ondisk_log_size":8591,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34250752,"num_objects":4181,"num_object_clones":0,"num_object_copies":16724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4181,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8591,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1961}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'8430","reported_seq":8535,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:52.013478+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:40:52.013478+0000","last_peered":"2024-09-05T16:40:52.013478+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:40:52.013478+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8430,"log_dups_size":0,"ondisk_log_size":8430,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34665632,"num_objects":4232,"num_object_clones":0,"num_object_copies":16928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4232,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8428,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2100}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'8579","reported_seq":8667,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:52.209785+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:40:52.209785+0000","last_peered":"2024-09-05T16:40:52.209785+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:40:52.209785+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8579,"log_dups_size":0,"ondisk_log_size":8579,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34403488,"num_objects":4200,"num_object_clones":0,"num_object_copies":16800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4200,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8577,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1968}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'8610","reported_seq":8710,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:52.257954+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:40:52.257954+0000","last_peered":"2024-09-05T16:40:52.257954+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:40:52.257954+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8610,"log_dups_size":0,"ondisk_log_size":8610,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34105664,"num_objects":4164,"num_object_clones":0,"num_object_copies":16656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4164,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8607,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1920}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'8625","reported_seq":8740,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:52.608268+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:40:52.608268+0000","last_peered":"2024-09-05T16:40:52.608268+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:40:52.608268+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8625,"log_dups_size":0,"ondisk_log_size":8625,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34280608,"num_objects":4185,"num_object_clones":0,"num_object_copies":16740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4185,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8623,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1937}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'8837","reported_seq":8961,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:51.829443+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:40:51.829443+0000","last_peered":"2024-09-05T16:40:51.829443+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:40:51.829443+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8837,"log_dups_size":0,"ondisk_log_size":8837,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35106544,"num_objects":4286,"num_object_clones":0,"num_object_copies":17144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4286,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8834,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2004}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'8601","reported_seq":8699,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:52.184892+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:40:52.184892+0000","last_peered":"2024-09-05T16:40:52.184892+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:40:52.184892+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8601,"log_dups_size":0,"ondisk_log_size":8601,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34060880,"num_objects":4158,"num_object_clones":0,"num_object_copies":16632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4158,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8600,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1920}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'8635","reported_seq":13370,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:40:51.189779+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:40:51.189779+0000","last_peered":"2024-09-05T16:40:51.189779+0000","last_clean":"2024-09-05T16:40:51.189779+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:40:51.189779+0000","last_undegraded":"2024-09-05T16:40:51.189779+0000","last_fullsized":"2024-09-05T16:40:51.189779+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":8635,"log_dups_size":0,"ondisk_log_size":8635,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34185216,"num_objects":4173,"num_object_clones":0,"num_object_copies":16692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4173,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":235,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'8882","reported_seq":8989,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:52.163649+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:40:52.163649+0000","last_peered":"2024-09-05T16:40:52.163649+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:40:52.163649+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8882,"log_dups_size":0,"ondisk_log_size":8882,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34763936,"num_objects":4244,"num_object_clones":0,"num_object_copies":16976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4244,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8880,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1910}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'8700","reported_seq":8811,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:52.555587+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:40:52.555587+0000","last_peered":"2024-09-05T16:40:52.555587+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:40:52.555587+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8700,"log_dups_size":0,"ondisk_log_size":8700,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33854624,"num_objects":4133,"num_object_clones":0,"num_object_copies":16532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4133,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8698,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1825}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'8584","reported_seq":8689,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:51.912495+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:40:51.912495+0000","last_peered":"2024-09-05T16:40:51.912495+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:40:51.912495+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8584,"log_dups_size":0,"ondisk_log_size":8584,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34329760,"num_objects":4191,"num_object_clones":0,"num_object_copies":16764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4191,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8582,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1970}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":549220840,"num_objects":67049,"num_object_clones":0,"num_object_copies":268196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31069,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67049,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":123788,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":568,"num_bytes_recovered":2293760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1095327744,"data_stored":1095327744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137912,"ondisk_log_size":137912,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084371,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":745864,"kb_used_data":268516,"kb_used_omap":4,"kb_used_meta":477307,"kb_avail":93625976,"statfs":{"total":96636764160,"available":95872999424,"internally_reserved":0,"allocated":274960384,"data_stored":274759110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":488763091},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,3,1,0,0,0,0,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986248,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":594832,"kb_used_data":266048,"kb_used_omap":4,"kb_used_meta":328763,"kb_avail":93777008,"statfs":{"total":96636764160,"available":96027656192,"internally_reserved":0,"allocated":272433152,"data_stored":272231878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":336654034},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673054,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":732944,"kb_used_data":268596,"kb_used_omap":1,"kb_used_meta":464318,"kb_avail":93638896,"statfs":{"total":96636764160,"available":95886229504,"internally_reserved":0,"allocated":275042304,"data_stored":274841030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":475462085},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,0,0,8,2,0,0,0,0,3],"upper_bound":4096},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182485,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":733528,"kb_used_data":268744,"kb_used_omap":4,"kb_used_meta":464763,"kb_avail":93638312,"statfs":{"total":96636764160,"available":95885631488,"internally_reserved":0,"allocated":275193856,"data_stored":274992582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":475918029},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,4,3,0,0,0,0,2],"upper_bound":4096},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":274526208,"data_stored":274526208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":272113664,"data_stored":272113664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":274206720,"data_stored":274206720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":274481152,"data_stored":274481152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:40:53.118 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-05T16:40:53.118 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'8670", 'reported_seq': 8773, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:51.921696+0000', 'last_change': '2024-09-05T16:34:56.833497+0000', 'last_active': '2024-09-05T16:40:51.921696+0000', 'last_peered': '2024-09-05T16:40:51.921696+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.828289+0000', 'last_became_peered': '2024-09-05T16:34:56.828289+0000', 'last_unstale': '2024-09-05T16:40:51.921696+0000', 'last_undegraded': '2024-09-05T16:34:56.612160+0000', 'last_fullsized': '2024-09-05T16:34:56.557632+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8670, 'log_dups_size': 0, 'ondisk_log_size': 8670, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34684928, 'num_objects': 4234, 'num_object_clones': 0, 'num_object_copies': 16936, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2236, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4234, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8670, 'num_write_kb': 51616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1998}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:40:53.119 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-05T16:40:53.119 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "37'8490", 'reported_seq': 9152, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:52.204450+0000', 'last_change': '2024-09-05T16:40:28.919189+0000', 'last_active': '2024-09-05T16:40:52.204450+0000', 'last_peered': '2024-09-05T16:40:52.204450+0000', 'last_clean': '2024-09-05T16:33:53.200762+0000', 'last_became_active': '2024-09-05T16:34:56.820292+0000', 'last_became_peered': '2024-09-05T16:34:56.820292+0000', 'last_unstale': '2024-09-05T16:40:52.204450+0000', 'last_undegraded': '2024-09-05T16:34:56.605776+0000', 'last_fullsized': '2024-09-05T16:34:56.559383+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8490, 'log_dups_size': 0, 'ondisk_log_size': 8490, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33289376, 'num_objects': 4064, 'num_object_clones': 0, 'num_object_copies': 16256, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1975, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4064, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8488, 'num_write_kb': 50212, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 296, 'num_bytes_recovered': 2293760, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2089}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:40:53.119 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:40:53.119 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'8587", 'reported_seq': 8709, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:52.557168+0000', 'last_change': '2024-09-05T16:34:56.867900+0000', 'last_active': '2024-09-05T16:40:52.557168+0000', 'last_peered': '2024-09-05T16:40:52.557168+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:40:52.557168+0000', 'last_undegraded': '2024-09-05T16:34:56.590972+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8587, 'log_dups_size': 0, 'ondisk_log_size': 8587, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34118248, 'num_objects': 4166, 'num_object_clones': 0, 'num_object_copies': 16664, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2233, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4166, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 8586, 'num_write_kb': 50999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1934}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:40:53.119 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:40:53.119 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'8405", 'reported_seq': 8503, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:52.186911+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:40:52.186911+0000', 'last_peered': '2024-09-05T16:40:52.186911+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:40:52.186911+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8405, 'log_dups_size': 0, 'ondisk_log_size': 8405, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34321568, 'num_objects': 4190, 'num_object_clones': 0, 'num_object_copies': 16760, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2147, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4190, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8403, 'num_write_kb': 50373, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2045}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:40:53.119 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:40:53.119 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'8591", 'reported_seq': 8702, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:52.245086+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:40:52.245086+0000', 'last_peered': '2024-09-05T16:40:52.245086+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:40:52.245086+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8591, 'log_dups_size': 0, 'ondisk_log_size': 8591, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34250752, 'num_objects': 4181, 'num_object_clones': 0, 'num_object_copies': 16724, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2220, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4181, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8591, 'num_write_kb': 51088, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 30, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1961}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:40:53.119 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:40:53.119 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'8430", 'reported_seq': 8535, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:52.013478+0000', 'last_change': '2024-09-05T16:34:56.903436+0000', 'last_active': '2024-09-05T16:40:52.013478+0000', 'last_peered': '2024-09-05T16:40:52.013478+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:40:52.013478+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8430, 'log_dups_size': 0, 'ondisk_log_size': 8430, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34665632, 'num_objects': 4232, 'num_object_clones': 0, 'num_object_copies': 16928, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2134, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4232, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8428, 'num_write_kb': 50636, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2100}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:40:53.119 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-05T16:40:53.119 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "37'8579", 'reported_seq': 8667, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:52.209785+0000', 'last_change': '2024-09-05T16:34:56.883319+0000', 'last_active': '2024-09-05T16:40:52.209785+0000', 'last_peered': '2024-09-05T16:40:52.209785+0000', 'last_clean': '2024-09-05T16:33:53.505305+0000', 'last_became_active': '2024-09-05T16:34:56.873389+0000', 'last_became_peered': '2024-09-05T16:34:56.873389+0000', 'last_unstale': '2024-09-05T16:40:52.209785+0000', 'last_undegraded': '2024-09-05T16:34:56.602567+0000', 'last_fullsized': '2024-09-05T16:34:56.559958+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8579, 'log_dups_size': 0, 'ondisk_log_size': 8579, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34403488, 'num_objects': 4200, 'num_object_clones': 0, 'num_object_copies': 16800, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2234, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4200, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8577, 'num_write_kb': 51108, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1968}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:40:53.120 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:40:53.120 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'8610", 'reported_seq': 8710, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:52.257954+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:40:52.257954+0000', 'last_peered': '2024-09-05T16:40:52.257954+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:40:52.257954+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8610, 'log_dups_size': 0, 'ondisk_log_size': 8610, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34105664, 'num_objects': 4164, 'num_object_clones': 0, 'num_object_copies': 16656, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4164, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8607, 'num_write_kb': 51081, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1920}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:40:53.120 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:40:53.120 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'8625", 'reported_seq': 8740, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:52.608268+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:40:52.608268+0000', 'last_peered': '2024-09-05T16:40:52.608268+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:40:52.608268+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8625, 'log_dups_size': 0, 'ondisk_log_size': 8625, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34280608, 'num_objects': 4185, 'num_object_clones': 0, 'num_object_copies': 16740, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4185, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8623, 'num_write_kb': 51230, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 16, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1937}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:40:53.120 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:40:53.120 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'8837", 'reported_seq': 8961, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:51.829443+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:40:51.829443+0000', 'last_peered': '2024-09-05T16:40:51.829443+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:40:51.829443+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8837, 'log_dups_size': 0, 'ondisk_log_size': 8837, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35106544, 'num_objects': 4286, 'num_object_clones': 0, 'num_object_copies': 17144, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2285, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4286, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8834, 'num_write_kb': 52476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2004}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:40:53.120 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:40:53.120 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'8601", 'reported_seq': 8699, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:52.184892+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:40:52.184892+0000', 'last_peered': '2024-09-05T16:40:52.184892+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:40:52.184892+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8601, 'log_dups_size': 0, 'ondisk_log_size': 8601, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34060880, 'num_objects': 4158, 'num_object_clones': 0, 'num_object_copies': 16632, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4158, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8600, 'num_write_kb': 51031, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1920}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:40:53.120 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:40:53.120 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'8882", 'reported_seq': 8989, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:52.163649+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:40:52.163649+0000', 'last_peered': '2024-09-05T16:40:52.163649+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:40:52.163649+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8882, 'log_dups_size': 0, 'ondisk_log_size': 8882, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34763936, 'num_objects': 4244, 'num_object_clones': 0, 'num_object_copies': 16976, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4244, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8880, 'num_write_kb': 52500, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1910}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:40:53.121 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:40:53.121 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'8700", 'reported_seq': 8811, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:52.555587+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:40:52.555587+0000', 'last_peered': '2024-09-05T16:40:52.555587+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:40:52.555587+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8700, 'log_dups_size': 0, 'ondisk_log_size': 8700, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33854624, 'num_objects': 4133, 'num_object_clones': 0, 'num_object_copies': 16532, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2310, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4133, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8698, 'num_write_kb': 51325, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1825}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:40:53.121 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:40:53.121 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'8584", 'reported_seq': 8689, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:51.912495+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:40:51.912495+0000', 'last_peered': '2024-09-05T16:40:51.912495+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:40:51.912495+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8584, 'log_dups_size': 0, 'ondisk_log_size': 8584, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34329760, 'num_objects': 4191, 'num_object_clones': 0, 'num_object_copies': 16764, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2223, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4191, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8582, 'num_write_kb': 51092, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1970}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:40:53.121 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:40:53.825 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:40:53.825 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 31069/268196 objects degraded (11.584%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":457,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":67049,"data_bytes":549220840,"bytes_used":2874540032,"bytes_avail":383672516608,"bytes_total":386547056640,"degraded_objects":31069,"degraded_total":268196,"degraded_ratio":0.1158443824665543,"recovering_objects_per_sec":18,"recovering_bytes_per_sec":128976,"recovering_keys_per_sec":0,"num_objects_recovered":222,"num_bytes_recovered":1548288,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":278},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:40:53.857 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:40:53.857 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:40:54.481 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:40:54.481 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:40:54.511 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":256,"stamp":"2024-09-05T16:40:52.950148+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":549220840,"num_objects":67049,"num_object_clones":0,"num_object_copies":268196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31069,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67049,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":123788,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":568,"num_bytes_recovered":2293760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137912,"ondisk_log_size":137912,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2807168,"kb_used_data":1071904,"kb_used_omap":14,"kb_used_meta":1735153,"kb_avail":374680192,"statfs":{"total":386547056640,"available":383672516608,"internally_reserved":0,"allocated":1097629696,"data_stored":1096824600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1776797239},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,1,5,15,6,0,0,0,0,5,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-27344896,"num_objects":-3338,"num_object_clones":0,"num_object_copies":-13352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3338,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3338,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":222,"num_bytes_recovered":1548288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004408"},"pg_stats":[{"pgid":"1.f","version":"37'8670","reported_seq":8773,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:51.921696+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:40:51.921696+0000","last_peered":"2024-09-05T16:40:51.921696+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:40:51.921696+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8670,"log_dups_size":0,"ondisk_log_size":8670,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34684928,"num_objects":4234,"num_object_clones":0,"num_object_copies":16936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4234,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8670,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1998}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'8490","reported_seq":9152,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:52.204450+0000","last_change":"2024-09-05T16:40:28.919189+0000","last_active":"2024-09-05T16:40:52.204450+0000","last_peered":"2024-09-05T16:40:52.204450+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:40:52.204450+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8490,"log_dups_size":0,"ondisk_log_size":8490,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33289376,"num_objects":4064,"num_object_clones":0,"num_object_copies":16256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1975,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4064,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8488,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":296,"num_bytes_recovered":2293760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2089}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'8587","reported_seq":8709,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:52.557168+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:40:52.557168+0000","last_peered":"2024-09-05T16:40:52.557168+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:40:52.557168+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8587,"log_dups_size":0,"ondisk_log_size":8587,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34118248,"num_objects":4166,"num_object_clones":0,"num_object_copies":16664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4166,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8586,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1934}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'8686","reported_seq":13414,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:40:51.359636+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:40:51.359636+0000","last_peered":"2024-09-05T16:40:51.359636+0000","last_clean":"2024-09-05T16:40:51.359636+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:40:51.359636+0000","last_undegraded":"2024-09-05T16:40:51.359636+0000","last_fullsized":"2024-09-05T16:40:51.359636+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":8686,"log_dups_size":0,"ondisk_log_size":8686,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34799616,"num_objects":4248,"num_object_clones":0,"num_object_copies":16992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4248,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2986,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'8405","reported_seq":8503,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:52.186911+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:40:52.186911+0000","last_peered":"2024-09-05T16:40:52.186911+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:40:52.186911+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8405,"log_dups_size":0,"ondisk_log_size":8405,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34321568,"num_objects":4190,"num_object_clones":0,"num_object_copies":16760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4190,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8403,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2045}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'8591","reported_seq":8702,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:52.245086+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:40:52.245086+0000","last_peered":"2024-09-05T16:40:52.245086+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:40:52.245086+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8591,"log_dups_size":0,"ondisk_log_size":8591,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34250752,"num_objects":4181,"num_object_clones":0,"num_object_copies":16724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4181,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8591,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1961}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'8430","reported_seq":8535,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:52.013478+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:40:52.013478+0000","last_peered":"2024-09-05T16:40:52.013478+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:40:52.013478+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8430,"log_dups_size":0,"ondisk_log_size":8430,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34665632,"num_objects":4232,"num_object_clones":0,"num_object_copies":16928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4232,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8428,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2100}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'8579","reported_seq":8667,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:52.209785+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:40:52.209785+0000","last_peered":"2024-09-05T16:40:52.209785+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:40:52.209785+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8579,"log_dups_size":0,"ondisk_log_size":8579,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34403488,"num_objects":4200,"num_object_clones":0,"num_object_copies":16800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4200,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8577,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1968}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'8610","reported_seq":8710,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:52.257954+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:40:52.257954+0000","last_peered":"2024-09-05T16:40:52.257954+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:40:52.257954+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8610,"log_dups_size":0,"ondisk_log_size":8610,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34105664,"num_objects":4164,"num_object_clones":0,"num_object_copies":16656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4164,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8607,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1920}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'8625","reported_seq":8740,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:52.608268+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:40:52.608268+0000","last_peered":"2024-09-05T16:40:52.608268+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:40:52.608268+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8625,"log_dups_size":0,"ondisk_log_size":8625,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34280608,"num_objects":4185,"num_object_clones":0,"num_object_copies":16740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4185,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8623,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1937}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'8837","reported_seq":8961,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:51.829443+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:40:51.829443+0000","last_peered":"2024-09-05T16:40:51.829443+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:40:51.829443+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8837,"log_dups_size":0,"ondisk_log_size":8837,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35106544,"num_objects":4286,"num_object_clones":0,"num_object_copies":17144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4286,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8834,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2004}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'8601","reported_seq":8699,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:52.184892+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:40:52.184892+0000","last_peered":"2024-09-05T16:40:52.184892+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:40:52.184892+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8601,"log_dups_size":0,"ondisk_log_size":8601,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34060880,"num_objects":4158,"num_object_clones":0,"num_object_copies":16632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4158,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8600,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1920}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'8635","reported_seq":13370,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:40:51.189779+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:40:51.189779+0000","last_peered":"2024-09-05T16:40:51.189779+0000","last_clean":"2024-09-05T16:40:51.189779+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:40:51.189779+0000","last_undegraded":"2024-09-05T16:40:51.189779+0000","last_fullsized":"2024-09-05T16:40:51.189779+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":8635,"log_dups_size":0,"ondisk_log_size":8635,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34185216,"num_objects":4173,"num_object_clones":0,"num_object_copies":16692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4173,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":235,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'8882","reported_seq":8989,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:52.163649+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:40:52.163649+0000","last_peered":"2024-09-05T16:40:52.163649+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:40:52.163649+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8882,"log_dups_size":0,"ondisk_log_size":8882,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34763936,"num_objects":4244,"num_object_clones":0,"num_object_copies":16976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4244,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8880,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1910}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'8700","reported_seq":8811,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:52.555587+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:40:52.555587+0000","last_peered":"2024-09-05T16:40:52.555587+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:40:52.555587+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8700,"log_dups_size":0,"ondisk_log_size":8700,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33854624,"num_objects":4133,"num_object_clones":0,"num_object_copies":16532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4133,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8698,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1825}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'8584","reported_seq":8689,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:51.912495+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:40:51.912495+0000","last_peered":"2024-09-05T16:40:51.912495+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:40:51.912495+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8584,"log_dups_size":0,"ondisk_log_size":8584,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34329760,"num_objects":4191,"num_object_clones":0,"num_object_copies":16764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4191,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8582,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1970}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":549220840,"num_objects":67049,"num_object_clones":0,"num_object_copies":268196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31069,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67049,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":123788,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":568,"num_bytes_recovered":2293760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1095327744,"data_stored":1095327744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137912,"ondisk_log_size":137912,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084371,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":745864,"kb_used_data":268516,"kb_used_omap":4,"kb_used_meta":477307,"kb_avail":93625976,"statfs":{"total":96636764160,"available":95872999424,"internally_reserved":0,"allocated":274960384,"data_stored":274759110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":488763091},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,3,1,0,0,0,0,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986248,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":594832,"kb_used_data":266048,"kb_used_omap":4,"kb_used_meta":328763,"kb_avail":93777008,"statfs":{"total":96636764160,"available":96027656192,"internally_reserved":0,"allocated":272433152,"data_stored":272231878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":336654034},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673054,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":732944,"kb_used_data":268596,"kb_used_omap":1,"kb_used_meta":464318,"kb_avail":93638896,"statfs":{"total":96636764160,"available":95886229504,"internally_reserved":0,"allocated":275042304,"data_stored":274841030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":475462085},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,0,0,8,2,0,0,0,0,3],"upper_bound":4096},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182485,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":733528,"kb_used_data":268744,"kb_used_omap":4,"kb_used_meta":464763,"kb_avail":93638312,"statfs":{"total":96636764160,"available":95885631488,"internally_reserved":0,"allocated":275193856,"data_stored":274992582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":475918029},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,4,3,0,0,0,0,2],"upper_bound":4096},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":274526208,"data_stored":274526208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":272113664,"data_stored":272113664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":274206720,"data_stored":274206720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":274481152,"data_stored":274481152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:40:57.513 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:40:58.157 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:40:58.157 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:40:58.183 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":258,"stamp":"2024-09-05T16:40:56.951966+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":547910120,"num_objects":66889,"num_object_clones":0,"num_object_copies":267556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31069,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66889,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":123948,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":568,"num_bytes_recovered":2293760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":138072,"ondisk_log_size":138072,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2793536,"kb_used_data":1070752,"kb_used_omap":14,"kb_used_meta":1722673,"kb_avail":374693824,"statfs":{"total":386547056640,"available":383686475776,"internally_reserved":0,"allocated":1096450048,"data_stored":1095644952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1764017719},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,1,2,4,18,6,0,0,0,0,5,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19251200,"num_objects":-2350,"num_object_clones":0,"num_object_copies":-9400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-165,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2350,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2350,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":165,"num_bytes_recovered":1089536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004701"},"pg_stats":[{"pgid":"1.f","version":"37'8670","reported_seq":8773,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:51.921696+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:40:51.921696+0000","last_peered":"2024-09-05T16:40:51.921696+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:40:51.921696+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8670,"log_dups_size":0,"ondisk_log_size":8670,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34684928,"num_objects":4234,"num_object_clones":0,"num_object_copies":16936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4234,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8670,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1998}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'8490","reported_seq":9152,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:52.204450+0000","last_change":"2024-09-05T16:40:28.919189+0000","last_active":"2024-09-05T16:40:52.204450+0000","last_peered":"2024-09-05T16:40:52.204450+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:40:52.204450+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8490,"log_dups_size":0,"ondisk_log_size":8490,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33289376,"num_objects":4064,"num_object_clones":0,"num_object_copies":16256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1975,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4064,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8488,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":296,"num_bytes_recovered":2293760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2089}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'8587","reported_seq":8709,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:52.557168+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:40:52.557168+0000","last_peered":"2024-09-05T16:40:52.557168+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:40:52.557168+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8587,"log_dups_size":0,"ondisk_log_size":8587,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34118248,"num_objects":4166,"num_object_clones":0,"num_object_copies":16664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4166,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8586,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1934}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'8769","reported_seq":13496,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:40:56.461426+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:40:56.461426+0000","last_peered":"2024-09-05T16:40:56.461426+0000","last_clean":"2024-09-05T16:40:56.461426+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:40:56.461426+0000","last_undegraded":"2024-09-05T16:40:56.461426+0000","last_fullsized":"2024-09-05T16:40:56.461426+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":8769,"log_dups_size":0,"ondisk_log_size":8769,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34119680,"num_objects":4165,"num_object_clones":0,"num_object_copies":16660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4165,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3069,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'8405","reported_seq":8503,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:52.186911+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:40:52.186911+0000","last_peered":"2024-09-05T16:40:52.186911+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:40:52.186911+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8405,"log_dups_size":0,"ondisk_log_size":8405,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34321568,"num_objects":4190,"num_object_clones":0,"num_object_copies":16760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4190,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8403,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2045}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'8591","reported_seq":8702,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:52.245086+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:40:52.245086+0000","last_peered":"2024-09-05T16:40:52.245086+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:40:52.245086+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8591,"log_dups_size":0,"ondisk_log_size":8591,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34250752,"num_objects":4181,"num_object_clones":0,"num_object_copies":16724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4181,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8591,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1961}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'8430","reported_seq":8535,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:52.013478+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:40:52.013478+0000","last_peered":"2024-09-05T16:40:52.013478+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:40:52.013478+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8430,"log_dups_size":0,"ondisk_log_size":8430,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34665632,"num_objects":4232,"num_object_clones":0,"num_object_copies":16928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4232,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8428,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2100}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'8579","reported_seq":8667,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:52.209785+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:40:52.209785+0000","last_peered":"2024-09-05T16:40:52.209785+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:40:52.209785+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8579,"log_dups_size":0,"ondisk_log_size":8579,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34403488,"num_objects":4200,"num_object_clones":0,"num_object_copies":16800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4200,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8577,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1968}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'8610","reported_seq":8710,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:52.257954+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:40:52.257954+0000","last_peered":"2024-09-05T16:40:52.257954+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:40:52.257954+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8610,"log_dups_size":0,"ondisk_log_size":8610,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34105664,"num_objects":4164,"num_object_clones":0,"num_object_copies":16656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4164,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8607,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1920}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'8625","reported_seq":8740,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:52.608268+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:40:52.608268+0000","last_peered":"2024-09-05T16:40:52.608268+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:40:52.608268+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8625,"log_dups_size":0,"ondisk_log_size":8625,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34280608,"num_objects":4185,"num_object_clones":0,"num_object_copies":16740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4185,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8623,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1937}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'8837","reported_seq":8961,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:51.829443+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:40:51.829443+0000","last_peered":"2024-09-05T16:40:51.829443+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:40:51.829443+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8837,"log_dups_size":0,"ondisk_log_size":8837,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35106544,"num_objects":4286,"num_object_clones":0,"num_object_copies":17144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4286,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8834,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2004}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'8601","reported_seq":8699,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:52.184892+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:40:52.184892+0000","last_peered":"2024-09-05T16:40:52.184892+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:40:52.184892+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8601,"log_dups_size":0,"ondisk_log_size":8601,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34060880,"num_objects":4158,"num_object_clones":0,"num_object_copies":16632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4158,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8600,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1920}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'8712","reported_seq":13446,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:40:56.452407+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:40:56.452407+0000","last_peered":"2024-09-05T16:40:56.452407+0000","last_clean":"2024-09-05T16:40:56.452407+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:40:56.452407+0000","last_undegraded":"2024-09-05T16:40:56.452407+0000","last_fullsized":"2024-09-05T16:40:56.452407+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":8712,"log_dups_size":0,"ondisk_log_size":8712,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33554432,"num_objects":4096,"num_object_clones":0,"num_object_copies":16384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4096,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":312,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'8882","reported_seq":8989,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:52.163649+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:40:52.163649+0000","last_peered":"2024-09-05T16:40:52.163649+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:40:52.163649+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8882,"log_dups_size":0,"ondisk_log_size":8882,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34763936,"num_objects":4244,"num_object_clones":0,"num_object_copies":16976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4244,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8880,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1910}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'8700","reported_seq":8811,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:52.555587+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:40:52.555587+0000","last_peered":"2024-09-05T16:40:52.555587+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:40:52.555587+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8700,"log_dups_size":0,"ondisk_log_size":8700,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33854624,"num_objects":4133,"num_object_clones":0,"num_object_copies":16532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4133,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8698,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1825}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'8584","reported_seq":8689,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:51.912495+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:40:51.912495+0000","last_peered":"2024-09-05T16:40:51.912495+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:40:51.912495+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8584,"log_dups_size":0,"ondisk_log_size":8584,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34329760,"num_objects":4191,"num_object_clones":0,"num_object_copies":16764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4191,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8582,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1970}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":547910120,"num_objects":66889,"num_object_clones":0,"num_object_copies":267556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31069,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66889,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":123948,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":568,"num_bytes_recovered":2293760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1094152192,"data_stored":1094152192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":138072,"ondisk_log_size":138072,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084371,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":745864,"kb_used_data":268516,"kb_used_omap":4,"kb_used_meta":477307,"kb_avail":93625976,"statfs":{"total":96636764160,"available":95872999424,"internally_reserved":0,"allocated":274960384,"data_stored":274759110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":488763091},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,3,1,0,0,0,0,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986249,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":581200,"kb_used_data":264896,"kb_used_omap":4,"kb_used_meta":316283,"kb_avail":93790640,"statfs":{"total":96636764160,"available":96041615360,"internally_reserved":0,"allocated":271253504,"data_stored":271052230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":323874514},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,2,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673054,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":732944,"kb_used_data":268596,"kb_used_omap":1,"kb_used_meta":464318,"kb_avail":93638896,"statfs":{"total":96636764160,"available":95886229504,"internally_reserved":0,"allocated":275042304,"data_stored":274841030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":475462085},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,0,0,8,2,0,0,0,0,3],"upper_bound":4096},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182485,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":733528,"kb_used_data":268744,"kb_used_omap":4,"kb_used_meta":464763,"kb_avail":93638312,"statfs":{"total":96636764160,"available":95885631488,"internally_reserved":0,"allocated":275193856,"data_stored":274992582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":475918029},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,4,3,0,0,0,0,2],"upper_bound":4096},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":274526208,"data_stored":274526208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":270938112,"data_stored":270938112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":274206720,"data_stored":274206720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":274481152,"data_stored":274481152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:40:58.184 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:40:58.803 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:40:58.804 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:40:58.827 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":258,"stamp":"2024-09-05T16:40:56.951966+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":547910120,"num_objects":66889,"num_object_clones":0,"num_object_copies":267556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31069,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66889,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":123948,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":568,"num_bytes_recovered":2293760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":138072,"ondisk_log_size":138072,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2793536,"kb_used_data":1070752,"kb_used_omap":14,"kb_used_meta":1722673,"kb_avail":374693824,"statfs":{"total":386547056640,"available":383686475776,"internally_reserved":0,"allocated":1096450048,"data_stored":1095644952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1764017719},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,1,2,4,18,6,0,0,0,0,5,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19251200,"num_objects":-2350,"num_object_clones":0,"num_object_copies":-9400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-165,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2350,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2350,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":165,"num_bytes_recovered":1089536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004701"},"pg_stats":[{"pgid":"1.f","version":"37'8670","reported_seq":8773,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:51.921696+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:40:51.921696+0000","last_peered":"2024-09-05T16:40:51.921696+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:40:51.921696+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8670,"log_dups_size":0,"ondisk_log_size":8670,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34684928,"num_objects":4234,"num_object_clones":0,"num_object_copies":16936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4234,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8670,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1998}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'8490","reported_seq":9152,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:52.204450+0000","last_change":"2024-09-05T16:40:28.919189+0000","last_active":"2024-09-05T16:40:52.204450+0000","last_peered":"2024-09-05T16:40:52.204450+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:40:52.204450+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8490,"log_dups_size":0,"ondisk_log_size":8490,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33289376,"num_objects":4064,"num_object_clones":0,"num_object_copies":16256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1975,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4064,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8488,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":296,"num_bytes_recovered":2293760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2089}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'8587","reported_seq":8709,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:52.557168+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:40:52.557168+0000","last_peered":"2024-09-05T16:40:52.557168+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:40:52.557168+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8587,"log_dups_size":0,"ondisk_log_size":8587,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34118248,"num_objects":4166,"num_object_clones":0,"num_object_copies":16664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4166,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8586,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1934}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'8769","reported_seq":13496,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:40:56.461426+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:40:56.461426+0000","last_peered":"2024-09-05T16:40:56.461426+0000","last_clean":"2024-09-05T16:40:56.461426+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:40:56.461426+0000","last_undegraded":"2024-09-05T16:40:56.461426+0000","last_fullsized":"2024-09-05T16:40:56.461426+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":8769,"log_dups_size":0,"ondisk_log_size":8769,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34119680,"num_objects":4165,"num_object_clones":0,"num_object_copies":16660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4165,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3069,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'8405","reported_seq":8503,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:52.186911+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:40:52.186911+0000","last_peered":"2024-09-05T16:40:52.186911+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:40:52.186911+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8405,"log_dups_size":0,"ondisk_log_size":8405,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34321568,"num_objects":4190,"num_object_clones":0,"num_object_copies":16760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4190,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8403,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2045}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'8591","reported_seq":8702,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:52.245086+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:40:52.245086+0000","last_peered":"2024-09-05T16:40:52.245086+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:40:52.245086+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8591,"log_dups_size":0,"ondisk_log_size":8591,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34250752,"num_objects":4181,"num_object_clones":0,"num_object_copies":16724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4181,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8591,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1961}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'8430","reported_seq":8535,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:52.013478+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:40:52.013478+0000","last_peered":"2024-09-05T16:40:52.013478+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:40:52.013478+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8430,"log_dups_size":0,"ondisk_log_size":8430,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34665632,"num_objects":4232,"num_object_clones":0,"num_object_copies":16928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4232,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8428,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2100}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'8579","reported_seq":8667,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:52.209785+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:40:52.209785+0000","last_peered":"2024-09-05T16:40:52.209785+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:40:52.209785+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8579,"log_dups_size":0,"ondisk_log_size":8579,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34403488,"num_objects":4200,"num_object_clones":0,"num_object_copies":16800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4200,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8577,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1968}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'8610","reported_seq":8710,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:52.257954+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:40:52.257954+0000","last_peered":"2024-09-05T16:40:52.257954+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:40:52.257954+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8610,"log_dups_size":0,"ondisk_log_size":8610,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34105664,"num_objects":4164,"num_object_clones":0,"num_object_copies":16656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4164,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8607,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1920}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'8625","reported_seq":8740,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:52.608268+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:40:52.608268+0000","last_peered":"2024-09-05T16:40:52.608268+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:40:52.608268+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8625,"log_dups_size":0,"ondisk_log_size":8625,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34280608,"num_objects":4185,"num_object_clones":0,"num_object_copies":16740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4185,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8623,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1937}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'8837","reported_seq":8961,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:51.829443+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:40:51.829443+0000","last_peered":"2024-09-05T16:40:51.829443+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:40:51.829443+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8837,"log_dups_size":0,"ondisk_log_size":8837,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35106544,"num_objects":4286,"num_object_clones":0,"num_object_copies":17144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4286,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8834,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2004}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'8601","reported_seq":8699,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:52.184892+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:40:52.184892+0000","last_peered":"2024-09-05T16:40:52.184892+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:40:52.184892+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8601,"log_dups_size":0,"ondisk_log_size":8601,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34060880,"num_objects":4158,"num_object_clones":0,"num_object_copies":16632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4158,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8600,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1920}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'8712","reported_seq":13446,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:40:56.452407+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:40:56.452407+0000","last_peered":"2024-09-05T16:40:56.452407+0000","last_clean":"2024-09-05T16:40:56.452407+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:40:56.452407+0000","last_undegraded":"2024-09-05T16:40:56.452407+0000","last_fullsized":"2024-09-05T16:40:56.452407+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":8712,"log_dups_size":0,"ondisk_log_size":8712,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33554432,"num_objects":4096,"num_object_clones":0,"num_object_copies":16384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4096,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":312,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'8882","reported_seq":8989,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:52.163649+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:40:52.163649+0000","last_peered":"2024-09-05T16:40:52.163649+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:40:52.163649+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8882,"log_dups_size":0,"ondisk_log_size":8882,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34763936,"num_objects":4244,"num_object_clones":0,"num_object_copies":16976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4244,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8880,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1910}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'8700","reported_seq":8811,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:52.555587+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:40:52.555587+0000","last_peered":"2024-09-05T16:40:52.555587+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:40:52.555587+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8700,"log_dups_size":0,"ondisk_log_size":8700,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33854624,"num_objects":4133,"num_object_clones":0,"num_object_copies":16532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4133,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8698,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1825}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'8584","reported_seq":8689,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:51.912495+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:40:51.912495+0000","last_peered":"2024-09-05T16:40:51.912495+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:40:51.912495+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8584,"log_dups_size":0,"ondisk_log_size":8584,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34329760,"num_objects":4191,"num_object_clones":0,"num_object_copies":16764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4191,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8582,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1970}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":547910120,"num_objects":66889,"num_object_clones":0,"num_object_copies":267556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31069,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66889,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":123948,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":568,"num_bytes_recovered":2293760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1094152192,"data_stored":1094152192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":138072,"ondisk_log_size":138072,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084371,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":745864,"kb_used_data":268516,"kb_used_omap":4,"kb_used_meta":477307,"kb_avail":93625976,"statfs":{"total":96636764160,"available":95872999424,"internally_reserved":0,"allocated":274960384,"data_stored":274759110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":488763091},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,3,1,0,0,0,0,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986249,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":581200,"kb_used_data":264896,"kb_used_omap":4,"kb_used_meta":316283,"kb_avail":93790640,"statfs":{"total":96636764160,"available":96041615360,"internally_reserved":0,"allocated":271253504,"data_stored":271052230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":323874514},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,2,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673054,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":732944,"kb_used_data":268596,"kb_used_omap":1,"kb_used_meta":464318,"kb_avail":93638896,"statfs":{"total":96636764160,"available":95886229504,"internally_reserved":0,"allocated":275042304,"data_stored":274841030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":475462085},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,0,0,8,2,0,0,0,0,3],"upper_bound":4096},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182485,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":733528,"kb_used_data":268744,"kb_used_omap":4,"kb_used_meta":464763,"kb_avail":93638312,"statfs":{"total":96636764160,"available":95885631488,"internally_reserved":0,"allocated":275193856,"data_stored":274992582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":475918029},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,4,3,0,0,0,0,2],"upper_bound":4096},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":274526208,"data_stored":274526208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":270938112,"data_stored":270938112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":274206720,"data_stored":274206720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":274481152,"data_stored":274481152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:40:58.828 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-05T16:40:58.828 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'8670", 'reported_seq': 8773, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:51.921696+0000', 'last_change': '2024-09-05T16:34:56.833497+0000', 'last_active': '2024-09-05T16:40:51.921696+0000', 'last_peered': '2024-09-05T16:40:51.921696+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.828289+0000', 'last_became_peered': '2024-09-05T16:34:56.828289+0000', 'last_unstale': '2024-09-05T16:40:51.921696+0000', 'last_undegraded': '2024-09-05T16:34:56.612160+0000', 'last_fullsized': '2024-09-05T16:34:56.557632+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8670, 'log_dups_size': 0, 'ondisk_log_size': 8670, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34684928, 'num_objects': 4234, 'num_object_clones': 0, 'num_object_copies': 16936, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2236, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4234, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8670, 'num_write_kb': 51616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1998}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:40:58.828 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-05T16:40:58.829 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "37'8490", 'reported_seq': 9152, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:52.204450+0000', 'last_change': '2024-09-05T16:40:28.919189+0000', 'last_active': '2024-09-05T16:40:52.204450+0000', 'last_peered': '2024-09-05T16:40:52.204450+0000', 'last_clean': '2024-09-05T16:33:53.200762+0000', 'last_became_active': '2024-09-05T16:34:56.820292+0000', 'last_became_peered': '2024-09-05T16:34:56.820292+0000', 'last_unstale': '2024-09-05T16:40:52.204450+0000', 'last_undegraded': '2024-09-05T16:34:56.605776+0000', 'last_fullsized': '2024-09-05T16:34:56.559383+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8490, 'log_dups_size': 0, 'ondisk_log_size': 8490, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33289376, 'num_objects': 4064, 'num_object_clones': 0, 'num_object_copies': 16256, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1975, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4064, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8488, 'num_write_kb': 50212, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 296, 'num_bytes_recovered': 2293760, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2089}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:40:58.829 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:40:58.829 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'8587", 'reported_seq': 8709, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:52.557168+0000', 'last_change': '2024-09-05T16:34:56.867900+0000', 'last_active': '2024-09-05T16:40:52.557168+0000', 'last_peered': '2024-09-05T16:40:52.557168+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:40:52.557168+0000', 'last_undegraded': '2024-09-05T16:34:56.590972+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8587, 'log_dups_size': 0, 'ondisk_log_size': 8587, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34118248, 'num_objects': 4166, 'num_object_clones': 0, 'num_object_copies': 16664, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2233, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4166, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 8586, 'num_write_kb': 50999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1934}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:40:58.829 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:40:58.829 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'8405", 'reported_seq': 8503, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:52.186911+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:40:52.186911+0000', 'last_peered': '2024-09-05T16:40:52.186911+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:40:52.186911+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8405, 'log_dups_size': 0, 'ondisk_log_size': 8405, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34321568, 'num_objects': 4190, 'num_object_clones': 0, 'num_object_copies': 16760, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2147, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4190, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8403, 'num_write_kb': 50373, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2045}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:40:58.829 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:40:58.829 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'8591", 'reported_seq': 8702, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:52.245086+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:40:52.245086+0000', 'last_peered': '2024-09-05T16:40:52.245086+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:40:52.245086+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8591, 'log_dups_size': 0, 'ondisk_log_size': 8591, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34250752, 'num_objects': 4181, 'num_object_clones': 0, 'num_object_copies': 16724, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2220, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4181, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8591, 'num_write_kb': 51088, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 30, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1961}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:40:58.829 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:40:58.829 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'8430", 'reported_seq': 8535, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:52.013478+0000', 'last_change': '2024-09-05T16:34:56.903436+0000', 'last_active': '2024-09-05T16:40:52.013478+0000', 'last_peered': '2024-09-05T16:40:52.013478+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:40:52.013478+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8430, 'log_dups_size': 0, 'ondisk_log_size': 8430, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34665632, 'num_objects': 4232, 'num_object_clones': 0, 'num_object_copies': 16928, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2134, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4232, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8428, 'num_write_kb': 50636, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2100}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:40:58.830 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-05T16:40:58.830 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "37'8579", 'reported_seq': 8667, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:52.209785+0000', 'last_change': '2024-09-05T16:34:56.883319+0000', 'last_active': '2024-09-05T16:40:52.209785+0000', 'last_peered': '2024-09-05T16:40:52.209785+0000', 'last_clean': '2024-09-05T16:33:53.505305+0000', 'last_became_active': '2024-09-05T16:34:56.873389+0000', 'last_became_peered': '2024-09-05T16:34:56.873389+0000', 'last_unstale': '2024-09-05T16:40:52.209785+0000', 'last_undegraded': '2024-09-05T16:34:56.602567+0000', 'last_fullsized': '2024-09-05T16:34:56.559958+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8579, 'log_dups_size': 0, 'ondisk_log_size': 8579, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34403488, 'num_objects': 4200, 'num_object_clones': 0, 'num_object_copies': 16800, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2234, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4200, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8577, 'num_write_kb': 51108, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1968}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:40:58.830 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:40:58.830 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'8610", 'reported_seq': 8710, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:52.257954+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:40:52.257954+0000', 'last_peered': '2024-09-05T16:40:52.257954+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:40:52.257954+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8610, 'log_dups_size': 0, 'ondisk_log_size': 8610, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34105664, 'num_objects': 4164, 'num_object_clones': 0, 'num_object_copies': 16656, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4164, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8607, 'num_write_kb': 51081, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1920}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:40:58.830 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:40:58.830 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'8625", 'reported_seq': 8740, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:52.608268+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:40:52.608268+0000', 'last_peered': '2024-09-05T16:40:52.608268+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:40:52.608268+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8625, 'log_dups_size': 0, 'ondisk_log_size': 8625, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34280608, 'num_objects': 4185, 'num_object_clones': 0, 'num_object_copies': 16740, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4185, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8623, 'num_write_kb': 51230, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 16, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1937}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:40:58.830 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:40:58.830 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'8837", 'reported_seq': 8961, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:51.829443+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:40:51.829443+0000', 'last_peered': '2024-09-05T16:40:51.829443+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:40:51.829443+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8837, 'log_dups_size': 0, 'ondisk_log_size': 8837, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35106544, 'num_objects': 4286, 'num_object_clones': 0, 'num_object_copies': 17144, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2285, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4286, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8834, 'num_write_kb': 52476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2004}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:40:58.831 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:40:58.831 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'8601", 'reported_seq': 8699, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:52.184892+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:40:52.184892+0000', 'last_peered': '2024-09-05T16:40:52.184892+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:40:52.184892+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8601, 'log_dups_size': 0, 'ondisk_log_size': 8601, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34060880, 'num_objects': 4158, 'num_object_clones': 0, 'num_object_copies': 16632, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4158, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8600, 'num_write_kb': 51031, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1920}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:40:58.831 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:40:58.831 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'8882", 'reported_seq': 8989, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:52.163649+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:40:52.163649+0000', 'last_peered': '2024-09-05T16:40:52.163649+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:40:52.163649+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8882, 'log_dups_size': 0, 'ondisk_log_size': 8882, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34763936, 'num_objects': 4244, 'num_object_clones': 0, 'num_object_copies': 16976, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4244, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8880, 'num_write_kb': 52500, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1910}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:40:58.831 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:40:58.831 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'8700", 'reported_seq': 8811, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:52.555587+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:40:52.555587+0000', 'last_peered': '2024-09-05T16:40:52.555587+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:40:52.555587+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8700, 'log_dups_size': 0, 'ondisk_log_size': 8700, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33854624, 'num_objects': 4133, 'num_object_clones': 0, 'num_object_copies': 16532, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2310, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4133, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8698, 'num_write_kb': 51325, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1825}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:40:58.831 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:40:58.831 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'8584", 'reported_seq': 8689, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:40:51.912495+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:40:51.912495+0000', 'last_peered': '2024-09-05T16:40:51.912495+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:40:51.912495+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8584, 'log_dups_size': 0, 'ondisk_log_size': 8584, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34329760, 'num_objects': 4191, 'num_object_clones': 0, 'num_object_copies': 16764, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2223, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4191, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8582, 'num_write_kb': 51092, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1970}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:40:58.832 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:40:59.551 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:40:59.551 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 31008/263772 objects degraded (11.756%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":462,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":65943,"data_bytes":540160488,"bytes_used":2843250688,"bytes_avail":383703805952,"bytes_total":386547056640,"degraded_objects":31008,"degraded_total":263772,"degraded_ratio":0.11755607115235886,"recovering_objects_per_sec":18,"recovering_bytes_per_sec":117370,"recovering_keys_per_sec":0,"num_objects_recovered":226,"num_bytes_recovered":1409024,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":263},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:40:59.573 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:40:59.574 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:41:00.127 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:41:00.127 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:41:00.155 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":259,"stamp":"2024-09-05T16:40:58.952848+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":540160488,"num_objects":65943,"num_object_clones":0,"num_object_copies":263772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31008,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65943,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":124894,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":629,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139018,"ondisk_log_size":139018,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2776612,"kb_used_data":1057980,"kb_used_omap":14,"kb_used_meta":1718513,"kb_avail":374710748,"statfs":{"total":386547056640,"available":383703805952,"internally_reserved":0,"allocated":1083371520,"data_stored":1082566424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1759757879},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,5,5,8,11,8,4,4,0,0,0,0,0,5],"upper_bound":16384},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25911296,"num_objects":-3163,"num_object_clones":0,"num_object_copies":-12652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3163,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3163,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":226,"num_bytes_recovered":1409024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004948"},"pg_stats":[{"pgid":"1.f","version":"37'8734","reported_seq":8837,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:57.306708+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:40:57.306708+0000","last_peered":"2024-09-05T16:40:57.306708+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:40:57.306708+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8734,"log_dups_size":0,"ondisk_log_size":8734,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34160640,"num_objects":4170,"num_object_clones":0,"num_object_copies":16680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4170,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8734,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1934}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'8563","reported_seq":9343,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:57.264100+0000","last_change":"2024-09-05T16:40:28.919189+0000","last_active":"2024-09-05T16:40:57.264100+0000","last_peered":"2024-09-05T16:40:57.264100+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:40:57.264100+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8563,"log_dups_size":0,"ondisk_log_size":8563,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32691360,"num_objects":3991,"num_object_clones":0,"num_object_copies":15964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1916,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3991,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8561,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":355,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2075}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'8664","reported_seq":8786,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:57.653995+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:40:57.653995+0000","last_peered":"2024-09-05T16:40:57.653995+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:40:57.653995+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8664,"log_dups_size":0,"ondisk_log_size":8664,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33487464,"num_objects":4089,"num_object_clones":0,"num_object_copies":16356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4089,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8663,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1857}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'8769","reported_seq":13496,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:40:56.461426+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:40:56.461426+0000","last_peered":"2024-09-05T16:40:56.461426+0000","last_clean":"2024-09-05T16:40:56.461426+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:40:56.461426+0000","last_undegraded":"2024-09-05T16:40:56.461426+0000","last_fullsized":"2024-09-05T16:40:56.461426+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":8769,"log_dups_size":0,"ondisk_log_size":8769,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34119680,"num_objects":4165,"num_object_clones":0,"num_object_copies":16660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4165,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3069,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'8482","reported_seq":8579,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:57.265475+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:40:57.265475+0000","last_peered":"2024-09-05T16:40:57.265475+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:40:57.265475+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8482,"log_dups_size":0,"ondisk_log_size":8482,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33690784,"num_objects":4113,"num_object_clones":0,"num_object_copies":16452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4113,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8480,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1968}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'8653","reported_seq":8766,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:57.117565+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:40:57.117565+0000","last_peered":"2024-09-05T16:40:57.117565+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:40:57.117565+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8653,"log_dups_size":0,"ondisk_log_size":8653,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33742848,"num_objects":4119,"num_object_clones":0,"num_object_copies":16476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4119,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8653,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1900}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'8504","reported_seq":8609,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:57.119559+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:40:57.119559+0000","last_peered":"2024-09-05T16:40:57.119559+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:40:57.119559+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8504,"log_dups_size":0,"ondisk_log_size":8504,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34059424,"num_objects":4158,"num_object_clones":0,"num_object_copies":16632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4158,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8502,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2026}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'8651","reported_seq":8739,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:57.048470+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:40:57.048470+0000","last_peered":"2024-09-05T16:40:57.048470+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:40:57.048470+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8651,"log_dups_size":0,"ondisk_log_size":8651,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33813664,"num_objects":4128,"num_object_clones":0,"num_object_copies":16512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4128,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8649,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1896}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'8665","reported_seq":8765,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:57.200958+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:40:57.200958+0000","last_peered":"2024-09-05T16:40:57.200958+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:40:57.200958+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8665,"log_dups_size":0,"ondisk_log_size":8665,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33655104,"num_objects":4109,"num_object_clones":0,"num_object_copies":16436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4109,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8662,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1865}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'8701","reported_seq":8814,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:57.651513+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:40:57.651513+0000","last_peered":"2024-09-05T16:40:57.651513+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:40:57.651513+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8701,"log_dups_size":0,"ondisk_log_size":8701,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33658016,"num_objects":4109,"num_object_clones":0,"num_object_copies":16436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4109,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8699,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1861}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'8898","reported_seq":9024,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:57.124978+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:40:57.124978+0000","last_peered":"2024-09-05T16:40:57.124978+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:40:57.124978+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8898,"log_dups_size":0,"ondisk_log_size":8898,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34606832,"num_objects":4225,"num_object_clones":0,"num_object_copies":16900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4225,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8895,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1944}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'8668","reported_seq":8766,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:57.215964+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:40:57.215964+0000","last_peered":"2024-09-05T16:40:57.215964+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:40:57.215964+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8668,"log_dups_size":0,"ondisk_log_size":8668,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33512016,"num_objects":4091,"num_object_clones":0,"num_object_copies":16364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4091,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8667,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1853}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'8712","reported_seq":13446,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:40:56.452407+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:40:56.452407+0000","last_peered":"2024-09-05T16:40:56.452407+0000","last_clean":"2024-09-05T16:40:56.452407+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:40:56.452407+0000","last_undegraded":"2024-09-05T16:40:56.452407+0000","last_fullsized":"2024-09-05T16:40:56.452407+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":8712,"log_dups_size":0,"ondisk_log_size":8712,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33554432,"num_objects":4096,"num_object_clones":0,"num_object_copies":16384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4096,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":312,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'8943","reported_seq":9049,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:57.332024+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:40:57.332024+0000","last_peered":"2024-09-05T16:40:57.332024+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:40:57.332024+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8943,"log_dups_size":0,"ondisk_log_size":8943,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34264224,"num_objects":4183,"num_object_clones":0,"num_object_copies":16732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4183,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8941,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1849}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'8765","reported_seq":8877,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:57.452090+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:40:57.452090+0000","last_peered":"2024-09-05T16:40:57.452090+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:40:57.452090+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8765,"log_dups_size":0,"ondisk_log_size":8765,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33322144,"num_objects":4068,"num_object_clones":0,"num_object_copies":16272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4068,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8763,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1760}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'8646","reported_seq":8751,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:40:57.179424+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:40:57.179424+0000","last_peered":"2024-09-05T16:40:57.179424+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:40:57.179424+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8646,"log_dups_size":0,"ondisk_log_size":8646,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33821856,"num_objects":4129,"num_object_clones":0,"num_object_copies":16516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4129,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8644,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1908}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":540160488,"num_objects":65943,"num_object_clones":0,"num_object_copies":263772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31008,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65943,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":124894,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":629,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1080725504,"data_stored":1080725504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139018,"ondisk_log_size":139018,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084372,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":729064,"kb_used_data":264196,"kb_used_omap":4,"kb_used_meta":464827,"kb_avail":93642776,"statfs":{"total":96636764160,"available":95890202624,"internally_reserved":0,"allocated":270536704,"data_stored":270335430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":475983571},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,3,1,2,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986249,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":581200,"kb_used_data":264896,"kb_used_omap":4,"kb_used_meta":316283,"kb_avail":93790640,"statfs":{"total":96636764160,"available":96041615360,"internally_reserved":0,"allocated":271253504,"data_stored":271052230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":323874514},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,2,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673055,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":732940,"kb_used_data":264432,"kb_used_omap":1,"kb_used_meta":468478,"kb_avail":93638900,"statfs":{"total":96636764160,"available":95886233600,"internally_reserved":0,"allocated":270778368,"data_stored":270577094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":479721925},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,2,0,1,2,0,0,0,0,0,3],"upper_bound":16384},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182486,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":733408,"kb_used_data":264456,"kb_used_omap":4,"kb_used_meta":468923,"kb_avail":93638432,"statfs":{"total":96636764160,"available":95885754368,"internally_reserved":0,"allocated":270802944,"data_stored":270601670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":480177869},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,1,1,6,3,1,0,0,0,0,0,0,2],"upper_bound":16384},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":270086144,"data_stored":270086144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":270938112,"data_stored":270938112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":269713408,"data_stored":269713408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":269987840,"data_stored":269987840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:41:03.157 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:41:03.788 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:41:03.789 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:41:03.808 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":261,"stamp":"2024-09-05T16:41:02.954989+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":531436008,"num_objects":64878,"num_object_clones":0,"num_object_copies":259512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30941,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64878,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":125959,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":696,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":140083,"ondisk_log_size":140083,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2831464,"kb_used_data":1042084,"kb_used_omap":14,"kb_used_meta":1789297,"kb_avail":374655896,"statfs":{"total":386547056640,"available":383647637504,"internally_reserved":0,"allocated":1067094016,"data_stored":1066288920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1832240695},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,4,6,15,7,6,1,0,0,0,0,0,5],"upper_bound":32768},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-26869760,"num_objects":-3280,"num_object_clones":0,"num_object_copies":-13120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-210,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3280,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3280,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":210,"num_bytes_recovered":811008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005609"},"pg_stats":[{"pgid":"1.f","version":"37'8792","reported_seq":8896,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:02.429401+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:41:02.429401+0000","last_peered":"2024-09-05T16:41:02.429401+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:41:02.429401+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8792,"log_dups_size":0,"ondisk_log_size":8792,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33685504,"num_objects":4112,"num_object_clones":0,"num_object_copies":16448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4112,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8792,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1876}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'8622","reported_seq":9537,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:02.455128+0000","last_change":"2024-09-05T16:40:28.919189+0000","last_active":"2024-09-05T16:41:02.455128+0000","last_peered":"2024-09-05T16:41:02.455128+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:41:02.455128+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8622,"log_dups_size":0,"ondisk_log_size":8622,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32208032,"num_objects":3932,"num_object_clones":0,"num_object_copies":15728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1849,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3932,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8620,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":422,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2083}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'8736","reported_seq":8858,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:02.675495+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:41:02.675495+0000","last_peered":"2024-09-05T16:41:02.675495+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:41:02.675495+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8736,"log_dups_size":0,"ondisk_log_size":8736,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32897640,"num_objects":4017,"num_object_clones":0,"num_object_copies":16068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4017,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8735,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1785}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'8823","reported_seq":13551,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:41:01.419562+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:41:01.419562+0000","last_peered":"2024-09-05T16:41:01.419562+0000","last_clean":"2024-09-05T16:41:01.419562+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:41:01.419562+0000","last_undegraded":"2024-09-05T16:41:01.419562+0000","last_fullsized":"2024-09-05T16:41:01.419562+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":8823,"log_dups_size":0,"ondisk_log_size":8823,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33677312,"num_objects":4111,"num_object_clones":0,"num_object_copies":16444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4111,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3123,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'8546","reported_seq":8644,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:02.105728+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:41:02.105728+0000","last_peered":"2024-09-05T16:41:02.105728+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:41:02.105728+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8546,"log_dups_size":0,"ondisk_log_size":8546,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33166496,"num_objects":4049,"num_object_clones":0,"num_object_copies":16196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4049,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8544,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1904}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'8730","reported_seq":8844,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:02.312075+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:41:02.312075+0000","last_peered":"2024-09-05T16:41:02.312075+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:41:02.312075+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8730,"log_dups_size":0,"ondisk_log_size":8730,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33112064,"num_objects":4042,"num_object_clones":0,"num_object_copies":16168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4042,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8730,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1823}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'8577","reported_seq":8682,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:02.322447+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:41:02.322447+0000","last_peered":"2024-09-05T16:41:02.322447+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:41:02.322447+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8577,"log_dups_size":0,"ondisk_log_size":8577,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33461408,"num_objects":4085,"num_object_clones":0,"num_object_copies":16340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4085,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8575,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1953}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'8734","reported_seq":8819,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:02.265041+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:41:02.265041+0000","last_peered":"2024-09-05T16:41:02.265041+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:41:02.265041+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8734,"log_dups_size":0,"ondisk_log_size":8734,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33133728,"num_objects":4045,"num_object_clones":0,"num_object_copies":16180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4045,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8732,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1813}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'8722","reported_seq":8822,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:02.271446+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:41:02.271446+0000","last_peered":"2024-09-05T16:41:02.271446+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:41:02.271446+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8722,"log_dups_size":0,"ondisk_log_size":8722,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33188160,"num_objects":4052,"num_object_clones":0,"num_object_copies":16208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4052,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8719,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1808}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'8773","reported_seq":8888,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:02.555472+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:41:02.555472+0000","last_peered":"2024-09-05T16:41:02.555472+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:41:02.555472+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8773,"log_dups_size":0,"ondisk_log_size":8773,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33068192,"num_objects":4037,"num_object_clones":0,"num_object_copies":16148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8771,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1789}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'8969","reported_seq":9095,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:02.350542+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:41:02.350542+0000","last_peered":"2024-09-05T16:41:02.350542+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:41:02.350542+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8969,"log_dups_size":0,"ondisk_log_size":8969,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34025200,"num_objects":4154,"num_object_clones":0,"num_object_copies":16616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4154,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8966,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1873}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'8745","reported_seq":8843,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:02.411947+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:41:02.411947+0000","last_peered":"2024-09-05T16:41:02.411947+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:41:02.411947+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8745,"log_dups_size":0,"ondisk_log_size":8745,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32881232,"num_objects":4014,"num_object_clones":0,"num_object_copies":16056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4014,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8744,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1776}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'8777","reported_seq":13511,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:41:01.484539+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:41:01.484539+0000","last_peered":"2024-09-05T16:41:01.484539+0000","last_clean":"2024-09-05T16:41:01.484539+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:41:01.484539+0000","last_undegraded":"2024-09-05T16:41:01.484539+0000","last_fullsized":"2024-09-05T16:41:01.484539+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":8777,"log_dups_size":0,"ondisk_log_size":8777,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33021952,"num_objects":4031,"num_object_clones":0,"num_object_copies":16124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4031,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":377,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'9011","reported_seq":9119,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:02.357982+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:41:02.357982+0000","last_peered":"2024-09-05T16:41:02.357982+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:41:02.357982+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9011,"log_dups_size":0,"ondisk_log_size":9011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33707168,"num_objects":4115,"num_object_clones":0,"num_object_copies":16460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4115,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9009,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1781}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'8819","reported_seq":8931,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:02.366875+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:41:02.366875+0000","last_peered":"2024-09-05T16:41:02.366875+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:41:02.366875+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8819,"log_dups_size":0,"ondisk_log_size":8819,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32879776,"num_objects":4014,"num_object_clones":0,"num_object_copies":16056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4014,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8817,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1706}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'8707","reported_seq":8813,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:02.270843+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:41:02.270843+0000","last_peered":"2024-09-05T16:41:02.270843+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:41:02.270843+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8707,"log_dups_size":0,"ondisk_log_size":8707,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33322144,"num_objects":4068,"num_object_clones":0,"num_object_copies":16272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4068,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8705,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1847}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":531436008,"num_objects":64878,"num_object_clones":0,"num_object_copies":259512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30941,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64878,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":125959,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":696,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1063170048,"data_stored":1063170048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":140083,"ondisk_log_size":140083,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084373,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":747156,"kb_used_data":260100,"kb_used_omap":4,"kb_used_meta":487035,"kb_avail":93624684,"statfs":{"total":96636764160,"available":95871676416,"internally_reserved":0,"allocated":266342400,"data_stored":266141126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":498724563},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986250,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":582080,"kb_used_data":261616,"kb_used_omap":4,"kb_used_meta":320443,"kb_avail":93789760,"statfs":{"total":96636764160,"available":96040714240,"internally_reserved":0,"allocated":267894784,"data_stored":267693510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":328134354},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2,7,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673056,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":750880,"kb_used_data":260172,"kb_used_omap":1,"kb_used_meta":490686,"kb_avail":93620960,"statfs":{"total":96636764160,"available":95867863040,"internally_reserved":0,"allocated":266416128,"data_stored":266214854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":502462917},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,3,3,2,0,0,0,0,0,0,0,3],"upper_bound":32768},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182487,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":751348,"kb_used_data":260196,"kb_used_omap":4,"kb_used_meta":491131,"kb_avail":93620492,"statfs":{"total":96636764160,"available":95867383808,"internally_reserved":0,"allocated":266440704,"data_stored":266239430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":502918861},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,1,4,1,4,1,0,0,0,0,0,2],"upper_bound":32768},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":265592832,"data_stored":265592832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":266616832,"data_stored":266616832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":265363456,"data_stored":265363456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":265596928,"data_stored":265596928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:41:03.809 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:41:04.455 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:41:04.582 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:41:04.583 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":261,"stamp":"2024-09-05T16:41:02.954989+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":531436008,"num_objects":64878,"num_object_clones":0,"num_object_copies":259512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30941,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64878,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":125959,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":696,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":140083,"ondisk_log_size":140083,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2831464,"kb_used_data":1042084,"kb_used_omap":14,"kb_used_meta":1789297,"kb_avail":374655896,"statfs":{"total":386547056640,"available":383647637504,"internally_reserved":0,"allocated":1067094016,"data_stored":1066288920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1832240695},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,4,6,15,7,6,1,0,0,0,0,0,5],"upper_bound":32768},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-26869760,"num_objects":-3280,"num_object_clones":0,"num_object_copies":-13120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-210,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3280,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3280,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":210,"num_bytes_recovered":811008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005609"},"pg_stats":[{"pgid":"1.f","version":"37'8792","reported_seq":8896,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:02.429401+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:41:02.429401+0000","last_peered":"2024-09-05T16:41:02.429401+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:41:02.429401+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8792,"log_dups_size":0,"ondisk_log_size":8792,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33685504,"num_objects":4112,"num_object_clones":0,"num_object_copies":16448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4112,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8792,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1876}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'8622","reported_seq":9537,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:02.455128+0000","last_change":"2024-09-05T16:40:28.919189+0000","last_active":"2024-09-05T16:41:02.455128+0000","last_peered":"2024-09-05T16:41:02.455128+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:41:02.455128+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8622,"log_dups_size":0,"ondisk_log_size":8622,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32208032,"num_objects":3932,"num_object_clones":0,"num_object_copies":15728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1849,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3932,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8620,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":422,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2083}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'8736","reported_seq":8858,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:02.675495+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:41:02.675495+0000","last_peered":"2024-09-05T16:41:02.675495+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:41:02.675495+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8736,"log_dups_size":0,"ondisk_log_size":8736,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32897640,"num_objects":4017,"num_object_clones":0,"num_object_copies":16068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4017,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8735,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1785}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'8823","reported_seq":13551,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:41:01.419562+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:41:01.419562+0000","last_peered":"2024-09-05T16:41:01.419562+0000","last_clean":"2024-09-05T16:41:01.419562+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:41:01.419562+0000","last_undegraded":"2024-09-05T16:41:01.419562+0000","last_fullsized":"2024-09-05T16:41:01.419562+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":8823,"log_dups_size":0,"ondisk_log_size":8823,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33677312,"num_objects":4111,"num_object_clones":0,"num_object_copies":16444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4111,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3123,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'8546","reported_seq":8644,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:02.105728+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:41:02.105728+0000","last_peered":"2024-09-05T16:41:02.105728+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:41:02.105728+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8546,"log_dups_size":0,"ondisk_log_size":8546,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33166496,"num_objects":4049,"num_object_clones":0,"num_object_copies":16196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4049,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8544,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1904}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'8730","reported_seq":8844,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:02.312075+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:41:02.312075+0000","last_peered":"2024-09-05T16:41:02.312075+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:41:02.312075+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8730,"log_dups_size":0,"ondisk_log_size":8730,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33112064,"num_objects":4042,"num_object_clones":0,"num_object_copies":16168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4042,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8730,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1823}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'8577","reported_seq":8682,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:02.322447+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:41:02.322447+0000","last_peered":"2024-09-05T16:41:02.322447+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:41:02.322447+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8577,"log_dups_size":0,"ondisk_log_size":8577,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33461408,"num_objects":4085,"num_object_clones":0,"num_object_copies":16340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4085,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8575,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1953}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'8734","reported_seq":8819,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:02.265041+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:41:02.265041+0000","last_peered":"2024-09-05T16:41:02.265041+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:41:02.265041+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8734,"log_dups_size":0,"ondisk_log_size":8734,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33133728,"num_objects":4045,"num_object_clones":0,"num_object_copies":16180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4045,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8732,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1813}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'8722","reported_seq":8822,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:02.271446+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:41:02.271446+0000","last_peered":"2024-09-05T16:41:02.271446+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:41:02.271446+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8722,"log_dups_size":0,"ondisk_log_size":8722,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33188160,"num_objects":4052,"num_object_clones":0,"num_object_copies":16208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4052,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8719,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1808}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'8773","reported_seq":8888,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:02.555472+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:41:02.555472+0000","last_peered":"2024-09-05T16:41:02.555472+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:41:02.555472+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8773,"log_dups_size":0,"ondisk_log_size":8773,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33068192,"num_objects":4037,"num_object_clones":0,"num_object_copies":16148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8771,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1789}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'8969","reported_seq":9095,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:02.350542+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:41:02.350542+0000","last_peered":"2024-09-05T16:41:02.350542+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:41:02.350542+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8969,"log_dups_size":0,"ondisk_log_size":8969,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34025200,"num_objects":4154,"num_object_clones":0,"num_object_copies":16616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4154,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8966,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1873}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'8745","reported_seq":8843,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:02.411947+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:41:02.411947+0000","last_peered":"2024-09-05T16:41:02.411947+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:41:02.411947+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8745,"log_dups_size":0,"ondisk_log_size":8745,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32881232,"num_objects":4014,"num_object_clones":0,"num_object_copies":16056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4014,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8744,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1776}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'8777","reported_seq":13511,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:41:01.484539+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:41:01.484539+0000","last_peered":"2024-09-05T16:41:01.484539+0000","last_clean":"2024-09-05T16:41:01.484539+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:41:01.484539+0000","last_undegraded":"2024-09-05T16:41:01.484539+0000","last_fullsized":"2024-09-05T16:41:01.484539+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":8777,"log_dups_size":0,"ondisk_log_size":8777,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33021952,"num_objects":4031,"num_object_clones":0,"num_object_copies":16124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4031,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":377,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'9011","reported_seq":9119,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:02.357982+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:41:02.357982+0000","last_peered":"2024-09-05T16:41:02.357982+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:41:02.357982+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9011,"log_dups_size":0,"ondisk_log_size":9011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33707168,"num_objects":4115,"num_object_clones":0,"num_object_copies":16460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4115,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9009,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1781}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'8819","reported_seq":8931,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:02.366875+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:41:02.366875+0000","last_peered":"2024-09-05T16:41:02.366875+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:41:02.366875+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8819,"log_dups_size":0,"ondisk_log_size":8819,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32879776,"num_objects":4014,"num_object_clones":0,"num_object_copies":16056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4014,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8817,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1706}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'8707","reported_seq":8813,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:02.270843+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:41:02.270843+0000","last_peered":"2024-09-05T16:41:02.270843+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:41:02.270843+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8707,"log_dups_size":0,"ondisk_log_size":8707,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33322144,"num_objects":4068,"num_object_clones":0,"num_object_copies":16272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4068,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8705,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1847}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":531436008,"num_objects":64878,"num_object_clones":0,"num_object_copies":259512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30941,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64878,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":125959,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":696,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1063170048,"data_stored":1063170048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":140083,"ondisk_log_size":140083,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084373,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":747156,"kb_used_data":260100,"kb_used_omap":4,"kb_used_meta":487035,"kb_avail":93624684,"statfs":{"total":96636764160,"available":95871676416,"internally_reserved":0,"allocated":266342400,"data_stored":266141126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":498724563},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986250,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":582080,"kb_used_data":261616,"kb_used_omap":4,"kb_used_meta":320443,"kb_avail":93789760,"statfs":{"total":96636764160,"available":96040714240,"internally_reserved":0,"allocated":267894784,"data_stored":267693510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":328134354},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2,7,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673056,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":750880,"kb_used_data":260172,"kb_used_omap":1,"kb_used_meta":490686,"kb_avail":93620960,"statfs":{"total":96636764160,"available":95867863040,"internally_reserved":0,"allocated":266416128,"data_stored":266214854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":502462917},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,3,3,2,0,0,0,0,0,0,0,3],"upper_bound":32768},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182487,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":751348,"kb_used_data":260196,"kb_used_omap":4,"kb_used_meta":491131,"kb_avail":93620492,"statfs":{"total":96636764160,"available":95867383808,"internally_reserved":0,"allocated":266440704,"data_stored":266239430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":502918861},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,1,4,1,4,1,0,0,0,0,0,2],"upper_bound":32768},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":265592832,"data_stored":265592832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":266616832,"data_stored":266616832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":265363456,"data_stored":265363456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":265596928,"data_stored":265596928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:41:04.584 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-05T16:41:04.584 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'8792", 'reported_seq': 8896, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:02.429401+0000', 'last_change': '2024-09-05T16:34:56.833497+0000', 'last_active': '2024-09-05T16:41:02.429401+0000', 'last_peered': '2024-09-05T16:41:02.429401+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.828289+0000', 'last_became_peered': '2024-09-05T16:34:56.828289+0000', 'last_unstale': '2024-09-05T16:41:02.429401+0000', 'last_undegraded': '2024-09-05T16:34:56.612160+0000', 'last_fullsized': '2024-09-05T16:34:56.557632+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8792, 'log_dups_size': 0, 'ondisk_log_size': 8792, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33685504, 'num_objects': 4112, 'num_object_clones': 0, 'num_object_copies': 16448, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2236, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4112, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8792, 'num_write_kb': 51616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1876}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:41:04.584 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-05T16:41:04.584 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "37'8622", 'reported_seq': 9537, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:02.455128+0000', 'last_change': '2024-09-05T16:40:28.919189+0000', 'last_active': '2024-09-05T16:41:02.455128+0000', 'last_peered': '2024-09-05T16:41:02.455128+0000', 'last_clean': '2024-09-05T16:33:53.200762+0000', 'last_became_active': '2024-09-05T16:34:56.820292+0000', 'last_became_peered': '2024-09-05T16:34:56.820292+0000', 'last_unstale': '2024-09-05T16:41:02.455128+0000', 'last_undegraded': '2024-09-05T16:34:56.605776+0000', 'last_fullsized': '2024-09-05T16:34:56.559383+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8622, 'log_dups_size': 0, 'ondisk_log_size': 8622, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32208032, 'num_objects': 3932, 'num_object_clones': 0, 'num_object_copies': 15728, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1849, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3932, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8620, 'num_write_kb': 50212, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 422, 'num_bytes_recovered': 2613248, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2083}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:41:04.584 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:41:04.584 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'8736", 'reported_seq': 8858, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:02.675495+0000', 'last_change': '2024-09-05T16:34:56.867900+0000', 'last_active': '2024-09-05T16:41:02.675495+0000', 'last_peered': '2024-09-05T16:41:02.675495+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:41:02.675495+0000', 'last_undegraded': '2024-09-05T16:34:56.590972+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8736, 'log_dups_size': 0, 'ondisk_log_size': 8736, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32897640, 'num_objects': 4017, 'num_object_clones': 0, 'num_object_copies': 16068, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2233, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4017, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 8735, 'num_write_kb': 50999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1785}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:41:04.584 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:41:04.584 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'8546", 'reported_seq': 8644, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:02.105728+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:41:02.105728+0000', 'last_peered': '2024-09-05T16:41:02.105728+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:41:02.105728+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8546, 'log_dups_size': 0, 'ondisk_log_size': 8546, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33166496, 'num_objects': 4049, 'num_object_clones': 0, 'num_object_copies': 16196, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2147, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4049, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8544, 'num_write_kb': 50373, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1904}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:41:04.584 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:41:04.585 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'8730", 'reported_seq': 8844, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:02.312075+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:41:02.312075+0000', 'last_peered': '2024-09-05T16:41:02.312075+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:41:02.312075+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8730, 'log_dups_size': 0, 'ondisk_log_size': 8730, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33112064, 'num_objects': 4042, 'num_object_clones': 0, 'num_object_copies': 16168, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4042, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8730, 'num_write_kb': 51088, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1823}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:41:04.585 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:41:04.585 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'8577", 'reported_seq': 8682, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:02.322447+0000', 'last_change': '2024-09-05T16:34:56.903436+0000', 'last_active': '2024-09-05T16:41:02.322447+0000', 'last_peered': '2024-09-05T16:41:02.322447+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:41:02.322447+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8577, 'log_dups_size': 0, 'ondisk_log_size': 8577, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33461408, 'num_objects': 4085, 'num_object_clones': 0, 'num_object_copies': 16340, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2134, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4085, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8575, 'num_write_kb': 50636, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1953}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:41:04.585 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-05T16:41:04.585 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "37'8734", 'reported_seq': 8819, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:02.265041+0000', 'last_change': '2024-09-05T16:34:56.883319+0000', 'last_active': '2024-09-05T16:41:02.265041+0000', 'last_peered': '2024-09-05T16:41:02.265041+0000', 'last_clean': '2024-09-05T16:33:53.505305+0000', 'last_became_active': '2024-09-05T16:34:56.873389+0000', 'last_became_peered': '2024-09-05T16:34:56.873389+0000', 'last_unstale': '2024-09-05T16:41:02.265041+0000', 'last_undegraded': '2024-09-05T16:34:56.602567+0000', 'last_fullsized': '2024-09-05T16:34:56.559958+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8734, 'log_dups_size': 0, 'ondisk_log_size': 8734, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33133728, 'num_objects': 4045, 'num_object_clones': 0, 'num_object_copies': 16180, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2234, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4045, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8732, 'num_write_kb': 51108, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1813}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:41:04.585 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:41:04.585 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'8722", 'reported_seq': 8822, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:02.271446+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:41:02.271446+0000', 'last_peered': '2024-09-05T16:41:02.271446+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:41:02.271446+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8722, 'log_dups_size': 0, 'ondisk_log_size': 8722, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33188160, 'num_objects': 4052, 'num_object_clones': 0, 'num_object_copies': 16208, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4052, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8719, 'num_write_kb': 51081, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1808}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:41:04.586 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:41:04.586 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'8773", 'reported_seq': 8888, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:02.555472+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:41:02.555472+0000', 'last_peered': '2024-09-05T16:41:02.555472+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:41:02.555472+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8773, 'log_dups_size': 0, 'ondisk_log_size': 8773, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33068192, 'num_objects': 4037, 'num_object_clones': 0, 'num_object_copies': 16148, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4037, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8771, 'num_write_kb': 51230, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 16, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1789}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:41:04.586 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:41:04.586 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'8969", 'reported_seq': 9095, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:02.350542+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:41:02.350542+0000', 'last_peered': '2024-09-05T16:41:02.350542+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:41:02.350542+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8969, 'log_dups_size': 0, 'ondisk_log_size': 8969, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34025200, 'num_objects': 4154, 'num_object_clones': 0, 'num_object_copies': 16616, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2284, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4154, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8966, 'num_write_kb': 52476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1873}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:41:04.586 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:41:04.586 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'8745", 'reported_seq': 8843, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:02.411947+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:41:02.411947+0000', 'last_peered': '2024-09-05T16:41:02.411947+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:41:02.411947+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8745, 'log_dups_size': 0, 'ondisk_log_size': 8745, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32881232, 'num_objects': 4014, 'num_object_clones': 0, 'num_object_copies': 16056, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4014, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8744, 'num_write_kb': 51031, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1776}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:41:04.586 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:41:04.586 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'9011", 'reported_seq': 9119, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:02.357982+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:41:02.357982+0000', 'last_peered': '2024-09-05T16:41:02.357982+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:41:02.357982+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9011, 'log_dups_size': 0, 'ondisk_log_size': 9011, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33707168, 'num_objects': 4115, 'num_object_clones': 0, 'num_object_copies': 16460, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4115, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9009, 'num_write_kb': 52500, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1781}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:41:04.586 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:41:04.586 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'8819", 'reported_seq': 8931, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:02.366875+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:41:02.366875+0000', 'last_peered': '2024-09-05T16:41:02.366875+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:41:02.366875+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8819, 'log_dups_size': 0, 'ondisk_log_size': 8819, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32879776, 'num_objects': 4014, 'num_object_clones': 0, 'num_object_copies': 16056, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2310, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4014, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8817, 'num_write_kb': 51325, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1706}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:41:04.586 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:41:04.587 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'8707", 'reported_seq': 8813, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:02.270843+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:41:02.270843+0000', 'last_peered': '2024-09-05T16:41:02.270843+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:41:02.270843+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8707, 'log_dups_size': 0, 'ondisk_log_size': 8707, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33322144, 'num_objects': 4068, 'num_object_clones': 0, 'num_object_copies': 16272, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2223, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4068, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8705, 'num_write_kb': 51092, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1847}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:41:04.587 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:41:05.343 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:41:05.343 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 30941/259512 objects degraded (11.923%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":468,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":64878,"data_bytes":531436008,"bytes_used":2899419136,"bytes_avail":383647637504,"bytes_total":386547056640,"degraded_objects":30941,"degraded_total":259512,"degraded_ratio":0.11922762723881747,"recovering_objects_per_sec":10,"recovering_bytes_per_sec":26611,"recovering_keys_per_sec":0,"num_objects_recovered":128,"num_bytes_recovered":319488,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":180},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:41:05.365 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:41:05.365 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:41:06.149 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:41:06.150 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:41:06.177 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":262,"stamp":"2024-09-05T16:41:04.955795+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":531436008,"num_objects":64878,"num_object_clones":0,"num_object_copies":259512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30941,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64878,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":125959,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":696,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":140083,"ondisk_log_size":140083,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2831464,"kb_used_data":1042084,"kb_used_omap":14,"kb_used_meta":1789297,"kb_avail":374655896,"statfs":{"total":386547056640,"available":383647637504,"internally_reserved":0,"allocated":1067094016,"data_stored":1066288920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1832240695},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,4,6,15,7,6,1,0,0,0,0,0,5],"upper_bound":32768},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17784832,"num_objects":-2171,"num_object_clones":0,"num_object_copies":-8684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-128,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2171,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2171,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":128,"num_bytes_recovered":319488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005647"},"pg_stats":[{"pgid":"1.f","version":"37'8792","reported_seq":8896,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:02.429401+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:41:02.429401+0000","last_peered":"2024-09-05T16:41:02.429401+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:41:02.429401+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8792,"log_dups_size":0,"ondisk_log_size":8792,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33685504,"num_objects":4112,"num_object_clones":0,"num_object_copies":16448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4112,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8792,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1876}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'8622","reported_seq":9537,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:02.455128+0000","last_change":"2024-09-05T16:40:28.919189+0000","last_active":"2024-09-05T16:41:02.455128+0000","last_peered":"2024-09-05T16:41:02.455128+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:41:02.455128+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8622,"log_dups_size":0,"ondisk_log_size":8622,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32208032,"num_objects":3932,"num_object_clones":0,"num_object_copies":15728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1849,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3932,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8620,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":422,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2083}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'8736","reported_seq":8858,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:02.675495+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:41:02.675495+0000","last_peered":"2024-09-05T16:41:02.675495+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:41:02.675495+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8736,"log_dups_size":0,"ondisk_log_size":8736,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32897640,"num_objects":4017,"num_object_clones":0,"num_object_copies":16068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4017,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8735,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1785}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'8823","reported_seq":13551,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:41:01.419562+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:41:01.419562+0000","last_peered":"2024-09-05T16:41:01.419562+0000","last_clean":"2024-09-05T16:41:01.419562+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:41:01.419562+0000","last_undegraded":"2024-09-05T16:41:01.419562+0000","last_fullsized":"2024-09-05T16:41:01.419562+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":8823,"log_dups_size":0,"ondisk_log_size":8823,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33677312,"num_objects":4111,"num_object_clones":0,"num_object_copies":16444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4111,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3123,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'8546","reported_seq":8644,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:02.105728+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:41:02.105728+0000","last_peered":"2024-09-05T16:41:02.105728+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:41:02.105728+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8546,"log_dups_size":0,"ondisk_log_size":8546,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33166496,"num_objects":4049,"num_object_clones":0,"num_object_copies":16196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4049,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8544,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1904}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'8730","reported_seq":8844,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:02.312075+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:41:02.312075+0000","last_peered":"2024-09-05T16:41:02.312075+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:41:02.312075+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8730,"log_dups_size":0,"ondisk_log_size":8730,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33112064,"num_objects":4042,"num_object_clones":0,"num_object_copies":16168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4042,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8730,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1823}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'8577","reported_seq":8682,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:02.322447+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:41:02.322447+0000","last_peered":"2024-09-05T16:41:02.322447+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:41:02.322447+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8577,"log_dups_size":0,"ondisk_log_size":8577,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33461408,"num_objects":4085,"num_object_clones":0,"num_object_copies":16340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4085,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8575,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1953}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'8734","reported_seq":8819,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:02.265041+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:41:02.265041+0000","last_peered":"2024-09-05T16:41:02.265041+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:41:02.265041+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8734,"log_dups_size":0,"ondisk_log_size":8734,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33133728,"num_objects":4045,"num_object_clones":0,"num_object_copies":16180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4045,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8732,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1813}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'8722","reported_seq":8822,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:02.271446+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:41:02.271446+0000","last_peered":"2024-09-05T16:41:02.271446+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:41:02.271446+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8722,"log_dups_size":0,"ondisk_log_size":8722,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33188160,"num_objects":4052,"num_object_clones":0,"num_object_copies":16208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4052,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8719,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1808}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'8773","reported_seq":8888,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:02.555472+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:41:02.555472+0000","last_peered":"2024-09-05T16:41:02.555472+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:41:02.555472+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8773,"log_dups_size":0,"ondisk_log_size":8773,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33068192,"num_objects":4037,"num_object_clones":0,"num_object_copies":16148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8771,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1789}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'8969","reported_seq":9095,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:02.350542+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:41:02.350542+0000","last_peered":"2024-09-05T16:41:02.350542+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:41:02.350542+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8969,"log_dups_size":0,"ondisk_log_size":8969,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34025200,"num_objects":4154,"num_object_clones":0,"num_object_copies":16616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4154,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8966,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1873}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'8745","reported_seq":8843,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:02.411947+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:41:02.411947+0000","last_peered":"2024-09-05T16:41:02.411947+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:41:02.411947+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8745,"log_dups_size":0,"ondisk_log_size":8745,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32881232,"num_objects":4014,"num_object_clones":0,"num_object_copies":16056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4014,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8744,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1776}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'8777","reported_seq":13511,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:41:01.484539+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:41:01.484539+0000","last_peered":"2024-09-05T16:41:01.484539+0000","last_clean":"2024-09-05T16:41:01.484539+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:41:01.484539+0000","last_undegraded":"2024-09-05T16:41:01.484539+0000","last_fullsized":"2024-09-05T16:41:01.484539+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":8777,"log_dups_size":0,"ondisk_log_size":8777,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33021952,"num_objects":4031,"num_object_clones":0,"num_object_copies":16124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4031,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":377,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'9011","reported_seq":9119,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:02.357982+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:41:02.357982+0000","last_peered":"2024-09-05T16:41:02.357982+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:41:02.357982+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9011,"log_dups_size":0,"ondisk_log_size":9011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33707168,"num_objects":4115,"num_object_clones":0,"num_object_copies":16460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4115,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9009,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1781}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'8819","reported_seq":8931,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:02.366875+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:41:02.366875+0000","last_peered":"2024-09-05T16:41:02.366875+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:41:02.366875+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8819,"log_dups_size":0,"ondisk_log_size":8819,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32879776,"num_objects":4014,"num_object_clones":0,"num_object_copies":16056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4014,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8817,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1706}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'8707","reported_seq":8813,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:02.270843+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:41:02.270843+0000","last_peered":"2024-09-05T16:41:02.270843+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:41:02.270843+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8707,"log_dups_size":0,"ondisk_log_size":8707,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33322144,"num_objects":4068,"num_object_clones":0,"num_object_copies":16272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4068,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8705,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1847}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":531436008,"num_objects":64878,"num_object_clones":0,"num_object_copies":259512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30941,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64878,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":125959,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":696,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1063170048,"data_stored":1063170048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":140083,"ondisk_log_size":140083,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084373,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":747156,"kb_used_data":260100,"kb_used_omap":4,"kb_used_meta":487035,"kb_avail":93624684,"statfs":{"total":96636764160,"available":95871676416,"internally_reserved":0,"allocated":266342400,"data_stored":266141126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":498724563},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986250,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":582080,"kb_used_data":261616,"kb_used_omap":4,"kb_used_meta":320443,"kb_avail":93789760,"statfs":{"total":96636764160,"available":96040714240,"internally_reserved":0,"allocated":267894784,"data_stored":267693510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":328134354},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2,7,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673056,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":750880,"kb_used_data":260172,"kb_used_omap":1,"kb_used_meta":490686,"kb_avail":93620960,"statfs":{"total":96636764160,"available":95867863040,"internally_reserved":0,"allocated":266416128,"data_stored":266214854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":502462917},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,3,3,2,0,0,0,0,0,0,0,3],"upper_bound":32768},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182487,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":751348,"kb_used_data":260196,"kb_used_omap":4,"kb_used_meta":491131,"kb_avail":93620492,"statfs":{"total":96636764160,"available":95867383808,"internally_reserved":0,"allocated":266440704,"data_stored":266239430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":502918861},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,1,4,1,4,1,0,0,0,0,0,2],"upper_bound":32768},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":265592832,"data_stored":265592832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":266616832,"data_stored":266616832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":265363456,"data_stored":265363456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":265596928,"data_stored":265596928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:41:09.180 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:41:10.850 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:41:10.850 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:41:10.872 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":264,"stamp":"2024-09-05T16:41:08.957331+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":522777064,"num_objects":63821,"num_object_clones":0,"num_object_copies":255284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30880,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63821,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":127016,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":757,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":141140,"ondisk_log_size":141140,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2815272,"kb_used_data":1024460,"kb_used_omap":14,"kb_used_meta":1790705,"kb_avail":374672088,"statfs":{"total":386547056640,"available":383664218112,"internally_reserved":0,"allocated":1049047040,"data_stored":1048241944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1833682487},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,14,14,17,7,0,0,0,0,0,0,3,2],"upper_bound":65536},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25133056,"num_objects":-3068,"num_object_clones":0,"num_object_copies":-12272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-189,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3068,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3068,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":189,"num_bytes_recovered":319488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005364"},"pg_stats":[{"pgid":"1.f","version":"37'8844","reported_seq":8948,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:07.369643+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:41:07.369643+0000","last_peered":"2024-09-05T16:41:07.369643+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:41:07.369643+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8844,"log_dups_size":0,"ondisk_log_size":8844,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33259520,"num_objects":4060,"num_object_clones":0,"num_object_copies":16240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8844,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1824}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'8680","reported_seq":9711,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:07.378816+0000","last_change":"2024-09-05T16:40:28.919189+0000","last_active":"2024-09-05T16:41:07.378816+0000","last_peered":"2024-09-05T16:41:07.378816+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:41:07.378816+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8680,"log_dups_size":0,"ondisk_log_size":8680,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31732896,"num_objects":3874,"num_object_clones":0,"num_object_copies":15496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1791,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3874,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8678,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":480,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2083}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'8798","reported_seq":8920,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:07.616556+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:41:07.616556+0000","last_peered":"2024-09-05T16:41:07.616556+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:41:07.616556+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8798,"log_dups_size":0,"ondisk_log_size":8798,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32389736,"num_objects":3955,"num_object_clones":0,"num_object_copies":15820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3955,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8797,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1723}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'8879","reported_seq":13606,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:41:06.383136+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:41:06.383136+0000","last_peered":"2024-09-05T16:41:06.383136+0000","last_clean":"2024-09-05T16:41:06.383136+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:41:06.383136+0000","last_undegraded":"2024-09-05T16:41:06.383136+0000","last_fullsized":"2024-09-05T16:41:06.383136+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":8879,"log_dups_size":0,"ondisk_log_size":8879,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33218560,"num_objects":4055,"num_object_clones":0,"num_object_copies":16220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4055,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3179,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'8612","reported_seq":8709,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:07.425913+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:41:07.425913+0000","last_peered":"2024-09-05T16:41:07.425913+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:41:07.425913+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8612,"log_dups_size":0,"ondisk_log_size":8612,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32625824,"num_objects":3983,"num_object_clones":0,"num_object_copies":15932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3983,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8610,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1838}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'8793","reported_seq":8907,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:07.383907+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:41:07.383907+0000","last_peered":"2024-09-05T16:41:07.383907+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:41:07.383907+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8793,"log_dups_size":0,"ondisk_log_size":8793,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32595968,"num_objects":3979,"num_object_clones":0,"num_object_copies":15916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3979,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8793,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1760}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'8643","reported_seq":8748,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:07.378184+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:41:07.378184+0000","last_peered":"2024-09-05T16:41:07.378184+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:41:07.378184+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8643,"log_dups_size":0,"ondisk_log_size":8643,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32920736,"num_objects":4019,"num_object_clones":0,"num_object_copies":16076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4019,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8641,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1887}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'8806","reported_seq":8894,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:07.446858+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:41:07.446858+0000","last_peered":"2024-09-05T16:41:07.446858+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:41:07.446858+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8806,"log_dups_size":0,"ondisk_log_size":8806,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32543904,"num_objects":3973,"num_object_clones":0,"num_object_copies":15892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3973,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8804,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1741}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'8792","reported_seq":8892,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:07.393155+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:41:07.393155+0000","last_peered":"2024-09-05T16:41:07.393155+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:41:07.393155+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8792,"log_dups_size":0,"ondisk_log_size":8792,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32614720,"num_objects":3982,"num_object_clones":0,"num_object_copies":15928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3982,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8789,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1738}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'8839","reported_seq":8960,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:07.633549+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:41:07.633549+0000","last_peered":"2024-09-05T16:41:07.633549+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:41:07.633549+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8839,"log_dups_size":0,"ondisk_log_size":8839,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32527520,"num_objects":3971,"num_object_clones":0,"num_object_copies":15884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8837,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1726}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'9039","reported_seq":9165,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:07.315140+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:41:07.315140+0000","last_peered":"2024-09-05T16:41:07.315140+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:41:07.315140+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9039,"log_dups_size":0,"ondisk_log_size":9039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33451760,"num_objects":4084,"num_object_clones":0,"num_object_copies":16336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4084,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9036,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1803}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'8819","reported_seq":8917,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:07.447581+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:41:07.447581+0000","last_peered":"2024-09-05T16:41:07.447581+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:41:07.447581+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8819,"log_dups_size":0,"ondisk_log_size":8819,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32275024,"num_objects":3940,"num_object_clones":0,"num_object_copies":15760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3940,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8818,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1702}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'8839","reported_seq":13571,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:41:06.445978+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:41:06.445978+0000","last_peered":"2024-09-05T16:41:06.445978+0000","last_clean":"2024-09-05T16:41:06.445978+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:41:06.445978+0000","last_undegraded":"2024-09-05T16:41:06.445978+0000","last_fullsized":"2024-09-05T16:41:06.445978+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":8839,"log_dups_size":0,"ondisk_log_size":8839,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32514048,"num_objects":3969,"num_object_clones":0,"num_object_copies":15876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3969,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":439,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'9088","reported_seq":9196,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:07.353929+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:41:07.353929+0000","last_peered":"2024-09-05T16:41:07.353929+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:41:07.353929+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9088,"log_dups_size":0,"ondisk_log_size":9088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33076384,"num_objects":4038,"num_object_clones":0,"num_object_copies":16152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4038,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9086,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1704}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'8886","reported_seq":8998,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:07.670621+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:41:07.670621+0000","last_peered":"2024-09-05T16:41:07.670621+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:41:07.670621+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8886,"log_dups_size":0,"ondisk_log_size":8886,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32330912,"num_objects":3947,"num_object_clones":0,"num_object_copies":15788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3947,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8884,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1639}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'8783","reported_seq":8889,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:07.448148+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:41:07.448148+0000","last_peered":"2024-09-05T16:41:07.448148+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:41:07.448148+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8783,"log_dups_size":0,"ondisk_log_size":8783,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32699552,"num_objects":3992,"num_object_clones":0,"num_object_copies":15968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3992,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8781,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1771}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":522777064,"num_objects":63821,"num_object_clones":0,"num_object_copies":255284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30880,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63821,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":127016,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":757,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1045815296,"data_stored":1045815296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":141140,"ondisk_log_size":141140,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084374,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":747128,"kb_used_data":255912,"kb_used_omap":4,"kb_used_meta":491195,"kb_avail":93624712,"statfs":{"total":96636764160,"available":95871705088,"internally_reserved":0,"allocated":262053888,"data_stored":261852614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":502984403},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,2,5,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986251,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":599212,"kb_used_data":256532,"kb_used_omap":4,"kb_used_meta":342651,"kb_avail":93772628,"statfs":{"total":96636764160,"available":96023171072,"internally_reserved":0,"allocated":262688768,"data_stored":262487494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":350875346},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,9,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673057,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":734216,"kb_used_data":255984,"kb_used_omap":1,"kb_used_meta":478206,"kb_avail":93637624,"statfs":{"total":96636764160,"available":95884926976,"internally_reserved":0,"allocated":262127616,"data_stored":261926342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":489683397},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,0,3,0,0,0,0,0,0,0,3],"upper_bound":32768},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182488,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":734716,"kb_used_data":256032,"kb_used_omap":4,"kb_used_meta":478651,"kb_avail":93637124,"statfs":{"total":96636764160,"available":95884414976,"internally_reserved":0,"allocated":262176768,"data_stored":261975494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":490139341},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,7,3,3,2,0,0,0,0,0,0,0,2],"upper_bound":65536},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":261283840,"data_stored":261283840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":262262784,"data_stored":262262784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":261001216,"data_stored":261001216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":261267456,"data_stored":261267456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:41:10.873 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:41:11.545 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:41:11.546 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:41:11.572 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":265,"stamp":"2024-09-05T16:41:10.958142+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":522777064,"num_objects":63821,"num_object_clones":0,"num_object_copies":255284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30880,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63821,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":127016,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":757,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":141140,"ondisk_log_size":141140,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2815272,"kb_used_data":1024460,"kb_used_omap":14,"kb_used_meta":1790705,"kb_avail":374672088,"statfs":{"total":386547056640,"available":383664218112,"internally_reserved":0,"allocated":1049047040,"data_stored":1048241944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1833682487},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,14,14,17,7,0,0,0,0,0,0,3,2],"upper_bound":65536},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17383424,"num_objects":-2122,"num_object_clones":0,"num_object_copies":-8488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-128,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2122,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2122,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":128,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005294"},"pg_stats":[{"pgid":"1.f","version":"37'8844","reported_seq":8948,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:07.369643+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:41:07.369643+0000","last_peered":"2024-09-05T16:41:07.369643+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:41:07.369643+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8844,"log_dups_size":0,"ondisk_log_size":8844,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33259520,"num_objects":4060,"num_object_clones":0,"num_object_copies":16240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8844,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1824}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'8680","reported_seq":9711,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:07.378816+0000","last_change":"2024-09-05T16:40:28.919189+0000","last_active":"2024-09-05T16:41:07.378816+0000","last_peered":"2024-09-05T16:41:07.378816+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:41:07.378816+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8680,"log_dups_size":0,"ondisk_log_size":8680,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31732896,"num_objects":3874,"num_object_clones":0,"num_object_copies":15496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1791,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3874,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8678,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":480,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2083}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'8798","reported_seq":8920,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:07.616556+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:41:07.616556+0000","last_peered":"2024-09-05T16:41:07.616556+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:41:07.616556+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8798,"log_dups_size":0,"ondisk_log_size":8798,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32389736,"num_objects":3955,"num_object_clones":0,"num_object_copies":15820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3955,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8797,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1723}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'8879","reported_seq":13606,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:41:06.383136+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:41:06.383136+0000","last_peered":"2024-09-05T16:41:06.383136+0000","last_clean":"2024-09-05T16:41:06.383136+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:41:06.383136+0000","last_undegraded":"2024-09-05T16:41:06.383136+0000","last_fullsized":"2024-09-05T16:41:06.383136+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":8879,"log_dups_size":0,"ondisk_log_size":8879,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33218560,"num_objects":4055,"num_object_clones":0,"num_object_copies":16220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4055,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3179,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'8612","reported_seq":8709,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:07.425913+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:41:07.425913+0000","last_peered":"2024-09-05T16:41:07.425913+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:41:07.425913+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8612,"log_dups_size":0,"ondisk_log_size":8612,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32625824,"num_objects":3983,"num_object_clones":0,"num_object_copies":15932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3983,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8610,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1838}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'8793","reported_seq":8907,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:07.383907+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:41:07.383907+0000","last_peered":"2024-09-05T16:41:07.383907+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:41:07.383907+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8793,"log_dups_size":0,"ondisk_log_size":8793,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32595968,"num_objects":3979,"num_object_clones":0,"num_object_copies":15916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3979,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8793,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1760}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'8643","reported_seq":8748,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:07.378184+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:41:07.378184+0000","last_peered":"2024-09-05T16:41:07.378184+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:41:07.378184+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8643,"log_dups_size":0,"ondisk_log_size":8643,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32920736,"num_objects":4019,"num_object_clones":0,"num_object_copies":16076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4019,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8641,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1887}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'8806","reported_seq":8894,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:07.446858+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:41:07.446858+0000","last_peered":"2024-09-05T16:41:07.446858+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:41:07.446858+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8806,"log_dups_size":0,"ondisk_log_size":8806,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32543904,"num_objects":3973,"num_object_clones":0,"num_object_copies":15892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3973,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8804,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1741}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'8792","reported_seq":8892,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:07.393155+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:41:07.393155+0000","last_peered":"2024-09-05T16:41:07.393155+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:41:07.393155+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8792,"log_dups_size":0,"ondisk_log_size":8792,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32614720,"num_objects":3982,"num_object_clones":0,"num_object_copies":15928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3982,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8789,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1738}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'8839","reported_seq":8960,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:07.633549+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:41:07.633549+0000","last_peered":"2024-09-05T16:41:07.633549+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:41:07.633549+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8839,"log_dups_size":0,"ondisk_log_size":8839,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32527520,"num_objects":3971,"num_object_clones":0,"num_object_copies":15884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8837,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1726}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'9039","reported_seq":9165,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:07.315140+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:41:07.315140+0000","last_peered":"2024-09-05T16:41:07.315140+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:41:07.315140+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9039,"log_dups_size":0,"ondisk_log_size":9039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33451760,"num_objects":4084,"num_object_clones":0,"num_object_copies":16336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4084,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9036,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1803}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'8819","reported_seq":8917,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:07.447581+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:41:07.447581+0000","last_peered":"2024-09-05T16:41:07.447581+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:41:07.447581+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8819,"log_dups_size":0,"ondisk_log_size":8819,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32275024,"num_objects":3940,"num_object_clones":0,"num_object_copies":15760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3940,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8818,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1702}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'8839","reported_seq":13571,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:41:06.445978+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:41:06.445978+0000","last_peered":"2024-09-05T16:41:06.445978+0000","last_clean":"2024-09-05T16:41:06.445978+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:41:06.445978+0000","last_undegraded":"2024-09-05T16:41:06.445978+0000","last_fullsized":"2024-09-05T16:41:06.445978+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":8839,"log_dups_size":0,"ondisk_log_size":8839,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32514048,"num_objects":3969,"num_object_clones":0,"num_object_copies":15876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3969,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":439,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'9088","reported_seq":9196,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:07.353929+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:41:07.353929+0000","last_peered":"2024-09-05T16:41:07.353929+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:41:07.353929+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9088,"log_dups_size":0,"ondisk_log_size":9088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33076384,"num_objects":4038,"num_object_clones":0,"num_object_copies":16152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4038,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9086,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1704}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'8886","reported_seq":8998,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:07.670621+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:41:07.670621+0000","last_peered":"2024-09-05T16:41:07.670621+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:41:07.670621+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8886,"log_dups_size":0,"ondisk_log_size":8886,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32330912,"num_objects":3947,"num_object_clones":0,"num_object_copies":15788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3947,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8884,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1639}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'8783","reported_seq":8889,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:07.448148+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:41:07.448148+0000","last_peered":"2024-09-05T16:41:07.448148+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:41:07.448148+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8783,"log_dups_size":0,"ondisk_log_size":8783,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32699552,"num_objects":3992,"num_object_clones":0,"num_object_copies":15968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3992,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8781,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1771}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":522777064,"num_objects":63821,"num_object_clones":0,"num_object_copies":255284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30880,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63821,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":127016,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":757,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1045815296,"data_stored":1045815296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":141140,"ondisk_log_size":141140,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084374,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":747128,"kb_used_data":255912,"kb_used_omap":4,"kb_used_meta":491195,"kb_avail":93624712,"statfs":{"total":96636764160,"available":95871705088,"internally_reserved":0,"allocated":262053888,"data_stored":261852614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":502984403},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,2,5,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986251,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":599212,"kb_used_data":256532,"kb_used_omap":4,"kb_used_meta":342651,"kb_avail":93772628,"statfs":{"total":96636764160,"available":96023171072,"internally_reserved":0,"allocated":262688768,"data_stored":262487494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":350875346},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,9,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673057,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":734216,"kb_used_data":255984,"kb_used_omap":1,"kb_used_meta":478206,"kb_avail":93637624,"statfs":{"total":96636764160,"available":95884926976,"internally_reserved":0,"allocated":262127616,"data_stored":261926342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":489683397},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,0,3,0,0,0,0,0,0,0,3],"upper_bound":32768},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182488,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":734716,"kb_used_data":256032,"kb_used_omap":4,"kb_used_meta":478651,"kb_avail":93637124,"statfs":{"total":96636764160,"available":95884414976,"internally_reserved":0,"allocated":262176768,"data_stored":261975494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":490139341},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,7,3,3,2,0,0,0,0,0,0,0,2],"upper_bound":65536},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":261283840,"data_stored":261283840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":262262784,"data_stored":262262784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":261001216,"data_stored":261001216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":261267456,"data_stored":261267456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:41:11.573 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-05T16:41:11.573 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'8844", 'reported_seq': 8948, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:07.369643+0000', 'last_change': '2024-09-05T16:34:56.833497+0000', 'last_active': '2024-09-05T16:41:07.369643+0000', 'last_peered': '2024-09-05T16:41:07.369643+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.828289+0000', 'last_became_peered': '2024-09-05T16:34:56.828289+0000', 'last_unstale': '2024-09-05T16:41:07.369643+0000', 'last_undegraded': '2024-09-05T16:34:56.612160+0000', 'last_fullsized': '2024-09-05T16:34:56.557632+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8844, 'log_dups_size': 0, 'ondisk_log_size': 8844, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33259520, 'num_objects': 4060, 'num_object_clones': 0, 'num_object_copies': 16240, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2236, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4060, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8844, 'num_write_kb': 51616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1824}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:41:11.573 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-05T16:41:11.574 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "37'8680", 'reported_seq': 9711, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:07.378816+0000', 'last_change': '2024-09-05T16:40:28.919189+0000', 'last_active': '2024-09-05T16:41:07.378816+0000', 'last_peered': '2024-09-05T16:41:07.378816+0000', 'last_clean': '2024-09-05T16:33:53.200762+0000', 'last_became_active': '2024-09-05T16:34:56.820292+0000', 'last_became_peered': '2024-09-05T16:34:56.820292+0000', 'last_unstale': '2024-09-05T16:41:07.378816+0000', 'last_undegraded': '2024-09-05T16:34:56.605776+0000', 'last_fullsized': '2024-09-05T16:34:56.559383+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8680, 'log_dups_size': 0, 'ondisk_log_size': 8680, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31732896, 'num_objects': 3874, 'num_object_clones': 0, 'num_object_copies': 15496, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1791, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3874, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8678, 'num_write_kb': 50212, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 480, 'num_bytes_recovered': 2613248, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2083}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:41:11.574 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:41:11.574 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'8798", 'reported_seq': 8920, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:07.616556+0000', 'last_change': '2024-09-05T16:34:56.867900+0000', 'last_active': '2024-09-05T16:41:07.616556+0000', 'last_peered': '2024-09-05T16:41:07.616556+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:41:07.616556+0000', 'last_undegraded': '2024-09-05T16:34:56.590972+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8798, 'log_dups_size': 0, 'ondisk_log_size': 8798, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32389736, 'num_objects': 3955, 'num_object_clones': 0, 'num_object_copies': 15820, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2233, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3955, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 8797, 'num_write_kb': 50999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1723}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:41:11.574 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:41:11.574 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'8612", 'reported_seq': 8709, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:07.425913+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:41:07.425913+0000', 'last_peered': '2024-09-05T16:41:07.425913+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:41:07.425913+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8612, 'log_dups_size': 0, 'ondisk_log_size': 8612, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32625824, 'num_objects': 3983, 'num_object_clones': 0, 'num_object_copies': 15932, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2147, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3983, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8610, 'num_write_kb': 50373, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1838}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:41:11.574 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:41:11.574 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'8793", 'reported_seq': 8907, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:07.383907+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:41:07.383907+0000', 'last_peered': '2024-09-05T16:41:07.383907+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:41:07.383907+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8793, 'log_dups_size': 0, 'ondisk_log_size': 8793, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32595968, 'num_objects': 3979, 'num_object_clones': 0, 'num_object_copies': 15916, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3979, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8793, 'num_write_kb': 51088, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1760}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:41:11.574 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:41:11.574 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'8643", 'reported_seq': 8748, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:07.378184+0000', 'last_change': '2024-09-05T16:34:56.903436+0000', 'last_active': '2024-09-05T16:41:07.378184+0000', 'last_peered': '2024-09-05T16:41:07.378184+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:41:07.378184+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8643, 'log_dups_size': 0, 'ondisk_log_size': 8643, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32920736, 'num_objects': 4019, 'num_object_clones': 0, 'num_object_copies': 16076, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2134, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4019, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8641, 'num_write_kb': 50636, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1887}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:41:11.574 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-05T16:41:11.574 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "37'8806", 'reported_seq': 8894, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:07.446858+0000', 'last_change': '2024-09-05T16:34:56.883319+0000', 'last_active': '2024-09-05T16:41:07.446858+0000', 'last_peered': '2024-09-05T16:41:07.446858+0000', 'last_clean': '2024-09-05T16:33:53.505305+0000', 'last_became_active': '2024-09-05T16:34:56.873389+0000', 'last_became_peered': '2024-09-05T16:34:56.873389+0000', 'last_unstale': '2024-09-05T16:41:07.446858+0000', 'last_undegraded': '2024-09-05T16:34:56.602567+0000', 'last_fullsized': '2024-09-05T16:34:56.559958+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8806, 'log_dups_size': 0, 'ondisk_log_size': 8806, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32543904, 'num_objects': 3973, 'num_object_clones': 0, 'num_object_copies': 15892, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2234, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3973, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8804, 'num_write_kb': 51108, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1741}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:41:11.575 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:41:11.575 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'8792", 'reported_seq': 8892, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:07.393155+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:41:07.393155+0000', 'last_peered': '2024-09-05T16:41:07.393155+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:41:07.393155+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8792, 'log_dups_size': 0, 'ondisk_log_size': 8792, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32614720, 'num_objects': 3982, 'num_object_clones': 0, 'num_object_copies': 15928, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3982, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8789, 'num_write_kb': 51081, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1738}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:41:11.575 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:41:11.575 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'8839", 'reported_seq': 8960, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:07.633549+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:41:07.633549+0000', 'last_peered': '2024-09-05T16:41:07.633549+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:41:07.633549+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8839, 'log_dups_size': 0, 'ondisk_log_size': 8839, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32527520, 'num_objects': 3971, 'num_object_clones': 0, 'num_object_copies': 15884, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3971, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8837, 'num_write_kb': 51230, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1726}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:41:11.575 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:41:11.575 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'9039", 'reported_seq': 9165, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:07.315140+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:41:07.315140+0000', 'last_peered': '2024-09-05T16:41:07.315140+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:41:07.315140+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9039, 'log_dups_size': 0, 'ondisk_log_size': 9039, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33451760, 'num_objects': 4084, 'num_object_clones': 0, 'num_object_copies': 16336, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2284, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4084, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9036, 'num_write_kb': 52476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1803}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:41:11.575 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:41:11.575 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'8819", 'reported_seq': 8917, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:07.447581+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:41:07.447581+0000', 'last_peered': '2024-09-05T16:41:07.447581+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:41:07.447581+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8819, 'log_dups_size': 0, 'ondisk_log_size': 8819, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32275024, 'num_objects': 3940, 'num_object_clones': 0, 'num_object_copies': 15760, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3940, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8818, 'num_write_kb': 51031, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1702}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:41:11.575 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:41:11.575 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'9088", 'reported_seq': 9196, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:07.353929+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:41:07.353929+0000', 'last_peered': '2024-09-05T16:41:07.353929+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:41:07.353929+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9088, 'log_dups_size': 0, 'ondisk_log_size': 9088, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33076384, 'num_objects': 4038, 'num_object_clones': 0, 'num_object_copies': 16152, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4038, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9086, 'num_write_kb': 52500, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1704}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:41:11.575 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:41:11.576 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'8886", 'reported_seq': 8998, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:07.670621+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:41:07.670621+0000', 'last_peered': '2024-09-05T16:41:07.670621+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:41:07.670621+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8886, 'log_dups_size': 0, 'ondisk_log_size': 8886, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32330912, 'num_objects': 3947, 'num_object_clones': 0, 'num_object_copies': 15788, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2310, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3947, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8884, 'num_write_kb': 51325, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1639}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:41:11.576 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:41:11.576 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'8783", 'reported_seq': 8889, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:07.448148+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:41:07.448148+0000', 'last_peered': '2024-09-05T16:41:07.448148+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:41:07.448148+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8783, 'log_dups_size': 0, 'ondisk_log_size': 8783, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32699552, 'num_objects': 3992, 'num_object_clones': 0, 'num_object_copies': 15968, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2223, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3992, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8781, 'num_write_kb': 51092, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1771}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:41:11.576 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:41:12.716 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:41:12.716 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 30880/255284 objects degraded (12.096%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":475,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":63821,"data_bytes":522777064,"bytes_used":2882838528,"bytes_avail":383664218112,"bytes_total":386547056640,"degraded_objects":30880,"degraded_total":255284,"degraded_ratio":0.12096331928362139,"recovering_objects_per_sec":10,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":128,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":176},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:41:12.758 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:41:12.758 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:41:13.400 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:41:13.400 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:41:13.423 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":266,"stamp":"2024-09-05T16:41:12.958959+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":513569256,"num_objects":62697,"num_object_clones":0,"num_object_copies":250788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30823,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62697,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":128140,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":814,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":142264,"ondisk_log_size":142264,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2868072,"kb_used_data":1006480,"kb_used_omap":14,"kb_used_meta":1861489,"kb_avail":374619288,"statfs":{"total":386547056640,"available":383610150912,"internally_reserved":0,"allocated":1030635520,"data_stored":1029830424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1906165303},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,8,5,5,15,15,10],"upper_bound":256},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-26591232,"num_objects":-3246,"num_object_clones":0,"num_object_copies":-12984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-185,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3246,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3246,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":185,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004563"},"pg_stats":[{"pgid":"1.f","version":"37'8909","reported_seq":9011,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:12.424490+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:41:12.424490+0000","last_peered":"2024-09-05T16:41:12.424490+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:41:12.424490+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8909,"log_dups_size":0,"ondisk_log_size":8909,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32727040,"num_objects":3995,"num_object_clones":0,"num_object_copies":15980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3995,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8909,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1759}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'8744","reported_seq":9885,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:12.515024+0000","last_change":"2024-09-05T16:40:28.919189+0000","last_active":"2024-09-05T16:41:12.515024+0000","last_peered":"2024-09-05T16:41:12.515024+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:41:12.515024+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8744,"log_dups_size":0,"ondisk_log_size":8744,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31208608,"num_objects":3810,"num_object_clones":0,"num_object_copies":15240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1736,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3810,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8742,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":535,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2074}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'8880","reported_seq":9002,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:12.781843+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:41:12.781843+0000","last_peered":"2024-09-05T16:41:12.781843+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:41:12.781843+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8880,"log_dups_size":0,"ondisk_log_size":8880,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31717992,"num_objects":3873,"num_object_clones":0,"num_object_copies":15492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3873,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8879,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1641}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'8944","reported_seq":13672,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:41:11.697429+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:41:11.697429+0000","last_peered":"2024-09-05T16:41:11.697429+0000","last_clean":"2024-09-05T16:41:11.697429+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:41:11.697429+0000","last_undegraded":"2024-09-05T16:41:11.697429+0000","last_fullsized":"2024-09-05T16:41:11.697429+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":8944,"log_dups_size":0,"ondisk_log_size":8944,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32686080,"num_objects":3990,"num_object_clones":0,"num_object_copies":15960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3990,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3244,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'8671","reported_seq":8769,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:12.298183+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:41:12.298183+0000","last_peered":"2024-09-05T16:41:12.298183+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:41:12.298183+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8671,"log_dups_size":0,"ondisk_log_size":8671,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32142496,"num_objects":3924,"num_object_clones":0,"num_object_copies":15696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3924,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8669,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1779}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'8861","reported_seq":8975,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:12.351364+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:41:12.351364+0000","last_peered":"2024-09-05T16:41:12.351364+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:41:12.351364+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8861,"log_dups_size":0,"ondisk_log_size":8861,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32038912,"num_objects":3911,"num_object_clones":0,"num_object_copies":15644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3911,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8861,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1692}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'8713","reported_seq":8820,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:12.326903+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:41:12.326903+0000","last_peered":"2024-09-05T16:41:12.326903+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:41:12.326903+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8713,"log_dups_size":0,"ondisk_log_size":8713,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32347296,"num_objects":3949,"num_object_clones":0,"num_object_copies":15796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3949,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8711,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1818}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'8880","reported_seq":8967,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:12.505474+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:41:12.505474+0000","last_peered":"2024-09-05T16:41:12.505474+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:41:12.505474+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8880,"log_dups_size":0,"ondisk_log_size":8880,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31937696,"num_objects":3899,"num_object_clones":0,"num_object_copies":15596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3899,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8878,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1667}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'8868","reported_seq":8966,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:12.352714+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:41:12.352714+0000","last_peered":"2024-09-05T16:41:12.352714+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:41:12.352714+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8868,"log_dups_size":0,"ondisk_log_size":8868,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31992128,"num_objects":3906,"num_object_clones":0,"num_object_copies":15624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3906,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8865,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1662}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'8900","reported_seq":9021,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:12.614163+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:41:12.614163+0000","last_peered":"2024-09-05T16:41:12.614163+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:41:12.614163+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8900,"log_dups_size":0,"ondisk_log_size":8900,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32027808,"num_objects":3910,"num_object_clones":0,"num_object_copies":15640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8898,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1665}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'9105","reported_seq":9231,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:12.235822+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:41:12.235822+0000","last_peered":"2024-09-05T16:41:12.235822+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:41:12.235822+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9105,"log_dups_size":0,"ondisk_log_size":9105,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32911088,"num_objects":4018,"num_object_clones":0,"num_object_copies":16072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4018,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9102,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1737}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'8882","reported_seq":8980,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:12.408282+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:41:12.408282+0000","last_peered":"2024-09-05T16:41:12.408282+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:41:12.408282+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8882,"log_dups_size":0,"ondisk_log_size":8882,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31758928,"num_objects":3877,"num_object_clones":0,"num_object_copies":15508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3877,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8881,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1639}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'8921","reported_seq":13656,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:41:11.637214+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:41:11.637214+0000","last_peered":"2024-09-05T16:41:11.637214+0000","last_clean":"2024-09-05T16:41:11.637214+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:41:11.637214+0000","last_undegraded":"2024-09-05T16:41:11.637214+0000","last_fullsized":"2024-09-05T16:41:11.637214+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":8921,"log_dups_size":0,"ondisk_log_size":8921,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31842304,"num_objects":3887,"num_object_clones":0,"num_object_copies":15548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3887,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":521,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'9164","reported_seq":9272,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:12.203390+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:41:12.203390+0000","last_peered":"2024-09-05T16:41:12.203390+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:41:12.203390+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9164,"log_dups_size":0,"ondisk_log_size":9164,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32453792,"num_objects":3962,"num_object_clones":0,"num_object_copies":15848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3962,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9162,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1628}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'8957","reported_seq":9069,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:12.757882+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:41:12.757882+0000","last_peered":"2024-09-05T16:41:12.757882+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:41:12.757882+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8957,"log_dups_size":0,"ondisk_log_size":8957,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31749280,"num_objects":3876,"num_object_clones":0,"num_object_copies":15504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3876,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8955,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1568}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'8865","reported_seq":8973,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:12.419228+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:41:12.419228+0000","last_peered":"2024-09-05T16:41:12.419228+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:41:12.419228+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8865,"log_dups_size":0,"ondisk_log_size":8865,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32027808,"num_objects":3910,"num_object_clones":0,"num_object_copies":15640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8863,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1690}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":513569256,"num_objects":62697,"num_object_clones":0,"num_object_copies":250788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30823,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62697,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":128140,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":814,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1027317760,"data_stored":1027317760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":142264,"ondisk_log_size":142264,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084375,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":764880,"kb_used_data":251432,"kb_used_omap":4,"kb_used_meta":513403,"kb_avail":93606960,"statfs":{"total":96636764160,"available":95853527040,"internally_reserved":0,"allocated":257466368,"data_stored":257265094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":525725395},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,0,1,10,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986252,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":598900,"kb_used_data":252068,"kb_used_omap":4,"kb_used_meta":346811,"kb_avail":93772940,"statfs":{"total":96636764160,"available":96023490560,"internally_reserved":0,"allocated":258117632,"data_stored":257916358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":355135186},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,2,1,3,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673058,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":751936,"kb_used_data":251504,"kb_used_omap":1,"kb_used_meta":500414,"kb_avail":93619904,"statfs":{"total":96636764160,"available":95866781696,"internally_reserved":0,"allocated":257540096,"data_stored":257338822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":512424389},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,1,3,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182489,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":752356,"kb_used_data":251476,"kb_used_omap":4,"kb_used_meta":500859,"kb_avail":93619484,"statfs":{"total":96636764160,"available":95866351616,"internally_reserved":0,"allocated":257511424,"data_stored":257310150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":512880333},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,1,1,5,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":256655360,"data_stored":256655360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":257540096,"data_stored":257540096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":256409600,"data_stored":256409600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":256712704,"data_stored":256712704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:41:16.425 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:41:16.975 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:41:17.023 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:41:17.024 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":268,"stamp":"2024-09-05T16:41:16.960630+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":512496104,"num_objects":62566,"num_object_clones":0,"num_object_copies":250264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30823,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62566,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":128271,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":814,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":142395,"ondisk_log_size":142395,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2851120,"kb_used_data":1001996,"kb_used_omap":14,"kb_used_meta":1849009,"kb_avail":374636240,"statfs":{"total":386547056640,"available":383627509760,"internally_reserved":0,"allocated":1026043904,"data_stored":1025238808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1893385783},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,4,9,3,16,14,12],"upper_bound":256},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18939904,"num_objects":-2312,"num_object_clones":0,"num_object_copies":-9248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2312,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2312,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":118,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004835"},"pg_stats":[{"pgid":"1.f","version":"37'8909","reported_seq":9011,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:12.424490+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:41:12.424490+0000","last_peered":"2024-09-05T16:41:12.424490+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:41:12.424490+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8909,"log_dups_size":0,"ondisk_log_size":8909,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32727040,"num_objects":3995,"num_object_clones":0,"num_object_copies":15980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3995,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8909,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1759}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'8744","reported_seq":9885,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:12.515024+0000","last_change":"2024-09-05T16:40:28.919189+0000","last_active":"2024-09-05T16:41:12.515024+0000","last_peered":"2024-09-05T16:41:12.515024+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:41:12.515024+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8744,"log_dups_size":0,"ondisk_log_size":8744,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31208608,"num_objects":3810,"num_object_clones":0,"num_object_copies":15240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1736,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3810,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8742,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":535,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2074}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'8880","reported_seq":9002,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:12.781843+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:41:12.781843+0000","last_peered":"2024-09-05T16:41:12.781843+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:41:12.781843+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8880,"log_dups_size":0,"ondisk_log_size":8880,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31717992,"num_objects":3873,"num_object_clones":0,"num_object_copies":15492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3873,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8879,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1641}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'9011","reported_seq":13738,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:41:16.840884+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:41:16.840884+0000","last_peered":"2024-09-05T16:41:16.840884+0000","last_clean":"2024-09-05T16:41:16.840884+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:41:16.840884+0000","last_undegraded":"2024-09-05T16:41:16.840884+0000","last_fullsized":"2024-09-05T16:41:16.840884+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":9011,"log_dups_size":0,"ondisk_log_size":9011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32137216,"num_objects":3923,"num_object_clones":0,"num_object_copies":15692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3311,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'8671","reported_seq":8769,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:12.298183+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:41:12.298183+0000","last_peered":"2024-09-05T16:41:12.298183+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:41:12.298183+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8671,"log_dups_size":0,"ondisk_log_size":8671,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32142496,"num_objects":3924,"num_object_clones":0,"num_object_copies":15696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3924,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8669,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1779}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'8861","reported_seq":8975,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:12.351364+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:41:12.351364+0000","last_peered":"2024-09-05T16:41:12.351364+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:41:12.351364+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8861,"log_dups_size":0,"ondisk_log_size":8861,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32038912,"num_objects":3911,"num_object_clones":0,"num_object_copies":15644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3911,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8861,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1692}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'8713","reported_seq":8820,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:12.326903+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:41:12.326903+0000","last_peered":"2024-09-05T16:41:12.326903+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:41:12.326903+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8713,"log_dups_size":0,"ondisk_log_size":8713,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32347296,"num_objects":3949,"num_object_clones":0,"num_object_copies":15796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3949,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8711,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1818}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'8880","reported_seq":8967,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:12.505474+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:41:12.505474+0000","last_peered":"2024-09-05T16:41:12.505474+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:41:12.505474+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8880,"log_dups_size":0,"ondisk_log_size":8880,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31937696,"num_objects":3899,"num_object_clones":0,"num_object_copies":15596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3899,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8878,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1667}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'8868","reported_seq":8966,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:12.352714+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:41:12.352714+0000","last_peered":"2024-09-05T16:41:12.352714+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:41:12.352714+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8868,"log_dups_size":0,"ondisk_log_size":8868,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31992128,"num_objects":3906,"num_object_clones":0,"num_object_copies":15624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3906,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8865,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1662}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'8900","reported_seq":9021,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:12.614163+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:41:12.614163+0000","last_peered":"2024-09-05T16:41:12.614163+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:41:12.614163+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8900,"log_dups_size":0,"ondisk_log_size":8900,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32027808,"num_objects":3910,"num_object_clones":0,"num_object_copies":15640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8898,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1665}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'9105","reported_seq":9231,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:12.235822+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:41:12.235822+0000","last_peered":"2024-09-05T16:41:12.235822+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:41:12.235822+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9105,"log_dups_size":0,"ondisk_log_size":9105,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32911088,"num_objects":4018,"num_object_clones":0,"num_object_copies":16072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4018,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9102,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1737}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'8882","reported_seq":8980,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:12.408282+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:41:12.408282+0000","last_peered":"2024-09-05T16:41:12.408282+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:41:12.408282+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8882,"log_dups_size":0,"ondisk_log_size":8882,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31758928,"num_objects":3877,"num_object_clones":0,"num_object_copies":15508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3877,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8881,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1639}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'8985","reported_seq":13720,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:41:16.746436+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:41:16.746436+0000","last_peered":"2024-09-05T16:41:16.746436+0000","last_clean":"2024-09-05T16:41:16.746436+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:41:16.746436+0000","last_undegraded":"2024-09-05T16:41:16.746436+0000","last_fullsized":"2024-09-05T16:41:16.746436+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":8985,"log_dups_size":0,"ondisk_log_size":8985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31318016,"num_objects":3823,"num_object_clones":0,"num_object_copies":15292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3823,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":585,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'9164","reported_seq":9272,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:12.203390+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:41:12.203390+0000","last_peered":"2024-09-05T16:41:12.203390+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:41:12.203390+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9164,"log_dups_size":0,"ondisk_log_size":9164,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32453792,"num_objects":3962,"num_object_clones":0,"num_object_copies":15848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3962,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9162,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1628}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'8957","reported_seq":9069,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:12.757882+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:41:12.757882+0000","last_peered":"2024-09-05T16:41:12.757882+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:41:12.757882+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8957,"log_dups_size":0,"ondisk_log_size":8957,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31749280,"num_objects":3876,"num_object_clones":0,"num_object_copies":15504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3876,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8955,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1568}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'8865","reported_seq":8973,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:12.419228+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:41:12.419228+0000","last_peered":"2024-09-05T16:41:12.419228+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:41:12.419228+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8865,"log_dups_size":0,"ondisk_log_size":8865,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32027808,"num_objects":3910,"num_object_clones":0,"num_object_copies":15640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8863,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1690}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":512496104,"num_objects":62566,"num_object_clones":0,"num_object_copies":250264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30823,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62566,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":128271,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":814,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1022681088,"data_stored":1022681088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":142395,"ondisk_log_size":142395,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084375,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":764880,"kb_used_data":251432,"kb_used_omap":4,"kb_used_meta":513403,"kb_avail":93606960,"statfs":{"total":96636764160,"available":95853527040,"internally_reserved":0,"allocated":257466368,"data_stored":257265094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":525725395},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,0,1,10,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986253,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":581948,"kb_used_data":247584,"kb_used_omap":4,"kb_used_meta":334331,"kb_avail":93789892,"statfs":{"total":96636764160,"available":96040849408,"internally_reserved":0,"allocated":253526016,"data_stored":253324742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":342355666},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,0,2,2,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673058,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":751936,"kb_used_data":251504,"kb_used_omap":1,"kb_used_meta":500414,"kb_avail":93619904,"statfs":{"total":96636764160,"available":95866781696,"internally_reserved":0,"allocated":257540096,"data_stored":257338822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":512424389},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,1,3,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182489,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":752356,"kb_used_data":251476,"kb_used_omap":4,"kb_used_meta":500859,"kb_avail":93619484,"statfs":{"total":96636764160,"available":95866351616,"internally_reserved":0,"allocated":257511424,"data_stored":257310150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":512880333},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,1,1,5,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":256655360,"data_stored":256655360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":252903424,"data_stored":252903424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":256409600,"data_stored":256409600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":256712704,"data_stored":256712704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:41:17.026 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:41:17.648 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:41:17.649 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:41:17.673 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":268,"stamp":"2024-09-05T16:41:16.960630+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":512496104,"num_objects":62566,"num_object_clones":0,"num_object_copies":250264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30823,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62566,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":128271,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":814,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":142395,"ondisk_log_size":142395,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2851120,"kb_used_data":1001996,"kb_used_omap":14,"kb_used_meta":1849009,"kb_avail":374636240,"statfs":{"total":386547056640,"available":383627509760,"internally_reserved":0,"allocated":1026043904,"data_stored":1025238808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1893385783},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,4,9,3,16,14,12],"upper_bound":256},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18939904,"num_objects":-2312,"num_object_clones":0,"num_object_copies":-9248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2312,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2312,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":118,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004835"},"pg_stats":[{"pgid":"1.f","version":"37'8909","reported_seq":9011,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:12.424490+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:41:12.424490+0000","last_peered":"2024-09-05T16:41:12.424490+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:41:12.424490+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8909,"log_dups_size":0,"ondisk_log_size":8909,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32727040,"num_objects":3995,"num_object_clones":0,"num_object_copies":15980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3995,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8909,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1759}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'8744","reported_seq":9885,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:12.515024+0000","last_change":"2024-09-05T16:40:28.919189+0000","last_active":"2024-09-05T16:41:12.515024+0000","last_peered":"2024-09-05T16:41:12.515024+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:41:12.515024+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8744,"log_dups_size":0,"ondisk_log_size":8744,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31208608,"num_objects":3810,"num_object_clones":0,"num_object_copies":15240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1736,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3810,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8742,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":535,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2074}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'8880","reported_seq":9002,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:12.781843+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:41:12.781843+0000","last_peered":"2024-09-05T16:41:12.781843+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:41:12.781843+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8880,"log_dups_size":0,"ondisk_log_size":8880,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31717992,"num_objects":3873,"num_object_clones":0,"num_object_copies":15492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3873,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8879,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1641}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'9011","reported_seq":13738,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:41:16.840884+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:41:16.840884+0000","last_peered":"2024-09-05T16:41:16.840884+0000","last_clean":"2024-09-05T16:41:16.840884+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:41:16.840884+0000","last_undegraded":"2024-09-05T16:41:16.840884+0000","last_fullsized":"2024-09-05T16:41:16.840884+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":9011,"log_dups_size":0,"ondisk_log_size":9011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32137216,"num_objects":3923,"num_object_clones":0,"num_object_copies":15692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3311,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'8671","reported_seq":8769,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:12.298183+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:41:12.298183+0000","last_peered":"2024-09-05T16:41:12.298183+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:41:12.298183+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8671,"log_dups_size":0,"ondisk_log_size":8671,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32142496,"num_objects":3924,"num_object_clones":0,"num_object_copies":15696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3924,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8669,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1779}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'8861","reported_seq":8975,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:12.351364+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:41:12.351364+0000","last_peered":"2024-09-05T16:41:12.351364+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:41:12.351364+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8861,"log_dups_size":0,"ondisk_log_size":8861,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32038912,"num_objects":3911,"num_object_clones":0,"num_object_copies":15644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3911,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8861,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1692}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'8713","reported_seq":8820,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:12.326903+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:41:12.326903+0000","last_peered":"2024-09-05T16:41:12.326903+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:41:12.326903+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8713,"log_dups_size":0,"ondisk_log_size":8713,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32347296,"num_objects":3949,"num_object_clones":0,"num_object_copies":15796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3949,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8711,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1818}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'8880","reported_seq":8967,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:12.505474+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:41:12.505474+0000","last_peered":"2024-09-05T16:41:12.505474+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:41:12.505474+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8880,"log_dups_size":0,"ondisk_log_size":8880,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31937696,"num_objects":3899,"num_object_clones":0,"num_object_copies":15596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3899,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8878,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1667}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'8868","reported_seq":8966,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:12.352714+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:41:12.352714+0000","last_peered":"2024-09-05T16:41:12.352714+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:41:12.352714+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8868,"log_dups_size":0,"ondisk_log_size":8868,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31992128,"num_objects":3906,"num_object_clones":0,"num_object_copies":15624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3906,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8865,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1662}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'8900","reported_seq":9021,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:12.614163+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:41:12.614163+0000","last_peered":"2024-09-05T16:41:12.614163+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:41:12.614163+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8900,"log_dups_size":0,"ondisk_log_size":8900,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32027808,"num_objects":3910,"num_object_clones":0,"num_object_copies":15640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8898,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1665}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'9105","reported_seq":9231,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:12.235822+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:41:12.235822+0000","last_peered":"2024-09-05T16:41:12.235822+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:41:12.235822+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9105,"log_dups_size":0,"ondisk_log_size":9105,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32911088,"num_objects":4018,"num_object_clones":0,"num_object_copies":16072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4018,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9102,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1737}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'8882","reported_seq":8980,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:12.408282+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:41:12.408282+0000","last_peered":"2024-09-05T16:41:12.408282+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:41:12.408282+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8882,"log_dups_size":0,"ondisk_log_size":8882,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31758928,"num_objects":3877,"num_object_clones":0,"num_object_copies":15508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3877,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8881,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1639}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'8985","reported_seq":13720,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:41:16.746436+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:41:16.746436+0000","last_peered":"2024-09-05T16:41:16.746436+0000","last_clean":"2024-09-05T16:41:16.746436+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:41:16.746436+0000","last_undegraded":"2024-09-05T16:41:16.746436+0000","last_fullsized":"2024-09-05T16:41:16.746436+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":8985,"log_dups_size":0,"ondisk_log_size":8985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31318016,"num_objects":3823,"num_object_clones":0,"num_object_copies":15292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3823,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":585,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'9164","reported_seq":9272,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:12.203390+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:41:12.203390+0000","last_peered":"2024-09-05T16:41:12.203390+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:41:12.203390+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9164,"log_dups_size":0,"ondisk_log_size":9164,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32453792,"num_objects":3962,"num_object_clones":0,"num_object_copies":15848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3962,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9162,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1628}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'8957","reported_seq":9069,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:12.757882+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:41:12.757882+0000","last_peered":"2024-09-05T16:41:12.757882+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:41:12.757882+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8957,"log_dups_size":0,"ondisk_log_size":8957,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31749280,"num_objects":3876,"num_object_clones":0,"num_object_copies":15504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3876,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8955,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1568}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'8865","reported_seq":8973,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:12.419228+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:41:12.419228+0000","last_peered":"2024-09-05T16:41:12.419228+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:41:12.419228+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8865,"log_dups_size":0,"ondisk_log_size":8865,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32027808,"num_objects":3910,"num_object_clones":0,"num_object_copies":15640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8863,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1690}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":512496104,"num_objects":62566,"num_object_clones":0,"num_object_copies":250264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30823,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62566,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":128271,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":814,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1022681088,"data_stored":1022681088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":142395,"ondisk_log_size":142395,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084375,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":764880,"kb_used_data":251432,"kb_used_omap":4,"kb_used_meta":513403,"kb_avail":93606960,"statfs":{"total":96636764160,"available":95853527040,"internally_reserved":0,"allocated":257466368,"data_stored":257265094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":525725395},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,0,1,10,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986253,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":581948,"kb_used_data":247584,"kb_used_omap":4,"kb_used_meta":334331,"kb_avail":93789892,"statfs":{"total":96636764160,"available":96040849408,"internally_reserved":0,"allocated":253526016,"data_stored":253324742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":342355666},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,0,2,2,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673058,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":751936,"kb_used_data":251504,"kb_used_omap":1,"kb_used_meta":500414,"kb_avail":93619904,"statfs":{"total":96636764160,"available":95866781696,"internally_reserved":0,"allocated":257540096,"data_stored":257338822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":512424389},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,1,3,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182489,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":752356,"kb_used_data":251476,"kb_used_omap":4,"kb_used_meta":500859,"kb_avail":93619484,"statfs":{"total":96636764160,"available":95866351616,"internally_reserved":0,"allocated":257511424,"data_stored":257310150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":512880333},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,1,1,5,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":256655360,"data_stored":256655360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":252903424,"data_stored":252903424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":256409600,"data_stored":256409600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":256712704,"data_stored":256712704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:41:17.674 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-05T16:41:17.674 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'8909", 'reported_seq': 9011, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:12.424490+0000', 'last_change': '2024-09-05T16:34:56.833497+0000', 'last_active': '2024-09-05T16:41:12.424490+0000', 'last_peered': '2024-09-05T16:41:12.424490+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.828289+0000', 'last_became_peered': '2024-09-05T16:34:56.828289+0000', 'last_unstale': '2024-09-05T16:41:12.424490+0000', 'last_undegraded': '2024-09-05T16:34:56.612160+0000', 'last_fullsized': '2024-09-05T16:34:56.557632+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8909, 'log_dups_size': 0, 'ondisk_log_size': 8909, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32727040, 'num_objects': 3995, 'num_object_clones': 0, 'num_object_copies': 15980, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2236, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3995, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8909, 'num_write_kb': 51616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1759}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:41:17.675 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-05T16:41:17.675 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "37'8744", 'reported_seq': 9885, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:12.515024+0000', 'last_change': '2024-09-05T16:40:28.919189+0000', 'last_active': '2024-09-05T16:41:12.515024+0000', 'last_peered': '2024-09-05T16:41:12.515024+0000', 'last_clean': '2024-09-05T16:33:53.200762+0000', 'last_became_active': '2024-09-05T16:34:56.820292+0000', 'last_became_peered': '2024-09-05T16:34:56.820292+0000', 'last_unstale': '2024-09-05T16:41:12.515024+0000', 'last_undegraded': '2024-09-05T16:34:56.605776+0000', 'last_fullsized': '2024-09-05T16:34:56.559383+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8744, 'log_dups_size': 0, 'ondisk_log_size': 8744, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31208608, 'num_objects': 3810, 'num_object_clones': 0, 'num_object_copies': 15240, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1736, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3810, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8742, 'num_write_kb': 50212, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 535, 'num_bytes_recovered': 2613248, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2074}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:41:17.675 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:41:17.675 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'8880", 'reported_seq': 9002, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:12.781843+0000', 'last_change': '2024-09-05T16:34:56.867900+0000', 'last_active': '2024-09-05T16:41:12.781843+0000', 'last_peered': '2024-09-05T16:41:12.781843+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:41:12.781843+0000', 'last_undegraded': '2024-09-05T16:34:56.590972+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8880, 'log_dups_size': 0, 'ondisk_log_size': 8880, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31717992, 'num_objects': 3873, 'num_object_clones': 0, 'num_object_copies': 15492, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2233, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3873, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 8879, 'num_write_kb': 50999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1641}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:41:17.675 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:41:17.675 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'8671", 'reported_seq': 8769, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:12.298183+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:41:12.298183+0000', 'last_peered': '2024-09-05T16:41:12.298183+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:41:12.298183+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8671, 'log_dups_size': 0, 'ondisk_log_size': 8671, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32142496, 'num_objects': 3924, 'num_object_clones': 0, 'num_object_copies': 15696, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2147, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3924, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8669, 'num_write_kb': 50373, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1779}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:41:17.675 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:41:17.675 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'8861", 'reported_seq': 8975, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:12.351364+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:41:12.351364+0000', 'last_peered': '2024-09-05T16:41:12.351364+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:41:12.351364+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8861, 'log_dups_size': 0, 'ondisk_log_size': 8861, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32038912, 'num_objects': 3911, 'num_object_clones': 0, 'num_object_copies': 15644, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3911, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8861, 'num_write_kb': 51088, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1692}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:41:17.675 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:41:17.675 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'8713", 'reported_seq': 8820, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:12.326903+0000', 'last_change': '2024-09-05T16:34:56.903436+0000', 'last_active': '2024-09-05T16:41:12.326903+0000', 'last_peered': '2024-09-05T16:41:12.326903+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:41:12.326903+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8713, 'log_dups_size': 0, 'ondisk_log_size': 8713, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32347296, 'num_objects': 3949, 'num_object_clones': 0, 'num_object_copies': 15796, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2133, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3949, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8711, 'num_write_kb': 50636, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1818}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:41:17.675 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-05T16:41:17.676 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "37'8880", 'reported_seq': 8967, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:12.505474+0000', 'last_change': '2024-09-05T16:34:56.883319+0000', 'last_active': '2024-09-05T16:41:12.505474+0000', 'last_peered': '2024-09-05T16:41:12.505474+0000', 'last_clean': '2024-09-05T16:33:53.505305+0000', 'last_became_active': '2024-09-05T16:34:56.873389+0000', 'last_became_peered': '2024-09-05T16:34:56.873389+0000', 'last_unstale': '2024-09-05T16:41:12.505474+0000', 'last_undegraded': '2024-09-05T16:34:56.602567+0000', 'last_fullsized': '2024-09-05T16:34:56.559958+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8880, 'log_dups_size': 0, 'ondisk_log_size': 8880, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31937696, 'num_objects': 3899, 'num_object_clones': 0, 'num_object_copies': 15596, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2234, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3899, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8878, 'num_write_kb': 51108, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1667}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:41:17.676 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:41:17.676 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'8868", 'reported_seq': 8966, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:12.352714+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:41:12.352714+0000', 'last_peered': '2024-09-05T16:41:12.352714+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:41:12.352714+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8868, 'log_dups_size': 0, 'ondisk_log_size': 8868, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31992128, 'num_objects': 3906, 'num_object_clones': 0, 'num_object_copies': 15624, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3906, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8865, 'num_write_kb': 51081, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1662}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:41:17.676 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:41:17.676 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'8900", 'reported_seq': 9021, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:12.614163+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:41:12.614163+0000', 'last_peered': '2024-09-05T16:41:12.614163+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:41:12.614163+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8900, 'log_dups_size': 0, 'ondisk_log_size': 8900, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32027808, 'num_objects': 3910, 'num_object_clones': 0, 'num_object_copies': 15640, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3910, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8898, 'num_write_kb': 51230, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1665}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:41:17.676 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:41:17.676 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'9105", 'reported_seq': 9231, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:12.235822+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:41:12.235822+0000', 'last_peered': '2024-09-05T16:41:12.235822+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:41:12.235822+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9105, 'log_dups_size': 0, 'ondisk_log_size': 9105, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32911088, 'num_objects': 4018, 'num_object_clones': 0, 'num_object_copies': 16072, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2284, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4018, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9102, 'num_write_kb': 52476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1737}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:41:17.676 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:41:17.676 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'8882", 'reported_seq': 8980, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:12.408282+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:41:12.408282+0000', 'last_peered': '2024-09-05T16:41:12.408282+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:41:12.408282+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8882, 'log_dups_size': 0, 'ondisk_log_size': 8882, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31758928, 'num_objects': 3877, 'num_object_clones': 0, 'num_object_copies': 15508, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3877, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8881, 'num_write_kb': 51031, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1639}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:41:17.676 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:41:17.676 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'9164", 'reported_seq': 9272, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:12.203390+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:41:12.203390+0000', 'last_peered': '2024-09-05T16:41:12.203390+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:41:12.203390+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9164, 'log_dups_size': 0, 'ondisk_log_size': 9164, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32453792, 'num_objects': 3962, 'num_object_clones': 0, 'num_object_copies': 15848, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3962, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9162, 'num_write_kb': 52500, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1628}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:41:17.677 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:41:17.677 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'8957", 'reported_seq': 9069, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:12.757882+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:41:12.757882+0000', 'last_peered': '2024-09-05T16:41:12.757882+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:41:12.757882+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8957, 'log_dups_size': 0, 'ondisk_log_size': 8957, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31749280, 'num_objects': 3876, 'num_object_clones': 0, 'num_object_copies': 15504, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2310, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3876, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8955, 'num_write_kb': 51325, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1568}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:41:17.677 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:41:17.677 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'8865", 'reported_seq': 8973, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:12.419228+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:41:12.419228+0000', 'last_peered': '2024-09-05T16:41:12.419228+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:41:12.419228+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8865, 'log_dups_size': 0, 'ondisk_log_size': 8865, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32027808, 'num_objects': 3910, 'num_object_clones': 0, 'num_object_copies': 15640, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3910, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8863, 'num_write_kb': 51092, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1690}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:41:17.677 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:41:18.425 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:41:18.425 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 30823/250264 objects degraded (12.316%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":481,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":62566,"data_bytes":512496104,"bytes_used":2919546880,"bytes_avail":383627509760,"bytes_total":386547056640,"degraded_objects":30823,"degraded_total":250264,"degraded_ratio":0.12316194099031423,"recovering_objects_per_sec":9,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":118,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":192},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:41:18.453 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:41:18.453 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:41:19.127 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:41:19.354 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:41:19.355 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":269,"stamp":"2024-09-05T16:41:18.961520+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":504295912,"num_objects":61565,"num_object_clones":0,"num_object_copies":246260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30775,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61565,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":129272,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":862,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":143396,"ondisk_log_size":143396,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2832820,"kb_used_data":987880,"kb_used_omap":14,"kb_used_meta":1844849,"kb_avail":374654540,"statfs":{"total":386547056640,"available":383646248960,"internally_reserved":0,"allocated":1011589120,"data_stored":1010784024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1889125943},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,9,4,10,10,13],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-26173440,"num_objects":-3195,"num_object_clones":0,"num_object_copies":-12780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-166,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3195,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3195,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":166,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005017"},"pg_stats":[{"pgid":"1.f","version":"37'8976","reported_seq":9079,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:17.434848+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:41:17.434848+0000","last_peered":"2024-09-05T16:41:17.434848+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:41:17.434848+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8976,"log_dups_size":0,"ondisk_log_size":8976,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32178176,"num_objects":3928,"num_object_clones":0,"num_object_copies":15712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3928,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8976,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1692}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'8819","reported_seq":10054,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:17.648431+0000","last_change":"2024-09-05T16:40:28.919189+0000","last_active":"2024-09-05T16:41:17.648431+0000","last_peered":"2024-09-05T16:41:17.648431+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:41:17.648431+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8819,"log_dups_size":0,"ondisk_log_size":8819,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30594208,"num_objects":3735,"num_object_clones":0,"num_object_copies":14940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1688,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3735,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8817,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":583,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2047}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'8949","reported_seq":9071,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:17.936403+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:41:17.936403+0000","last_peered":"2024-09-05T16:41:17.936403+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:41:17.936403+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8949,"log_dups_size":0,"ondisk_log_size":8949,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31152744,"num_objects":3804,"num_object_clones":0,"num_object_copies":15216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3804,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8948,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1572}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'9011","reported_seq":13738,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:41:16.840884+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:41:16.840884+0000","last_peered":"2024-09-05T16:41:16.840884+0000","last_clean":"2024-09-05T16:41:16.840884+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:41:16.840884+0000","last_undegraded":"2024-09-05T16:41:16.840884+0000","last_fullsized":"2024-09-05T16:41:16.840884+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":9011,"log_dups_size":0,"ondisk_log_size":9011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32137216,"num_objects":3923,"num_object_clones":0,"num_object_copies":15692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3311,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'8751","reported_seq":8849,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:17.536852+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:41:17.536852+0000","last_peered":"2024-09-05T16:41:17.536852+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:41:17.536852+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8751,"log_dups_size":0,"ondisk_log_size":8751,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31487136,"num_objects":3844,"num_object_clones":0,"num_object_copies":15376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3844,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8749,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1699}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'8925","reported_seq":9039,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:17.444008+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:41:17.444008+0000","last_peered":"2024-09-05T16:41:17.444008+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:41:17.444008+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8925,"log_dups_size":0,"ondisk_log_size":8925,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31514624,"num_objects":3847,"num_object_clones":0,"num_object_copies":15388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3847,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8925,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1628}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'8779","reported_seq":8886,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:17.374499+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:41:17.374499+0000","last_peered":"2024-09-05T16:41:17.374499+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:41:17.374499+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8779,"log_dups_size":0,"ondisk_log_size":8779,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31806624,"num_objects":3883,"num_object_clones":0,"num_object_copies":15532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3883,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8777,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1752}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'8954","reported_seq":9041,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:17.446416+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:41:17.446416+0000","last_peered":"2024-09-05T16:41:17.446416+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:41:17.446416+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8954,"log_dups_size":0,"ondisk_log_size":8954,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31331488,"num_objects":3825,"num_object_clones":0,"num_object_copies":15300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3825,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8952,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1593}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'8940","reported_seq":9040,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:17.286774+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:41:17.286774+0000","last_peered":"2024-09-05T16:41:17.286774+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:41:17.286774+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8940,"log_dups_size":0,"ondisk_log_size":8940,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31402304,"num_objects":3834,"num_object_clones":0,"num_object_copies":15336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3834,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8937,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1590}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'8983","reported_seq":9104,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:17.955014+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:41:17.955014+0000","last_peered":"2024-09-05T16:41:17.955014+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:41:17.955014+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8983,"log_dups_size":0,"ondisk_log_size":8983,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31347872,"num_objects":3827,"num_object_clones":0,"num_object_copies":15308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3827,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8981,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1582}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'9174","reported_seq":9298,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:17.394770+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:41:17.394770+0000","last_peered":"2024-09-05T16:41:17.394770+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:41:17.394770+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9174,"log_dups_size":0,"ondisk_log_size":9174,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32345840,"num_objects":3949,"num_object_clones":0,"num_object_copies":15796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3949,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9171,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1668}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'8941","reported_seq":9039,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:17.434270+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:41:17.434270+0000","last_peered":"2024-09-05T16:41:17.434270+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:41:17.434270+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8941,"log_dups_size":0,"ondisk_log_size":8941,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31275600,"num_objects":3818,"num_object_clones":0,"num_object_copies":15272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3818,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8940,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1580}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'8985","reported_seq":13720,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:41:16.746436+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:41:16.746436+0000","last_peered":"2024-09-05T16:41:16.746436+0000","last_clean":"2024-09-05T16:41:16.746436+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:41:16.746436+0000","last_undegraded":"2024-09-05T16:41:16.746436+0000","last_fullsized":"2024-09-05T16:41:16.746436+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":8985,"log_dups_size":0,"ondisk_log_size":8985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31318016,"num_objects":3823,"num_object_clones":0,"num_object_copies":15292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3823,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":585,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'9233","reported_seq":9340,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:17.538034+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:41:17.538034+0000","last_peered":"2024-09-05T16:41:17.538034+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:41:17.538034+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9233,"log_dups_size":0,"ondisk_log_size":9233,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31888544,"num_objects":3893,"num_object_clones":0,"num_object_copies":15572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3893,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9231,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1559}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'9031","reported_seq":9143,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:17.631351+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:41:17.631351+0000","last_peered":"2024-09-05T16:41:17.631351+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:41:17.631351+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9031,"log_dups_size":0,"ondisk_log_size":9031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31143072,"num_objects":3802,"num_object_clones":0,"num_object_copies":15208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3802,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9029,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1494}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'8945","reported_seq":9050,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:17.477332+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:41:17.477332+0000","last_peered":"2024-09-05T16:41:17.477332+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:41:17.477332+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8945,"log_dups_size":0,"ondisk_log_size":8945,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31372448,"num_objects":3830,"num_object_clones":0,"num_object_copies":15320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3830,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8943,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1610}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":504295912,"num_objects":61565,"num_object_clones":0,"num_object_copies":246260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30775,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61565,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":129272,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":862,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1008971776,"data_stored":1008971776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":143396,"ondisk_log_size":143396,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084376,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":747888,"kb_used_data":246944,"kb_used_omap":4,"kb_used_meta":500923,"kb_avail":93623952,"statfs":{"total":96636764160,"available":95870926848,"internally_reserved":0,"allocated":252870656,"data_stored":252669382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":512945875},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,3,7,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986253,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":581948,"kb_used_data":247584,"kb_used_omap":4,"kb_used_meta":334331,"kb_avail":93789892,"statfs":{"total":96636764160,"available":96040849408,"internally_reserved":0,"allocated":253526016,"data_stored":253324742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":342355666},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,0,2,2,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673059,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":750872,"kb_used_data":246280,"kb_used_omap":1,"kb_used_meta":504574,"kb_avail":93620968,"statfs":{"total":96636764160,"available":95867871232,"internally_reserved":0,"allocated":252190720,"data_stored":251989446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":516684229},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182490,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":752112,"kb_used_data":247072,"kb_used_omap":4,"kb_used_meta":505019,"kb_avail":93619728,"statfs":{"total":96636764160,"available":95866601472,"internally_reserved":0,"allocated":253001728,"data_stored":252800454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":517140173},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,1,1,5,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":252108800,"data_stored":252108800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":252903424,"data_stored":252903424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":251793408,"data_stored":251793408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":252166144,"data_stored":252166144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:41:22.357 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:41:22.977 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:41:22.979 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:41:23.006 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":271,"stamp":"2024-09-05T16:41:22.963266+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":496873960,"num_objects":60659,"num_object_clones":0,"num_object_copies":242636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30720,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60659,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":130178,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":917,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":144302,"ondisk_log_size":144302,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2831864,"kb_used_data":974448,"kb_used_omap":14,"kb_used_meta":1857329,"kb_avail":374655496,"statfs":{"total":386547056640,"available":383647227904,"internally_reserved":0,"allocated":997834752,"data_stored":997029656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1901905463},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,1,12,11,17,12,7,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25903104,"num_objects":-3162,"num_object_clones":0,"num_object_copies":-12648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3162,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3162,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":160,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005123"},"pg_stats":[{"pgid":"1.f","version":"37'9053","reported_seq":9157,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:22.542098+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:41:22.542098+0000","last_peered":"2024-09-05T16:41:22.542098+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:41:22.542098+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9053,"log_dups_size":0,"ondisk_log_size":9053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31547392,"num_objects":3851,"num_object_clones":0,"num_object_copies":15404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3851,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9053,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1615}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'8883","reported_seq":10230,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:22.713812+0000","last_change":"2024-09-05T16:40:28.919189+0000","last_active":"2024-09-05T16:41:22.713812+0000","last_peered":"2024-09-05T16:41:22.713812+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:41:22.713812+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8883,"log_dups_size":0,"ondisk_log_size":8883,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30069920,"num_objects":3671,"num_object_clones":0,"num_object_copies":14684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1633,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3671,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8881,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":638,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2038}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'8949","reported_seq":9071,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:17.936403+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:41:17.936403+0000","last_peered":"2024-09-05T16:41:17.936403+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:41:17.936403+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8949,"log_dups_size":0,"ondisk_log_size":8949,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31152744,"num_objects":3804,"num_object_clones":0,"num_object_copies":15216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3804,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8948,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1572}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'9068","reported_seq":13796,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:41:21.814505+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:41:21.814505+0000","last_peered":"2024-09-05T16:41:21.814505+0000","last_clean":"2024-09-05T16:41:21.814505+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:41:21.814505+0000","last_undegraded":"2024-09-05T16:41:21.814505+0000","last_fullsized":"2024-09-05T16:41:21.814505+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":9068,"log_dups_size":0,"ondisk_log_size":9068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31670272,"num_objects":3866,"num_object_clones":0,"num_object_copies":15464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3866,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3368,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'8829","reported_seq":8927,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:22.404637+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:41:22.404637+0000","last_peered":"2024-09-05T16:41:22.404637+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:41:22.404637+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8829,"log_dups_size":0,"ondisk_log_size":8829,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30848160,"num_objects":3766,"num_object_clones":0,"num_object_copies":15064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3766,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8827,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1621}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'9002","reported_seq":9116,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:22.538942+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:41:22.538942+0000","last_peered":"2024-09-05T16:41:22.538942+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:41:22.538942+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9002,"log_dups_size":0,"ondisk_log_size":9002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30883840,"num_objects":3770,"num_object_clones":0,"num_object_copies":15080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3770,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9002,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1551}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'8841","reported_seq":8948,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:22.703002+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:41:22.703002+0000","last_peered":"2024-09-05T16:41:22.703002+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:41:22.703002+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8841,"log_dups_size":0,"ondisk_log_size":8841,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31298720,"num_objects":3821,"num_object_clones":0,"num_object_copies":15284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3821,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8839,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1690}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'9022","reported_seq":9108,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:22.500408+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:41:22.500408+0000","last_peered":"2024-09-05T16:41:22.500408+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:41:22.500408+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9022,"log_dups_size":0,"ondisk_log_size":9022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30774432,"num_objects":3757,"num_object_clones":0,"num_object_copies":15028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3757,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9020,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'9018","reported_seq":9117,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:22.589677+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:41:22.589677+0000","last_peered":"2024-09-05T16:41:22.589677+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:41:22.589677+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9018,"log_dups_size":0,"ondisk_log_size":9018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30763328,"num_objects":3756,"num_object_clones":0,"num_object_copies":15024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3756,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9015,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1512}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'8983","reported_seq":9104,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:17.955014+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:41:17.955014+0000","last_peered":"2024-09-05T16:41:17.955014+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:41:17.955014+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8983,"log_dups_size":0,"ondisk_log_size":8983,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31347872,"num_objects":3827,"num_object_clones":0,"num_object_copies":15308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3827,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8981,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1582}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'9243","reported_seq":9369,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:22.598347+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:41:22.598347+0000","last_peered":"2024-09-05T16:41:22.598347+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:41:22.598347+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9243,"log_dups_size":0,"ondisk_log_size":9243,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31780592,"num_objects":3880,"num_object_clones":0,"num_object_copies":15520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3880,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9240,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1599}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'9014","reported_seq":9112,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:22.540864+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:41:22.540864+0000","last_peered":"2024-09-05T16:41:22.540864+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:41:22.540864+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9014,"log_dups_size":0,"ondisk_log_size":9014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30677584,"num_objects":3745,"num_object_clones":0,"num_object_copies":14980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3745,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9013,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1507}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'9048","reported_seq":13782,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:41:21.921778+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:41:21.921778+0000","last_peered":"2024-09-05T16:41:21.921778+0000","last_clean":"2024-09-05T16:41:21.921778+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:41:21.921778+0000","last_undegraded":"2024-09-05T16:41:21.921778+0000","last_fullsized":"2024-09-05T16:41:21.921778+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":9048,"log_dups_size":0,"ondisk_log_size":9048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30801920,"num_objects":3760,"num_object_clones":0,"num_object_copies":15040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3760,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":648,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'9312","reported_seq":9420,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:22.539595+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:41:22.539595+0000","last_peered":"2024-09-05T16:41:22.539595+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:41:22.539595+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9312,"log_dups_size":0,"ondisk_log_size":9312,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31241376,"num_objects":3814,"num_object_clones":0,"num_object_copies":15256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3814,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9310,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1480}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'9031","reported_seq":9143,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:17.631351+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:41:17.631351+0000","last_peered":"2024-09-05T16:41:17.631351+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:41:17.631351+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9031,"log_dups_size":0,"ondisk_log_size":9031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31143072,"num_objects":3802,"num_object_clones":0,"num_object_copies":15208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3802,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9029,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1494}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'9006","reported_seq":9114,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:22.550391+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:41:22.550391+0000","last_peered":"2024-09-05T16:41:22.550391+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:41:22.550391+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9006,"log_dups_size":0,"ondisk_log_size":9006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30872736,"num_objects":3769,"num_object_clones":0,"num_object_copies":15076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3769,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9004,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1549}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":496873960,"num_objects":60659,"num_object_clones":0,"num_object_copies":242636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30720,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60659,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":130178,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":917,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":994975744,"data_stored":994975744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":144302,"ondisk_log_size":144302,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084377,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":747600,"kb_used_data":242496,"kb_used_omap":4,"kb_used_meta":505083,"kb_avail":93624240,"statfs":{"total":96636764160,"available":95871221760,"internally_reserved":0,"allocated":248315904,"data_stored":248114630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":517205715},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,3,7,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986254,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":581600,"kb_used_data":243080,"kb_used_omap":4,"kb_used_meta":338491,"kb_avail":93790240,"statfs":{"total":96636764160,"available":96041205760,"internally_reserved":0,"allocated":248913920,"data_stored":248712646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":346615506},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,3,0,2,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673059,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":750872,"kb_used_data":246280,"kb_used_omap":1,"kb_used_meta":504574,"kb_avail":93620968,"statfs":{"total":96636764160,"available":95867871232,"internally_reserved":0,"allocated":252190720,"data_stored":251989446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":516684229},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182491,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":751792,"kb_used_data":242592,"kb_used_omap":4,"kb_used_meta":509179,"kb_avail":93620048,"statfs":{"total":96636764160,"available":95866929152,"internally_reserved":0,"allocated":248414208,"data_stored":248212934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":521400013},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,8,8,4,4,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":247422976,"data_stored":247422976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":248287232,"data_stored":248287232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":251793408,"data_stored":251793408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":247472128,"data_stored":247472128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:41:23.007 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:41:23.679 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:41:23.680 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:41:23.702 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":271,"stamp":"2024-09-05T16:41:22.963266+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":496873960,"num_objects":60659,"num_object_clones":0,"num_object_copies":242636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30720,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60659,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":130178,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":917,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":144302,"ondisk_log_size":144302,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2831864,"kb_used_data":974448,"kb_used_omap":14,"kb_used_meta":1857329,"kb_avail":374655496,"statfs":{"total":386547056640,"available":383647227904,"internally_reserved":0,"allocated":997834752,"data_stored":997029656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1901905463},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,1,12,11,17,12,7,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25903104,"num_objects":-3162,"num_object_clones":0,"num_object_copies":-12648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3162,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3162,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":160,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005123"},"pg_stats":[{"pgid":"1.f","version":"37'9053","reported_seq":9157,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:22.542098+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:41:22.542098+0000","last_peered":"2024-09-05T16:41:22.542098+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:41:22.542098+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9053,"log_dups_size":0,"ondisk_log_size":9053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31547392,"num_objects":3851,"num_object_clones":0,"num_object_copies":15404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3851,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9053,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1615}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'8883","reported_seq":10230,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:22.713812+0000","last_change":"2024-09-05T16:40:28.919189+0000","last_active":"2024-09-05T16:41:22.713812+0000","last_peered":"2024-09-05T16:41:22.713812+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:41:22.713812+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8883,"log_dups_size":0,"ondisk_log_size":8883,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30069920,"num_objects":3671,"num_object_clones":0,"num_object_copies":14684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1633,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3671,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8881,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":638,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2038}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'8949","reported_seq":9071,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:17.936403+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:41:17.936403+0000","last_peered":"2024-09-05T16:41:17.936403+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:41:17.936403+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8949,"log_dups_size":0,"ondisk_log_size":8949,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31152744,"num_objects":3804,"num_object_clones":0,"num_object_copies":15216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3804,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8948,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1572}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'9068","reported_seq":13796,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:41:21.814505+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:41:21.814505+0000","last_peered":"2024-09-05T16:41:21.814505+0000","last_clean":"2024-09-05T16:41:21.814505+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:41:21.814505+0000","last_undegraded":"2024-09-05T16:41:21.814505+0000","last_fullsized":"2024-09-05T16:41:21.814505+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":9068,"log_dups_size":0,"ondisk_log_size":9068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31670272,"num_objects":3866,"num_object_clones":0,"num_object_copies":15464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3866,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3368,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'8829","reported_seq":8927,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:22.404637+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:41:22.404637+0000","last_peered":"2024-09-05T16:41:22.404637+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:41:22.404637+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8829,"log_dups_size":0,"ondisk_log_size":8829,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30848160,"num_objects":3766,"num_object_clones":0,"num_object_copies":15064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3766,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8827,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1621}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'9002","reported_seq":9116,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:22.538942+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:41:22.538942+0000","last_peered":"2024-09-05T16:41:22.538942+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:41:22.538942+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9002,"log_dups_size":0,"ondisk_log_size":9002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30883840,"num_objects":3770,"num_object_clones":0,"num_object_copies":15080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3770,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9002,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1551}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'8841","reported_seq":8948,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:22.703002+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:41:22.703002+0000","last_peered":"2024-09-05T16:41:22.703002+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:41:22.703002+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8841,"log_dups_size":0,"ondisk_log_size":8841,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31298720,"num_objects":3821,"num_object_clones":0,"num_object_copies":15284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3821,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8839,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1690}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'9022","reported_seq":9108,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:22.500408+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:41:22.500408+0000","last_peered":"2024-09-05T16:41:22.500408+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:41:22.500408+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9022,"log_dups_size":0,"ondisk_log_size":9022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30774432,"num_objects":3757,"num_object_clones":0,"num_object_copies":15028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3757,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9020,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'9018","reported_seq":9117,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:22.589677+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:41:22.589677+0000","last_peered":"2024-09-05T16:41:22.589677+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:41:22.589677+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9018,"log_dups_size":0,"ondisk_log_size":9018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30763328,"num_objects":3756,"num_object_clones":0,"num_object_copies":15024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3756,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9015,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1512}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'8983","reported_seq":9104,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:17.955014+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:41:17.955014+0000","last_peered":"2024-09-05T16:41:17.955014+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:41:17.955014+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8983,"log_dups_size":0,"ondisk_log_size":8983,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31347872,"num_objects":3827,"num_object_clones":0,"num_object_copies":15308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3827,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8981,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1582}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'9243","reported_seq":9369,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:22.598347+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:41:22.598347+0000","last_peered":"2024-09-05T16:41:22.598347+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:41:22.598347+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9243,"log_dups_size":0,"ondisk_log_size":9243,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31780592,"num_objects":3880,"num_object_clones":0,"num_object_copies":15520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3880,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9240,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1599}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'9014","reported_seq":9112,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:22.540864+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:41:22.540864+0000","last_peered":"2024-09-05T16:41:22.540864+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:41:22.540864+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9014,"log_dups_size":0,"ondisk_log_size":9014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30677584,"num_objects":3745,"num_object_clones":0,"num_object_copies":14980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3745,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9013,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1507}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'9048","reported_seq":13782,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:41:21.921778+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:41:21.921778+0000","last_peered":"2024-09-05T16:41:21.921778+0000","last_clean":"2024-09-05T16:41:21.921778+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:41:21.921778+0000","last_undegraded":"2024-09-05T16:41:21.921778+0000","last_fullsized":"2024-09-05T16:41:21.921778+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":9048,"log_dups_size":0,"ondisk_log_size":9048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30801920,"num_objects":3760,"num_object_clones":0,"num_object_copies":15040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3760,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":648,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'9312","reported_seq":9420,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:22.539595+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:41:22.539595+0000","last_peered":"2024-09-05T16:41:22.539595+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:41:22.539595+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9312,"log_dups_size":0,"ondisk_log_size":9312,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31241376,"num_objects":3814,"num_object_clones":0,"num_object_copies":15256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3814,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9310,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1480}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'9031","reported_seq":9143,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:17.631351+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:41:17.631351+0000","last_peered":"2024-09-05T16:41:17.631351+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:41:17.631351+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9031,"log_dups_size":0,"ondisk_log_size":9031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31143072,"num_objects":3802,"num_object_clones":0,"num_object_copies":15208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3802,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9029,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1494}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'9006","reported_seq":9114,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:22.550391+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:41:22.550391+0000","last_peered":"2024-09-05T16:41:22.550391+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:41:22.550391+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9006,"log_dups_size":0,"ondisk_log_size":9006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30872736,"num_objects":3769,"num_object_clones":0,"num_object_copies":15076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3769,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9004,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1549}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":496873960,"num_objects":60659,"num_object_clones":0,"num_object_copies":242636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30720,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60659,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":130178,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":917,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":994975744,"data_stored":994975744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":144302,"ondisk_log_size":144302,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084377,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":747600,"kb_used_data":242496,"kb_used_omap":4,"kb_used_meta":505083,"kb_avail":93624240,"statfs":{"total":96636764160,"available":95871221760,"internally_reserved":0,"allocated":248315904,"data_stored":248114630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":517205715},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,3,7,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986254,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":581600,"kb_used_data":243080,"kb_used_omap":4,"kb_used_meta":338491,"kb_avail":93790240,"statfs":{"total":96636764160,"available":96041205760,"internally_reserved":0,"allocated":248913920,"data_stored":248712646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":346615506},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,3,0,2,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673059,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":750872,"kb_used_data":246280,"kb_used_omap":1,"kb_used_meta":504574,"kb_avail":93620968,"statfs":{"total":96636764160,"available":95867871232,"internally_reserved":0,"allocated":252190720,"data_stored":251989446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":516684229},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182491,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":751792,"kb_used_data":242592,"kb_used_omap":4,"kb_used_meta":509179,"kb_avail":93620048,"statfs":{"total":96636764160,"available":95866929152,"internally_reserved":0,"allocated":248414208,"data_stored":248212934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":521400013},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,8,8,4,4,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":247422976,"data_stored":247422976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":248287232,"data_stored":248287232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":251793408,"data_stored":251793408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":247472128,"data_stored":247472128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:41:23.703 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-05T16:41:23.703 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'9053", 'reported_seq': 9157, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:22.542098+0000', 'last_change': '2024-09-05T16:34:56.833497+0000', 'last_active': '2024-09-05T16:41:22.542098+0000', 'last_peered': '2024-09-05T16:41:22.542098+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.828289+0000', 'last_became_peered': '2024-09-05T16:34:56.828289+0000', 'last_unstale': '2024-09-05T16:41:22.542098+0000', 'last_undegraded': '2024-09-05T16:34:56.612160+0000', 'last_fullsized': '2024-09-05T16:34:56.557632+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9053, 'log_dups_size': 0, 'ondisk_log_size': 9053, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31547392, 'num_objects': 3851, 'num_object_clones': 0, 'num_object_copies': 15404, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2236, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3851, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9053, 'num_write_kb': 51616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1615}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:41:23.704 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-05T16:41:23.704 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "37'8883", 'reported_seq': 10230, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:22.713812+0000', 'last_change': '2024-09-05T16:40:28.919189+0000', 'last_active': '2024-09-05T16:41:22.713812+0000', 'last_peered': '2024-09-05T16:41:22.713812+0000', 'last_clean': '2024-09-05T16:33:53.200762+0000', 'last_became_active': '2024-09-05T16:34:56.820292+0000', 'last_became_peered': '2024-09-05T16:34:56.820292+0000', 'last_unstale': '2024-09-05T16:41:22.713812+0000', 'last_undegraded': '2024-09-05T16:34:56.605776+0000', 'last_fullsized': '2024-09-05T16:34:56.559383+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8883, 'log_dups_size': 0, 'ondisk_log_size': 8883, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30069920, 'num_objects': 3671, 'num_object_clones': 0, 'num_object_copies': 14684, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1633, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3671, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8881, 'num_write_kb': 50212, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 638, 'num_bytes_recovered': 2613248, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2038}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:41:23.704 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:41:23.704 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'8949", 'reported_seq': 9071, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:17.936403+0000', 'last_change': '2024-09-05T16:34:56.867900+0000', 'last_active': '2024-09-05T16:41:17.936403+0000', 'last_peered': '2024-09-05T16:41:17.936403+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:41:17.936403+0000', 'last_undegraded': '2024-09-05T16:34:56.590972+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8949, 'log_dups_size': 0, 'ondisk_log_size': 8949, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31152744, 'num_objects': 3804, 'num_object_clones': 0, 'num_object_copies': 15216, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2233, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3804, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 8948, 'num_write_kb': 50999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1572}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:41:23.704 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:41:23.704 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'8829", 'reported_seq': 8927, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:22.404637+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:41:22.404637+0000', 'last_peered': '2024-09-05T16:41:22.404637+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:41:22.404637+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8829, 'log_dups_size': 0, 'ondisk_log_size': 8829, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30848160, 'num_objects': 3766, 'num_object_clones': 0, 'num_object_copies': 15064, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2147, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3766, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8827, 'num_write_kb': 50373, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1621}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:41:23.704 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:41:23.704 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'9002", 'reported_seq': 9116, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:22.538942+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:41:22.538942+0000', 'last_peered': '2024-09-05T16:41:22.538942+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:41:22.538942+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9002, 'log_dups_size': 0, 'ondisk_log_size': 9002, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30883840, 'num_objects': 3770, 'num_object_clones': 0, 'num_object_copies': 15080, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3770, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9002, 'num_write_kb': 51088, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1551}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:41:23.704 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:41:23.704 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'8841", 'reported_seq': 8948, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:22.703002+0000', 'last_change': '2024-09-05T16:34:56.903436+0000', 'last_active': '2024-09-05T16:41:22.703002+0000', 'last_peered': '2024-09-05T16:41:22.703002+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:41:22.703002+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8841, 'log_dups_size': 0, 'ondisk_log_size': 8841, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31298720, 'num_objects': 3821, 'num_object_clones': 0, 'num_object_copies': 15284, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2133, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3821, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8839, 'num_write_kb': 50636, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1690}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:41:23.705 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-05T16:41:23.705 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "37'9022", 'reported_seq': 9108, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:22.500408+0000', 'last_change': '2024-09-05T16:34:56.883319+0000', 'last_active': '2024-09-05T16:41:22.500408+0000', 'last_peered': '2024-09-05T16:41:22.500408+0000', 'last_clean': '2024-09-05T16:33:53.505305+0000', 'last_became_active': '2024-09-05T16:34:56.873389+0000', 'last_became_peered': '2024-09-05T16:34:56.873389+0000', 'last_unstale': '2024-09-05T16:41:22.500408+0000', 'last_undegraded': '2024-09-05T16:34:56.602567+0000', 'last_fullsized': '2024-09-05T16:34:56.559958+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9022, 'log_dups_size': 0, 'ondisk_log_size': 9022, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30774432, 'num_objects': 3757, 'num_object_clones': 0, 'num_object_copies': 15028, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2234, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3757, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9020, 'num_write_kb': 51108, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1525}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:41:23.705 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:41:23.705 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'9018", 'reported_seq': 9117, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:22.589677+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:41:22.589677+0000', 'last_peered': '2024-09-05T16:41:22.589677+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:41:22.589677+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9018, 'log_dups_size': 0, 'ondisk_log_size': 9018, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30763328, 'num_objects': 3756, 'num_object_clones': 0, 'num_object_copies': 15024, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3756, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9015, 'num_write_kb': 51081, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1512}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:41:23.705 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:41:23.705 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'8983", 'reported_seq': 9104, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:17.955014+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:41:17.955014+0000', 'last_peered': '2024-09-05T16:41:17.955014+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:41:17.955014+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8983, 'log_dups_size': 0, 'ondisk_log_size': 8983, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31347872, 'num_objects': 3827, 'num_object_clones': 0, 'num_object_copies': 15308, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3827, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8981, 'num_write_kb': 51230, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1582}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:41:23.705 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:41:23.705 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'9243", 'reported_seq': 9369, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:22.598347+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:41:22.598347+0000', 'last_peered': '2024-09-05T16:41:22.598347+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:41:22.598347+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9243, 'log_dups_size': 0, 'ondisk_log_size': 9243, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31780592, 'num_objects': 3880, 'num_object_clones': 0, 'num_object_copies': 15520, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2284, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3880, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9240, 'num_write_kb': 52476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1599}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:41:23.705 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:41:23.705 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'9014", 'reported_seq': 9112, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:22.540864+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:41:22.540864+0000', 'last_peered': '2024-09-05T16:41:22.540864+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:41:22.540864+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9014, 'log_dups_size': 0, 'ondisk_log_size': 9014, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30677584, 'num_objects': 3745, 'num_object_clones': 0, 'num_object_copies': 14980, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3745, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9013, 'num_write_kb': 51031, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1507}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:41:23.706 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:41:23.706 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'9312", 'reported_seq': 9420, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:22.539595+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:41:22.539595+0000', 'last_peered': '2024-09-05T16:41:22.539595+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:41:22.539595+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9312, 'log_dups_size': 0, 'ondisk_log_size': 9312, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31241376, 'num_objects': 3814, 'num_object_clones': 0, 'num_object_copies': 15256, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3814, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9310, 'num_write_kb': 52500, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1480}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:41:23.706 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:41:23.706 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'9031", 'reported_seq': 9143, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:17.631351+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:41:17.631351+0000', 'last_peered': '2024-09-05T16:41:17.631351+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:41:17.631351+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9031, 'log_dups_size': 0, 'ondisk_log_size': 9031, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31143072, 'num_objects': 3802, 'num_object_clones': 0, 'num_object_copies': 15208, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2310, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3802, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9029, 'num_write_kb': 51325, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1494}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:41:23.706 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:41:23.706 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'9006", 'reported_seq': 9114, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:22.550391+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:41:22.550391+0000', 'last_peered': '2024-09-05T16:41:22.550391+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:41:22.550391+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9006, 'log_dups_size': 0, 'ondisk_log_size': 9006, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30872736, 'num_objects': 3769, 'num_object_clones': 0, 'num_object_copies': 15076, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3769, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9004, 'num_write_kb': 51092, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1549}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:41:23.706 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:41:24.459 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:41:24.459 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 30720/242636 objects degraded (12.661%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":487,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":60659,"data_bytes":496873960,"bytes_used":2899828736,"bytes_avail":383647227904,"bytes_total":386547056640,"degraded_objects":30720,"degraded_total":242636,"degraded_ratio":0.12660940668326218,"recovering_objects_per_sec":13,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":160,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":263},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:41:24.492 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:41:24.492 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:41:25.061 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:41:25.062 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:41:25.090 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":272,"stamp":"2024-09-05T16:41:24.964156+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":495030760,"num_objects":60434,"num_object_clones":0,"num_object_copies":241736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30720,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60434,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":130403,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":917,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":144527,"ondisk_log_size":144527,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2831472,"kb_used_data":969860,"kb_used_omap":14,"kb_used_meta":1861489,"kb_avail":374655888,"statfs":{"total":386547056640,"available":383647629312,"internally_reserved":0,"allocated":993136640,"data_stored":992331544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1906165303},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,1,12,13,18,13,7,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18538496,"num_objects":-2263,"num_object_clones":0,"num_object_copies":-9052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2263,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2263,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005196"},"pg_stats":[{"pgid":"1.f","version":"37'9053","reported_seq":9157,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:22.542098+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:41:22.542098+0000","last_peered":"2024-09-05T16:41:22.542098+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:41:22.542098+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9053,"log_dups_size":0,"ondisk_log_size":9053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31547392,"num_objects":3851,"num_object_clones":0,"num_object_copies":15404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3851,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9053,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1615}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'8883","reported_seq":10230,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:22.713812+0000","last_change":"2024-09-05T16:40:28.919189+0000","last_active":"2024-09-05T16:41:22.713812+0000","last_peered":"2024-09-05T16:41:22.713812+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:41:22.713812+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8883,"log_dups_size":0,"ondisk_log_size":8883,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30069920,"num_objects":3671,"num_object_clones":0,"num_object_copies":14684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1633,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3671,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8881,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":638,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2038}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'9019","reported_seq":9141,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:22.908495+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:41:22.908495+0000","last_peered":"2024-09-05T16:41:22.908495+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:41:22.908495+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9019,"log_dups_size":0,"ondisk_log_size":9019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30579304,"num_objects":3734,"num_object_clones":0,"num_object_copies":14936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3734,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9018,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1502}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'9068","reported_seq":13796,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:41:21.814505+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:41:21.814505+0000","last_peered":"2024-09-05T16:41:21.814505+0000","last_clean":"2024-09-05T16:41:21.814505+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:41:21.814505+0000","last_undegraded":"2024-09-05T16:41:21.814505+0000","last_fullsized":"2024-09-05T16:41:21.814505+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":9068,"log_dups_size":0,"ondisk_log_size":9068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31670272,"num_objects":3866,"num_object_clones":0,"num_object_copies":15464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3866,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3368,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'8829","reported_seq":8927,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:22.404637+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:41:22.404637+0000","last_peered":"2024-09-05T16:41:22.404637+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:41:22.404637+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8829,"log_dups_size":0,"ondisk_log_size":8829,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30848160,"num_objects":3766,"num_object_clones":0,"num_object_copies":15064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3766,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8827,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1621}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'9002","reported_seq":9116,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:22.538942+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:41:22.538942+0000","last_peered":"2024-09-05T16:41:22.538942+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:41:22.538942+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9002,"log_dups_size":0,"ondisk_log_size":9002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30883840,"num_objects":3770,"num_object_clones":0,"num_object_copies":15080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3770,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9002,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1551}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'8841","reported_seq":8948,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:22.703002+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:41:22.703002+0000","last_peered":"2024-09-05T16:41:22.703002+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:41:22.703002+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8841,"log_dups_size":0,"ondisk_log_size":8841,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31298720,"num_objects":3821,"num_object_clones":0,"num_object_copies":15284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3821,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8839,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1690}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'9022","reported_seq":9108,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:22.500408+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:41:22.500408+0000","last_peered":"2024-09-05T16:41:22.500408+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:41:22.500408+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9022,"log_dups_size":0,"ondisk_log_size":9022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30774432,"num_objects":3757,"num_object_clones":0,"num_object_copies":15028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3757,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9020,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'9018","reported_seq":9117,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:22.589677+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:41:22.589677+0000","last_peered":"2024-09-05T16:41:22.589677+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:41:22.589677+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9018,"log_dups_size":0,"ondisk_log_size":9018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30763328,"num_objects":3756,"num_object_clones":0,"num_object_copies":15024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3756,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9015,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1512}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'9058","reported_seq":9179,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:22.438481+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:41:22.438481+0000","last_peered":"2024-09-05T16:41:22.438481+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:41:22.438481+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9058,"log_dups_size":0,"ondisk_log_size":9058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30733472,"num_objects":3752,"num_object_clones":0,"num_object_copies":15008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9056,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1507}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'9243","reported_seq":9369,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:22.598347+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:41:22.598347+0000","last_peered":"2024-09-05T16:41:22.598347+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:41:22.598347+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9243,"log_dups_size":0,"ondisk_log_size":9243,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31780592,"num_objects":3880,"num_object_clones":0,"num_object_copies":15520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3880,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9240,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1599}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'9014","reported_seq":9112,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:22.540864+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:41:22.540864+0000","last_peered":"2024-09-05T16:41:22.540864+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:41:22.540864+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9014,"log_dups_size":0,"ondisk_log_size":9014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30677584,"num_objects":3745,"num_object_clones":0,"num_object_copies":14980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3745,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9013,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1507}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'9048","reported_seq":13782,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:41:21.921778+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:41:21.921778+0000","last_peered":"2024-09-05T16:41:21.921778+0000","last_clean":"2024-09-05T16:41:21.921778+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:41:21.921778+0000","last_undegraded":"2024-09-05T16:41:21.921778+0000","last_fullsized":"2024-09-05T16:41:21.921778+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":9048,"log_dups_size":0,"ondisk_log_size":9048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30801920,"num_objects":3760,"num_object_clones":0,"num_object_copies":15040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3760,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":648,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'9312","reported_seq":9420,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:22.539595+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:41:22.539595+0000","last_peered":"2024-09-05T16:41:22.539595+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:41:22.539595+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9312,"log_dups_size":0,"ondisk_log_size":9312,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31241376,"num_objects":3814,"num_object_clones":0,"num_object_copies":15256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3814,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9310,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1480}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'9111","reported_seq":9221,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:23.019662+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:41:23.019662+0000","last_peered":"2024-09-05T16:41:23.019662+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:41:23.019662+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9111,"log_dups_size":0,"ondisk_log_size":9111,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30487712,"num_objects":3722,"num_object_clones":0,"num_object_copies":14888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3722,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9109,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1414}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'9006","reported_seq":9114,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:22.550391+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:41:22.550391+0000","last_peered":"2024-09-05T16:41:22.550391+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:41:22.550391+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9006,"log_dups_size":0,"ondisk_log_size":9006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30872736,"num_objects":3769,"num_object_clones":0,"num_object_copies":15076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3769,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9004,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1549}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":495030760,"num_objects":60434,"num_object_clones":0,"num_object_copies":241736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30720,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60434,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":130403,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":917,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":990314496,"data_stored":990314496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":144527,"ondisk_log_size":144527,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084377,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":747600,"kb_used_data":242496,"kb_used_omap":4,"kb_used_meta":505083,"kb_avail":93624240,"statfs":{"total":96636764160,"available":95871221760,"internally_reserved":0,"allocated":248315904,"data_stored":248114630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":517205715},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,3,7,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986254,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":581600,"kb_used_data":243080,"kb_used_omap":4,"kb_used_meta":338491,"kb_avail":93790240,"statfs":{"total":96636764160,"available":96041205760,"internally_reserved":0,"allocated":248913920,"data_stored":248712646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":346615506},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,3,0,2,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673060,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":750480,"kb_used_data":241692,"kb_used_omap":1,"kb_used_meta":508734,"kb_avail":93621360,"statfs":{"total":96636764160,"available":95868272640,"internally_reserved":0,"allocated":247492608,"data_stored":247291334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":520944069},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,2,1,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182491,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":751792,"kb_used_data":242592,"kb_used_omap":4,"kb_used_meta":509179,"kb_avail":93620048,"statfs":{"total":96636764160,"available":95866929152,"internally_reserved":0,"allocated":248414208,"data_stored":248212934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":521400013},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,8,8,4,4,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":247422976,"data_stored":247422976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":248287232,"data_stored":248287232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":247132160,"data_stored":247132160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":247472128,"data_stored":247472128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:41:28.092 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:41:28.732 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:41:28.732 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:41:28.758 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":273,"stamp":"2024-09-05T16:41:26.965036+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":495030760,"num_objects":60434,"num_object_clones":0,"num_object_copies":241736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30720,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60434,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":130403,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":917,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":144527,"ondisk_log_size":144527,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2831472,"kb_used_data":969860,"kb_used_omap":14,"kb_used_meta":1861489,"kb_avail":374655888,"statfs":{"total":386547056640,"available":383647629312,"internally_reserved":0,"allocated":993136640,"data_stored":992331544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1906165303},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,1,12,13,18,13,7,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18538496,"num_objects":-2263,"num_object_clones":0,"num_object_copies":-9052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2263,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2263,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005229"},"pg_stats":[{"pgid":"1.f","version":"37'9053","reported_seq":9157,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:22.542098+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:41:22.542098+0000","last_peered":"2024-09-05T16:41:22.542098+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:41:22.542098+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9053,"log_dups_size":0,"ondisk_log_size":9053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31547392,"num_objects":3851,"num_object_clones":0,"num_object_copies":15404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3851,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9053,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1615}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'8883","reported_seq":10230,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:22.713812+0000","last_change":"2024-09-05T16:40:28.919189+0000","last_active":"2024-09-05T16:41:22.713812+0000","last_peered":"2024-09-05T16:41:22.713812+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:41:22.713812+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8883,"log_dups_size":0,"ondisk_log_size":8883,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30069920,"num_objects":3671,"num_object_clones":0,"num_object_copies":14684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1633,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3671,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8881,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":638,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2038}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'9019","reported_seq":9141,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:22.908495+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:41:22.908495+0000","last_peered":"2024-09-05T16:41:22.908495+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:41:22.908495+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9019,"log_dups_size":0,"ondisk_log_size":9019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30579304,"num_objects":3734,"num_object_clones":0,"num_object_copies":14936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3734,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9018,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1502}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'9068","reported_seq":13796,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:41:21.814505+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:41:21.814505+0000","last_peered":"2024-09-05T16:41:21.814505+0000","last_clean":"2024-09-05T16:41:21.814505+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:41:21.814505+0000","last_undegraded":"2024-09-05T16:41:21.814505+0000","last_fullsized":"2024-09-05T16:41:21.814505+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":9068,"log_dups_size":0,"ondisk_log_size":9068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31670272,"num_objects":3866,"num_object_clones":0,"num_object_copies":15464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3866,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3368,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'8829","reported_seq":8927,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:22.404637+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:41:22.404637+0000","last_peered":"2024-09-05T16:41:22.404637+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:41:22.404637+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8829,"log_dups_size":0,"ondisk_log_size":8829,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30848160,"num_objects":3766,"num_object_clones":0,"num_object_copies":15064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3766,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8827,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1621}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'9002","reported_seq":9116,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:22.538942+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:41:22.538942+0000","last_peered":"2024-09-05T16:41:22.538942+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:41:22.538942+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9002,"log_dups_size":0,"ondisk_log_size":9002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30883840,"num_objects":3770,"num_object_clones":0,"num_object_copies":15080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3770,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9002,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1551}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'8841","reported_seq":8948,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:22.703002+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:41:22.703002+0000","last_peered":"2024-09-05T16:41:22.703002+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:41:22.703002+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8841,"log_dups_size":0,"ondisk_log_size":8841,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31298720,"num_objects":3821,"num_object_clones":0,"num_object_copies":15284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3821,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8839,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1690}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'9022","reported_seq":9108,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:22.500408+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:41:22.500408+0000","last_peered":"2024-09-05T16:41:22.500408+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:41:22.500408+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9022,"log_dups_size":0,"ondisk_log_size":9022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30774432,"num_objects":3757,"num_object_clones":0,"num_object_copies":15028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3757,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9020,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'9018","reported_seq":9117,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:22.589677+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:41:22.589677+0000","last_peered":"2024-09-05T16:41:22.589677+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:41:22.589677+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9018,"log_dups_size":0,"ondisk_log_size":9018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30763328,"num_objects":3756,"num_object_clones":0,"num_object_copies":15024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3756,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9015,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1512}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'9058","reported_seq":9179,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:22.438481+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:41:22.438481+0000","last_peered":"2024-09-05T16:41:22.438481+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:41:22.438481+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9058,"log_dups_size":0,"ondisk_log_size":9058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30733472,"num_objects":3752,"num_object_clones":0,"num_object_copies":15008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9056,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1507}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'9243","reported_seq":9369,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:22.598347+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:41:22.598347+0000","last_peered":"2024-09-05T16:41:22.598347+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:41:22.598347+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9243,"log_dups_size":0,"ondisk_log_size":9243,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31780592,"num_objects":3880,"num_object_clones":0,"num_object_copies":15520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3880,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9240,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1599}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'9014","reported_seq":9112,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:22.540864+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:41:22.540864+0000","last_peered":"2024-09-05T16:41:22.540864+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:41:22.540864+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9014,"log_dups_size":0,"ondisk_log_size":9014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30677584,"num_objects":3745,"num_object_clones":0,"num_object_copies":14980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3745,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9013,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1507}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'9048","reported_seq":13782,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:41:21.921778+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:41:21.921778+0000","last_peered":"2024-09-05T16:41:21.921778+0000","last_clean":"2024-09-05T16:41:21.921778+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:41:21.921778+0000","last_undegraded":"2024-09-05T16:41:21.921778+0000","last_fullsized":"2024-09-05T16:41:21.921778+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":9048,"log_dups_size":0,"ondisk_log_size":9048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30801920,"num_objects":3760,"num_object_clones":0,"num_object_copies":15040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3760,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":648,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'9312","reported_seq":9420,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:22.539595+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:41:22.539595+0000","last_peered":"2024-09-05T16:41:22.539595+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:41:22.539595+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9312,"log_dups_size":0,"ondisk_log_size":9312,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31241376,"num_objects":3814,"num_object_clones":0,"num_object_copies":15256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3814,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9310,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1480}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'9111","reported_seq":9221,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:23.019662+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:41:23.019662+0000","last_peered":"2024-09-05T16:41:23.019662+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:41:23.019662+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9111,"log_dups_size":0,"ondisk_log_size":9111,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30487712,"num_objects":3722,"num_object_clones":0,"num_object_copies":14888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3722,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9109,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1414}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'9006","reported_seq":9114,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:22.550391+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:41:22.550391+0000","last_peered":"2024-09-05T16:41:22.550391+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:41:22.550391+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9006,"log_dups_size":0,"ondisk_log_size":9006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30872736,"num_objects":3769,"num_object_clones":0,"num_object_copies":15076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3769,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9004,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1549}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":495030760,"num_objects":60434,"num_object_clones":0,"num_object_copies":241736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30720,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60434,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":130403,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":917,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":990314496,"data_stored":990314496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":144527,"ondisk_log_size":144527,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084377,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":747600,"kb_used_data":242496,"kb_used_omap":4,"kb_used_meta":505083,"kb_avail":93624240,"statfs":{"total":96636764160,"available":95871221760,"internally_reserved":0,"allocated":248315904,"data_stored":248114630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":517205715},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,3,7,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986254,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":581600,"kb_used_data":243080,"kb_used_omap":4,"kb_used_meta":338491,"kb_avail":93790240,"statfs":{"total":96636764160,"available":96041205760,"internally_reserved":0,"allocated":248913920,"data_stored":248712646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":346615506},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,3,0,2,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673060,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":750480,"kb_used_data":241692,"kb_used_omap":1,"kb_used_meta":508734,"kb_avail":93621360,"statfs":{"total":96636764160,"available":95868272640,"internally_reserved":0,"allocated":247492608,"data_stored":247291334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":520944069},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,2,1,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182491,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":751792,"kb_used_data":242592,"kb_used_omap":4,"kb_used_meta":509179,"kb_avail":93620048,"statfs":{"total":96636764160,"available":95866929152,"internally_reserved":0,"allocated":248414208,"data_stored":248212934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":521400013},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,8,8,4,4,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":247422976,"data_stored":247422976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":248287232,"data_stored":248287232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":247132160,"data_stored":247132160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":247472128,"data_stored":247472128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:41:28.759 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:41:29.400 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:41:29.401 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:41:29.423 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":274,"stamp":"2024-09-05T16:41:28.965870+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":485945832,"num_objects":59325,"num_object_clones":0,"num_object_copies":237300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30669,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59325,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":131512,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":968,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":145636,"ondisk_log_size":145636,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2797328,"kb_used_data":952340,"kb_used_omap":14,"kb_used_meta":1844849,"kb_avail":374690032,"statfs":{"total":386547056640,"available":383682592768,"internally_reserved":0,"allocated":975196160,"data_stored":974391064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1889125943},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,12,19,18,10,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-26550272,"num_objects":-3241,"num_object_clones":0,"num_object_copies":-12964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3241,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":154,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005239"},"pg_stats":[{"pgid":"1.f","version":"37'9130","reported_seq":9234,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:27.489685+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:41:27.489685+0000","last_peered":"2024-09-05T16:41:27.489685+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:41:27.489685+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9130,"log_dups_size":0,"ondisk_log_size":9130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30916608,"num_objects":3774,"num_object_clones":0,"num_object_copies":15096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3774,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9130,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1538}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'8952","reported_seq":10399,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:27.809708+0000","last_change":"2024-09-05T16:40:28.919189+0000","last_active":"2024-09-05T16:41:27.809708+0000","last_peered":"2024-09-05T16:41:27.809708+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:41:27.809708+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8952,"log_dups_size":0,"ondisk_log_size":8952,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29504672,"num_objects":3602,"num_object_clones":0,"num_object_copies":14408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1582,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3602,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8950,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":689,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2020}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'9094","reported_seq":9214,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:27.926406+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:41:27.926406+0000","last_peered":"2024-09-05T16:41:27.926406+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:41:27.926406+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9094,"log_dups_size":0,"ondisk_log_size":9094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29964904,"num_objects":3659,"num_object_clones":0,"num_object_copies":14636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3659,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9093,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1427}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'9142","reported_seq":13870,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:41:26.938159+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:41:26.938159+0000","last_peered":"2024-09-05T16:41:26.938159+0000","last_clean":"2024-09-05T16:41:26.938159+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:41:26.938159+0000","last_undegraded":"2024-09-05T16:41:26.938159+0000","last_fullsized":"2024-09-05T16:41:26.938159+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":9142,"log_dups_size":0,"ondisk_log_size":9142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31064064,"num_objects":3792,"num_object_clones":0,"num_object_copies":15168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3792,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3442,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'8905","reported_seq":9003,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:27.558278+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:41:27.558278+0000","last_peered":"2024-09-05T16:41:27.558278+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:41:27.558278+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8905,"log_dups_size":0,"ondisk_log_size":8905,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30225568,"num_objects":3690,"num_object_clones":0,"num_object_copies":14760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3690,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8903,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1545}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'9071","reported_seq":9185,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:27.670899+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:41:27.670899+0000","last_peered":"2024-09-05T16:41:27.670899+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:41:27.670899+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9071,"log_dups_size":0,"ondisk_log_size":9071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30318592,"num_objects":3701,"num_object_clones":0,"num_object_copies":14804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3701,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9071,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1482}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'8910","reported_seq":9017,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:27.521737+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:41:27.521737+0000","last_peered":"2024-09-05T16:41:27.521737+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:41:27.521737+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8910,"log_dups_size":0,"ondisk_log_size":8910,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30733472,"num_objects":3752,"num_object_clones":0,"num_object_copies":15008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8908,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1621}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'9085","reported_seq":9173,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:27.641947+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:41:27.641947+0000","last_peered":"2024-09-05T16:41:27.641947+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:41:27.641947+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9085,"log_dups_size":0,"ondisk_log_size":9085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30258336,"num_objects":3694,"num_object_clones":0,"num_object_copies":14776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3694,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9083,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1462}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'9096","reported_seq":9196,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:27.531507+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:41:27.531507+0000","last_peered":"2024-09-05T16:41:27.531507+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:41:27.531507+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9096,"log_dups_size":0,"ondisk_log_size":9096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30124352,"num_objects":3678,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3678,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9093,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1434}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'9132","reported_seq":9251,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:28.072846+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:41:28.072846+0000","last_peered":"2024-09-05T16:41:28.072846+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:41:28.072846+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9132,"log_dups_size":0,"ondisk_log_size":9132,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30127264,"num_objects":3678,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3678,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9130,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1433}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'9310","reported_seq":9436,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:27.596534+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:41:27.596534+0000","last_peered":"2024-09-05T16:41:27.596534+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:41:27.596534+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9310,"log_dups_size":0,"ondisk_log_size":9310,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31231728,"num_objects":3813,"num_object_clones":0,"num_object_copies":15252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3813,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9307,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1532}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'9071","reported_seq":9169,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:27.550052+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:41:27.550052+0000","last_peered":"2024-09-05T16:41:27.550052+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:41:27.550052+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9071,"log_dups_size":0,"ondisk_log_size":9071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30210640,"num_objects":3688,"num_object_clones":0,"num_object_copies":14752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9070,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1450}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'9116","reported_seq":13851,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:41:26.938597+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:41:26.938597+0000","last_peered":"2024-09-05T16:41:26.938597+0000","last_clean":"2024-09-05T16:41:26.938597+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:41:26.938597+0000","last_undegraded":"2024-09-05T16:41:26.938597+0000","last_fullsized":"2024-09-05T16:41:26.938597+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":9116,"log_dups_size":0,"ondisk_log_size":9116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30244864,"num_objects":3692,"num_object_clones":0,"num_object_copies":14768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3692,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":716,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'9377","reported_seq":9485,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:27.601580+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:41:27.601580+0000","last_peered":"2024-09-05T16:41:27.601580+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:41:27.601580+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9377,"log_dups_size":0,"ondisk_log_size":9377,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30708896,"num_objects":3749,"num_object_clones":0,"num_object_copies":14996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3749,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9375,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1415}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'9173","reported_seq":9285,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:28.001903+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:41:28.001903+0000","last_peered":"2024-09-05T16:41:28.001903+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:41:28.001903+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9173,"log_dups_size":0,"ondisk_log_size":9173,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29979808,"num_objects":3660,"num_object_clones":0,"num_object_copies":14640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3660,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9171,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1352}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'9072","reported_seq":9180,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:27.547582+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:41:27.547582+0000","last_peered":"2024-09-05T16:41:27.547582+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:41:27.547582+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9072,"log_dups_size":0,"ondisk_log_size":9072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30332064,"num_objects":3703,"num_object_clones":0,"num_object_copies":14812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3703,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9070,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1483}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":485945832,"num_objects":59325,"num_object_clones":0,"num_object_copies":237300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30669,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59325,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":131512,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":968,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":972148736,"data_stored":972148736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":145636,"ondisk_log_size":145636,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084378,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":747412,"kb_used_data":238128,"kb_used_omap":4,"kb_used_meta":509243,"kb_avail":93624428,"statfs":{"total":96636764160,"available":95871414272,"internally_reserved":0,"allocated":243843072,"data_stored":243641798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":521465555},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,7,4,11,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986255,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":581492,"kb_used_data":238796,"kb_used_omap":4,"kb_used_meta":342651,"kb_avail":93790348,"statfs":{"total":96636764160,"available":96041316352,"internally_reserved":0,"allocated":244527104,"data_stored":244325830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":350875346},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,5,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673061,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":733576,"kb_used_data":237300,"kb_used_omap":1,"kb_used_meta":496254,"kb_avail":93638264,"statfs":{"total":96636764160,"available":95885582336,"internally_reserved":0,"allocated":242995200,"data_stored":242793926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":508164549},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,7,2,2,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182492,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":734848,"kb_used_data":238116,"kb_used_omap":4,"kb_used_meta":496699,"kb_avail":93636992,"statfs":{"total":96636764160,"available":95884279808,"internally_reserved":0,"allocated":243830784,"data_stored":243629510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":508620493},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":242880512,"data_stored":242880512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":243642368,"data_stored":243642368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":242642944,"data_stored":242642944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":242982912,"data_stored":242982912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:41:29.424 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-05T16:41:29.424 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'9130", 'reported_seq': 9234, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:27.489685+0000', 'last_change': '2024-09-05T16:34:56.833497+0000', 'last_active': '2024-09-05T16:41:27.489685+0000', 'last_peered': '2024-09-05T16:41:27.489685+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.828289+0000', 'last_became_peered': '2024-09-05T16:34:56.828289+0000', 'last_unstale': '2024-09-05T16:41:27.489685+0000', 'last_undegraded': '2024-09-05T16:34:56.612160+0000', 'last_fullsized': '2024-09-05T16:34:56.557632+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9130, 'log_dups_size': 0, 'ondisk_log_size': 9130, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30916608, 'num_objects': 3774, 'num_object_clones': 0, 'num_object_copies': 15096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2236, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3774, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9130, 'num_write_kb': 51616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1538}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:41:29.425 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-05T16:41:29.425 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "37'8952", 'reported_seq': 10399, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:27.809708+0000', 'last_change': '2024-09-05T16:40:28.919189+0000', 'last_active': '2024-09-05T16:41:27.809708+0000', 'last_peered': '2024-09-05T16:41:27.809708+0000', 'last_clean': '2024-09-05T16:33:53.200762+0000', 'last_became_active': '2024-09-05T16:34:56.820292+0000', 'last_became_peered': '2024-09-05T16:34:56.820292+0000', 'last_unstale': '2024-09-05T16:41:27.809708+0000', 'last_undegraded': '2024-09-05T16:34:56.605776+0000', 'last_fullsized': '2024-09-05T16:34:56.559383+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8952, 'log_dups_size': 0, 'ondisk_log_size': 8952, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29504672, 'num_objects': 3602, 'num_object_clones': 0, 'num_object_copies': 14408, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1582, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3602, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8950, 'num_write_kb': 50212, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 689, 'num_bytes_recovered': 2613248, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2020}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:41:29.425 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:41:29.425 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'9094", 'reported_seq': 9214, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:27.926406+0000', 'last_change': '2024-09-05T16:34:56.867900+0000', 'last_active': '2024-09-05T16:41:27.926406+0000', 'last_peered': '2024-09-05T16:41:27.926406+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:41:27.926406+0000', 'last_undegraded': '2024-09-05T16:34:56.590972+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9094, 'log_dups_size': 0, 'ondisk_log_size': 9094, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29964904, 'num_objects': 3659, 'num_object_clones': 0, 'num_object_copies': 14636, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2233, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3659, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 9093, 'num_write_kb': 50999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1427}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:41:29.425 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:41:29.425 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'8905", 'reported_seq': 9003, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:27.558278+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:41:27.558278+0000', 'last_peered': '2024-09-05T16:41:27.558278+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:41:27.558278+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8905, 'log_dups_size': 0, 'ondisk_log_size': 8905, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30225568, 'num_objects': 3690, 'num_object_clones': 0, 'num_object_copies': 14760, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2147, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3690, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8903, 'num_write_kb': 50373, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1545}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:41:29.425 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:41:29.425 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'9071", 'reported_seq': 9185, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:27.670899+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:41:27.670899+0000', 'last_peered': '2024-09-05T16:41:27.670899+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:41:27.670899+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9071, 'log_dups_size': 0, 'ondisk_log_size': 9071, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30318592, 'num_objects': 3701, 'num_object_clones': 0, 'num_object_copies': 14804, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3701, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9071, 'num_write_kb': 51088, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1482}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:41:29.426 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:41:29.426 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'8910", 'reported_seq': 9017, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:27.521737+0000', 'last_change': '2024-09-05T16:34:56.903436+0000', 'last_active': '2024-09-05T16:41:27.521737+0000', 'last_peered': '2024-09-05T16:41:27.521737+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:41:27.521737+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8910, 'log_dups_size': 0, 'ondisk_log_size': 8910, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30733472, 'num_objects': 3752, 'num_object_clones': 0, 'num_object_copies': 15008, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2133, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3752, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8908, 'num_write_kb': 50636, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1621}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:41:29.426 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-05T16:41:29.426 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "37'9085", 'reported_seq': 9173, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:27.641947+0000', 'last_change': '2024-09-05T16:34:56.883319+0000', 'last_active': '2024-09-05T16:41:27.641947+0000', 'last_peered': '2024-09-05T16:41:27.641947+0000', 'last_clean': '2024-09-05T16:33:53.505305+0000', 'last_became_active': '2024-09-05T16:34:56.873389+0000', 'last_became_peered': '2024-09-05T16:34:56.873389+0000', 'last_unstale': '2024-09-05T16:41:27.641947+0000', 'last_undegraded': '2024-09-05T16:34:56.602567+0000', 'last_fullsized': '2024-09-05T16:34:56.559958+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9085, 'log_dups_size': 0, 'ondisk_log_size': 9085, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30258336, 'num_objects': 3694, 'num_object_clones': 0, 'num_object_copies': 14776, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2234, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3694, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9083, 'num_write_kb': 51108, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1462}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:41:29.426 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:41:29.426 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'9096", 'reported_seq': 9196, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:27.531507+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:41:27.531507+0000', 'last_peered': '2024-09-05T16:41:27.531507+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:41:27.531507+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9096, 'log_dups_size': 0, 'ondisk_log_size': 9096, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30124352, 'num_objects': 3678, 'num_object_clones': 0, 'num_object_copies': 14712, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3678, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9093, 'num_write_kb': 51081, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1434}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:41:29.426 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:41:29.426 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'9132", 'reported_seq': 9251, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:28.072846+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:41:28.072846+0000', 'last_peered': '2024-09-05T16:41:28.072846+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:41:28.072846+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9132, 'log_dups_size': 0, 'ondisk_log_size': 9132, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30127264, 'num_objects': 3678, 'num_object_clones': 0, 'num_object_copies': 14712, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3678, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9130, 'num_write_kb': 51230, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1433}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:41:29.427 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:41:29.427 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'9310", 'reported_seq': 9436, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:27.596534+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:41:27.596534+0000', 'last_peered': '2024-09-05T16:41:27.596534+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:41:27.596534+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9310, 'log_dups_size': 0, 'ondisk_log_size': 9310, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31231728, 'num_objects': 3813, 'num_object_clones': 0, 'num_object_copies': 15252, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2284, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3813, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9307, 'num_write_kb': 52476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1532}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:41:29.427 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:41:29.427 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'9071", 'reported_seq': 9169, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:27.550052+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:41:27.550052+0000', 'last_peered': '2024-09-05T16:41:27.550052+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:41:27.550052+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9071, 'log_dups_size': 0, 'ondisk_log_size': 9071, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30210640, 'num_objects': 3688, 'num_object_clones': 0, 'num_object_copies': 14752, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3688, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9070, 'num_write_kb': 51031, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1450}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:41:29.427 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:41:29.427 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'9377", 'reported_seq': 9485, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:27.601580+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:41:27.601580+0000', 'last_peered': '2024-09-05T16:41:27.601580+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:41:27.601580+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9377, 'log_dups_size': 0, 'ondisk_log_size': 9377, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30708896, 'num_objects': 3749, 'num_object_clones': 0, 'num_object_copies': 14996, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3749, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9375, 'num_write_kb': 52500, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1415}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:41:29.427 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:41:29.427 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'9173", 'reported_seq': 9285, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:28.001903+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:41:28.001903+0000', 'last_peered': '2024-09-05T16:41:28.001903+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:41:28.001903+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9173, 'log_dups_size': 0, 'ondisk_log_size': 9173, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29979808, 'num_objects': 3660, 'num_object_clones': 0, 'num_object_copies': 14640, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2310, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3660, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9171, 'num_write_kb': 51325, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1352}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:41:29.427 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:41:29.427 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'9072", 'reported_seq': 9180, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:27.547582+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:41:27.547582+0000', 'last_peered': '2024-09-05T16:41:27.547582+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:41:27.547582+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9072, 'log_dups_size': 0, 'ondisk_log_size': 9072, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30332064, 'num_objects': 3703, 'num_object_clones': 0, 'num_object_copies': 14812, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3703, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9070, 'num_write_kb': 51092, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1483}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:41:29.428 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:41:30.193 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:41:30.251 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 30669/237300 objects degraded (12.924%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":493,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":59325,"data_bytes":485945832,"bytes_used":2864463872,"bytes_avail":383682592768,"bytes_total":386547056640,"degraded_objects":30669,"degraded_total":237300,"degraded_ratio":0.12924146649810367,"recovering_objects_per_sec":12,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":154,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":269},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:41:30.253 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:41:30.253 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:41:30.875 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:41:30.876 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:41:30.904 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":274,"stamp":"2024-09-05T16:41:28.965870+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":485945832,"num_objects":59325,"num_object_clones":0,"num_object_copies":237300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30669,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59325,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":131512,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":968,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":145636,"ondisk_log_size":145636,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2797328,"kb_used_data":952340,"kb_used_omap":14,"kb_used_meta":1844849,"kb_avail":374690032,"statfs":{"total":386547056640,"available":383682592768,"internally_reserved":0,"allocated":975196160,"data_stored":974391064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1889125943},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,12,19,18,10,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-26550272,"num_objects":-3241,"num_object_clones":0,"num_object_copies":-12964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3241,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":154,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005239"},"pg_stats":[{"pgid":"1.f","version":"37'9130","reported_seq":9234,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:27.489685+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:41:27.489685+0000","last_peered":"2024-09-05T16:41:27.489685+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:41:27.489685+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9130,"log_dups_size":0,"ondisk_log_size":9130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30916608,"num_objects":3774,"num_object_clones":0,"num_object_copies":15096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3774,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9130,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1538}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'8952","reported_seq":10399,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:27.809708+0000","last_change":"2024-09-05T16:40:28.919189+0000","last_active":"2024-09-05T16:41:27.809708+0000","last_peered":"2024-09-05T16:41:27.809708+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:41:27.809708+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8952,"log_dups_size":0,"ondisk_log_size":8952,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29504672,"num_objects":3602,"num_object_clones":0,"num_object_copies":14408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1582,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3602,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8950,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":689,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2020}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'9094","reported_seq":9214,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:27.926406+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:41:27.926406+0000","last_peered":"2024-09-05T16:41:27.926406+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:41:27.926406+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9094,"log_dups_size":0,"ondisk_log_size":9094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29964904,"num_objects":3659,"num_object_clones":0,"num_object_copies":14636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3659,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9093,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1427}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'9142","reported_seq":13870,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:41:26.938159+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:41:26.938159+0000","last_peered":"2024-09-05T16:41:26.938159+0000","last_clean":"2024-09-05T16:41:26.938159+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:41:26.938159+0000","last_undegraded":"2024-09-05T16:41:26.938159+0000","last_fullsized":"2024-09-05T16:41:26.938159+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":9142,"log_dups_size":0,"ondisk_log_size":9142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31064064,"num_objects":3792,"num_object_clones":0,"num_object_copies":15168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3792,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3442,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'8905","reported_seq":9003,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:27.558278+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:41:27.558278+0000","last_peered":"2024-09-05T16:41:27.558278+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:41:27.558278+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8905,"log_dups_size":0,"ondisk_log_size":8905,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30225568,"num_objects":3690,"num_object_clones":0,"num_object_copies":14760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3690,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8903,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1545}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'9071","reported_seq":9185,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:27.670899+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:41:27.670899+0000","last_peered":"2024-09-05T16:41:27.670899+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:41:27.670899+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9071,"log_dups_size":0,"ondisk_log_size":9071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30318592,"num_objects":3701,"num_object_clones":0,"num_object_copies":14804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3701,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9071,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1482}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'8910","reported_seq":9017,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:27.521737+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:41:27.521737+0000","last_peered":"2024-09-05T16:41:27.521737+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:41:27.521737+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8910,"log_dups_size":0,"ondisk_log_size":8910,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30733472,"num_objects":3752,"num_object_clones":0,"num_object_copies":15008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8908,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1621}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'9085","reported_seq":9173,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:27.641947+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:41:27.641947+0000","last_peered":"2024-09-05T16:41:27.641947+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:41:27.641947+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9085,"log_dups_size":0,"ondisk_log_size":9085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30258336,"num_objects":3694,"num_object_clones":0,"num_object_copies":14776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3694,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9083,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1462}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'9096","reported_seq":9196,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:27.531507+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:41:27.531507+0000","last_peered":"2024-09-05T16:41:27.531507+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:41:27.531507+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9096,"log_dups_size":0,"ondisk_log_size":9096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30124352,"num_objects":3678,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3678,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9093,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1434}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'9132","reported_seq":9251,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:28.072846+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:41:28.072846+0000","last_peered":"2024-09-05T16:41:28.072846+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:41:28.072846+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9132,"log_dups_size":0,"ondisk_log_size":9132,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30127264,"num_objects":3678,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3678,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9130,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1433}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'9310","reported_seq":9436,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:27.596534+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:41:27.596534+0000","last_peered":"2024-09-05T16:41:27.596534+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:41:27.596534+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9310,"log_dups_size":0,"ondisk_log_size":9310,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31231728,"num_objects":3813,"num_object_clones":0,"num_object_copies":15252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3813,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9307,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1532}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'9071","reported_seq":9169,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:27.550052+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:41:27.550052+0000","last_peered":"2024-09-05T16:41:27.550052+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:41:27.550052+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9071,"log_dups_size":0,"ondisk_log_size":9071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30210640,"num_objects":3688,"num_object_clones":0,"num_object_copies":14752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9070,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1450}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'9116","reported_seq":13851,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:41:26.938597+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:41:26.938597+0000","last_peered":"2024-09-05T16:41:26.938597+0000","last_clean":"2024-09-05T16:41:26.938597+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:41:26.938597+0000","last_undegraded":"2024-09-05T16:41:26.938597+0000","last_fullsized":"2024-09-05T16:41:26.938597+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":9116,"log_dups_size":0,"ondisk_log_size":9116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30244864,"num_objects":3692,"num_object_clones":0,"num_object_copies":14768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3692,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":716,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'9377","reported_seq":9485,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:27.601580+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:41:27.601580+0000","last_peered":"2024-09-05T16:41:27.601580+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:41:27.601580+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9377,"log_dups_size":0,"ondisk_log_size":9377,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30708896,"num_objects":3749,"num_object_clones":0,"num_object_copies":14996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3749,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9375,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1415}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'9173","reported_seq":9285,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:28.001903+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:41:28.001903+0000","last_peered":"2024-09-05T16:41:28.001903+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:41:28.001903+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9173,"log_dups_size":0,"ondisk_log_size":9173,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29979808,"num_objects":3660,"num_object_clones":0,"num_object_copies":14640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3660,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9171,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1352}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'9072","reported_seq":9180,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:27.547582+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:41:27.547582+0000","last_peered":"2024-09-05T16:41:27.547582+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:41:27.547582+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9072,"log_dups_size":0,"ondisk_log_size":9072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30332064,"num_objects":3703,"num_object_clones":0,"num_object_copies":14812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3703,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9070,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1483}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":485945832,"num_objects":59325,"num_object_clones":0,"num_object_copies":237300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30669,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59325,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":131512,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":968,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":972148736,"data_stored":972148736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":145636,"ondisk_log_size":145636,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084378,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":747412,"kb_used_data":238128,"kb_used_omap":4,"kb_used_meta":509243,"kb_avail":93624428,"statfs":{"total":96636764160,"available":95871414272,"internally_reserved":0,"allocated":243843072,"data_stored":243641798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":521465555},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,7,4,11,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986255,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":581492,"kb_used_data":238796,"kb_used_omap":4,"kb_used_meta":342651,"kb_avail":93790348,"statfs":{"total":96636764160,"available":96041316352,"internally_reserved":0,"allocated":244527104,"data_stored":244325830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":350875346},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,5,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673061,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":733576,"kb_used_data":237300,"kb_used_omap":1,"kb_used_meta":496254,"kb_avail":93638264,"statfs":{"total":96636764160,"available":95885582336,"internally_reserved":0,"allocated":242995200,"data_stored":242793926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":508164549},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,7,2,2,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182492,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":734848,"kb_used_data":238116,"kb_used_omap":4,"kb_used_meta":496699,"kb_avail":93636992,"statfs":{"total":96636764160,"available":95884279808,"internally_reserved":0,"allocated":243830784,"data_stored":243629510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":508620493},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":242880512,"data_stored":242880512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":243642368,"data_stored":243642368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":242642944,"data_stored":242642944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":242982912,"data_stored":242982912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:41:33.906 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:41:34.524 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:41:34.525 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:41:34.551 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":276,"stamp":"2024-09-05T16:41:32.967249+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":478949864,"num_objects":58471,"num_object_clones":0,"num_object_copies":233884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30613,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58471,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":132366,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1024,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":146490,"ondisk_log_size":146490,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2833200,"kb_used_data":939668,"kb_used_omap":14,"kb_used_meta":1893425,"kb_avail":374654160,"statfs":{"total":386547056640,"available":383645859840,"internally_reserved":0,"allocated":962220032,"data_stored":961414936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1938867767},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,3,1,5,16,7,11,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25346048,"num_objects":-3094,"num_object_clones":0,"num_object_copies":-12376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-162,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3094,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3094,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":162,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004813"},"pg_stats":[{"pgid":"1.f","version":"37'9205","reported_seq":9309,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:32.624897+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:41:32.624897+0000","last_peered":"2024-09-05T16:41:32.624897+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:41:32.624897+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9205,"log_dups_size":0,"ondisk_log_size":9205,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30302208,"num_objects":3699,"num_object_clones":0,"num_object_copies":14796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3699,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9205,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1463}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'9018","reported_seq":10579,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:32.804629+0000","last_change":"2024-09-05T16:40:28.919189+0000","last_active":"2024-09-05T16:41:32.804629+0000","last_peered":"2024-09-05T16:41:32.804629+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:41:32.804629+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9018,"log_dups_size":0,"ondisk_log_size":9018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28964000,"num_objects":3536,"num_object_clones":0,"num_object_copies":14144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3536,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9016,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":745,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2010}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'9094","reported_seq":9214,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:27.926406+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:41:27.926406+0000","last_peered":"2024-09-05T16:41:27.926406+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:41:27.926406+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9094,"log_dups_size":0,"ondisk_log_size":9094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29964904,"num_objects":3659,"num_object_clones":0,"num_object_copies":14636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3659,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9093,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1427}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'9203","reported_seq":13931,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:41:32.100387+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:41:32.100387+0000","last_peered":"2024-09-05T16:41:32.100387+0000","last_clean":"2024-09-05T16:41:32.100387+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:41:32.100387+0000","last_undegraded":"2024-09-05T16:41:32.100387+0000","last_fullsized":"2024-09-05T16:41:32.100387+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":9203,"log_dups_size":0,"ondisk_log_size":9203,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30564352,"num_objects":3731,"num_object_clones":0,"num_object_copies":14924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3731,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3503,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'8975","reported_seq":9073,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:32.639159+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:41:32.639159+0000","last_peered":"2024-09-05T16:41:32.639159+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:41:32.639159+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8975,"log_dups_size":0,"ondisk_log_size":8975,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29652128,"num_objects":3620,"num_object_clones":0,"num_object_copies":14480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3620,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8973,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1475}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'9126","reported_seq":9240,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:32.731987+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:41:32.731987+0000","last_peered":"2024-09-05T16:41:32.731987+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:41:32.731987+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9126,"log_dups_size":0,"ondisk_log_size":9126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29868032,"num_objects":3646,"num_object_clones":0,"num_object_copies":14584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3646,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9126,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1427}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'8972","reported_seq":9078,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:32.766616+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:41:32.766616+0000","last_peered":"2024-09-05T16:41:32.766616+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:41:32.766616+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8972,"log_dups_size":0,"ondisk_log_size":8972,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30225568,"num_objects":3690,"num_object_clones":0,"num_object_copies":14760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3690,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8970,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1559}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'9143","reported_seq":9231,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:32.534889+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:41:32.534889+0000","last_peered":"2024-09-05T16:41:32.534889+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:41:32.534889+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9143,"log_dups_size":0,"ondisk_log_size":9143,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29783200,"num_objects":3636,"num_object_clones":0,"num_object_copies":14544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9141,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1404}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'9172","reported_seq":9272,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:32.536454+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:41:32.536454+0000","last_peered":"2024-09-05T16:41:32.536454+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:41:32.536454+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9172,"log_dups_size":0,"ondisk_log_size":9172,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29501760,"num_objects":3602,"num_object_clones":0,"num_object_copies":14408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3602,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9169,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1358}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'9132","reported_seq":9251,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:28.072846+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:41:28.072846+0000","last_peered":"2024-09-05T16:41:28.072846+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:41:28.072846+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9132,"log_dups_size":0,"ondisk_log_size":9132,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30127264,"num_objects":3678,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3678,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9130,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1433}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'9380","reported_seq":9506,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:32.536947+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:41:32.536947+0000","last_peered":"2024-09-05T16:41:32.536947+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:41:32.536947+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9380,"log_dups_size":0,"ondisk_log_size":9380,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30658288,"num_objects":3743,"num_object_clones":0,"num_object_copies":14972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3743,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9377,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1462}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'9146","reported_seq":9244,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:32.569893+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:41:32.569893+0000","last_peered":"2024-09-05T16:41:32.569893+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:41:32.569893+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9146,"log_dups_size":0,"ondisk_log_size":9146,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29596240,"num_objects":3613,"num_object_clones":0,"num_object_copies":14452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3613,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9145,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1375}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'9167","reported_seq":13902,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:41:31.989986+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:41:31.989986+0000","last_peered":"2024-09-05T16:41:31.989986+0000","last_clean":"2024-09-05T16:41:31.989986+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:41:31.989986+0000","last_undegraded":"2024-09-05T16:41:31.989986+0000","last_fullsized":"2024-09-05T16:41:31.989986+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":9167,"log_dups_size":0,"ondisk_log_size":9167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29827072,"num_objects":3641,"num_object_clones":0,"num_object_copies":14564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3641,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":767,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'9431","reported_seq":9539,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:32.605712+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:41:32.605712+0000","last_peered":"2024-09-05T16:41:32.605712+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:41:32.605712+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9431,"log_dups_size":0,"ondisk_log_size":9431,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30266528,"num_objects":3695,"num_object_clones":0,"num_object_copies":14780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3695,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9429,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1361}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'9173","reported_seq":9285,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:28.001903+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:41:28.001903+0000","last_peered":"2024-09-05T16:41:28.001903+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:41:28.001903+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9173,"log_dups_size":0,"ondisk_log_size":9173,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29979808,"num_objects":3660,"num_object_clones":0,"num_object_copies":14640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3660,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9171,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1352}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'9153","reported_seq":9261,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:32.439586+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:41:32.439586+0000","last_peered":"2024-09-05T16:41:32.439586+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:41:32.439586+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9153,"log_dups_size":0,"ondisk_log_size":9153,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29668512,"num_objects":3622,"num_object_clones":0,"num_object_copies":14488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3622,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9151,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1402}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":478949864,"num_objects":58471,"num_object_clones":0,"num_object_copies":233884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30613,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58471,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":132366,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1024,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":958836736,"data_stored":958836736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":146490,"ondisk_log_size":146490,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084379,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":747428,"kb_used_data":234004,"kb_used_omap":4,"kb_used_meta":513403,"kb_avail":93624412,"statfs":{"total":96636764160,"available":95871397888,"internally_reserved":0,"allocated":239620096,"data_stored":239418822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":525725395},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,1,0,1,1,1,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986256,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":617428,"kb_used_data":234480,"kb_used_omap":4,"kb_used_meta":382907,"kb_avail":93754412,"statfs":{"total":96636764160,"available":96004517888,"internally_reserved":0,"allocated":240107520,"data_stored":239906246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":392097490},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,5,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673061,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":733576,"kb_used_data":237300,"kb_used_omap":1,"kb_used_meta":496254,"kb_avail":93638264,"statfs":{"total":96636764160,"available":95885582336,"internally_reserved":0,"allocated":242995200,"data_stored":242793926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":508164549},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,7,2,2,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182493,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":734768,"kb_used_data":233884,"kb_used_omap":4,"kb_used_meta":500859,"kb_avail":93637072,"statfs":{"total":96636764160,"available":95884361728,"internally_reserved":0,"allocated":239497216,"data_stored":239295942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":512880333},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":238432256,"data_stored":238432256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":239226880,"data_stored":239226880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":242642944,"data_stored":242642944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":238534656,"data_stored":238534656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:41:34.552 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:41:35.209 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:41:35.210 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:41:35.239 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":277,"stamp":"2024-09-05T16:41:34.968195+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":477237736,"num_objects":58262,"num_object_clones":0,"num_object_copies":233048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30613,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58262,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":132575,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1024,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":146699,"ondisk_log_size":146699,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2833092,"kb_used_data":935404,"kb_used_omap":14,"kb_used_meta":1897585,"kb_avail":374654268,"statfs":{"total":386547056640,"available":383645970432,"internally_reserved":0,"allocated":957853696,"data_stored":957048600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1943127607},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,2,1,11,10,9,9,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19636224,"num_objects":-2397,"num_object_clones":0,"num_object_copies":-9588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-107,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2397,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2397,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":107,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004929"},"pg_stats":[{"pgid":"1.f","version":"37'9205","reported_seq":9309,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:32.624897+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:41:32.624897+0000","last_peered":"2024-09-05T16:41:32.624897+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:41:32.624897+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9205,"log_dups_size":0,"ondisk_log_size":9205,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30302208,"num_objects":3699,"num_object_clones":0,"num_object_copies":14796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3699,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9205,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1463}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'9018","reported_seq":10579,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:32.804629+0000","last_change":"2024-09-05T16:40:28.919189+0000","last_active":"2024-09-05T16:41:32.804629+0000","last_peered":"2024-09-05T16:41:32.804629+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:41:32.804629+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9018,"log_dups_size":0,"ondisk_log_size":9018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28964000,"num_objects":3536,"num_object_clones":0,"num_object_copies":14144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3536,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9016,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":745,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2010}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'9153","reported_seq":9275,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:32.980662+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:41:32.980662+0000","last_peered":"2024-09-05T16:41:32.980662+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:41:32.980662+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9153,"log_dups_size":0,"ondisk_log_size":9153,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29481576,"num_objects":3600,"num_object_clones":0,"num_object_copies":14400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3600,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9152,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1368}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'9203","reported_seq":13931,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:41:32.100387+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:41:32.100387+0000","last_peered":"2024-09-05T16:41:32.100387+0000","last_clean":"2024-09-05T16:41:32.100387+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:41:32.100387+0000","last_undegraded":"2024-09-05T16:41:32.100387+0000","last_fullsized":"2024-09-05T16:41:32.100387+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":9203,"log_dups_size":0,"ondisk_log_size":9203,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30564352,"num_objects":3731,"num_object_clones":0,"num_object_copies":14924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3731,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3503,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'8975","reported_seq":9073,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:32.639159+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:41:32.639159+0000","last_peered":"2024-09-05T16:41:32.639159+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:41:32.639159+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8975,"log_dups_size":0,"ondisk_log_size":8975,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29652128,"num_objects":3620,"num_object_clones":0,"num_object_copies":14480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3620,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8973,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1475}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'9126","reported_seq":9240,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:32.731987+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:41:32.731987+0000","last_peered":"2024-09-05T16:41:32.731987+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:41:32.731987+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9126,"log_dups_size":0,"ondisk_log_size":9126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29868032,"num_objects":3646,"num_object_clones":0,"num_object_copies":14584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3646,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9126,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1427}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'8972","reported_seq":9078,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:32.766616+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:41:32.766616+0000","last_peered":"2024-09-05T16:41:32.766616+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:41:32.766616+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8972,"log_dups_size":0,"ondisk_log_size":8972,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30225568,"num_objects":3690,"num_object_clones":0,"num_object_copies":14760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3690,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8970,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1559}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'9143","reported_seq":9231,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:32.534889+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:41:32.534889+0000","last_peered":"2024-09-05T16:41:32.534889+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:41:32.534889+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9143,"log_dups_size":0,"ondisk_log_size":9143,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29783200,"num_objects":3636,"num_object_clones":0,"num_object_copies":14544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9141,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1404}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'9172","reported_seq":9272,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:32.536454+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:41:32.536454+0000","last_peered":"2024-09-05T16:41:32.536454+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:41:32.536454+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9172,"log_dups_size":0,"ondisk_log_size":9172,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29501760,"num_objects":3602,"num_object_clones":0,"num_object_copies":14408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3602,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9169,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1358}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'9226","reported_seq":9347,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:32.882667+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:41:32.882667+0000","last_peered":"2024-09-05T16:41:32.882667+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:41:32.882667+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9226,"log_dups_size":0,"ondisk_log_size":9226,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29357216,"num_objects":3584,"num_object_clones":0,"num_object_copies":14336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3584,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9224,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1339}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'9380","reported_seq":9506,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:32.536947+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:41:32.536947+0000","last_peered":"2024-09-05T16:41:32.536947+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:41:32.536947+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9380,"log_dups_size":0,"ondisk_log_size":9380,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30658288,"num_objects":3743,"num_object_clones":0,"num_object_copies":14972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3743,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9377,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1462}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'9146","reported_seq":9244,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:32.569893+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:41:32.569893+0000","last_peered":"2024-09-05T16:41:32.569893+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:41:32.569893+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9146,"log_dups_size":0,"ondisk_log_size":9146,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29596240,"num_objects":3613,"num_object_clones":0,"num_object_copies":14452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3613,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9145,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1375}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'9167","reported_seq":13902,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:41:31.989986+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:41:31.989986+0000","last_peered":"2024-09-05T16:41:31.989986+0000","last_clean":"2024-09-05T16:41:31.989986+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:41:31.989986+0000","last_undegraded":"2024-09-05T16:41:31.989986+0000","last_fullsized":"2024-09-05T16:41:31.989986+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":9167,"log_dups_size":0,"ondisk_log_size":9167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29827072,"num_objects":3641,"num_object_clones":0,"num_object_copies":14564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3641,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":767,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'9431","reported_seq":9539,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:32.605712+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:41:32.605712+0000","last_peered":"2024-09-05T16:41:32.605712+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:41:32.605712+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9431,"log_dups_size":0,"ondisk_log_size":9431,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30266528,"num_objects":3695,"num_object_clones":0,"num_object_copies":14780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3695,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9429,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1361}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'9229","reported_seq":9341,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:32.940584+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:41:32.940584+0000","last_peered":"2024-09-05T16:41:32.940584+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:41:32.940584+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9229,"log_dups_size":0,"ondisk_log_size":9229,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29521056,"num_objects":3604,"num_object_clones":0,"num_object_copies":14416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3604,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9227,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1296}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'9153","reported_seq":9261,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:32.439586+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:41:32.439586+0000","last_peered":"2024-09-05T16:41:32.439586+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:41:32.439586+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9153,"log_dups_size":0,"ondisk_log_size":9153,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29668512,"num_objects":3622,"num_object_clones":0,"num_object_copies":14488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3622,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9151,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1402}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":477237736,"num_objects":58262,"num_object_clones":0,"num_object_copies":233048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30613,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58262,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":132575,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1024,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":954417152,"data_stored":954417152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":146699,"ondisk_log_size":146699,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084379,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":747428,"kb_used_data":234004,"kb_used_omap":4,"kb_used_meta":513403,"kb_avail":93624412,"statfs":{"total":96636764160,"available":95871397888,"internally_reserved":0,"allocated":239620096,"data_stored":239418822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":525725395},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,1,0,1,1,1,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986256,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":617428,"kb_used_data":234480,"kb_used_omap":4,"kb_used_meta":382907,"kb_avail":93754412,"statfs":{"total":96636764160,"available":96004517888,"internally_reserved":0,"allocated":240107520,"data_stored":239906246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":392097490},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,5,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673062,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":733468,"kb_used_data":233036,"kb_used_omap":1,"kb_used_meta":500414,"kb_avail":93638372,"statfs":{"total":96636764160,"available":95885692928,"internally_reserved":0,"allocated":238628864,"data_stored":238427590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":512424389},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,7,1,4,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182493,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":734768,"kb_used_data":233884,"kb_used_omap":4,"kb_used_meta":500859,"kb_avail":93637072,"statfs":{"total":96636764160,"available":95884361728,"internally_reserved":0,"allocated":239497216,"data_stored":239295942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":512880333},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":238432256,"data_stored":238432256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":239226880,"data_stored":239226880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":238223360,"data_stored":238223360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":238534656,"data_stored":238534656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:41:35.241 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-05T16:41:35.241 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'9205", 'reported_seq': 9309, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:32.624897+0000', 'last_change': '2024-09-05T16:34:56.833497+0000', 'last_active': '2024-09-05T16:41:32.624897+0000', 'last_peered': '2024-09-05T16:41:32.624897+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.828289+0000', 'last_became_peered': '2024-09-05T16:34:56.828289+0000', 'last_unstale': '2024-09-05T16:41:32.624897+0000', 'last_undegraded': '2024-09-05T16:34:56.612160+0000', 'last_fullsized': '2024-09-05T16:34:56.557632+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9205, 'log_dups_size': 0, 'ondisk_log_size': 9205, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30302208, 'num_objects': 3699, 'num_object_clones': 0, 'num_object_copies': 14796, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2236, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3699, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9205, 'num_write_kb': 51616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1463}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:41:35.241 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-05T16:41:35.241 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "37'9018", 'reported_seq': 10579, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:32.804629+0000', 'last_change': '2024-09-05T16:40:28.919189+0000', 'last_active': '2024-09-05T16:41:32.804629+0000', 'last_peered': '2024-09-05T16:41:32.804629+0000', 'last_clean': '2024-09-05T16:33:53.200762+0000', 'last_became_active': '2024-09-05T16:34:56.820292+0000', 'last_became_peered': '2024-09-05T16:34:56.820292+0000', 'last_unstale': '2024-09-05T16:41:32.804629+0000', 'last_undegraded': '2024-09-05T16:34:56.605776+0000', 'last_fullsized': '2024-09-05T16:34:56.559383+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9018, 'log_dups_size': 0, 'ondisk_log_size': 9018, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28964000, 'num_objects': 3536, 'num_object_clones': 0, 'num_object_copies': 14144, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1526, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3536, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9016, 'num_write_kb': 50212, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 745, 'num_bytes_recovered': 2613248, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2010}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:41:35.241 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:41:35.241 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'9153", 'reported_seq': 9275, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:32.980662+0000', 'last_change': '2024-09-05T16:34:56.867900+0000', 'last_active': '2024-09-05T16:41:32.980662+0000', 'last_peered': '2024-09-05T16:41:32.980662+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:41:32.980662+0000', 'last_undegraded': '2024-09-05T16:34:56.590972+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9153, 'log_dups_size': 0, 'ondisk_log_size': 9153, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29481576, 'num_objects': 3600, 'num_object_clones': 0, 'num_object_copies': 14400, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2233, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3600, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 9152, 'num_write_kb': 50999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1368}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:41:35.241 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:41:35.242 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'8975", 'reported_seq': 9073, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:32.639159+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:41:32.639159+0000', 'last_peered': '2024-09-05T16:41:32.639159+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:41:32.639159+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8975, 'log_dups_size': 0, 'ondisk_log_size': 8975, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29652128, 'num_objects': 3620, 'num_object_clones': 0, 'num_object_copies': 14480, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2147, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3620, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8973, 'num_write_kb': 50373, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1475}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:41:35.242 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:41:35.242 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'9126", 'reported_seq': 9240, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:32.731987+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:41:32.731987+0000', 'last_peered': '2024-09-05T16:41:32.731987+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:41:32.731987+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9126, 'log_dups_size': 0, 'ondisk_log_size': 9126, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29868032, 'num_objects': 3646, 'num_object_clones': 0, 'num_object_copies': 14584, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3646, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9126, 'num_write_kb': 51088, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1427}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:41:35.242 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:41:35.242 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'8972", 'reported_seq': 9078, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:32.766616+0000', 'last_change': '2024-09-05T16:34:56.903436+0000', 'last_active': '2024-09-05T16:41:32.766616+0000', 'last_peered': '2024-09-05T16:41:32.766616+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:41:32.766616+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 8972, 'log_dups_size': 0, 'ondisk_log_size': 8972, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30225568, 'num_objects': 3690, 'num_object_clones': 0, 'num_object_copies': 14760, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2133, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3690, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8970, 'num_write_kb': 50636, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1559}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:41:35.242 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-05T16:41:35.242 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "37'9143", 'reported_seq': 9231, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:32.534889+0000', 'last_change': '2024-09-05T16:34:56.883319+0000', 'last_active': '2024-09-05T16:41:32.534889+0000', 'last_peered': '2024-09-05T16:41:32.534889+0000', 'last_clean': '2024-09-05T16:33:53.505305+0000', 'last_became_active': '2024-09-05T16:34:56.873389+0000', 'last_became_peered': '2024-09-05T16:34:56.873389+0000', 'last_unstale': '2024-09-05T16:41:32.534889+0000', 'last_undegraded': '2024-09-05T16:34:56.602567+0000', 'last_fullsized': '2024-09-05T16:34:56.559958+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9143, 'log_dups_size': 0, 'ondisk_log_size': 9143, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29783200, 'num_objects': 3636, 'num_object_clones': 0, 'num_object_copies': 14544, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2234, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3636, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9141, 'num_write_kb': 51108, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1404}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:41:35.243 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:41:35.243 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'9172", 'reported_seq': 9272, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:32.536454+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:41:32.536454+0000', 'last_peered': '2024-09-05T16:41:32.536454+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:41:32.536454+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9172, 'log_dups_size': 0, 'ondisk_log_size': 9172, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29501760, 'num_objects': 3602, 'num_object_clones': 0, 'num_object_copies': 14408, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3602, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9169, 'num_write_kb': 51081, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1358}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:41:35.243 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:41:35.243 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'9226", 'reported_seq': 9347, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:32.882667+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:41:32.882667+0000', 'last_peered': '2024-09-05T16:41:32.882667+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:41:32.882667+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9226, 'log_dups_size': 0, 'ondisk_log_size': 9226, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29357216, 'num_objects': 3584, 'num_object_clones': 0, 'num_object_copies': 14336, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3584, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9224, 'num_write_kb': 51230, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1339}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:41:35.243 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:41:35.243 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'9380", 'reported_seq': 9506, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:32.536947+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:41:32.536947+0000', 'last_peered': '2024-09-05T16:41:32.536947+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:41:32.536947+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9380, 'log_dups_size': 0, 'ondisk_log_size': 9380, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30658288, 'num_objects': 3743, 'num_object_clones': 0, 'num_object_copies': 14972, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2284, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3743, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9377, 'num_write_kb': 52476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1462}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:41:35.243 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:41:35.243 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'9146", 'reported_seq': 9244, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:32.569893+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:41:32.569893+0000', 'last_peered': '2024-09-05T16:41:32.569893+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:41:32.569893+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9146, 'log_dups_size': 0, 'ondisk_log_size': 9146, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29596240, 'num_objects': 3613, 'num_object_clones': 0, 'num_object_copies': 14452, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3613, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9145, 'num_write_kb': 51031, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1375}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:41:35.243 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:41:35.244 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'9431", 'reported_seq': 9539, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:32.605712+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:41:32.605712+0000', 'last_peered': '2024-09-05T16:41:32.605712+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:41:32.605712+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9431, 'log_dups_size': 0, 'ondisk_log_size': 9431, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30266528, 'num_objects': 3695, 'num_object_clones': 0, 'num_object_copies': 14780, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3695, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9429, 'num_write_kb': 52500, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1361}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:41:35.245 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:41:35.245 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'9229", 'reported_seq': 9341, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:32.940584+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:41:32.940584+0000', 'last_peered': '2024-09-05T16:41:32.940584+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:41:32.940584+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9229, 'log_dups_size': 0, 'ondisk_log_size': 9229, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29521056, 'num_objects': 3604, 'num_object_clones': 0, 'num_object_copies': 14416, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2310, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3604, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9227, 'num_write_kb': 51325, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1296}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:41:35.245 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:41:35.245 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'9153", 'reported_seq': 9261, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:32.439586+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:41:32.439586+0000', 'last_peered': '2024-09-05T16:41:32.439586+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:41:32.439586+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9153, 'log_dups_size': 0, 'ondisk_log_size': 9153, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29668512, 'num_objects': 3622, 'num_object_clones': 0, 'num_object_copies': 14488, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3622, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9151, 'num_write_kb': 51092, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1402}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:41:35.246 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:41:36.008 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:41:36.008 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 30613/233048 objects degraded (13.136%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":499,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":58262,"data_bytes":477237736,"bytes_used":2901086208,"bytes_avail":383645970432,"bytes_total":386547056640,"degraded_objects":30613,"degraded_total":233048,"degraded_ratio":0.13135920497064982,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":107,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":199},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:41:36.036 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:41:36.037 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:41:36.650 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:41:36.651 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:41:36.684 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":277,"stamp":"2024-09-05T16:41:34.968195+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":477237736,"num_objects":58262,"num_object_clones":0,"num_object_copies":233048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30613,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58262,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":132575,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1024,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":146699,"ondisk_log_size":146699,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2833092,"kb_used_data":935404,"kb_used_omap":14,"kb_used_meta":1897585,"kb_avail":374654268,"statfs":{"total":386547056640,"available":383645970432,"internally_reserved":0,"allocated":957853696,"data_stored":957048600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1943127607},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,2,1,11,10,9,9,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19636224,"num_objects":-2397,"num_object_clones":0,"num_object_copies":-9588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-107,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2397,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2397,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":107,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004929"},"pg_stats":[{"pgid":"1.f","version":"37'9205","reported_seq":9309,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:32.624897+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:41:32.624897+0000","last_peered":"2024-09-05T16:41:32.624897+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:41:32.624897+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9205,"log_dups_size":0,"ondisk_log_size":9205,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30302208,"num_objects":3699,"num_object_clones":0,"num_object_copies":14796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3699,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9205,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1463}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'9018","reported_seq":10579,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:32.804629+0000","last_change":"2024-09-05T16:40:28.919189+0000","last_active":"2024-09-05T16:41:32.804629+0000","last_peered":"2024-09-05T16:41:32.804629+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:41:32.804629+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9018,"log_dups_size":0,"ondisk_log_size":9018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28964000,"num_objects":3536,"num_object_clones":0,"num_object_copies":14144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3536,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9016,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":745,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2010}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'9153","reported_seq":9275,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:32.980662+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:41:32.980662+0000","last_peered":"2024-09-05T16:41:32.980662+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:41:32.980662+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9153,"log_dups_size":0,"ondisk_log_size":9153,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29481576,"num_objects":3600,"num_object_clones":0,"num_object_copies":14400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3600,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9152,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1368}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'9203","reported_seq":13931,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:41:32.100387+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:41:32.100387+0000","last_peered":"2024-09-05T16:41:32.100387+0000","last_clean":"2024-09-05T16:41:32.100387+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:41:32.100387+0000","last_undegraded":"2024-09-05T16:41:32.100387+0000","last_fullsized":"2024-09-05T16:41:32.100387+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":9203,"log_dups_size":0,"ondisk_log_size":9203,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30564352,"num_objects":3731,"num_object_clones":0,"num_object_copies":14924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3731,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3503,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'8975","reported_seq":9073,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:32.639159+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:41:32.639159+0000","last_peered":"2024-09-05T16:41:32.639159+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:41:32.639159+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8975,"log_dups_size":0,"ondisk_log_size":8975,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29652128,"num_objects":3620,"num_object_clones":0,"num_object_copies":14480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3620,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8973,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1475}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'9126","reported_seq":9240,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:32.731987+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:41:32.731987+0000","last_peered":"2024-09-05T16:41:32.731987+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:41:32.731987+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9126,"log_dups_size":0,"ondisk_log_size":9126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29868032,"num_objects":3646,"num_object_clones":0,"num_object_copies":14584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3646,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9126,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1427}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'8972","reported_seq":9078,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:32.766616+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:41:32.766616+0000","last_peered":"2024-09-05T16:41:32.766616+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:41:32.766616+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":8972,"log_dups_size":0,"ondisk_log_size":8972,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30225568,"num_objects":3690,"num_object_clones":0,"num_object_copies":14760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3690,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8970,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1559}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'9143","reported_seq":9231,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:32.534889+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:41:32.534889+0000","last_peered":"2024-09-05T16:41:32.534889+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:41:32.534889+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9143,"log_dups_size":0,"ondisk_log_size":9143,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29783200,"num_objects":3636,"num_object_clones":0,"num_object_copies":14544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9141,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1404}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'9172","reported_seq":9272,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:32.536454+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:41:32.536454+0000","last_peered":"2024-09-05T16:41:32.536454+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:41:32.536454+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9172,"log_dups_size":0,"ondisk_log_size":9172,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29501760,"num_objects":3602,"num_object_clones":0,"num_object_copies":14408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3602,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9169,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1358}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'9226","reported_seq":9347,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:32.882667+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:41:32.882667+0000","last_peered":"2024-09-05T16:41:32.882667+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:41:32.882667+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9226,"log_dups_size":0,"ondisk_log_size":9226,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29357216,"num_objects":3584,"num_object_clones":0,"num_object_copies":14336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3584,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9224,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1339}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'9380","reported_seq":9506,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:32.536947+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:41:32.536947+0000","last_peered":"2024-09-05T16:41:32.536947+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:41:32.536947+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9380,"log_dups_size":0,"ondisk_log_size":9380,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30658288,"num_objects":3743,"num_object_clones":0,"num_object_copies":14972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3743,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9377,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1462}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'9146","reported_seq":9244,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:32.569893+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:41:32.569893+0000","last_peered":"2024-09-05T16:41:32.569893+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:41:32.569893+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9146,"log_dups_size":0,"ondisk_log_size":9146,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29596240,"num_objects":3613,"num_object_clones":0,"num_object_copies":14452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3613,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9145,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1375}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'9167","reported_seq":13902,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:41:31.989986+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:41:31.989986+0000","last_peered":"2024-09-05T16:41:31.989986+0000","last_clean":"2024-09-05T16:41:31.989986+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:41:31.989986+0000","last_undegraded":"2024-09-05T16:41:31.989986+0000","last_fullsized":"2024-09-05T16:41:31.989986+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":9167,"log_dups_size":0,"ondisk_log_size":9167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29827072,"num_objects":3641,"num_object_clones":0,"num_object_copies":14564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3641,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":767,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'9431","reported_seq":9539,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:32.605712+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:41:32.605712+0000","last_peered":"2024-09-05T16:41:32.605712+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:41:32.605712+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9431,"log_dups_size":0,"ondisk_log_size":9431,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30266528,"num_objects":3695,"num_object_clones":0,"num_object_copies":14780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3695,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9429,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1361}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'9229","reported_seq":9341,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:32.940584+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:41:32.940584+0000","last_peered":"2024-09-05T16:41:32.940584+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:41:32.940584+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9229,"log_dups_size":0,"ondisk_log_size":9229,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29521056,"num_objects":3604,"num_object_clones":0,"num_object_copies":14416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3604,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9227,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1296}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'9153","reported_seq":9261,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:32.439586+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:41:32.439586+0000","last_peered":"2024-09-05T16:41:32.439586+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:41:32.439586+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9153,"log_dups_size":0,"ondisk_log_size":9153,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29668512,"num_objects":3622,"num_object_clones":0,"num_object_copies":14488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3622,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9151,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1402}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":477237736,"num_objects":58262,"num_object_clones":0,"num_object_copies":233048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30613,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58262,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":132575,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1024,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":954417152,"data_stored":954417152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":146699,"ondisk_log_size":146699,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084379,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":747428,"kb_used_data":234004,"kb_used_omap":4,"kb_used_meta":513403,"kb_avail":93624412,"statfs":{"total":96636764160,"available":95871397888,"internally_reserved":0,"allocated":239620096,"data_stored":239418822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":525725395},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,1,0,1,1,1,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986256,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":617428,"kb_used_data":234480,"kb_used_omap":4,"kb_used_meta":382907,"kb_avail":93754412,"statfs":{"total":96636764160,"available":96004517888,"internally_reserved":0,"allocated":240107520,"data_stored":239906246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":392097490},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,5,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673062,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":733468,"kb_used_data":233036,"kb_used_omap":1,"kb_used_meta":500414,"kb_avail":93638372,"statfs":{"total":96636764160,"available":95885692928,"internally_reserved":0,"allocated":238628864,"data_stored":238427590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":512424389},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,7,1,4,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182493,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":734768,"kb_used_data":233884,"kb_used_omap":4,"kb_used_meta":500859,"kb_avail":93637072,"statfs":{"total":96636764160,"available":95884361728,"internally_reserved":0,"allocated":239497216,"data_stored":239295942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":512880333},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":238432256,"data_stored":238432256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":239226880,"data_stored":239226880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":238223360,"data_stored":238223360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":238534656,"data_stored":238534656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:41:39.686 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:41:40.333 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:41:40.333 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:41:40.367 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":279,"stamp":"2024-09-05T16:41:38.969793+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":468234728,"num_objects":57163,"num_object_clones":0,"num_object_copies":228652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30557,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57163,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":133674,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1080,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":147798,"ondisk_log_size":147798,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2817520,"kb_used_data":918420,"kb_used_omap":14,"kb_used_meta":1898993,"kb_avail":374669840,"statfs":{"total":386547056640,"available":383661916160,"internally_reserved":0,"allocated":940462080,"data_stored":939656984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1944569399},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,6,0,4,9,11,18,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-26796032,"num_objects":-3271,"num_object_clones":0,"num_object_copies":-13084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-163,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3271,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3271,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":163,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004757"},"pg_stats":[{"pgid":"1.f","version":"37'9271","reported_seq":9375,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:37.727685+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:41:37.727685+0000","last_peered":"2024-09-05T16:41:37.727685+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:41:37.727685+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9271,"log_dups_size":0,"ondisk_log_size":9271,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29761536,"num_objects":3633,"num_object_clones":0,"num_object_copies":14532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3633,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9271,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1397}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'9083","reported_seq":10756,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:37.846973+0000","last_change":"2024-09-05T16:40:28.919189+0000","last_active":"2024-09-05T16:41:37.846973+0000","last_peered":"2024-09-05T16:41:37.846973+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:41:37.846973+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9083,"log_dups_size":0,"ondisk_log_size":9083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28431520,"num_objects":3471,"num_object_clones":0,"num_object_copies":13884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1470,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3471,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9081,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":801,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2001}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'9231","reported_seq":9353,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:37.972851+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:41:37.972851+0000","last_peered":"2024-09-05T16:41:37.972851+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:41:37.972851+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9231,"log_dups_size":0,"ondisk_log_size":9231,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28842600,"num_objects":3522,"num_object_clones":0,"num_object_copies":14088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3522,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9230,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1290}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'9269","reported_seq":13996,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:41:37.271959+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:41:37.271959+0000","last_peered":"2024-09-05T16:41:37.271959+0000","last_clean":"2024-09-05T16:41:37.271959+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:41:37.271959+0000","last_undegraded":"2024-09-05T16:41:37.271959+0000","last_fullsized":"2024-09-05T16:41:37.271959+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":9269,"log_dups_size":0,"ondisk_log_size":9269,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30023680,"num_objects":3665,"num_object_clones":0,"num_object_copies":14660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3569,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'9039","reported_seq":9137,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:37.553922+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:41:37.553922+0000","last_peered":"2024-09-05T16:41:37.553922+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:41:37.553922+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9039,"log_dups_size":0,"ondisk_log_size":9039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29127840,"num_objects":3556,"num_object_clones":0,"num_object_copies":14224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3556,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9037,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1411}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'9200","reported_seq":9314,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:37.807360+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:41:37.807360+0000","last_peered":"2024-09-05T16:41:37.807360+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:41:37.807360+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9200,"log_dups_size":0,"ondisk_log_size":9200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29261824,"num_objects":3572,"num_object_clones":0,"num_object_copies":14288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3572,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9200,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1353}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'9034","reported_seq":9140,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:37.791543+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:41:37.791543+0000","last_peered":"2024-09-05T16:41:37.791543+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:41:37.791543+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9034,"log_dups_size":0,"ondisk_log_size":9034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29717664,"num_objects":3628,"num_object_clones":0,"num_object_copies":14512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9032,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1497}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'9216","reported_seq":9304,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:37.742406+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:41:37.742406+0000","last_peered":"2024-09-05T16:41:37.742406+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:41:37.742406+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9216,"log_dups_size":0,"ondisk_log_size":9216,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29185184,"num_objects":3563,"num_object_clones":0,"num_object_copies":14252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3563,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9214,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1331}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'9230","reported_seq":9330,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:37.748420+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:41:37.748420+0000","last_peered":"2024-09-05T16:41:37.748420+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:41:37.748420+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9230,"log_dups_size":0,"ondisk_log_size":9230,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29026624,"num_objects":3544,"num_object_clones":0,"num_object_copies":14176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3544,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9227,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1300}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'9309","reported_seq":9430,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:38.170297+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:41:38.170297+0000","last_peered":"2024-09-05T16:41:38.170297+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:41:38.170297+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9309,"log_dups_size":0,"ondisk_log_size":9309,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28677280,"num_objects":3501,"num_object_clones":0,"num_object_copies":14004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3501,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9307,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1256}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'9450","reported_seq":9576,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:37.647430+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:41:37.647430+0000","last_peered":"2024-09-05T16:41:37.647430+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:41:37.647430+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9450,"log_dups_size":0,"ondisk_log_size":9450,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30084848,"num_objects":3673,"num_object_clones":0,"num_object_copies":14692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3673,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9447,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1392}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'9218","reported_seq":9316,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:37.728703+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:41:37.728703+0000","last_peered":"2024-09-05T16:41:37.728703+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:41:37.728703+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9218,"log_dups_size":0,"ondisk_log_size":9218,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29006416,"num_objects":3541,"num_object_clones":0,"num_object_copies":14164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3541,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9217,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1303}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'9230","reported_seq":13965,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:41:37.057074+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:41:37.057074+0000","last_peered":"2024-09-05T16:41:37.057074+0000","last_clean":"2024-09-05T16:41:37.057074+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:41:37.057074+0000","last_undegraded":"2024-09-05T16:41:37.057074+0000","last_fullsized":"2024-09-05T16:41:37.057074+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":9230,"log_dups_size":0,"ondisk_log_size":9230,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29310976,"num_objects":3578,"num_object_clones":0,"num_object_copies":14312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3578,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":830,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'9496","reported_seq":9604,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:37.754466+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:41:37.754466+0000","last_peered":"2024-09-05T16:41:37.754466+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:41:37.754466+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9496,"log_dups_size":0,"ondisk_log_size":9496,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29734048,"num_objects":3630,"num_object_clones":0,"num_object_copies":14520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3630,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9494,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1296}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'9301","reported_seq":9412,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:38.181667+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:41:38.181667+0000","last_peered":"2024-09-05T16:41:38.181667+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:41:38.181667+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9301,"log_dups_size":0,"ondisk_log_size":9301,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28931232,"num_objects":3532,"num_object_clones":0,"num_object_copies":14128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3532,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9299,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1224}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'9221","reported_seq":9329,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:37.879160+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:41:37.879160+0000","last_peered":"2024-09-05T16:41:37.879160+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:41:37.879160+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9221,"log_dups_size":0,"ondisk_log_size":9221,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29111456,"num_objects":3554,"num_object_clones":0,"num_object_copies":14216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3554,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9219,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1334}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":468234728,"num_objects":57163,"num_object_clones":0,"num_object_copies":228652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30557,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57163,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":133674,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1080,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":936779776,"data_stored":936779776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":147798,"ondisk_log_size":147798,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084380,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":730624,"kb_used_data":229668,"kb_used_omap":4,"kb_used_meta":500923,"kb_avail":93641216,"statfs":{"total":96636764160,"available":95888605184,"internally_reserved":0,"allocated":235180032,"data_stored":234978758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":512945875},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986257,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":618704,"kb_used_data":230208,"kb_used_omap":4,"kb_used_meta":388475,"kb_avail":93753136,"statfs":{"total":96636764160,"available":96003211264,"internally_reserved":0,"allocated":235732992,"data_stored":235531718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":397799122},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,3,0,4,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673063,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":733428,"kb_used_data":228828,"kb_used_omap":1,"kb_used_meta":504574,"kb_avail":93638412,"statfs":{"total":96636764160,"available":95885733888,"internally_reserved":0,"allocated":234319872,"data_stored":234118598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":516684229},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,0,2,4,6,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182494,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":734764,"kb_used_data":229716,"kb_used_omap":4,"kb_used_meta":505019,"kb_avail":93637076,"statfs":{"total":96636764160,"available":95884365824,"internally_reserved":0,"allocated":235229184,"data_stored":235027910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":517140173},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,4,3,5,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":234074112,"data_stored":234074112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":234807296,"data_stored":234807296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":233787392,"data_stored":233787392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":234110976,"data_stored":234110976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:41:40.368 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:41:40.976 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:41:40.977 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:41:41.013 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":280,"stamp":"2024-09-05T16:41:40.970669+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":468234728,"num_objects":57163,"num_object_clones":0,"num_object_copies":228652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30557,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57163,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":133674,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1080,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":147798,"ondisk_log_size":147798,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2817520,"kb_used_data":918420,"kb_used_omap":14,"kb_used_meta":1898993,"kb_avail":374669840,"statfs":{"total":386547056640,"available":383661916160,"internally_reserved":0,"allocated":940462080,"data_stored":939656984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1944569399},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,6,0,4,9,11,18,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17711104,"num_objects":-2162,"num_object_clones":0,"num_object_copies":-8648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2162,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2162,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":112,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004799"},"pg_stats":[{"pgid":"1.f","version":"37'9271","reported_seq":9375,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:37.727685+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:41:37.727685+0000","last_peered":"2024-09-05T16:41:37.727685+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:41:37.727685+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9271,"log_dups_size":0,"ondisk_log_size":9271,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29761536,"num_objects":3633,"num_object_clones":0,"num_object_copies":14532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3633,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9271,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1397}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'9083","reported_seq":10756,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:37.846973+0000","last_change":"2024-09-05T16:40:28.919189+0000","last_active":"2024-09-05T16:41:37.846973+0000","last_peered":"2024-09-05T16:41:37.846973+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:41:37.846973+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9083,"log_dups_size":0,"ondisk_log_size":9083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28431520,"num_objects":3471,"num_object_clones":0,"num_object_copies":13884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1470,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3471,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9081,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":801,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2001}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'9231","reported_seq":9353,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:37.972851+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:41:37.972851+0000","last_peered":"2024-09-05T16:41:37.972851+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:41:37.972851+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9231,"log_dups_size":0,"ondisk_log_size":9231,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28842600,"num_objects":3522,"num_object_clones":0,"num_object_copies":14088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3522,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9230,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1290}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'9269","reported_seq":13996,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:41:37.271959+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:41:37.271959+0000","last_peered":"2024-09-05T16:41:37.271959+0000","last_clean":"2024-09-05T16:41:37.271959+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:41:37.271959+0000","last_undegraded":"2024-09-05T16:41:37.271959+0000","last_fullsized":"2024-09-05T16:41:37.271959+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":9269,"log_dups_size":0,"ondisk_log_size":9269,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30023680,"num_objects":3665,"num_object_clones":0,"num_object_copies":14660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3569,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'9039","reported_seq":9137,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:37.553922+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:41:37.553922+0000","last_peered":"2024-09-05T16:41:37.553922+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:41:37.553922+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9039,"log_dups_size":0,"ondisk_log_size":9039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29127840,"num_objects":3556,"num_object_clones":0,"num_object_copies":14224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3556,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9037,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1411}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'9200","reported_seq":9314,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:37.807360+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:41:37.807360+0000","last_peered":"2024-09-05T16:41:37.807360+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:41:37.807360+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9200,"log_dups_size":0,"ondisk_log_size":9200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29261824,"num_objects":3572,"num_object_clones":0,"num_object_copies":14288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3572,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9200,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1353}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'9034","reported_seq":9140,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:37.791543+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:41:37.791543+0000","last_peered":"2024-09-05T16:41:37.791543+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:41:37.791543+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9034,"log_dups_size":0,"ondisk_log_size":9034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29717664,"num_objects":3628,"num_object_clones":0,"num_object_copies":14512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9032,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1497}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'9216","reported_seq":9304,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:37.742406+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:41:37.742406+0000","last_peered":"2024-09-05T16:41:37.742406+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:41:37.742406+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9216,"log_dups_size":0,"ondisk_log_size":9216,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29185184,"num_objects":3563,"num_object_clones":0,"num_object_copies":14252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3563,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9214,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1331}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'9230","reported_seq":9330,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:37.748420+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:41:37.748420+0000","last_peered":"2024-09-05T16:41:37.748420+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:41:37.748420+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9230,"log_dups_size":0,"ondisk_log_size":9230,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29026624,"num_objects":3544,"num_object_clones":0,"num_object_copies":14176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3544,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9227,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1300}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'9309","reported_seq":9430,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:38.170297+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:41:38.170297+0000","last_peered":"2024-09-05T16:41:38.170297+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:41:38.170297+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9309,"log_dups_size":0,"ondisk_log_size":9309,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28677280,"num_objects":3501,"num_object_clones":0,"num_object_copies":14004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3501,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9307,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1256}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'9450","reported_seq":9576,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:37.647430+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:41:37.647430+0000","last_peered":"2024-09-05T16:41:37.647430+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:41:37.647430+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9450,"log_dups_size":0,"ondisk_log_size":9450,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30084848,"num_objects":3673,"num_object_clones":0,"num_object_copies":14692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3673,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9447,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1392}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'9218","reported_seq":9316,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:37.728703+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:41:37.728703+0000","last_peered":"2024-09-05T16:41:37.728703+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:41:37.728703+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9218,"log_dups_size":0,"ondisk_log_size":9218,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29006416,"num_objects":3541,"num_object_clones":0,"num_object_copies":14164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3541,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9217,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1303}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'9230","reported_seq":13965,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:41:37.057074+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:41:37.057074+0000","last_peered":"2024-09-05T16:41:37.057074+0000","last_clean":"2024-09-05T16:41:37.057074+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:41:37.057074+0000","last_undegraded":"2024-09-05T16:41:37.057074+0000","last_fullsized":"2024-09-05T16:41:37.057074+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":9230,"log_dups_size":0,"ondisk_log_size":9230,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29310976,"num_objects":3578,"num_object_clones":0,"num_object_copies":14312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3578,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":830,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'9496","reported_seq":9604,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:37.754466+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:41:37.754466+0000","last_peered":"2024-09-05T16:41:37.754466+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:41:37.754466+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9496,"log_dups_size":0,"ondisk_log_size":9496,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29734048,"num_objects":3630,"num_object_clones":0,"num_object_copies":14520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3630,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9494,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1296}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'9301","reported_seq":9412,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:38.181667+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:41:38.181667+0000","last_peered":"2024-09-05T16:41:38.181667+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:41:38.181667+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9301,"log_dups_size":0,"ondisk_log_size":9301,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28931232,"num_objects":3532,"num_object_clones":0,"num_object_copies":14128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3532,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9299,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1224}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'9221","reported_seq":9329,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:37.879160+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:41:37.879160+0000","last_peered":"2024-09-05T16:41:37.879160+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:41:37.879160+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9221,"log_dups_size":0,"ondisk_log_size":9221,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29111456,"num_objects":3554,"num_object_clones":0,"num_object_copies":14216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3554,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9219,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1334}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":468234728,"num_objects":57163,"num_object_clones":0,"num_object_copies":228652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30557,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57163,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":133674,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1080,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":936779776,"data_stored":936779776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":147798,"ondisk_log_size":147798,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084380,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":730624,"kb_used_data":229668,"kb_used_omap":4,"kb_used_meta":500923,"kb_avail":93641216,"statfs":{"total":96636764160,"available":95888605184,"internally_reserved":0,"allocated":235180032,"data_stored":234978758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":512945875},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986257,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":618704,"kb_used_data":230208,"kb_used_omap":4,"kb_used_meta":388475,"kb_avail":93753136,"statfs":{"total":96636764160,"available":96003211264,"internally_reserved":0,"allocated":235732992,"data_stored":235531718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":397799122},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,3,0,4,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673063,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":733428,"kb_used_data":228828,"kb_used_omap":1,"kb_used_meta":504574,"kb_avail":93638412,"statfs":{"total":96636764160,"available":95885733888,"internally_reserved":0,"allocated":234319872,"data_stored":234118598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":516684229},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,0,2,4,6,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182494,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":734764,"kb_used_data":229716,"kb_used_omap":4,"kb_used_meta":505019,"kb_avail":93637076,"statfs":{"total":96636764160,"available":95884365824,"internally_reserved":0,"allocated":235229184,"data_stored":235027910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":517140173},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,4,3,5,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":234074112,"data_stored":234074112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":234807296,"data_stored":234807296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":233787392,"data_stored":233787392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":234110976,"data_stored":234110976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:41:41.014 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-05T16:41:41.014 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'9271", 'reported_seq': 9375, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:37.727685+0000', 'last_change': '2024-09-05T16:34:56.833497+0000', 'last_active': '2024-09-05T16:41:37.727685+0000', 'last_peered': '2024-09-05T16:41:37.727685+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.828289+0000', 'last_became_peered': '2024-09-05T16:34:56.828289+0000', 'last_unstale': '2024-09-05T16:41:37.727685+0000', 'last_undegraded': '2024-09-05T16:34:56.612160+0000', 'last_fullsized': '2024-09-05T16:34:56.557632+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9271, 'log_dups_size': 0, 'ondisk_log_size': 9271, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29761536, 'num_objects': 3633, 'num_object_clones': 0, 'num_object_copies': 14532, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2236, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3633, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9271, 'num_write_kb': 51616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1397}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:41:41.015 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-05T16:41:41.015 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "37'9083", 'reported_seq': 10756, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:37.846973+0000', 'last_change': '2024-09-05T16:40:28.919189+0000', 'last_active': '2024-09-05T16:41:37.846973+0000', 'last_peered': '2024-09-05T16:41:37.846973+0000', 'last_clean': '2024-09-05T16:33:53.200762+0000', 'last_became_active': '2024-09-05T16:34:56.820292+0000', 'last_became_peered': '2024-09-05T16:34:56.820292+0000', 'last_unstale': '2024-09-05T16:41:37.846973+0000', 'last_undegraded': '2024-09-05T16:34:56.605776+0000', 'last_fullsized': '2024-09-05T16:34:56.559383+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9083, 'log_dups_size': 0, 'ondisk_log_size': 9083, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28431520, 'num_objects': 3471, 'num_object_clones': 0, 'num_object_copies': 13884, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1470, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3471, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9081, 'num_write_kb': 50212, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 801, 'num_bytes_recovered': 2613248, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2001}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:41:41.015 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:41:41.015 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'9231", 'reported_seq': 9353, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:37.972851+0000', 'last_change': '2024-09-05T16:34:56.867900+0000', 'last_active': '2024-09-05T16:41:37.972851+0000', 'last_peered': '2024-09-05T16:41:37.972851+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:41:37.972851+0000', 'last_undegraded': '2024-09-05T16:34:56.590972+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9231, 'log_dups_size': 0, 'ondisk_log_size': 9231, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28842600, 'num_objects': 3522, 'num_object_clones': 0, 'num_object_copies': 14088, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2233, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3522, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 9230, 'num_write_kb': 50999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1290}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:41:41.015 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:41:41.015 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'9039", 'reported_seq': 9137, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:37.553922+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:41:37.553922+0000', 'last_peered': '2024-09-05T16:41:37.553922+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:41:37.553922+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9039, 'log_dups_size': 0, 'ondisk_log_size': 9039, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29127840, 'num_objects': 3556, 'num_object_clones': 0, 'num_object_copies': 14224, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2147, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3556, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9037, 'num_write_kb': 50373, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1411}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:41:41.015 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:41:41.015 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'9200", 'reported_seq': 9314, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:37.807360+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:41:37.807360+0000', 'last_peered': '2024-09-05T16:41:37.807360+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:41:37.807360+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9200, 'log_dups_size': 0, 'ondisk_log_size': 9200, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29261824, 'num_objects': 3572, 'num_object_clones': 0, 'num_object_copies': 14288, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3572, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9200, 'num_write_kb': 51088, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1353}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:41:41.015 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:41:41.015 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'9034", 'reported_seq': 9140, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:37.791543+0000', 'last_change': '2024-09-05T16:34:56.903436+0000', 'last_active': '2024-09-05T16:41:37.791543+0000', 'last_peered': '2024-09-05T16:41:37.791543+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:41:37.791543+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9034, 'log_dups_size': 0, 'ondisk_log_size': 9034, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29717664, 'num_objects': 3628, 'num_object_clones': 0, 'num_object_copies': 14512, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2133, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3628, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9032, 'num_write_kb': 50636, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1497}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:41:41.015 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-05T16:41:41.016 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "37'9216", 'reported_seq': 9304, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:37.742406+0000', 'last_change': '2024-09-05T16:34:56.883319+0000', 'last_active': '2024-09-05T16:41:37.742406+0000', 'last_peered': '2024-09-05T16:41:37.742406+0000', 'last_clean': '2024-09-05T16:33:53.505305+0000', 'last_became_active': '2024-09-05T16:34:56.873389+0000', 'last_became_peered': '2024-09-05T16:34:56.873389+0000', 'last_unstale': '2024-09-05T16:41:37.742406+0000', 'last_undegraded': '2024-09-05T16:34:56.602567+0000', 'last_fullsized': '2024-09-05T16:34:56.559958+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9216, 'log_dups_size': 0, 'ondisk_log_size': 9216, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29185184, 'num_objects': 3563, 'num_object_clones': 0, 'num_object_copies': 14252, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2234, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3563, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9214, 'num_write_kb': 51108, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1331}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:41:41.016 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:41:41.016 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'9230", 'reported_seq': 9330, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:37.748420+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:41:37.748420+0000', 'last_peered': '2024-09-05T16:41:37.748420+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:41:37.748420+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9230, 'log_dups_size': 0, 'ondisk_log_size': 9230, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29026624, 'num_objects': 3544, 'num_object_clones': 0, 'num_object_copies': 14176, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3544, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9227, 'num_write_kb': 51081, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1300}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:41:41.016 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:41:41.016 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'9309", 'reported_seq': 9430, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:38.170297+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:41:38.170297+0000', 'last_peered': '2024-09-05T16:41:38.170297+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:41:38.170297+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9309, 'log_dups_size': 0, 'ondisk_log_size': 9309, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28677280, 'num_objects': 3501, 'num_object_clones': 0, 'num_object_copies': 14004, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3501, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9307, 'num_write_kb': 51230, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1256}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:41:41.016 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:41:41.016 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'9450", 'reported_seq': 9576, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:37.647430+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:41:37.647430+0000', 'last_peered': '2024-09-05T16:41:37.647430+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:41:37.647430+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9450, 'log_dups_size': 0, 'ondisk_log_size': 9450, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30084848, 'num_objects': 3673, 'num_object_clones': 0, 'num_object_copies': 14692, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2284, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3673, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9447, 'num_write_kb': 52476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1392}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:41:41.016 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:41:41.016 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'9218", 'reported_seq': 9316, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:37.728703+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:41:37.728703+0000', 'last_peered': '2024-09-05T16:41:37.728703+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:41:37.728703+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9218, 'log_dups_size': 0, 'ondisk_log_size': 9218, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29006416, 'num_objects': 3541, 'num_object_clones': 0, 'num_object_copies': 14164, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3541, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9217, 'num_write_kb': 51031, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1303}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:41:41.016 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:41:41.017 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'9496", 'reported_seq': 9604, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:37.754466+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:41:37.754466+0000', 'last_peered': '2024-09-05T16:41:37.754466+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:41:37.754466+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9496, 'log_dups_size': 0, 'ondisk_log_size': 9496, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29734048, 'num_objects': 3630, 'num_object_clones': 0, 'num_object_copies': 14520, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3630, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9494, 'num_write_kb': 52500, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1296}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:41:41.017 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:41:41.017 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'9301", 'reported_seq': 9412, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:38.181667+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:41:38.181667+0000', 'last_peered': '2024-09-05T16:41:38.181667+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:41:38.181667+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9301, 'log_dups_size': 0, 'ondisk_log_size': 9301, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28931232, 'num_objects': 3532, 'num_object_clones': 0, 'num_object_copies': 14128, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2310, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3532, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9299, 'num_write_kb': 51325, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1224}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:41:41.017 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:41:41.017 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'9221", 'reported_seq': 9329, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:37.879160+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:41:37.879160+0000', 'last_peered': '2024-09-05T16:41:37.879160+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:41:37.879160+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9221, 'log_dups_size': 0, 'ondisk_log_size': 9221, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29111456, 'num_objects': 3554, 'num_object_clones': 0, 'num_object_copies': 14216, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3554, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9219, 'num_write_kb': 51092, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1334}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:41:41.017 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:41:41.767 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:41:41.770 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 30557/228652 objects degraded (13.364%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":504,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":57163,"data_bytes":468234728,"bytes_used":2885140480,"bytes_avail":383661916160,"bytes_total":386547056640,"degraded_objects":30557,"degraded_total":228652,"degraded_ratio":0.13363976698213881,"recovering_objects_per_sec":9,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":112,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":180},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:41:41.792 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:41:41.792 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:41:42.386 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:41:42.386 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:41:42.422 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":280,"stamp":"2024-09-05T16:41:40.970669+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":468234728,"num_objects":57163,"num_object_clones":0,"num_object_copies":228652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30557,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57163,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":133674,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1080,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":147798,"ondisk_log_size":147798,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2817520,"kb_used_data":918420,"kb_used_omap":14,"kb_used_meta":1898993,"kb_avail":374669840,"statfs":{"total":386547056640,"available":383661916160,"internally_reserved":0,"allocated":940462080,"data_stored":939656984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1944569399},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,6,0,4,9,11,18,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17711104,"num_objects":-2162,"num_object_clones":0,"num_object_copies":-8648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2162,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2162,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":112,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004799"},"pg_stats":[{"pgid":"1.f","version":"37'9271","reported_seq":9375,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:37.727685+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:41:37.727685+0000","last_peered":"2024-09-05T16:41:37.727685+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:41:37.727685+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9271,"log_dups_size":0,"ondisk_log_size":9271,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29761536,"num_objects":3633,"num_object_clones":0,"num_object_copies":14532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3633,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9271,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1397}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'9083","reported_seq":10756,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:37.846973+0000","last_change":"2024-09-05T16:40:28.919189+0000","last_active":"2024-09-05T16:41:37.846973+0000","last_peered":"2024-09-05T16:41:37.846973+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:41:37.846973+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9083,"log_dups_size":0,"ondisk_log_size":9083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28431520,"num_objects":3471,"num_object_clones":0,"num_object_copies":13884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1470,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3471,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9081,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":801,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2001}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'9231","reported_seq":9353,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:37.972851+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:41:37.972851+0000","last_peered":"2024-09-05T16:41:37.972851+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:41:37.972851+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9231,"log_dups_size":0,"ondisk_log_size":9231,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28842600,"num_objects":3522,"num_object_clones":0,"num_object_copies":14088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3522,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9230,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1290}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'9269","reported_seq":13996,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:41:37.271959+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:41:37.271959+0000","last_peered":"2024-09-05T16:41:37.271959+0000","last_clean":"2024-09-05T16:41:37.271959+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:41:37.271959+0000","last_undegraded":"2024-09-05T16:41:37.271959+0000","last_fullsized":"2024-09-05T16:41:37.271959+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":9269,"log_dups_size":0,"ondisk_log_size":9269,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30023680,"num_objects":3665,"num_object_clones":0,"num_object_copies":14660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3569,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'9039","reported_seq":9137,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:37.553922+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:41:37.553922+0000","last_peered":"2024-09-05T16:41:37.553922+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:41:37.553922+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9039,"log_dups_size":0,"ondisk_log_size":9039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29127840,"num_objects":3556,"num_object_clones":0,"num_object_copies":14224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3556,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9037,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1411}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'9200","reported_seq":9314,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:37.807360+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:41:37.807360+0000","last_peered":"2024-09-05T16:41:37.807360+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:41:37.807360+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9200,"log_dups_size":0,"ondisk_log_size":9200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29261824,"num_objects":3572,"num_object_clones":0,"num_object_copies":14288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3572,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9200,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1353}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'9034","reported_seq":9140,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:37.791543+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:41:37.791543+0000","last_peered":"2024-09-05T16:41:37.791543+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:41:37.791543+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9034,"log_dups_size":0,"ondisk_log_size":9034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29717664,"num_objects":3628,"num_object_clones":0,"num_object_copies":14512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9032,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1497}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'9216","reported_seq":9304,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:37.742406+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:41:37.742406+0000","last_peered":"2024-09-05T16:41:37.742406+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:41:37.742406+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9216,"log_dups_size":0,"ondisk_log_size":9216,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29185184,"num_objects":3563,"num_object_clones":0,"num_object_copies":14252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3563,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9214,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1331}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'9230","reported_seq":9330,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:37.748420+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:41:37.748420+0000","last_peered":"2024-09-05T16:41:37.748420+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:41:37.748420+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9230,"log_dups_size":0,"ondisk_log_size":9230,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29026624,"num_objects":3544,"num_object_clones":0,"num_object_copies":14176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3544,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9227,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1300}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'9309","reported_seq":9430,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:38.170297+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:41:38.170297+0000","last_peered":"2024-09-05T16:41:38.170297+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:41:38.170297+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9309,"log_dups_size":0,"ondisk_log_size":9309,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28677280,"num_objects":3501,"num_object_clones":0,"num_object_copies":14004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3501,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9307,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1256}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'9450","reported_seq":9576,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:37.647430+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:41:37.647430+0000","last_peered":"2024-09-05T16:41:37.647430+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:41:37.647430+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9450,"log_dups_size":0,"ondisk_log_size":9450,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30084848,"num_objects":3673,"num_object_clones":0,"num_object_copies":14692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3673,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9447,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1392}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'9218","reported_seq":9316,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:37.728703+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:41:37.728703+0000","last_peered":"2024-09-05T16:41:37.728703+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:41:37.728703+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9218,"log_dups_size":0,"ondisk_log_size":9218,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29006416,"num_objects":3541,"num_object_clones":0,"num_object_copies":14164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3541,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9217,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1303}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'9230","reported_seq":13965,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:41:37.057074+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:41:37.057074+0000","last_peered":"2024-09-05T16:41:37.057074+0000","last_clean":"2024-09-05T16:41:37.057074+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:41:37.057074+0000","last_undegraded":"2024-09-05T16:41:37.057074+0000","last_fullsized":"2024-09-05T16:41:37.057074+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":9230,"log_dups_size":0,"ondisk_log_size":9230,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29310976,"num_objects":3578,"num_object_clones":0,"num_object_copies":14312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3578,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":830,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'9496","reported_seq":9604,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:37.754466+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:41:37.754466+0000","last_peered":"2024-09-05T16:41:37.754466+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:41:37.754466+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9496,"log_dups_size":0,"ondisk_log_size":9496,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29734048,"num_objects":3630,"num_object_clones":0,"num_object_copies":14520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3630,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9494,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1296}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'9301","reported_seq":9412,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:38.181667+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:41:38.181667+0000","last_peered":"2024-09-05T16:41:38.181667+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:41:38.181667+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9301,"log_dups_size":0,"ondisk_log_size":9301,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28931232,"num_objects":3532,"num_object_clones":0,"num_object_copies":14128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3532,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9299,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1224}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'9221","reported_seq":9329,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:37.879160+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:41:37.879160+0000","last_peered":"2024-09-05T16:41:37.879160+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:41:37.879160+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9221,"log_dups_size":0,"ondisk_log_size":9221,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29111456,"num_objects":3554,"num_object_clones":0,"num_object_copies":14216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3554,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9219,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1334}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":468234728,"num_objects":57163,"num_object_clones":0,"num_object_copies":228652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30557,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57163,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":133674,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1080,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":936779776,"data_stored":936779776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":147798,"ondisk_log_size":147798,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084380,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":730624,"kb_used_data":229668,"kb_used_omap":4,"kb_used_meta":500923,"kb_avail":93641216,"statfs":{"total":96636764160,"available":95888605184,"internally_reserved":0,"allocated":235180032,"data_stored":234978758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":512945875},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986257,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":618704,"kb_used_data":230208,"kb_used_omap":4,"kb_used_meta":388475,"kb_avail":93753136,"statfs":{"total":96636764160,"available":96003211264,"internally_reserved":0,"allocated":235732992,"data_stored":235531718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":397799122},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,3,0,4,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673063,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":733428,"kb_used_data":228828,"kb_used_omap":1,"kb_used_meta":504574,"kb_avail":93638412,"statfs":{"total":96636764160,"available":95885733888,"internally_reserved":0,"allocated":234319872,"data_stored":234118598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":516684229},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,0,2,4,6,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182494,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":734764,"kb_used_data":229716,"kb_used_omap":4,"kb_used_meta":505019,"kb_avail":93637076,"statfs":{"total":96636764160,"available":95884365824,"internally_reserved":0,"allocated":235229184,"data_stored":235027910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":517140173},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,4,3,5,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":234074112,"data_stored":234074112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":234807296,"data_stored":234807296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":233787392,"data_stored":233787392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":234110976,"data_stored":234110976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:41:45.424 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:41:46.026 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:41:46.027 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:41:46.057 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":282,"stamp":"2024-09-05T16:41:44.972309+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459551208,"num_objects":56103,"num_object_clones":0,"num_object_copies":224412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30499,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56103,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":134734,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1138,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":148858,"ondisk_log_size":148858,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2816864,"kb_used_data":901108,"kb_used_omap":14,"kb_used_meta":1915633,"kb_avail":374670496,"statfs":{"total":386547056640,"available":383662587904,"internally_reserved":0,"allocated":922734592,"data_stored":921929496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1961608759},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,1,3,17,14,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19398656,"num_objects":-2368,"num_object_clones":0,"num_object_copies":-9472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-114,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2368,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2368,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":114,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005060"},"pg_stats":[{"pgid":"1.f","version":"37'9339","reported_seq":9443,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:42.846468+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:41:42.846468+0000","last_peered":"2024-09-05T16:41:42.846468+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:41:42.846468+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9339,"log_dups_size":0,"ondisk_log_size":9339,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29204480,"num_objects":3565,"num_object_clones":0,"num_object_copies":14260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3565,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9339,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1329}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'9145","reported_seq":10934,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:42.915656+0000","last_change":"2024-09-05T16:40:28.919189+0000","last_active":"2024-09-05T16:41:42.915656+0000","last_peered":"2024-09-05T16:41:42.915656+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:41:42.915656+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9145,"log_dups_size":0,"ondisk_log_size":9145,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27923616,"num_objects":3409,"num_object_clones":0,"num_object_copies":13636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1412,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3409,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9143,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":859,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1997}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'9285","reported_seq":9407,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:42.942955+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:41:42.942955+0000","last_peered":"2024-09-05T16:41:42.942955+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:41:42.942955+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9285,"log_dups_size":0,"ondisk_log_size":9285,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28400232,"num_objects":3468,"num_object_clones":0,"num_object_copies":13872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3468,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9284,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1236}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'9335","reported_seq":14064,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:41:42.137646+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:41:42.137646+0000","last_peered":"2024-09-05T16:41:42.137646+0000","last_clean":"2024-09-05T16:41:42.137646+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:41:42.137646+0000","last_undegraded":"2024-09-05T16:41:42.137646+0000","last_fullsized":"2024-09-05T16:41:42.137646+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":9335,"log_dups_size":0,"ondisk_log_size":9335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29483008,"num_objects":3599,"num_object_clones":0,"num_object_copies":14396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3635,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'9104","reported_seq":9202,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:42.831370+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:41:42.831370+0000","last_peered":"2024-09-05T16:41:42.831370+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:41:42.831370+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9104,"log_dups_size":0,"ondisk_log_size":9104,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28595360,"num_objects":3491,"num_object_clones":0,"num_object_copies":13964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3491,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9102,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1346}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'9281","reported_seq":9394,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:42.850297+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:41:42.850297+0000","last_peered":"2024-09-05T16:41:42.850297+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:41:42.850297+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9281,"log_dups_size":0,"ondisk_log_size":9281,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28598272,"num_objects":3491,"num_object_clones":0,"num_object_copies":13964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3491,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9281,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1272}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'9109","reported_seq":9216,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:42.892228+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:41:42.892228+0000","last_peered":"2024-09-05T16:41:42.892228+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:41:42.892228+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9109,"log_dups_size":0,"ondisk_log_size":9109,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29103264,"num_objects":3553,"num_object_clones":0,"num_object_copies":14212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3553,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9107,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1422}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'9283","reported_seq":9371,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:42.749463+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:41:42.749463+0000","last_peered":"2024-09-05T16:41:42.749463+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:41:42.749463+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9283,"log_dups_size":0,"ondisk_log_size":9283,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28636320,"num_objects":3496,"num_object_clones":0,"num_object_copies":13984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3496,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9281,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1264}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'9307","reported_seq":9407,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:42.661220+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:41:42.661220+0000","last_peered":"2024-09-05T16:41:42.661220+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:41:42.661220+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9307,"log_dups_size":0,"ondisk_log_size":9307,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28395840,"num_objects":3467,"num_object_clones":0,"num_object_copies":13868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3467,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9304,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1223}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'9370","reported_seq":9491,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:42.942916+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:41:42.942916+0000","last_peered":"2024-09-05T16:41:42.942916+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:41:42.942916+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9370,"log_dups_size":0,"ondisk_log_size":9370,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28177568,"num_objects":3440,"num_object_clones":0,"num_object_copies":13760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3440,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9368,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1195}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'9505","reported_seq":9631,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:42.864760+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:41:42.864760+0000","last_peered":"2024-09-05T16:41:42.864760+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:41:42.864760+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9505,"log_dups_size":0,"ondisk_log_size":9505,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29634288,"num_objects":3618,"num_object_clones":0,"num_object_copies":14472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3618,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9502,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1337}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'9278","reported_seq":9376,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:42.711506+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:41:42.711506+0000","last_peered":"2024-09-05T16:41:42.711506+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:41:42.711506+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9278,"log_dups_size":0,"ondisk_log_size":9278,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28514896,"num_objects":3481,"num_object_clones":0,"num_object_copies":13924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3481,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9277,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1243}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'9291","reported_seq":14024,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:41:41.884909+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:41:41.884909+0000","last_peered":"2024-09-05T16:41:41.884909+0000","last_clean":"2024-09-05T16:41:41.884909+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:41:41.884909+0000","last_undegraded":"2024-09-05T16:41:41.884909+0000","last_fullsized":"2024-09-05T16:41:41.884909+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":9291,"log_dups_size":0,"ondisk_log_size":9291,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28811264,"num_objects":3517,"num_object_clones":0,"num_object_copies":14068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3517,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":891,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'9565","reported_seq":9673,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:42.817680+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:41:42.817680+0000","last_peered":"2024-09-05T16:41:42.817680+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:41:42.817680+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9565,"log_dups_size":0,"ondisk_log_size":9565,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29168800,"num_objects":3561,"num_object_clones":0,"num_object_copies":14244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3561,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9563,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1227}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'9376","reported_seq":9488,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:43.100969+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:41:43.100969+0000","last_peered":"2024-09-05T16:41:43.100969+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:41:43.100969+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9376,"log_dups_size":0,"ondisk_log_size":9376,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28316832,"num_objects":3457,"num_object_clones":0,"num_object_copies":13828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3457,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9374,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1149}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'9285","reported_seq":9393,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:42.854123+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:41:42.854123+0000","last_peered":"2024-09-05T16:41:42.854123+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:41:42.854123+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9285,"log_dups_size":0,"ondisk_log_size":9285,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28587168,"num_objects":3490,"num_object_clones":0,"num_object_copies":13960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9283,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1270}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":459551208,"num_objects":56103,"num_object_clones":0,"num_object_copies":224412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30499,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56103,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":134734,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1138,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":919064576,"data_stored":919064576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":148858,"ondisk_log_size":148858,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084381,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":730460,"kb_used_data":225356,"kb_used_omap":4,"kb_used_meta":505083,"kb_avail":93641380,"statfs":{"total":96636764160,"available":95888773120,"internally_reserved":0,"allocated":230764544,"data_stored":230563270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":517205715},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,0,0,2,6,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986258,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":618572,"kb_used_data":225876,"kb_used_omap":4,"kb_used_meta":392635,"kb_avail":93753268,"statfs":{"total":96636764160,"available":96003346432,"internally_reserved":0,"allocated":231297024,"data_stored":231095750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":402058962},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,4,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673064,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":733272,"kb_used_data":224516,"kb_used_omap":1,"kb_used_meta":508734,"kb_avail":93638568,"statfs":{"total":96636764160,"available":95885893632,"internally_reserved":0,"allocated":229904384,"data_stored":229703110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":520944069},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,2,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182495,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":734560,"kb_used_data":225360,"kb_used_omap":4,"kb_used_meta":509179,"kb_avail":93637280,"statfs":{"total":96636764160,"available":95884574720,"internally_reserved":0,"allocated":230768640,"data_stored":230567366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":521400013},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,9,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":229638144,"data_stored":229638144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":230379520,"data_stored":230379520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":229388288,"data_stored":229388288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":229658624,"data_stored":229658624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:41:46.059 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:41:46.684 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:41:46.685 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:41:46.701 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":282,"stamp":"2024-09-05T16:41:44.972309+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459551208,"num_objects":56103,"num_object_clones":0,"num_object_copies":224412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30499,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56103,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":134734,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1138,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":148858,"ondisk_log_size":148858,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2816864,"kb_used_data":901108,"kb_used_omap":14,"kb_used_meta":1915633,"kb_avail":374670496,"statfs":{"total":386547056640,"available":383662587904,"internally_reserved":0,"allocated":922734592,"data_stored":921929496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1961608759},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,1,3,17,14,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19398656,"num_objects":-2368,"num_object_clones":0,"num_object_copies":-9472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-114,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2368,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2368,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":114,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005060"},"pg_stats":[{"pgid":"1.f","version":"37'9339","reported_seq":9443,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:42.846468+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:41:42.846468+0000","last_peered":"2024-09-05T16:41:42.846468+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:41:42.846468+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9339,"log_dups_size":0,"ondisk_log_size":9339,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29204480,"num_objects":3565,"num_object_clones":0,"num_object_copies":14260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3565,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9339,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1329}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'9145","reported_seq":10934,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:42.915656+0000","last_change":"2024-09-05T16:40:28.919189+0000","last_active":"2024-09-05T16:41:42.915656+0000","last_peered":"2024-09-05T16:41:42.915656+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:41:42.915656+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9145,"log_dups_size":0,"ondisk_log_size":9145,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27923616,"num_objects":3409,"num_object_clones":0,"num_object_copies":13636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1412,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3409,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9143,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":859,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1997}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'9285","reported_seq":9407,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:42.942955+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:41:42.942955+0000","last_peered":"2024-09-05T16:41:42.942955+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:41:42.942955+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9285,"log_dups_size":0,"ondisk_log_size":9285,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28400232,"num_objects":3468,"num_object_clones":0,"num_object_copies":13872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3468,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9284,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1236}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'9335","reported_seq":14064,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:41:42.137646+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:41:42.137646+0000","last_peered":"2024-09-05T16:41:42.137646+0000","last_clean":"2024-09-05T16:41:42.137646+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:41:42.137646+0000","last_undegraded":"2024-09-05T16:41:42.137646+0000","last_fullsized":"2024-09-05T16:41:42.137646+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":9335,"log_dups_size":0,"ondisk_log_size":9335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29483008,"num_objects":3599,"num_object_clones":0,"num_object_copies":14396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3635,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'9104","reported_seq":9202,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:42.831370+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:41:42.831370+0000","last_peered":"2024-09-05T16:41:42.831370+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:41:42.831370+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9104,"log_dups_size":0,"ondisk_log_size":9104,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28595360,"num_objects":3491,"num_object_clones":0,"num_object_copies":13964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3491,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9102,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1346}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'9281","reported_seq":9394,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:42.850297+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:41:42.850297+0000","last_peered":"2024-09-05T16:41:42.850297+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:41:42.850297+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9281,"log_dups_size":0,"ondisk_log_size":9281,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28598272,"num_objects":3491,"num_object_clones":0,"num_object_copies":13964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3491,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9281,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1272}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'9109","reported_seq":9216,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:42.892228+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:41:42.892228+0000","last_peered":"2024-09-05T16:41:42.892228+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:41:42.892228+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9109,"log_dups_size":0,"ondisk_log_size":9109,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29103264,"num_objects":3553,"num_object_clones":0,"num_object_copies":14212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3553,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9107,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1422}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'9283","reported_seq":9371,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:42.749463+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:41:42.749463+0000","last_peered":"2024-09-05T16:41:42.749463+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:41:42.749463+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9283,"log_dups_size":0,"ondisk_log_size":9283,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28636320,"num_objects":3496,"num_object_clones":0,"num_object_copies":13984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3496,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9281,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1264}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'9307","reported_seq":9407,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:42.661220+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:41:42.661220+0000","last_peered":"2024-09-05T16:41:42.661220+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:41:42.661220+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9307,"log_dups_size":0,"ondisk_log_size":9307,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28395840,"num_objects":3467,"num_object_clones":0,"num_object_copies":13868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3467,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9304,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1223}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'9370","reported_seq":9491,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:42.942916+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:41:42.942916+0000","last_peered":"2024-09-05T16:41:42.942916+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:41:42.942916+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9370,"log_dups_size":0,"ondisk_log_size":9370,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28177568,"num_objects":3440,"num_object_clones":0,"num_object_copies":13760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3440,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9368,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1195}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'9505","reported_seq":9631,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:42.864760+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:41:42.864760+0000","last_peered":"2024-09-05T16:41:42.864760+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:41:42.864760+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9505,"log_dups_size":0,"ondisk_log_size":9505,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29634288,"num_objects":3618,"num_object_clones":0,"num_object_copies":14472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3618,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9502,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1337}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'9278","reported_seq":9376,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:42.711506+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:41:42.711506+0000","last_peered":"2024-09-05T16:41:42.711506+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:41:42.711506+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9278,"log_dups_size":0,"ondisk_log_size":9278,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28514896,"num_objects":3481,"num_object_clones":0,"num_object_copies":13924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3481,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9277,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1243}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'9291","reported_seq":14024,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:41:41.884909+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:41:41.884909+0000","last_peered":"2024-09-05T16:41:41.884909+0000","last_clean":"2024-09-05T16:41:41.884909+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:41:41.884909+0000","last_undegraded":"2024-09-05T16:41:41.884909+0000","last_fullsized":"2024-09-05T16:41:41.884909+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":9291,"log_dups_size":0,"ondisk_log_size":9291,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28811264,"num_objects":3517,"num_object_clones":0,"num_object_copies":14068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3517,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":891,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'9565","reported_seq":9673,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:42.817680+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:41:42.817680+0000","last_peered":"2024-09-05T16:41:42.817680+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:41:42.817680+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9565,"log_dups_size":0,"ondisk_log_size":9565,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29168800,"num_objects":3561,"num_object_clones":0,"num_object_copies":14244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3561,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9563,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1227}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'9376","reported_seq":9488,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:43.100969+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:41:43.100969+0000","last_peered":"2024-09-05T16:41:43.100969+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:41:43.100969+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9376,"log_dups_size":0,"ondisk_log_size":9376,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28316832,"num_objects":3457,"num_object_clones":0,"num_object_copies":13828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3457,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9374,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1149}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'9285","reported_seq":9393,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:42.854123+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:41:42.854123+0000","last_peered":"2024-09-05T16:41:42.854123+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:41:42.854123+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9285,"log_dups_size":0,"ondisk_log_size":9285,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28587168,"num_objects":3490,"num_object_clones":0,"num_object_copies":13960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9283,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1270}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":459551208,"num_objects":56103,"num_object_clones":0,"num_object_copies":224412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30499,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56103,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":134734,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1138,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":919064576,"data_stored":919064576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":148858,"ondisk_log_size":148858,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084381,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":730460,"kb_used_data":225356,"kb_used_omap":4,"kb_used_meta":505083,"kb_avail":93641380,"statfs":{"total":96636764160,"available":95888773120,"internally_reserved":0,"allocated":230764544,"data_stored":230563270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":517205715},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,0,0,2,6,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986258,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":618572,"kb_used_data":225876,"kb_used_omap":4,"kb_used_meta":392635,"kb_avail":93753268,"statfs":{"total":96636764160,"available":96003346432,"internally_reserved":0,"allocated":231297024,"data_stored":231095750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":402058962},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,4,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673064,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":733272,"kb_used_data":224516,"kb_used_omap":1,"kb_used_meta":508734,"kb_avail":93638568,"statfs":{"total":96636764160,"available":95885893632,"internally_reserved":0,"allocated":229904384,"data_stored":229703110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":520944069},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,2,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182495,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":734560,"kb_used_data":225360,"kb_used_omap":4,"kb_used_meta":509179,"kb_avail":93637280,"statfs":{"total":96636764160,"available":95884574720,"internally_reserved":0,"allocated":230768640,"data_stored":230567366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":521400013},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,9,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":229638144,"data_stored":229638144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":230379520,"data_stored":230379520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":229388288,"data_stored":229388288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":229658624,"data_stored":229658624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:41:46.702 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-05T16:41:46.702 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'9339", 'reported_seq': 9443, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:42.846468+0000', 'last_change': '2024-09-05T16:34:56.833497+0000', 'last_active': '2024-09-05T16:41:42.846468+0000', 'last_peered': '2024-09-05T16:41:42.846468+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.828289+0000', 'last_became_peered': '2024-09-05T16:34:56.828289+0000', 'last_unstale': '2024-09-05T16:41:42.846468+0000', 'last_undegraded': '2024-09-05T16:34:56.612160+0000', 'last_fullsized': '2024-09-05T16:34:56.557632+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9339, 'log_dups_size': 0, 'ondisk_log_size': 9339, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29204480, 'num_objects': 3565, 'num_object_clones': 0, 'num_object_copies': 14260, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2236, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3565, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9339, 'num_write_kb': 51616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1329}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:41:46.702 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-05T16:41:46.702 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "37'9145", 'reported_seq': 10934, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:42.915656+0000', 'last_change': '2024-09-05T16:40:28.919189+0000', 'last_active': '2024-09-05T16:41:42.915656+0000', 'last_peered': '2024-09-05T16:41:42.915656+0000', 'last_clean': '2024-09-05T16:33:53.200762+0000', 'last_became_active': '2024-09-05T16:34:56.820292+0000', 'last_became_peered': '2024-09-05T16:34:56.820292+0000', 'last_unstale': '2024-09-05T16:41:42.915656+0000', 'last_undegraded': '2024-09-05T16:34:56.605776+0000', 'last_fullsized': '2024-09-05T16:34:56.559383+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9145, 'log_dups_size': 0, 'ondisk_log_size': 9145, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27923616, 'num_objects': 3409, 'num_object_clones': 0, 'num_object_copies': 13636, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1412, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3409, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9143, 'num_write_kb': 50212, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 859, 'num_bytes_recovered': 2613248, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 1997}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:41:46.702 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:41:46.702 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'9285", 'reported_seq': 9407, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:42.942955+0000', 'last_change': '2024-09-05T16:34:56.867900+0000', 'last_active': '2024-09-05T16:41:42.942955+0000', 'last_peered': '2024-09-05T16:41:42.942955+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:41:42.942955+0000', 'last_undegraded': '2024-09-05T16:34:56.590972+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9285, 'log_dups_size': 0, 'ondisk_log_size': 9285, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28400232, 'num_objects': 3468, 'num_object_clones': 0, 'num_object_copies': 13872, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2233, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3468, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 9284, 'num_write_kb': 50999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1236}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:41:46.702 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:41:46.702 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'9104", 'reported_seq': 9202, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:42.831370+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:41:42.831370+0000', 'last_peered': '2024-09-05T16:41:42.831370+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:41:42.831370+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9104, 'log_dups_size': 0, 'ondisk_log_size': 9104, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28595360, 'num_objects': 3491, 'num_object_clones': 0, 'num_object_copies': 13964, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2147, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3491, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9102, 'num_write_kb': 50373, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1346}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:41:46.703 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:41:46.703 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'9281", 'reported_seq': 9394, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:42.850297+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:41:42.850297+0000', 'last_peered': '2024-09-05T16:41:42.850297+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:41:42.850297+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9281, 'log_dups_size': 0, 'ondisk_log_size': 9281, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28598272, 'num_objects': 3491, 'num_object_clones': 0, 'num_object_copies': 13964, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3491, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9281, 'num_write_kb': 51088, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1272}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:41:46.703 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:41:46.703 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'9109", 'reported_seq': 9216, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:42.892228+0000', 'last_change': '2024-09-05T16:34:56.903436+0000', 'last_active': '2024-09-05T16:41:42.892228+0000', 'last_peered': '2024-09-05T16:41:42.892228+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:41:42.892228+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9109, 'log_dups_size': 0, 'ondisk_log_size': 9109, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29103264, 'num_objects': 3553, 'num_object_clones': 0, 'num_object_copies': 14212, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2133, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3553, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9107, 'num_write_kb': 50636, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1422}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:41:46.703 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-05T16:41:46.703 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "37'9283", 'reported_seq': 9371, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:42.749463+0000', 'last_change': '2024-09-05T16:34:56.883319+0000', 'last_active': '2024-09-05T16:41:42.749463+0000', 'last_peered': '2024-09-05T16:41:42.749463+0000', 'last_clean': '2024-09-05T16:33:53.505305+0000', 'last_became_active': '2024-09-05T16:34:56.873389+0000', 'last_became_peered': '2024-09-05T16:34:56.873389+0000', 'last_unstale': '2024-09-05T16:41:42.749463+0000', 'last_undegraded': '2024-09-05T16:34:56.602567+0000', 'last_fullsized': '2024-09-05T16:34:56.559958+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9283, 'log_dups_size': 0, 'ondisk_log_size': 9283, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28636320, 'num_objects': 3496, 'num_object_clones': 0, 'num_object_copies': 13984, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2234, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3496, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9281, 'num_write_kb': 51108, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1264}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:41:46.703 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:41:46.703 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'9307", 'reported_seq': 9407, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:42.661220+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:41:42.661220+0000', 'last_peered': '2024-09-05T16:41:42.661220+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:41:42.661220+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9307, 'log_dups_size': 0, 'ondisk_log_size': 9307, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28395840, 'num_objects': 3467, 'num_object_clones': 0, 'num_object_copies': 13868, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3467, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9304, 'num_write_kb': 51081, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1223}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:41:46.703 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:41:46.703 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'9370", 'reported_seq': 9491, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:42.942916+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:41:42.942916+0000', 'last_peered': '2024-09-05T16:41:42.942916+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:41:42.942916+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9370, 'log_dups_size': 0, 'ondisk_log_size': 9370, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28177568, 'num_objects': 3440, 'num_object_clones': 0, 'num_object_copies': 13760, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3440, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9368, 'num_write_kb': 51230, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1195}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:41:46.703 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:41:46.704 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'9505", 'reported_seq': 9631, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:42.864760+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:41:42.864760+0000', 'last_peered': '2024-09-05T16:41:42.864760+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:41:42.864760+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9505, 'log_dups_size': 0, 'ondisk_log_size': 9505, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29634288, 'num_objects': 3618, 'num_object_clones': 0, 'num_object_copies': 14472, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2284, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3618, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9502, 'num_write_kb': 52476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1337}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:41:46.704 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:41:46.704 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'9278", 'reported_seq': 9376, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:42.711506+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:41:42.711506+0000', 'last_peered': '2024-09-05T16:41:42.711506+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:41:42.711506+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9278, 'log_dups_size': 0, 'ondisk_log_size': 9278, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28514896, 'num_objects': 3481, 'num_object_clones': 0, 'num_object_copies': 13924, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3481, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9277, 'num_write_kb': 51031, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1243}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:41:46.704 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:41:46.704 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'9565", 'reported_seq': 9673, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:42.817680+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:41:42.817680+0000', 'last_peered': '2024-09-05T16:41:42.817680+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:41:42.817680+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9565, 'log_dups_size': 0, 'ondisk_log_size': 9565, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29168800, 'num_objects': 3561, 'num_object_clones': 0, 'num_object_copies': 14244, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3561, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9563, 'num_write_kb': 52500, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1227}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:41:46.704 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:41:46.704 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'9376", 'reported_seq': 9488, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:43.100969+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:41:43.100969+0000', 'last_peered': '2024-09-05T16:41:43.100969+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:41:43.100969+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9376, 'log_dups_size': 0, 'ondisk_log_size': 9376, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28316832, 'num_objects': 3457, 'num_object_clones': 0, 'num_object_copies': 13828, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2310, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3457, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9374, 'num_write_kb': 51325, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1149}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:41:46.704 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:41:46.704 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'9285", 'reported_seq': 9393, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:42.854123+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:41:42.854123+0000', 'last_peered': '2024-09-05T16:41:42.854123+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:41:42.854123+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9285, 'log_dups_size': 0, 'ondisk_log_size': 9285, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28587168, 'num_objects': 3490, 'num_object_clones': 0, 'num_object_copies': 13960, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3490, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9283, 'num_write_kb': 51092, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1270}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:41:46.705 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:41:47.422 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:41:47.422 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 30499/224412 objects degraded (13.591%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":510,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":56103,"data_bytes":459551208,"bytes_used":2884468736,"bytes_avail":383662587904,"bytes_total":386547056640,"degraded_objects":30499,"degraded_total":224412,"degraded_ratio":0.1359062795215942,"recovering_objects_per_sec":9,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":114,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":179},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:41:47.455 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:41:47.455 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:41:48.043 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:41:48.044 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:41:48.077 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":283,"stamp":"2024-09-05T16:41:46.973054+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459551208,"num_objects":56103,"num_object_clones":0,"num_object_copies":224412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30499,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56103,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":134734,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1138,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":148858,"ondisk_log_size":148858,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2816864,"kb_used_data":901108,"kb_used_omap":14,"kb_used_meta":1915633,"kb_avail":374670496,"statfs":{"total":386547056640,"available":383662587904,"internally_reserved":0,"allocated":922734592,"data_stored":921929496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1961608759},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,1,3,17,14,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17686528,"num_objects":-2159,"num_object_clones":0,"num_object_copies":-8636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-114,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2159,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2159,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":114,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004858"},"pg_stats":[{"pgid":"1.f","version":"37'9339","reported_seq":9443,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:42.846468+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:41:42.846468+0000","last_peered":"2024-09-05T16:41:42.846468+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:41:42.846468+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9339,"log_dups_size":0,"ondisk_log_size":9339,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29204480,"num_objects":3565,"num_object_clones":0,"num_object_copies":14260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3565,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9339,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1329}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'9145","reported_seq":10934,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:42.915656+0000","last_change":"2024-09-05T16:40:28.919189+0000","last_active":"2024-09-05T16:41:42.915656+0000","last_peered":"2024-09-05T16:41:42.915656+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:41:42.915656+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9145,"log_dups_size":0,"ondisk_log_size":9145,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27923616,"num_objects":3409,"num_object_clones":0,"num_object_copies":13636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1412,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3409,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9143,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":859,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1997}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'9285","reported_seq":9407,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:42.942955+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:41:42.942955+0000","last_peered":"2024-09-05T16:41:42.942955+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:41:42.942955+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9285,"log_dups_size":0,"ondisk_log_size":9285,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28400232,"num_objects":3468,"num_object_clones":0,"num_object_copies":13872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3468,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9284,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1236}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'9335","reported_seq":14064,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:41:42.137646+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:41:42.137646+0000","last_peered":"2024-09-05T16:41:42.137646+0000","last_clean":"2024-09-05T16:41:42.137646+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:41:42.137646+0000","last_undegraded":"2024-09-05T16:41:42.137646+0000","last_fullsized":"2024-09-05T16:41:42.137646+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":9335,"log_dups_size":0,"ondisk_log_size":9335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29483008,"num_objects":3599,"num_object_clones":0,"num_object_copies":14396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3635,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'9104","reported_seq":9202,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:42.831370+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:41:42.831370+0000","last_peered":"2024-09-05T16:41:42.831370+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:41:42.831370+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9104,"log_dups_size":0,"ondisk_log_size":9104,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28595360,"num_objects":3491,"num_object_clones":0,"num_object_copies":13964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3491,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9102,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1346}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'9281","reported_seq":9394,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:42.850297+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:41:42.850297+0000","last_peered":"2024-09-05T16:41:42.850297+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:41:42.850297+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9281,"log_dups_size":0,"ondisk_log_size":9281,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28598272,"num_objects":3491,"num_object_clones":0,"num_object_copies":13964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3491,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9281,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1272}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'9109","reported_seq":9216,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:42.892228+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:41:42.892228+0000","last_peered":"2024-09-05T16:41:42.892228+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:41:42.892228+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9109,"log_dups_size":0,"ondisk_log_size":9109,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29103264,"num_objects":3553,"num_object_clones":0,"num_object_copies":14212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3553,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9107,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1422}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'9283","reported_seq":9371,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:42.749463+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:41:42.749463+0000","last_peered":"2024-09-05T16:41:42.749463+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:41:42.749463+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9283,"log_dups_size":0,"ondisk_log_size":9283,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28636320,"num_objects":3496,"num_object_clones":0,"num_object_copies":13984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3496,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9281,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1264}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'9307","reported_seq":9407,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:42.661220+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:41:42.661220+0000","last_peered":"2024-09-05T16:41:42.661220+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:41:42.661220+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9307,"log_dups_size":0,"ondisk_log_size":9307,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28395840,"num_objects":3467,"num_object_clones":0,"num_object_copies":13868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3467,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9304,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1223}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'9370","reported_seq":9491,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:42.942916+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:41:42.942916+0000","last_peered":"2024-09-05T16:41:42.942916+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:41:42.942916+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9370,"log_dups_size":0,"ondisk_log_size":9370,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28177568,"num_objects":3440,"num_object_clones":0,"num_object_copies":13760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3440,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9368,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1195}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'9505","reported_seq":9631,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:42.864760+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:41:42.864760+0000","last_peered":"2024-09-05T16:41:42.864760+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:41:42.864760+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9505,"log_dups_size":0,"ondisk_log_size":9505,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29634288,"num_objects":3618,"num_object_clones":0,"num_object_copies":14472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3618,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9502,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1337}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'9278","reported_seq":9376,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:42.711506+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:41:42.711506+0000","last_peered":"2024-09-05T16:41:42.711506+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:41:42.711506+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9278,"log_dups_size":0,"ondisk_log_size":9278,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28514896,"num_objects":3481,"num_object_clones":0,"num_object_copies":13924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3481,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9277,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1243}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'9291","reported_seq":14024,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:41:41.884909+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:41:41.884909+0000","last_peered":"2024-09-05T16:41:41.884909+0000","last_clean":"2024-09-05T16:41:41.884909+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:41:41.884909+0000","last_undegraded":"2024-09-05T16:41:41.884909+0000","last_fullsized":"2024-09-05T16:41:41.884909+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":9291,"log_dups_size":0,"ondisk_log_size":9291,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28811264,"num_objects":3517,"num_object_clones":0,"num_object_copies":14068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3517,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":891,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'9565","reported_seq":9673,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:42.817680+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:41:42.817680+0000","last_peered":"2024-09-05T16:41:42.817680+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:41:42.817680+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9565,"log_dups_size":0,"ondisk_log_size":9565,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29168800,"num_objects":3561,"num_object_clones":0,"num_object_copies":14244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3561,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9563,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1227}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'9376","reported_seq":9488,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:43.100969+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:41:43.100969+0000","last_peered":"2024-09-05T16:41:43.100969+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:41:43.100969+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9376,"log_dups_size":0,"ondisk_log_size":9376,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28316832,"num_objects":3457,"num_object_clones":0,"num_object_copies":13828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3457,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9374,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1149}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'9285","reported_seq":9393,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:42.854123+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:41:42.854123+0000","last_peered":"2024-09-05T16:41:42.854123+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:41:42.854123+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9285,"log_dups_size":0,"ondisk_log_size":9285,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28587168,"num_objects":3490,"num_object_clones":0,"num_object_copies":13960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9283,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1270}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":459551208,"num_objects":56103,"num_object_clones":0,"num_object_copies":224412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30499,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56103,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":134734,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1138,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":919064576,"data_stored":919064576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":148858,"ondisk_log_size":148858,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084381,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":730460,"kb_used_data":225356,"kb_used_omap":4,"kb_used_meta":505083,"kb_avail":93641380,"statfs":{"total":96636764160,"available":95888773120,"internally_reserved":0,"allocated":230764544,"data_stored":230563270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":517205715},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,0,0,2,6,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986258,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":618572,"kb_used_data":225876,"kb_used_omap":4,"kb_used_meta":392635,"kb_avail":93753268,"statfs":{"total":96636764160,"available":96003346432,"internally_reserved":0,"allocated":231297024,"data_stored":231095750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":402058962},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,4,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673064,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":733272,"kb_used_data":224516,"kb_used_omap":1,"kb_used_meta":508734,"kb_avail":93638568,"statfs":{"total":96636764160,"available":95885893632,"internally_reserved":0,"allocated":229904384,"data_stored":229703110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":520944069},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,2,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182495,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":734560,"kb_used_data":225360,"kb_used_omap":4,"kb_used_meta":509179,"kb_avail":93637280,"statfs":{"total":96636764160,"available":95884574720,"internally_reserved":0,"allocated":230768640,"data_stored":230567366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":521400013},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,9,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":229638144,"data_stored":229638144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":230379520,"data_stored":230379520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":229388288,"data_stored":229388288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":229658624,"data_stored":229658624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:41:51.079 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:41:51.891 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:41:51.892 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:41:51.925 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":285,"stamp":"2024-09-05T16:41:50.974605+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":450769384,"num_objects":55031,"num_object_clones":0,"num_object_copies":220124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30443,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55031,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":135806,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1194,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149930,"ondisk_log_size":149930,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2782252,"kb_used_data":883168,"kb_used_omap":14,"kb_used_meta":1898993,"kb_avail":374705108,"statfs":{"total":386547056640,"available":383698030592,"internally_reserved":0,"allocated":904364032,"data_stored":903558936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1944569399},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,1,10,14,17,9,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17465344,"num_objects":-2132,"num_object_clones":0,"num_object_copies":-8528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-114,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2132,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2132,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":114,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004812"},"pg_stats":[{"pgid":"1.f","version":"37'9418","reported_seq":9522,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:47.837770+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:41:47.837770+0000","last_peered":"2024-09-05T16:41:47.837770+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:41:47.837770+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9418,"log_dups_size":0,"ondisk_log_size":9418,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28557312,"num_objects":3486,"num_object_clones":0,"num_object_copies":13944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3486,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9418,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1250}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'9205","reported_seq":11103,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:47.974668+0000","last_change":"2024-09-05T16:40:28.919189+0000","last_active":"2024-09-05T16:41:47.974668+0000","last_peered":"2024-09-05T16:41:47.974668+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:41:47.974668+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9205,"log_dups_size":0,"ondisk_log_size":9205,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27432096,"num_objects":3349,"num_object_clones":0,"num_object_copies":13396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1356,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3349,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9203,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":915,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1993}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'9357","reported_seq":9478,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:47.990834+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:41:47.990834+0000","last_peered":"2024-09-05T16:41:47.990834+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:41:47.990834+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9357,"log_dups_size":0,"ondisk_log_size":9357,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27810408,"num_objects":3396,"num_object_clones":0,"num_object_copies":13584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3396,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9356,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1164}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'9393","reported_seq":14122,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:41:47.091318+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:41:47.091318+0000","last_peered":"2024-09-05T16:41:47.091318+0000","last_clean":"2024-09-05T16:41:47.091318+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:41:47.091318+0000","last_undegraded":"2024-09-05T16:41:47.091318+0000","last_fullsized":"2024-09-05T16:41:47.091318+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":9393,"log_dups_size":0,"ondisk_log_size":9393,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29007872,"num_objects":3541,"num_object_clones":0,"num_object_copies":14164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3541,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3693,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'9175","reported_seq":9273,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:47.748799+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:41:47.748799+0000","last_peered":"2024-09-05T16:41:47.748799+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:41:47.748799+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9175,"log_dups_size":0,"ondisk_log_size":9175,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28013728,"num_objects":3420,"num_object_clones":0,"num_object_copies":13680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3420,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9173,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1275}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'9352","reported_seq":9466,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:47.835607+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:41:47.835607+0000","last_peered":"2024-09-05T16:41:47.835607+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:41:47.835607+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9352,"log_dups_size":0,"ondisk_log_size":9352,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28016640,"num_objects":3420,"num_object_clones":0,"num_object_copies":13680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3420,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9352,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1201}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'9170","reported_seq":9277,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:47.815255+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:41:47.815255+0000","last_peered":"2024-09-05T16:41:47.815255+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:41:47.815255+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9170,"log_dups_size":0,"ondisk_log_size":9170,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28603552,"num_objects":3492,"num_object_clones":0,"num_object_copies":13968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3492,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9168,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1361}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'9355","reported_seq":9443,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:47.684466+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:41:47.684466+0000","last_peered":"2024-09-05T16:41:47.684466+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:41:47.684466+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9355,"log_dups_size":0,"ondisk_log_size":9355,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28046496,"num_objects":3424,"num_object_clones":0,"num_object_copies":13696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3424,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9353,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1192}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'9365","reported_seq":9465,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:47.837109+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:41:47.837109+0000","last_peered":"2024-09-05T16:41:47.837109+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:41:47.837109+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9365,"log_dups_size":0,"ondisk_log_size":9365,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27920704,"num_objects":3409,"num_object_clones":0,"num_object_copies":13636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3409,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9362,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1165}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'9431","reported_seq":9552,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:48.112923+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:41:48.112923+0000","last_peered":"2024-09-05T16:41:48.112923+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:41:48.112923+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9431,"log_dups_size":0,"ondisk_log_size":9431,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27677856,"num_objects":3379,"num_object_clones":0,"num_object_copies":13516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3379,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9429,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1134}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'9580","reported_seq":9706,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:47.736332+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:41:47.736332+0000","last_peered":"2024-09-05T16:41:47.736332+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:41:47.736332+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9580,"log_dups_size":0,"ondisk_log_size":9580,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29019888,"num_objects":3543,"num_object_clones":0,"num_object_copies":14172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3543,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9577,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1262}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'9335","reported_seq":9433,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:47.844891+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:41:47.844891+0000","last_peered":"2024-09-05T16:41:47.844891+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:41:47.844891+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9335,"log_dups_size":0,"ondisk_log_size":9335,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28047952,"num_objects":3424,"num_object_clones":0,"num_object_copies":13696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3424,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9334,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1186}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'9366","reported_seq":14101,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:41:47.142525+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:41:47.142525+0000","last_peered":"2024-09-05T16:41:47.142525+0000","last_clean":"2024-09-05T16:41:47.142525+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:41:47.142525+0000","last_undegraded":"2024-09-05T16:41:47.142525+0000","last_fullsized":"2024-09-05T16:41:47.142525+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":9366,"log_dups_size":0,"ondisk_log_size":9366,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28196864,"num_objects":3442,"num_object_clones":0,"num_object_copies":13768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3442,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":966,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'9638","reported_seq":9746,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:47.963770+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:41:47.963770+0000","last_peered":"2024-09-05T16:41:47.963770+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:41:47.963770+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9638,"log_dups_size":0,"ondisk_log_size":9638,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28570784,"num_objects":3488,"num_object_clones":0,"num_object_copies":13952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3488,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9636,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1154}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'9443","reported_seq":9555,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:48.162012+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:41:48.162012+0000","last_peered":"2024-09-05T16:41:48.162012+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:41:48.162012+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9443,"log_dups_size":0,"ondisk_log_size":9443,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27767968,"num_objects":3390,"num_object_clones":0,"num_object_copies":13560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3390,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9441,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1082}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'9347","reported_seq":9455,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:47.711982+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:41:47.711982+0000","last_peered":"2024-09-05T16:41:47.711982+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:41:47.711982+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9347,"log_dups_size":0,"ondisk_log_size":9347,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28079264,"num_objects":3428,"num_object_clones":0,"num_object_copies":13712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3428,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9345,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1208}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":450769384,"num_objects":55031,"num_object_clones":0,"num_object_copies":220124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30443,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55031,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":135806,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1194,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":901574656,"data_stored":901574656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149930,"ondisk_log_size":149930,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084382,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":730284,"kb_used_data":221012,"kb_used_omap":4,"kb_used_meta":509243,"kb_avail":93641556,"statfs":{"total":96636764160,"available":95888953344,"internally_reserved":0,"allocated":226316288,"data_stored":226115014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":521465555},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,9,0,4,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986259,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":618344,"kb_used_data":221528,"kb_used_omap":4,"kb_used_meta":396795,"kb_avail":93753496,"statfs":{"total":96636764160,"available":96003579904,"internally_reserved":0,"allocated":226844672,"data_stored":226643398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":406318802},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,9,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673065,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":716564,"kb_used_data":220292,"kb_used_omap":1,"kb_used_meta":496254,"kb_avail":93655276,"statfs":{"total":96636764160,"available":95903002624,"internally_reserved":0,"allocated":225579008,"data_stored":225377734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":508164549},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,3,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182496,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":717060,"kb_used_data":220336,"kb_used_omap":4,"kb_used_meta":496699,"kb_avail":93654780,"statfs":{"total":96636764160,"available":95902494720,"internally_reserved":0,"allocated":225624064,"data_stored":225422790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":508620493},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,9,3,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":225312768,"data_stored":225312768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":225931264,"data_stored":225931264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":225034240,"data_stored":225034240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":225296384,"data_stored":225296384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:41:51.926 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:41:52.533 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:41:52.535 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:41:52.555 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":285,"stamp":"2024-09-05T16:41:50.974605+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":450769384,"num_objects":55031,"num_object_clones":0,"num_object_copies":220124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30443,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55031,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":135806,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1194,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149930,"ondisk_log_size":149930,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2782252,"kb_used_data":883168,"kb_used_omap":14,"kb_used_meta":1898993,"kb_avail":374705108,"statfs":{"total":386547056640,"available":383698030592,"internally_reserved":0,"allocated":904364032,"data_stored":903558936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1944569399},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,1,10,14,17,9,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17465344,"num_objects":-2132,"num_object_clones":0,"num_object_copies":-8528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-114,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2132,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2132,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":114,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004812"},"pg_stats":[{"pgid":"1.f","version":"37'9418","reported_seq":9522,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:47.837770+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:41:47.837770+0000","last_peered":"2024-09-05T16:41:47.837770+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:41:47.837770+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9418,"log_dups_size":0,"ondisk_log_size":9418,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28557312,"num_objects":3486,"num_object_clones":0,"num_object_copies":13944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3486,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9418,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1250}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'9205","reported_seq":11103,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:47.974668+0000","last_change":"2024-09-05T16:40:28.919189+0000","last_active":"2024-09-05T16:41:47.974668+0000","last_peered":"2024-09-05T16:41:47.974668+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:41:47.974668+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9205,"log_dups_size":0,"ondisk_log_size":9205,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27432096,"num_objects":3349,"num_object_clones":0,"num_object_copies":13396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1356,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3349,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9203,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":915,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1993}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'9357","reported_seq":9478,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:47.990834+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:41:47.990834+0000","last_peered":"2024-09-05T16:41:47.990834+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:41:47.990834+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9357,"log_dups_size":0,"ondisk_log_size":9357,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27810408,"num_objects":3396,"num_object_clones":0,"num_object_copies":13584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3396,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9356,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1164}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'9393","reported_seq":14122,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:41:47.091318+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:41:47.091318+0000","last_peered":"2024-09-05T16:41:47.091318+0000","last_clean":"2024-09-05T16:41:47.091318+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:41:47.091318+0000","last_undegraded":"2024-09-05T16:41:47.091318+0000","last_fullsized":"2024-09-05T16:41:47.091318+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":9393,"log_dups_size":0,"ondisk_log_size":9393,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29007872,"num_objects":3541,"num_object_clones":0,"num_object_copies":14164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3541,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3693,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'9175","reported_seq":9273,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:47.748799+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:41:47.748799+0000","last_peered":"2024-09-05T16:41:47.748799+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:41:47.748799+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9175,"log_dups_size":0,"ondisk_log_size":9175,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28013728,"num_objects":3420,"num_object_clones":0,"num_object_copies":13680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3420,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9173,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1275}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'9352","reported_seq":9466,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:47.835607+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:41:47.835607+0000","last_peered":"2024-09-05T16:41:47.835607+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:41:47.835607+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9352,"log_dups_size":0,"ondisk_log_size":9352,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28016640,"num_objects":3420,"num_object_clones":0,"num_object_copies":13680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3420,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9352,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1201}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'9170","reported_seq":9277,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:47.815255+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:41:47.815255+0000","last_peered":"2024-09-05T16:41:47.815255+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:41:47.815255+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9170,"log_dups_size":0,"ondisk_log_size":9170,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28603552,"num_objects":3492,"num_object_clones":0,"num_object_copies":13968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3492,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9168,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1361}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'9355","reported_seq":9443,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:47.684466+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:41:47.684466+0000","last_peered":"2024-09-05T16:41:47.684466+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:41:47.684466+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9355,"log_dups_size":0,"ondisk_log_size":9355,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28046496,"num_objects":3424,"num_object_clones":0,"num_object_copies":13696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3424,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9353,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1192}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'9365","reported_seq":9465,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:47.837109+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:41:47.837109+0000","last_peered":"2024-09-05T16:41:47.837109+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:41:47.837109+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9365,"log_dups_size":0,"ondisk_log_size":9365,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27920704,"num_objects":3409,"num_object_clones":0,"num_object_copies":13636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3409,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9362,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1165}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'9431","reported_seq":9552,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:48.112923+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:41:48.112923+0000","last_peered":"2024-09-05T16:41:48.112923+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:41:48.112923+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9431,"log_dups_size":0,"ondisk_log_size":9431,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27677856,"num_objects":3379,"num_object_clones":0,"num_object_copies":13516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3379,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9429,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1134}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'9580","reported_seq":9706,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:47.736332+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:41:47.736332+0000","last_peered":"2024-09-05T16:41:47.736332+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:41:47.736332+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9580,"log_dups_size":0,"ondisk_log_size":9580,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29019888,"num_objects":3543,"num_object_clones":0,"num_object_copies":14172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3543,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9577,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1262}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'9335","reported_seq":9433,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:47.844891+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:41:47.844891+0000","last_peered":"2024-09-05T16:41:47.844891+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:41:47.844891+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9335,"log_dups_size":0,"ondisk_log_size":9335,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28047952,"num_objects":3424,"num_object_clones":0,"num_object_copies":13696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3424,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9334,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1186}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'9366","reported_seq":14101,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:41:47.142525+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:41:47.142525+0000","last_peered":"2024-09-05T16:41:47.142525+0000","last_clean":"2024-09-05T16:41:47.142525+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:41:47.142525+0000","last_undegraded":"2024-09-05T16:41:47.142525+0000","last_fullsized":"2024-09-05T16:41:47.142525+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":9366,"log_dups_size":0,"ondisk_log_size":9366,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28196864,"num_objects":3442,"num_object_clones":0,"num_object_copies":13768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3442,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":966,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'9638","reported_seq":9746,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:47.963770+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:41:47.963770+0000","last_peered":"2024-09-05T16:41:47.963770+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:41:47.963770+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9638,"log_dups_size":0,"ondisk_log_size":9638,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28570784,"num_objects":3488,"num_object_clones":0,"num_object_copies":13952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3488,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9636,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1154}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'9443","reported_seq":9555,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:48.162012+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:41:48.162012+0000","last_peered":"2024-09-05T16:41:48.162012+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:41:48.162012+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9443,"log_dups_size":0,"ondisk_log_size":9443,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27767968,"num_objects":3390,"num_object_clones":0,"num_object_copies":13560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3390,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9441,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1082}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'9347","reported_seq":9455,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:47.711982+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:41:47.711982+0000","last_peered":"2024-09-05T16:41:47.711982+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:41:47.711982+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9347,"log_dups_size":0,"ondisk_log_size":9347,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28079264,"num_objects":3428,"num_object_clones":0,"num_object_copies":13712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3428,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9345,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1208}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":450769384,"num_objects":55031,"num_object_clones":0,"num_object_copies":220124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30443,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55031,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":135806,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1194,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":901574656,"data_stored":901574656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149930,"ondisk_log_size":149930,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084382,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":730284,"kb_used_data":221012,"kb_used_omap":4,"kb_used_meta":509243,"kb_avail":93641556,"statfs":{"total":96636764160,"available":95888953344,"internally_reserved":0,"allocated":226316288,"data_stored":226115014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":521465555},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,9,0,4,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986259,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":618344,"kb_used_data":221528,"kb_used_omap":4,"kb_used_meta":396795,"kb_avail":93753496,"statfs":{"total":96636764160,"available":96003579904,"internally_reserved":0,"allocated":226844672,"data_stored":226643398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":406318802},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,9,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673065,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":716564,"kb_used_data":220292,"kb_used_omap":1,"kb_used_meta":496254,"kb_avail":93655276,"statfs":{"total":96636764160,"available":95903002624,"internally_reserved":0,"allocated":225579008,"data_stored":225377734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":508164549},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,3,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182496,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":717060,"kb_used_data":220336,"kb_used_omap":4,"kb_used_meta":496699,"kb_avail":93654780,"statfs":{"total":96636764160,"available":95902494720,"internally_reserved":0,"allocated":225624064,"data_stored":225422790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":508620493},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,9,3,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":225312768,"data_stored":225312768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":225931264,"data_stored":225931264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":225034240,"data_stored":225034240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":225296384,"data_stored":225296384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:41:52.557 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-05T16:41:52.557 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'9418", 'reported_seq': 9522, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:47.837770+0000', 'last_change': '2024-09-05T16:34:56.833497+0000', 'last_active': '2024-09-05T16:41:47.837770+0000', 'last_peered': '2024-09-05T16:41:47.837770+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.828289+0000', 'last_became_peered': '2024-09-05T16:34:56.828289+0000', 'last_unstale': '2024-09-05T16:41:47.837770+0000', 'last_undegraded': '2024-09-05T16:34:56.612160+0000', 'last_fullsized': '2024-09-05T16:34:56.557632+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9418, 'log_dups_size': 0, 'ondisk_log_size': 9418, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28557312, 'num_objects': 3486, 'num_object_clones': 0, 'num_object_copies': 13944, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2236, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3486, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9418, 'num_write_kb': 51616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1250}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:41:52.557 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-05T16:41:52.558 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "37'9205", 'reported_seq': 11103, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:47.974668+0000', 'last_change': '2024-09-05T16:40:28.919189+0000', 'last_active': '2024-09-05T16:41:47.974668+0000', 'last_peered': '2024-09-05T16:41:47.974668+0000', 'last_clean': '2024-09-05T16:33:53.200762+0000', 'last_became_active': '2024-09-05T16:34:56.820292+0000', 'last_became_peered': '2024-09-05T16:34:56.820292+0000', 'last_unstale': '2024-09-05T16:41:47.974668+0000', 'last_undegraded': '2024-09-05T16:34:56.605776+0000', 'last_fullsized': '2024-09-05T16:34:56.559383+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9205, 'log_dups_size': 0, 'ondisk_log_size': 9205, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27432096, 'num_objects': 3349, 'num_object_clones': 0, 'num_object_copies': 13396, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1356, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3349, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9203, 'num_write_kb': 50212, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 915, 'num_bytes_recovered': 2613248, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 1993}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:41:52.558 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:41:52.558 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'9357", 'reported_seq': 9478, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:47.990834+0000', 'last_change': '2024-09-05T16:34:56.867900+0000', 'last_active': '2024-09-05T16:41:47.990834+0000', 'last_peered': '2024-09-05T16:41:47.990834+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:41:47.990834+0000', 'last_undegraded': '2024-09-05T16:34:56.590972+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9357, 'log_dups_size': 0, 'ondisk_log_size': 9357, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27810408, 'num_objects': 3396, 'num_object_clones': 0, 'num_object_copies': 13584, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2233, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3396, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 9356, 'num_write_kb': 50999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1164}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:41:52.559 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:41:52.559 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'9175", 'reported_seq': 9273, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:47.748799+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:41:47.748799+0000', 'last_peered': '2024-09-05T16:41:47.748799+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:41:47.748799+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9175, 'log_dups_size': 0, 'ondisk_log_size': 9175, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28013728, 'num_objects': 3420, 'num_object_clones': 0, 'num_object_copies': 13680, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2147, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3420, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9173, 'num_write_kb': 50373, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1275}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:41:52.559 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:41:52.559 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'9352", 'reported_seq': 9466, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:47.835607+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:41:47.835607+0000', 'last_peered': '2024-09-05T16:41:47.835607+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:41:47.835607+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9352, 'log_dups_size': 0, 'ondisk_log_size': 9352, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28016640, 'num_objects': 3420, 'num_object_clones': 0, 'num_object_copies': 13680, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3420, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9352, 'num_write_kb': 51088, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1201}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:41:52.560 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:41:52.560 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'9170", 'reported_seq': 9277, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:47.815255+0000', 'last_change': '2024-09-05T16:34:56.903436+0000', 'last_active': '2024-09-05T16:41:47.815255+0000', 'last_peered': '2024-09-05T16:41:47.815255+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:41:47.815255+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9170, 'log_dups_size': 0, 'ondisk_log_size': 9170, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28603552, 'num_objects': 3492, 'num_object_clones': 0, 'num_object_copies': 13968, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2133, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3492, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9168, 'num_write_kb': 50636, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1361}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:41:52.560 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-05T16:41:52.560 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "37'9355", 'reported_seq': 9443, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:47.684466+0000', 'last_change': '2024-09-05T16:34:56.883319+0000', 'last_active': '2024-09-05T16:41:47.684466+0000', 'last_peered': '2024-09-05T16:41:47.684466+0000', 'last_clean': '2024-09-05T16:33:53.505305+0000', 'last_became_active': '2024-09-05T16:34:56.873389+0000', 'last_became_peered': '2024-09-05T16:34:56.873389+0000', 'last_unstale': '2024-09-05T16:41:47.684466+0000', 'last_undegraded': '2024-09-05T16:34:56.602567+0000', 'last_fullsized': '2024-09-05T16:34:56.559958+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9355, 'log_dups_size': 0, 'ondisk_log_size': 9355, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28046496, 'num_objects': 3424, 'num_object_clones': 0, 'num_object_copies': 13696, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2234, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3424, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9353, 'num_write_kb': 51108, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1192}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:41:52.561 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:41:52.561 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'9365", 'reported_seq': 9465, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:47.837109+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:41:47.837109+0000', 'last_peered': '2024-09-05T16:41:47.837109+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:41:47.837109+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9365, 'log_dups_size': 0, 'ondisk_log_size': 9365, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27920704, 'num_objects': 3409, 'num_object_clones': 0, 'num_object_copies': 13636, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3409, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9362, 'num_write_kb': 51081, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1165}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:41:52.561 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:41:52.561 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'9431", 'reported_seq': 9552, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:48.112923+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:41:48.112923+0000', 'last_peered': '2024-09-05T16:41:48.112923+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:41:48.112923+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9431, 'log_dups_size': 0, 'ondisk_log_size': 9431, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27677856, 'num_objects': 3379, 'num_object_clones': 0, 'num_object_copies': 13516, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3379, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9429, 'num_write_kb': 51230, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1134}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:41:52.561 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:41:52.562 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'9580", 'reported_seq': 9706, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:47.736332+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:41:47.736332+0000', 'last_peered': '2024-09-05T16:41:47.736332+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:41:47.736332+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9580, 'log_dups_size': 0, 'ondisk_log_size': 9580, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29019888, 'num_objects': 3543, 'num_object_clones': 0, 'num_object_copies': 14172, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2284, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3543, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9577, 'num_write_kb': 52476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1262}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:41:52.562 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:41:52.562 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'9335", 'reported_seq': 9433, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:47.844891+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:41:47.844891+0000', 'last_peered': '2024-09-05T16:41:47.844891+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:41:47.844891+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9335, 'log_dups_size': 0, 'ondisk_log_size': 9335, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28047952, 'num_objects': 3424, 'num_object_clones': 0, 'num_object_copies': 13696, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3424, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9334, 'num_write_kb': 51031, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1186}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:41:52.562 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:41:52.562 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'9638", 'reported_seq': 9746, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:47.963770+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:41:47.963770+0000', 'last_peered': '2024-09-05T16:41:47.963770+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:41:47.963770+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9638, 'log_dups_size': 0, 'ondisk_log_size': 9638, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28570784, 'num_objects': 3488, 'num_object_clones': 0, 'num_object_copies': 13952, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3488, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9636, 'num_write_kb': 52500, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1154}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:41:52.563 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:41:52.563 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'9443", 'reported_seq': 9555, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:48.162012+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:41:48.162012+0000', 'last_peered': '2024-09-05T16:41:48.162012+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:41:48.162012+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9443, 'log_dups_size': 0, 'ondisk_log_size': 9443, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27767968, 'num_objects': 3390, 'num_object_clones': 0, 'num_object_copies': 13560, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2310, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3390, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9441, 'num_write_kb': 51325, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1082}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:41:52.563 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:41:52.563 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'9347", 'reported_seq': 9455, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:47.711982+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:41:47.711982+0000', 'last_peered': '2024-09-05T16:41:47.711982+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:41:47.711982+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9347, 'log_dups_size': 0, 'ondisk_log_size': 9347, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28079264, 'num_objects': 3428, 'num_object_clones': 0, 'num_object_copies': 13712, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3428, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9345, 'num_write_kb': 51092, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1208}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:41:52.564 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:41:53.337 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:41:53.337 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 30443/219468 objects degraded (13.871%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":516,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":54867,"data_bytes":449425896,"bytes_used":2848882688,"bytes_avail":383698173952,"bytes_total":386547056640,"degraded_objects":30443,"degraded_total":219468,"degraded_ratio":0.13871270526910529,"recovering_objects_per_sec":9,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":114,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":191},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:41:53.360 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:41:53.360 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:41:53.979 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:41:53.980 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:41:53.999 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":286,"stamp":"2024-09-05T16:41:52.975221+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":449425896,"num_objects":54867,"num_object_clones":0,"num_object_copies":219468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30443,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54867,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":135970,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1194,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150094,"ondisk_log_size":150094,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2782112,"kb_used_data":878840,"kb_used_omap":14,"kb_used_meta":1903153,"kb_avail":374705248,"statfs":{"total":386547056640,"available":383698173952,"internally_reserved":0,"allocated":899932160,"data_stored":899127064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1948829239},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,1,9,12,13,11,11],"upper_bound":256},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18808832,"num_objects":-2296,"num_object_clones":0,"num_object_copies":-9184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-114,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2296,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2296,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":114,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004551"},"pg_stats":[{"pgid":"1.f","version":"37'9418","reported_seq":9522,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:47.837770+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:41:47.837770+0000","last_peered":"2024-09-05T16:41:47.837770+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:41:47.837770+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9418,"log_dups_size":0,"ondisk_log_size":9418,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28557312,"num_objects":3486,"num_object_clones":0,"num_object_copies":13944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3486,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9418,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1250}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'9205","reported_seq":11103,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:47.974668+0000","last_change":"2024-09-05T16:40:28.919189+0000","last_active":"2024-09-05T16:41:47.974668+0000","last_peered":"2024-09-05T16:41:47.974668+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:41:47.974668+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9205,"log_dups_size":0,"ondisk_log_size":9205,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27432096,"num_objects":3349,"num_object_clones":0,"num_object_copies":13396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1356,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3349,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9203,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":915,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1993}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'9357","reported_seq":9478,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:47.990834+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:41:47.990834+0000","last_peered":"2024-09-05T16:41:47.990834+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:41:47.990834+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9357,"log_dups_size":0,"ondisk_log_size":9357,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27810408,"num_objects":3396,"num_object_clones":0,"num_object_copies":13584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3396,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9356,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1164}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'9476","reported_seq":14205,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:41:52.320143+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:41:52.320143+0000","last_peered":"2024-09-05T16:41:52.320143+0000","last_clean":"2024-09-05T16:41:52.320143+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:41:52.320143+0000","last_undegraded":"2024-09-05T16:41:52.320143+0000","last_fullsized":"2024-09-05T16:41:52.320143+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":9476,"log_dups_size":0,"ondisk_log_size":9476,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28327936,"num_objects":3458,"num_object_clones":0,"num_object_copies":13832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3458,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3776,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'9175","reported_seq":9273,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:47.748799+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:41:47.748799+0000","last_peered":"2024-09-05T16:41:47.748799+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:41:47.748799+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9175,"log_dups_size":0,"ondisk_log_size":9175,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28013728,"num_objects":3420,"num_object_clones":0,"num_object_copies":13680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3420,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9173,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1275}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'9352","reported_seq":9466,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:47.835607+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:41:47.835607+0000","last_peered":"2024-09-05T16:41:47.835607+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:41:47.835607+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9352,"log_dups_size":0,"ondisk_log_size":9352,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28016640,"num_objects":3420,"num_object_clones":0,"num_object_copies":13680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3420,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9352,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1201}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'9170","reported_seq":9277,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:47.815255+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:41:47.815255+0000","last_peered":"2024-09-05T16:41:47.815255+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:41:47.815255+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9170,"log_dups_size":0,"ondisk_log_size":9170,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28603552,"num_objects":3492,"num_object_clones":0,"num_object_copies":13968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3492,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9168,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1361}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'9355","reported_seq":9443,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:47.684466+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:41:47.684466+0000","last_peered":"2024-09-05T16:41:47.684466+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:41:47.684466+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9355,"log_dups_size":0,"ondisk_log_size":9355,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28046496,"num_objects":3424,"num_object_clones":0,"num_object_copies":13696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3424,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9353,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1192}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'9365","reported_seq":9465,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:47.837109+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:41:47.837109+0000","last_peered":"2024-09-05T16:41:47.837109+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:41:47.837109+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9365,"log_dups_size":0,"ondisk_log_size":9365,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27920704,"num_objects":3409,"num_object_clones":0,"num_object_copies":13636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3409,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9362,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1165}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'9431","reported_seq":9552,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:48.112923+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:41:48.112923+0000","last_peered":"2024-09-05T16:41:48.112923+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:41:48.112923+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9431,"log_dups_size":0,"ondisk_log_size":9431,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27677856,"num_objects":3379,"num_object_clones":0,"num_object_copies":13516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3379,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9429,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1134}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'9580","reported_seq":9706,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:47.736332+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:41:47.736332+0000","last_peered":"2024-09-05T16:41:47.736332+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:41:47.736332+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9580,"log_dups_size":0,"ondisk_log_size":9580,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29019888,"num_objects":3543,"num_object_clones":0,"num_object_copies":14172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3543,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9577,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1262}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'9335","reported_seq":9433,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:47.844891+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:41:47.844891+0000","last_peered":"2024-09-05T16:41:47.844891+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:41:47.844891+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9335,"log_dups_size":0,"ondisk_log_size":9335,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28047952,"num_objects":3424,"num_object_clones":0,"num_object_copies":13696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3424,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9334,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1186}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'9447","reported_seq":14182,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:41:52.315158+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:41:52.315158+0000","last_peered":"2024-09-05T16:41:52.315158+0000","last_clean":"2024-09-05T16:41:52.315158+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:41:52.315158+0000","last_undegraded":"2024-09-05T16:41:52.315158+0000","last_fullsized":"2024-09-05T16:41:52.315158+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":9447,"log_dups_size":0,"ondisk_log_size":9447,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27533312,"num_objects":3361,"num_object_clones":0,"num_object_copies":13444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3361,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1047,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'9638","reported_seq":9746,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:47.963770+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:41:47.963770+0000","last_peered":"2024-09-05T16:41:47.963770+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:41:47.963770+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9638,"log_dups_size":0,"ondisk_log_size":9638,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28570784,"num_objects":3488,"num_object_clones":0,"num_object_copies":13952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3488,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9636,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1154}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'9443","reported_seq":9555,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:48.162012+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:41:48.162012+0000","last_peered":"2024-09-05T16:41:48.162012+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:41:48.162012+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9443,"log_dups_size":0,"ondisk_log_size":9443,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27767968,"num_objects":3390,"num_object_clones":0,"num_object_copies":13560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3390,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9441,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1082}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'9347","reported_seq":9455,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:47.711982+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:41:47.711982+0000","last_peered":"2024-09-05T16:41:47.711982+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:41:47.711982+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9347,"log_dups_size":0,"ondisk_log_size":9347,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28079264,"num_objects":3428,"num_object_clones":0,"num_object_copies":13712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3428,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9345,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1208}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":449425896,"num_objects":54867,"num_object_clones":0,"num_object_copies":219468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30443,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54867,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":135970,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1194,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":897130496,"data_stored":897130496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150094,"ondisk_log_size":150094,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084382,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":730284,"kb_used_data":221012,"kb_used_omap":4,"kb_used_meta":509243,"kb_avail":93641556,"statfs":{"total":96636764160,"available":95888953344,"internally_reserved":0,"allocated":226316288,"data_stored":226115014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":521465555},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,9,0,4,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986260,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":618204,"kb_used_data":217200,"kb_used_omap":4,"kb_used_meta":400955,"kb_avail":93753636,"statfs":{"total":96636764160,"available":96003723264,"internally_reserved":0,"allocated":222412800,"data_stored":222211526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":410578642},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,5,5,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673065,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":716564,"kb_used_data":220292,"kb_used_omap":1,"kb_used_meta":496254,"kb_avail":93655276,"statfs":{"total":96636764160,"available":95903002624,"internally_reserved":0,"allocated":225579008,"data_stored":225377734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":508164549},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,3,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182496,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":717060,"kb_used_data":220336,"kb_used_omap":4,"kb_used_meta":496699,"kb_avail":93654780,"statfs":{"total":96636764160,"available":95902494720,"internally_reserved":0,"allocated":225624064,"data_stored":225422790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":508620493},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,9,3,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":225312768,"data_stored":225312768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":221487104,"data_stored":221487104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":225034240,"data_stored":225034240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":225296384,"data_stored":225296384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:41:57.001 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:41:57.625 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:41:57.625 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:41:57.647 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":288,"stamp":"2024-09-05T16:41:56.976950+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":441848296,"num_objects":53942,"num_object_clones":0,"num_object_copies":215768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30394,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53942,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":136895,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1243,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":151019,"ondisk_log_size":151019,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2799032,"kb_used_data":865216,"kb_used_omap":14,"kb_used_meta":1933681,"kb_avail":374688328,"statfs":{"total":386547056640,"available":383680847872,"internally_reserved":0,"allocated":885981184,"data_stored":885176088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1980089911},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[8,0,6,2,1,20,10,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17702912,"num_objects":-2161,"num_object_clones":0,"num_object_copies":-8644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-105,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2161,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2161,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":105,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004640"},"pg_stats":[{"pgid":"1.f","version":"37'9477","reported_seq":9581,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:52.817979+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:41:52.817979+0000","last_peered":"2024-09-05T16:41:52.817979+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:41:52.817979+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9477,"log_dups_size":0,"ondisk_log_size":9477,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28073984,"num_objects":3427,"num_object_clones":0,"num_object_copies":13708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3427,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9477,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1191}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'9274","reported_seq":11272,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:52.937999+0000","last_change":"2024-09-05T16:40:28.919189+0000","last_active":"2024-09-05T16:41:52.937999+0000","last_peered":"2024-09-05T16:41:52.937999+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:41:52.937999+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9274,"log_dups_size":0,"ondisk_log_size":9274,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26866848,"num_objects":3280,"num_object_clones":0,"num_object_copies":13120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1307,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3280,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9272,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":964,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1973}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'9406","reported_seq":9528,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:53.200339+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:41:53.200339+0000","last_peered":"2024-09-05T16:41:53.200339+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:41:53.200339+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9406,"log_dups_size":0,"ondisk_log_size":9406,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27409000,"num_objects":3347,"num_object_clones":0,"num_object_copies":13388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3347,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9405,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1115}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'9476","reported_seq":14205,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:41:52.320143+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:41:52.320143+0000","last_peered":"2024-09-05T16:41:52.320143+0000","last_clean":"2024-09-05T16:41:52.320143+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:41:52.320143+0000","last_undegraded":"2024-09-05T16:41:52.320143+0000","last_fullsized":"2024-09-05T16:41:52.320143+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":9476,"log_dups_size":0,"ondisk_log_size":9476,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28327936,"num_objects":3458,"num_object_clones":0,"num_object_copies":13832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3458,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3776,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'9236","reported_seq":9334,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:52.968098+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:41:52.968098+0000","last_peered":"2024-09-05T16:41:52.968098+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:41:52.968098+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9236,"log_dups_size":0,"ondisk_log_size":9236,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27514016,"num_objects":3359,"num_object_clones":0,"num_object_copies":13436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3359,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9234,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1214}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'9398","reported_seq":9512,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:52.977937+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:41:52.977937+0000","last_peered":"2024-09-05T16:41:52.977937+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:41:52.977937+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9398,"log_dups_size":0,"ondisk_log_size":9398,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27639808,"num_objects":3374,"num_object_clones":0,"num_object_copies":13496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3374,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9398,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1155}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'9274","reported_seq":9381,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:52.892760+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:41:52.892760+0000","last_peered":"2024-09-05T16:41:52.892760+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:41:52.892760+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9274,"log_dups_size":0,"ondisk_log_size":9274,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27751584,"num_objects":3388,"num_object_clones":0,"num_object_copies":13552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3388,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9272,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1257}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'9425","reported_seq":9513,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:52.806380+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:41:52.806380+0000","last_peered":"2024-09-05T16:41:52.806380+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:41:52.806380+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9425,"log_dups_size":0,"ondisk_log_size":9425,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27473056,"num_objects":3354,"num_object_clones":0,"num_object_copies":13416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3354,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9423,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1122}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'9430","reported_seq":9530,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:53.023766+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:41:53.023766+0000","last_peered":"2024-09-05T16:41:53.023766+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:41:53.023766+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9430,"log_dups_size":0,"ondisk_log_size":9430,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27388224,"num_objects":3344,"num_object_clones":0,"num_object_copies":13376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3344,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9427,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1100}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'9484","reported_seq":9605,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:53.253872+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:41:53.253872+0000","last_peered":"2024-09-05T16:41:53.253872+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:41:53.253872+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9484,"log_dups_size":0,"ondisk_log_size":9484,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27243680,"num_objects":3326,"num_object_clones":0,"num_object_copies":13304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3326,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9482,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1081}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'9664","reported_seq":9790,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:52.817893+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:41:52.817893+0000","last_peered":"2024-09-05T16:41:52.817893+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:41:52.817893+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9664,"log_dups_size":0,"ondisk_log_size":9664,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28331760,"num_objects":3459,"num_object_clones":0,"num_object_copies":13836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3459,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9661,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1178}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'9410","reported_seq":9505,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:53.111502+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:41:53.111502+0000","last_peered":"2024-09-05T16:41:53.111502+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:41:53.111502+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9410,"log_dups_size":0,"ondisk_log_size":9410,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27433552,"num_objects":3349,"num_object_clones":0,"num_object_copies":13396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3349,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9409,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1111}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'9447","reported_seq":14182,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:41:52.315158+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:41:52.315158+0000","last_peered":"2024-09-05T16:41:52.315158+0000","last_clean":"2024-09-05T16:41:52.315158+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:41:52.315158+0000","last_undegraded":"2024-09-05T16:41:52.315158+0000","last_fullsized":"2024-09-05T16:41:52.315158+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":9447,"log_dups_size":0,"ondisk_log_size":9447,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27533312,"num_objects":3361,"num_object_clones":0,"num_object_copies":13444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3361,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1047,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'9706","reported_seq":9814,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:52.959881+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:41:52.959881+0000","last_peered":"2024-09-05T16:41:52.959881+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:41:52.959881+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9706,"log_dups_size":0,"ondisk_log_size":9706,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28013728,"num_objects":3420,"num_object_clones":0,"num_object_copies":13680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3420,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9704,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1086}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'9498","reported_seq":9610,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:53.138669+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:41:53.138669+0000","last_peered":"2024-09-05T16:41:53.138669+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:41:53.138669+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9498,"log_dups_size":0,"ondisk_log_size":9498,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27317408,"num_objects":3335,"num_object_clones":0,"num_object_copies":13340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3335,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9496,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1027}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'9414","reported_seq":9522,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:52.940095+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:41:52.940095+0000","last_peered":"2024-09-05T16:41:52.940095+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:41:52.940095+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9414,"log_dups_size":0,"ondisk_log_size":9414,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27530400,"num_objects":3361,"num_object_clones":0,"num_object_copies":13444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3361,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9412,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1141}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":441848296,"num_objects":53942,"num_object_clones":0,"num_object_copies":215768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30394,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53942,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":136895,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1243,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":883847168,"data_stored":883847168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":151019,"ondisk_log_size":151019,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084383,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":729436,"kb_used_data":216012,"kb_used_omap":4,"kb_used_meta":513403,"kb_avail":93642404,"statfs":{"total":96636764160,"available":95889821696,"internally_reserved":0,"allocated":221196288,"data_stored":220995014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":525725395},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[8,0,1,0,1,9,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986260,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":618204,"kb_used_data":217200,"kb_used_omap":4,"kb_used_meta":400955,"kb_avail":93753636,"statfs":{"total":96636764160,"available":96003723264,"internally_reserved":0,"allocated":222412800,"data_stored":222211526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":410578642},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,5,5,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673066,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":734456,"kb_used_data":215976,"kb_used_omap":1,"kb_used_meta":518462,"kb_avail":93637384,"statfs":{"total":96636764160,"available":95884681216,"internally_reserved":0,"allocated":221159424,"data_stored":220958150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":530905541},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,0,0,4,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182497,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":716936,"kb_used_data":216028,"kb_used_omap":4,"kb_used_meta":500859,"kb_avail":93654904,"statfs":{"total":96636764160,"available":95902621696,"internally_reserved":0,"allocated":221212672,"data_stored":221011398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":512880333},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":220897280,"data_stored":220897280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":221487104,"data_stored":221487104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":220622848,"data_stored":220622848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":220839936,"data_stored":220839936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:41:57.648 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:41:58.310 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:41:58.311 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:41:58.342 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":288,"stamp":"2024-09-05T16:41:56.976950+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":441848296,"num_objects":53942,"num_object_clones":0,"num_object_copies":215768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30394,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53942,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":136895,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1243,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":151019,"ondisk_log_size":151019,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2799032,"kb_used_data":865216,"kb_used_omap":14,"kb_used_meta":1933681,"kb_avail":374688328,"statfs":{"total":386547056640,"available":383680847872,"internally_reserved":0,"allocated":885981184,"data_stored":885176088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":1980089911},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[8,0,6,2,1,20,10,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17702912,"num_objects":-2161,"num_object_clones":0,"num_object_copies":-8644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-105,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2161,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2161,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":105,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004640"},"pg_stats":[{"pgid":"1.f","version":"37'9477","reported_seq":9581,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:52.817979+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:41:52.817979+0000","last_peered":"2024-09-05T16:41:52.817979+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:41:52.817979+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9477,"log_dups_size":0,"ondisk_log_size":9477,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28073984,"num_objects":3427,"num_object_clones":0,"num_object_copies":13708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3427,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9477,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1191}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'9274","reported_seq":11272,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:52.937999+0000","last_change":"2024-09-05T16:40:28.919189+0000","last_active":"2024-09-05T16:41:52.937999+0000","last_peered":"2024-09-05T16:41:52.937999+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:41:52.937999+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9274,"log_dups_size":0,"ondisk_log_size":9274,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26866848,"num_objects":3280,"num_object_clones":0,"num_object_copies":13120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1307,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3280,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9272,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":964,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1973}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'9406","reported_seq":9528,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:53.200339+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:41:53.200339+0000","last_peered":"2024-09-05T16:41:53.200339+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:41:53.200339+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9406,"log_dups_size":0,"ondisk_log_size":9406,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27409000,"num_objects":3347,"num_object_clones":0,"num_object_copies":13388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3347,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9405,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1115}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'9476","reported_seq":14205,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:41:52.320143+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:41:52.320143+0000","last_peered":"2024-09-05T16:41:52.320143+0000","last_clean":"2024-09-05T16:41:52.320143+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:41:52.320143+0000","last_undegraded":"2024-09-05T16:41:52.320143+0000","last_fullsized":"2024-09-05T16:41:52.320143+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":9476,"log_dups_size":0,"ondisk_log_size":9476,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28327936,"num_objects":3458,"num_object_clones":0,"num_object_copies":13832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3458,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3776,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'9236","reported_seq":9334,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:52.968098+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:41:52.968098+0000","last_peered":"2024-09-05T16:41:52.968098+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:41:52.968098+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9236,"log_dups_size":0,"ondisk_log_size":9236,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27514016,"num_objects":3359,"num_object_clones":0,"num_object_copies":13436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3359,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9234,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1214}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'9398","reported_seq":9512,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:52.977937+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:41:52.977937+0000","last_peered":"2024-09-05T16:41:52.977937+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:41:52.977937+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9398,"log_dups_size":0,"ondisk_log_size":9398,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27639808,"num_objects":3374,"num_object_clones":0,"num_object_copies":13496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3374,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9398,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1155}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'9274","reported_seq":9381,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:52.892760+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:41:52.892760+0000","last_peered":"2024-09-05T16:41:52.892760+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:41:52.892760+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9274,"log_dups_size":0,"ondisk_log_size":9274,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27751584,"num_objects":3388,"num_object_clones":0,"num_object_copies":13552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3388,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9272,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1257}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'9425","reported_seq":9513,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:52.806380+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:41:52.806380+0000","last_peered":"2024-09-05T16:41:52.806380+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:41:52.806380+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9425,"log_dups_size":0,"ondisk_log_size":9425,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27473056,"num_objects":3354,"num_object_clones":0,"num_object_copies":13416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3354,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9423,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1122}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'9430","reported_seq":9530,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:53.023766+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:41:53.023766+0000","last_peered":"2024-09-05T16:41:53.023766+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:41:53.023766+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9430,"log_dups_size":0,"ondisk_log_size":9430,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27388224,"num_objects":3344,"num_object_clones":0,"num_object_copies":13376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3344,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9427,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1100}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'9484","reported_seq":9605,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:53.253872+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:41:53.253872+0000","last_peered":"2024-09-05T16:41:53.253872+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:41:53.253872+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9484,"log_dups_size":0,"ondisk_log_size":9484,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27243680,"num_objects":3326,"num_object_clones":0,"num_object_copies":13304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3326,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9482,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1081}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'9664","reported_seq":9790,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:52.817893+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:41:52.817893+0000","last_peered":"2024-09-05T16:41:52.817893+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:41:52.817893+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9664,"log_dups_size":0,"ondisk_log_size":9664,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28331760,"num_objects":3459,"num_object_clones":0,"num_object_copies":13836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3459,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9661,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1178}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'9410","reported_seq":9505,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:53.111502+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:41:53.111502+0000","last_peered":"2024-09-05T16:41:53.111502+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:41:53.111502+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9410,"log_dups_size":0,"ondisk_log_size":9410,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27433552,"num_objects":3349,"num_object_clones":0,"num_object_copies":13396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3349,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9409,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1111}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'9447","reported_seq":14182,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:41:52.315158+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:41:52.315158+0000","last_peered":"2024-09-05T16:41:52.315158+0000","last_clean":"2024-09-05T16:41:52.315158+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:41:52.315158+0000","last_undegraded":"2024-09-05T16:41:52.315158+0000","last_fullsized":"2024-09-05T16:41:52.315158+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":9447,"log_dups_size":0,"ondisk_log_size":9447,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27533312,"num_objects":3361,"num_object_clones":0,"num_object_copies":13444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3361,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1047,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'9706","reported_seq":9814,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:52.959881+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:41:52.959881+0000","last_peered":"2024-09-05T16:41:52.959881+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:41:52.959881+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9706,"log_dups_size":0,"ondisk_log_size":9706,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28013728,"num_objects":3420,"num_object_clones":0,"num_object_copies":13680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3420,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9704,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1086}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'9498","reported_seq":9610,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:53.138669+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:41:53.138669+0000","last_peered":"2024-09-05T16:41:53.138669+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:41:53.138669+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9498,"log_dups_size":0,"ondisk_log_size":9498,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27317408,"num_objects":3335,"num_object_clones":0,"num_object_copies":13340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3335,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9496,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1027}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'9414","reported_seq":9522,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:52.940095+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:41:52.940095+0000","last_peered":"2024-09-05T16:41:52.940095+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:41:52.940095+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9414,"log_dups_size":0,"ondisk_log_size":9414,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27530400,"num_objects":3361,"num_object_clones":0,"num_object_copies":13444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3361,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9412,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1141}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":441848296,"num_objects":53942,"num_object_clones":0,"num_object_copies":215768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30394,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53942,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":136895,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1243,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":883847168,"data_stored":883847168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":151019,"ondisk_log_size":151019,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084383,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":729436,"kb_used_data":216012,"kb_used_omap":4,"kb_used_meta":513403,"kb_avail":93642404,"statfs":{"total":96636764160,"available":95889821696,"internally_reserved":0,"allocated":221196288,"data_stored":220995014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":525725395},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[8,0,1,0,1,9,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986260,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":618204,"kb_used_data":217200,"kb_used_omap":4,"kb_used_meta":400955,"kb_avail":93753636,"statfs":{"total":96636764160,"available":96003723264,"internally_reserved":0,"allocated":222412800,"data_stored":222211526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":410578642},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,5,5,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673066,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":734456,"kb_used_data":215976,"kb_used_omap":1,"kb_used_meta":518462,"kb_avail":93637384,"statfs":{"total":96636764160,"available":95884681216,"internally_reserved":0,"allocated":221159424,"data_stored":220958150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":530905541},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,0,0,4,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182497,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":716936,"kb_used_data":216028,"kb_used_omap":4,"kb_used_meta":500859,"kb_avail":93654904,"statfs":{"total":96636764160,"available":95902621696,"internally_reserved":0,"allocated":221212672,"data_stored":221011398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":512880333},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":220897280,"data_stored":220897280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":221487104,"data_stored":221487104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":220622848,"data_stored":220622848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":220839936,"data_stored":220839936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:41:58.343 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-05T16:41:58.344 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'9477", 'reported_seq': 9581, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:52.817979+0000', 'last_change': '2024-09-05T16:34:56.833497+0000', 'last_active': '2024-09-05T16:41:52.817979+0000', 'last_peered': '2024-09-05T16:41:52.817979+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.828289+0000', 'last_became_peered': '2024-09-05T16:34:56.828289+0000', 'last_unstale': '2024-09-05T16:41:52.817979+0000', 'last_undegraded': '2024-09-05T16:34:56.612160+0000', 'last_fullsized': '2024-09-05T16:34:56.557632+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9477, 'log_dups_size': 0, 'ondisk_log_size': 9477, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28073984, 'num_objects': 3427, 'num_object_clones': 0, 'num_object_copies': 13708, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2236, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3427, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9477, 'num_write_kb': 51616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1191}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:41:58.344 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-05T16:41:58.344 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "37'9274", 'reported_seq': 11272, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:52.937999+0000', 'last_change': '2024-09-05T16:40:28.919189+0000', 'last_active': '2024-09-05T16:41:52.937999+0000', 'last_peered': '2024-09-05T16:41:52.937999+0000', 'last_clean': '2024-09-05T16:33:53.200762+0000', 'last_became_active': '2024-09-05T16:34:56.820292+0000', 'last_became_peered': '2024-09-05T16:34:56.820292+0000', 'last_unstale': '2024-09-05T16:41:52.937999+0000', 'last_undegraded': '2024-09-05T16:34:56.605776+0000', 'last_fullsized': '2024-09-05T16:34:56.559383+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9274, 'log_dups_size': 0, 'ondisk_log_size': 9274, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26866848, 'num_objects': 3280, 'num_object_clones': 0, 'num_object_copies': 13120, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1307, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3280, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9272, 'num_write_kb': 50212, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 964, 'num_bytes_recovered': 2613248, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 1973}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:41:58.344 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:41:58.344 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'9406", 'reported_seq': 9528, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:53.200339+0000', 'last_change': '2024-09-05T16:34:56.867900+0000', 'last_active': '2024-09-05T16:41:53.200339+0000', 'last_peered': '2024-09-05T16:41:53.200339+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:41:53.200339+0000', 'last_undegraded': '2024-09-05T16:34:56.590972+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9406, 'log_dups_size': 0, 'ondisk_log_size': 9406, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27409000, 'num_objects': 3347, 'num_object_clones': 0, 'num_object_copies': 13388, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2233, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3347, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 9405, 'num_write_kb': 50999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1115}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:41:58.344 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:41:58.344 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'9236", 'reported_seq': 9334, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:52.968098+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:41:52.968098+0000', 'last_peered': '2024-09-05T16:41:52.968098+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:41:52.968098+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9236, 'log_dups_size': 0, 'ondisk_log_size': 9236, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27514016, 'num_objects': 3359, 'num_object_clones': 0, 'num_object_copies': 13436, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2147, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3359, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9234, 'num_write_kb': 50373, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1214}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:41:58.345 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:41:58.345 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'9398", 'reported_seq': 9512, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:52.977937+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:41:52.977937+0000', 'last_peered': '2024-09-05T16:41:52.977937+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:41:52.977937+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9398, 'log_dups_size': 0, 'ondisk_log_size': 9398, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27639808, 'num_objects': 3374, 'num_object_clones': 0, 'num_object_copies': 13496, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3374, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9398, 'num_write_kb': 51088, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1155}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:41:58.345 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:41:58.345 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'9274", 'reported_seq': 9381, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:52.892760+0000', 'last_change': '2024-09-05T16:34:56.903436+0000', 'last_active': '2024-09-05T16:41:52.892760+0000', 'last_peered': '2024-09-05T16:41:52.892760+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:41:52.892760+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9274, 'log_dups_size': 0, 'ondisk_log_size': 9274, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27751584, 'num_objects': 3388, 'num_object_clones': 0, 'num_object_copies': 13552, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2133, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3388, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9272, 'num_write_kb': 50636, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1257}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:41:58.345 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-05T16:41:58.345 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "37'9425", 'reported_seq': 9513, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:52.806380+0000', 'last_change': '2024-09-05T16:34:56.883319+0000', 'last_active': '2024-09-05T16:41:52.806380+0000', 'last_peered': '2024-09-05T16:41:52.806380+0000', 'last_clean': '2024-09-05T16:33:53.505305+0000', 'last_became_active': '2024-09-05T16:34:56.873389+0000', 'last_became_peered': '2024-09-05T16:34:56.873389+0000', 'last_unstale': '2024-09-05T16:41:52.806380+0000', 'last_undegraded': '2024-09-05T16:34:56.602567+0000', 'last_fullsized': '2024-09-05T16:34:56.559958+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9425, 'log_dups_size': 0, 'ondisk_log_size': 9425, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27473056, 'num_objects': 3354, 'num_object_clones': 0, 'num_object_copies': 13416, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2234, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3354, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9423, 'num_write_kb': 51108, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1122}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:41:58.345 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:41:58.345 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'9430", 'reported_seq': 9530, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:53.023766+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:41:53.023766+0000', 'last_peered': '2024-09-05T16:41:53.023766+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:41:53.023766+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9430, 'log_dups_size': 0, 'ondisk_log_size': 9430, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27388224, 'num_objects': 3344, 'num_object_clones': 0, 'num_object_copies': 13376, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3344, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9427, 'num_write_kb': 51081, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1100}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:41:58.345 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:41:58.345 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'9484", 'reported_seq': 9605, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:53.253872+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:41:53.253872+0000', 'last_peered': '2024-09-05T16:41:53.253872+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:41:53.253872+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9484, 'log_dups_size': 0, 'ondisk_log_size': 9484, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27243680, 'num_objects': 3326, 'num_object_clones': 0, 'num_object_copies': 13304, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3326, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9482, 'num_write_kb': 51230, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1081}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:41:58.345 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:41:58.346 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'9664", 'reported_seq': 9790, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:52.817893+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:41:52.817893+0000', 'last_peered': '2024-09-05T16:41:52.817893+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:41:52.817893+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9664, 'log_dups_size': 0, 'ondisk_log_size': 9664, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28331760, 'num_objects': 3459, 'num_object_clones': 0, 'num_object_copies': 13836, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2284, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3459, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9661, 'num_write_kb': 52476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1178}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:41:58.346 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:41:58.346 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'9410", 'reported_seq': 9505, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:53.111502+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:41:53.111502+0000', 'last_peered': '2024-09-05T16:41:53.111502+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:41:53.111502+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9410, 'log_dups_size': 0, 'ondisk_log_size': 9410, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27433552, 'num_objects': 3349, 'num_object_clones': 0, 'num_object_copies': 13396, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3349, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9409, 'num_write_kb': 51031, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1111}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:41:58.346 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:41:58.346 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'9706", 'reported_seq': 9814, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:52.959881+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:41:52.959881+0000', 'last_peered': '2024-09-05T16:41:52.959881+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:41:52.959881+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9706, 'log_dups_size': 0, 'ondisk_log_size': 9706, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28013728, 'num_objects': 3420, 'num_object_clones': 0, 'num_object_copies': 13680, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3420, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9704, 'num_write_kb': 52500, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1086}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:41:58.346 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:41:58.346 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'9498", 'reported_seq': 9610, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:53.138669+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:41:53.138669+0000', 'last_peered': '2024-09-05T16:41:53.138669+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:41:53.138669+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9498, 'log_dups_size': 0, 'ondisk_log_size': 9498, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27317408, 'num_objects': 3335, 'num_object_clones': 0, 'num_object_copies': 13340, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2310, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3335, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9496, 'num_write_kb': 51325, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1027}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:41:58.346 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:41:58.346 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'9414", 'reported_seq': 9522, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:52.940095+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:41:52.940095+0000', 'last_peered': '2024-09-05T16:41:52.940095+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:41:52.940095+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9414, 'log_dups_size': 0, 'ondisk_log_size': 9414, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27530400, 'num_objects': 3361, 'num_object_clones': 0, 'num_object_copies': 13444, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3361, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9412, 'num_write_kb': 51092, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1141}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:41:58.347 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:41:59.073 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:41:59.074 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 30394/215768 objects degraded (14.086%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":522,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":53942,"data_bytes":441848296,"bytes_used":2866208768,"bytes_avail":383680847872,"bytes_total":386547056640,"degraded_objects":30394,"degraded_total":215768,"degraded_ratio":0.1408642616143265,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":105,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":180},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:41:59.103 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:41:59.103 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:41:59.758 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:41:59.759 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:41:59.782 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":289,"stamp":"2024-09-05T16:41:58.977779+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":433009128,"num_objects":52863,"num_object_clones":0,"num_object_copies":211452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30343,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52863,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":137974,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1294,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":152098,"ondisk_log_size":152098,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2819740,"kb_used_data":848440,"kb_used_omap":14,"kb_used_meta":1971185,"kb_avail":374667620,"statfs":{"total":386547056640,"available":383659642880,"internally_reserved":0,"allocated":868802560,"data_stored":867997464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2018494007},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,7,5,7,12,8,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-26542080,"num_objects":-3240,"num_object_clones":0,"num_object_copies":-12960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-156,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3240,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3240,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":156,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004725"},"pg_stats":[{"pgid":"1.f","version":"37'9536","reported_seq":9640,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:58.066444+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:41:58.066444+0000","last_peered":"2024-09-05T16:41:58.066444+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:41:58.066444+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9536,"log_dups_size":0,"ondisk_log_size":9536,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27590656,"num_objects":3368,"num_object_clones":0,"num_object_copies":13472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3368,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9536,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1132}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'9346","reported_seq":11447,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:58.015542+0000","last_change":"2024-09-05T16:40:28.919189+0000","last_active":"2024-09-05T16:41:58.015542+0000","last_peered":"2024-09-05T16:41:58.015542+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:41:58.015542+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9346,"log_dups_size":0,"ondisk_log_size":9346,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26277024,"num_objects":3208,"num_object_clones":0,"num_object_copies":12832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1256,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3208,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9344,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1015,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1952}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'9467","reported_seq":9589,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:58.427005+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:41:58.427005+0000","last_peered":"2024-09-05T16:41:58.427005+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:41:58.427005+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9467,"log_dups_size":0,"ondisk_log_size":9467,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26909288,"num_objects":3286,"num_object_clones":0,"num_object_copies":13144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3286,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9466,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1054}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'9552","reported_seq":14281,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:41:57.436529+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:41:57.436529+0000","last_peered":"2024-09-05T16:41:57.436529+0000","last_clean":"2024-09-05T16:41:57.436529+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:41:57.436529+0000","last_undegraded":"2024-09-05T16:41:57.436529+0000","last_fullsized":"2024-09-05T16:41:57.436529+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":9552,"log_dups_size":0,"ondisk_log_size":9552,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27705344,"num_objects":3382,"num_object_clones":0,"num_object_copies":13528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3382,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3852,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'9301","reported_seq":9399,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:57.998091+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:41:57.998091+0000","last_peered":"2024-09-05T16:41:57.998091+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:41:57.998091+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9301,"log_dups_size":0,"ondisk_log_size":9301,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26981536,"num_objects":3294,"num_object_clones":0,"num_object_copies":13176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3294,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9299,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1149}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'9465","reported_seq":9579,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:58.145013+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:41:58.145013+0000","last_peered":"2024-09-05T16:41:58.145013+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:41:58.145013+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9465,"log_dups_size":0,"ondisk_log_size":9465,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27090944,"num_objects":3307,"num_object_clones":0,"num_object_copies":13228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3307,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9465,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1088}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'9329","reported_seq":9436,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:58.045018+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:41:58.045018+0000","last_peered":"2024-09-05T16:41:58.045018+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:41:58.045018+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9329,"log_dups_size":0,"ondisk_log_size":9329,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27301024,"num_objects":3333,"num_object_clones":0,"num_object_copies":13332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3333,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9327,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1202}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'9487","reported_seq":9575,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:57.910799+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:41:57.910799+0000","last_peered":"2024-09-05T16:41:57.910799+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:41:57.910799+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9487,"log_dups_size":0,"ondisk_log_size":9487,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26965152,"num_objects":3292,"num_object_clones":0,"num_object_copies":13168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9485,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1060}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'9496","reported_seq":9596,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:58.139294+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:41:58.139294+0000","last_peered":"2024-09-05T16:41:58.139294+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:41:58.139294+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9496,"log_dups_size":0,"ondisk_log_size":9496,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26847552,"num_objects":3278,"num_object_clones":0,"num_object_copies":13112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3278,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9493,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1034}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'9551","reported_seq":9672,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:58.261433+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:41:58.261433+0000","last_peered":"2024-09-05T16:41:58.261433+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:41:58.261433+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9551,"log_dups_size":0,"ondisk_log_size":9551,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26694816,"num_objects":3259,"num_object_clones":0,"num_object_copies":13036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3259,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9549,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1014}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'9742","reported_seq":9868,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:58.001018+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:41:58.001018+0000","last_peered":"2024-09-05T16:41:58.001018+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:41:58.001018+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9742,"log_dups_size":0,"ondisk_log_size":9742,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27692784,"num_objects":3381,"num_object_clones":0,"num_object_copies":13524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3381,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9739,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1100}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'9483","reported_seq":9581,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:57.713112+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:41:57.713112+0000","last_peered":"2024-09-05T16:41:57.713112+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:41:57.713112+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9483,"log_dups_size":0,"ondisk_log_size":9483,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835536,"num_objects":3276,"num_object_clones":0,"num_object_copies":13104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3276,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9482,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1038}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'9512","reported_seq":14247,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:41:57.301121+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:41:57.301121+0000","last_peered":"2024-09-05T16:41:57.301121+0000","last_clean":"2024-09-05T16:41:57.301121+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:41:57.301121+0000","last_undegraded":"2024-09-05T16:41:57.301121+0000","last_fullsized":"2024-09-05T16:41:57.301121+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":9512,"log_dups_size":0,"ondisk_log_size":9512,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27000832,"num_objects":3296,"num_object_clones":0,"num_object_copies":13184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3296,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1112,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'9777","reported_seq":9885,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:58.106964+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:41:58.106964+0000","last_peered":"2024-09-05T16:41:58.106964+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:41:58.106964+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9777,"log_dups_size":0,"ondisk_log_size":9777,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27432096,"num_objects":3349,"num_object_clones":0,"num_object_copies":13396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3349,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9775,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1015}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'9585","reported_seq":9697,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:58.353420+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:41:58.353420+0000","last_peered":"2024-09-05T16:41:58.353420+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:41:58.353420+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9585,"log_dups_size":0,"ondisk_log_size":9585,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26604704,"num_objects":3248,"num_object_clones":0,"num_object_copies":12992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3248,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9583,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":940}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'9469","reported_seq":9577,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:57.819845+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:41:57.819845+0000","last_peered":"2024-09-05T16:41:57.819845+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:41:57.819845+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9469,"log_dups_size":0,"ondisk_log_size":9469,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27079840,"num_objects":3306,"num_object_clones":0,"num_object_copies":13224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3306,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9467,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1086}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":433009128,"num_objects":52863,"num_object_clones":0,"num_object_copies":211452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30343,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52863,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":137974,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1294,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":866189312,"data_stored":866189312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":152098,"ondisk_log_size":152098,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084384,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":730764,"kb_used_data":211756,"kb_used_omap":4,"kb_used_meta":518971,"kb_avail":93641076,"statfs":{"total":96636764160,"available":95888461824,"internally_reserved":0,"allocated":216838144,"data_stored":216636870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":531427027},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,2,4,6,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986261,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":619620,"kb_used_data":213076,"kb_used_omap":4,"kb_used_meta":406523,"kb_avail":93752220,"statfs":{"total":96636764160,"available":96002273280,"internally_reserved":0,"allocated":218189824,"data_stored":217988550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":416280274},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673067,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":734424,"kb_used_data":211784,"kb_used_omap":1,"kb_used_meta":522622,"kb_avail":93637416,"statfs":{"total":96636764160,"available":95884713984,"internally_reserved":0,"allocated":216866816,"data_stored":216665542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":535165381},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,0,2,1,5,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182498,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":734932,"kb_used_data":211824,"kb_used_omap":4,"kb_used_meta":523067,"kb_avail":93636908,"statfs":{"total":96636764160,"available":95884193792,"internally_reserved":0,"allocated":216907776,"data_stored":216706502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":535621325},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,0,0,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":216535040,"data_stored":216535040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":217071616,"data_stored":217071616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":216203264,"data_stored":216203264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":216379392,"data_stored":216379392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:42:02.784 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:42:03.372 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:42:03.372 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:42:03.393 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":291,"stamp":"2024-09-05T16:42:02.981117+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":431886824,"num_objects":52726,"num_object_clones":0,"num_object_copies":210904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30343,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52726,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":138111,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1294,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":152235,"ondisk_log_size":152235,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2818896,"kb_used_data":843436,"kb_used_omap":14,"kb_used_meta":1975345,"kb_avail":374668464,"statfs":{"total":386547056640,"available":383660507136,"internally_reserved":0,"allocated":863678464,"data_stored":862873368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2022753847},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,6,6,5,12,16,8,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18882560,"num_objects":-2305,"num_object_clones":0,"num_object_copies":-9220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-100,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2305,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2305,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":100,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006512"},"pg_stats":[{"pgid":"1.f","version":"37'9536","reported_seq":9640,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:58.066444+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:41:58.066444+0000","last_peered":"2024-09-05T16:41:58.066444+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:41:58.066444+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9536,"log_dups_size":0,"ondisk_log_size":9536,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27590656,"num_objects":3368,"num_object_clones":0,"num_object_copies":13472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3368,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9536,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1132}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'9346","reported_seq":11447,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:58.015542+0000","last_change":"2024-09-05T16:40:28.919189+0000","last_active":"2024-09-05T16:41:58.015542+0000","last_peered":"2024-09-05T16:41:58.015542+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:41:58.015542+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9346,"log_dups_size":0,"ondisk_log_size":9346,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26277024,"num_objects":3208,"num_object_clones":0,"num_object_copies":12832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1256,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3208,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9344,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1015,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1952}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'9467","reported_seq":9589,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:58.427005+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:41:58.427005+0000","last_peered":"2024-09-05T16:41:58.427005+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:41:58.427005+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9467,"log_dups_size":0,"ondisk_log_size":9467,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26909288,"num_objects":3286,"num_object_clones":0,"num_object_copies":13144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3286,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9466,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1054}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'9637","reported_seq":14366,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:42:02.676545+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:42:02.676545+0000","last_peered":"2024-09-05T16:42:02.676545+0000","last_clean":"2024-09-05T16:42:02.676545+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:42:02.676545+0000","last_undegraded":"2024-09-05T16:42:02.676545+0000","last_fullsized":"2024-09-05T16:42:02.676545+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":9637,"log_dups_size":0,"ondisk_log_size":9637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27009024,"num_objects":3297,"num_object_clones":0,"num_object_copies":13188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3297,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3937,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'9301","reported_seq":9399,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:57.998091+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:41:57.998091+0000","last_peered":"2024-09-05T16:41:57.998091+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:41:57.998091+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9301,"log_dups_size":0,"ondisk_log_size":9301,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26981536,"num_objects":3294,"num_object_clones":0,"num_object_copies":13176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3294,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9299,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1149}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'9465","reported_seq":9579,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:58.145013+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:41:58.145013+0000","last_peered":"2024-09-05T16:41:58.145013+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:41:58.145013+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9465,"log_dups_size":0,"ondisk_log_size":9465,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27090944,"num_objects":3307,"num_object_clones":0,"num_object_copies":13228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3307,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9465,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1088}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'9329","reported_seq":9436,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:58.045018+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:41:58.045018+0000","last_peered":"2024-09-05T16:41:58.045018+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:41:58.045018+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9329,"log_dups_size":0,"ondisk_log_size":9329,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27301024,"num_objects":3333,"num_object_clones":0,"num_object_copies":13332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3333,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9327,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1202}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'9487","reported_seq":9575,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:57.910799+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:41:57.910799+0000","last_peered":"2024-09-05T16:41:57.910799+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:41:57.910799+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9487,"log_dups_size":0,"ondisk_log_size":9487,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26965152,"num_objects":3292,"num_object_clones":0,"num_object_copies":13168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9485,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1060}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'9496","reported_seq":9596,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:58.139294+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:41:58.139294+0000","last_peered":"2024-09-05T16:41:58.139294+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:41:58.139294+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9496,"log_dups_size":0,"ondisk_log_size":9496,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26847552,"num_objects":3278,"num_object_clones":0,"num_object_copies":13112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3278,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9493,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1034}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'9551","reported_seq":9672,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:58.261433+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:41:58.261433+0000","last_peered":"2024-09-05T16:41:58.261433+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:41:58.261433+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9551,"log_dups_size":0,"ondisk_log_size":9551,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26694816,"num_objects":3259,"num_object_clones":0,"num_object_copies":13036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3259,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9549,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1014}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'9742","reported_seq":9868,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:58.001018+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:41:58.001018+0000","last_peered":"2024-09-05T16:41:58.001018+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:41:58.001018+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9742,"log_dups_size":0,"ondisk_log_size":9742,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27692784,"num_objects":3381,"num_object_clones":0,"num_object_copies":13524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3381,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9739,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1100}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'9483","reported_seq":9581,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:57.713112+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:41:57.713112+0000","last_peered":"2024-09-05T16:41:57.713112+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:41:57.713112+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9483,"log_dups_size":0,"ondisk_log_size":9483,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835536,"num_objects":3276,"num_object_clones":0,"num_object_copies":13104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3276,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9482,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1038}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'9564","reported_seq":14299,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:42:02.561035+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:42:02.561035+0000","last_peered":"2024-09-05T16:42:02.561035+0000","last_clean":"2024-09-05T16:42:02.561035+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:42:02.561035+0000","last_undegraded":"2024-09-05T16:42:02.561035+0000","last_fullsized":"2024-09-05T16:42:02.561035+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":9564,"log_dups_size":0,"ondisk_log_size":9564,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26574848,"num_objects":3244,"num_object_clones":0,"num_object_copies":12976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3244,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1164,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'9777","reported_seq":9885,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:58.106964+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:41:58.106964+0000","last_peered":"2024-09-05T16:41:58.106964+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:41:58.106964+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9777,"log_dups_size":0,"ondisk_log_size":9777,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27432096,"num_objects":3349,"num_object_clones":0,"num_object_copies":13396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3349,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9775,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1015}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'9585","reported_seq":9697,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:58.353420+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:41:58.353420+0000","last_peered":"2024-09-05T16:41:58.353420+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:41:58.353420+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9585,"log_dups_size":0,"ondisk_log_size":9585,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26604704,"num_objects":3248,"num_object_clones":0,"num_object_copies":12992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3248,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9583,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":940}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'9469","reported_seq":9577,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:57.819845+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:41:57.819845+0000","last_peered":"2024-09-05T16:41:57.819845+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:41:57.819845+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9469,"log_dups_size":0,"ondisk_log_size":9469,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27079840,"num_objects":3306,"num_object_clones":0,"num_object_copies":13224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3306,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9467,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1086}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":431886824,"num_objects":52726,"num_object_clones":0,"num_object_copies":210904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30343,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52726,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":138111,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1294,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":861802496,"data_stored":861802496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":152235,"ondisk_log_size":152235,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084384,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":730764,"kb_used_data":211756,"kb_used_omap":4,"kb_used_meta":518971,"kb_avail":93641076,"statfs":{"total":96636764160,"available":95888461824,"internally_reserved":0,"allocated":216838144,"data_stored":216636870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":531427027},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,2,4,6,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986262,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":618776,"kb_used_data":208072,"kb_used_omap":4,"kb_used_meta":410683,"kb_avail":93753064,"statfs":{"total":96636764160,"available":96003137536,"internally_reserved":0,"allocated":213065728,"data_stored":212864454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":420540114},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,3,6,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673067,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":734424,"kb_used_data":211784,"kb_used_omap":1,"kb_used_meta":522622,"kb_avail":93637416,"statfs":{"total":96636764160,"available":95884713984,"internally_reserved":0,"allocated":216866816,"data_stored":216665542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":535165381},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,0,2,1,5,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182498,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":734932,"kb_used_data":211824,"kb_used_omap":4,"kb_used_meta":523067,"kb_avail":93636908,"statfs":{"total":96636764160,"available":95884193792,"internally_reserved":0,"allocated":216907776,"data_stored":216706502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":535621325},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,0,0,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":216535040,"data_stored":216535040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":212684800,"data_stored":212684800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":216203264,"data_stored":216203264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":216379392,"data_stored":216379392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:42:03.395 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:42:04.049 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:42:04.050 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:42:04.075 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":291,"stamp":"2024-09-05T16:42:02.981117+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":431886824,"num_objects":52726,"num_object_clones":0,"num_object_copies":210904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30343,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52726,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":138111,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1294,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":152235,"ondisk_log_size":152235,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2818896,"kb_used_data":843436,"kb_used_omap":14,"kb_used_meta":1975345,"kb_avail":374668464,"statfs":{"total":386547056640,"available":383660507136,"internally_reserved":0,"allocated":863678464,"data_stored":862873368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2022753847},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,6,6,5,12,16,8,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18882560,"num_objects":-2305,"num_object_clones":0,"num_object_copies":-9220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-100,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2305,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2305,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":100,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006512"},"pg_stats":[{"pgid":"1.f","version":"37'9536","reported_seq":9640,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:58.066444+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:41:58.066444+0000","last_peered":"2024-09-05T16:41:58.066444+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:41:58.066444+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9536,"log_dups_size":0,"ondisk_log_size":9536,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27590656,"num_objects":3368,"num_object_clones":0,"num_object_copies":13472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3368,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9536,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1132}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'9346","reported_seq":11447,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:58.015542+0000","last_change":"2024-09-05T16:40:28.919189+0000","last_active":"2024-09-05T16:41:58.015542+0000","last_peered":"2024-09-05T16:41:58.015542+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:41:58.015542+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9346,"log_dups_size":0,"ondisk_log_size":9346,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26277024,"num_objects":3208,"num_object_clones":0,"num_object_copies":12832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1256,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3208,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9344,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1015,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1952}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'9467","reported_seq":9589,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:58.427005+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:41:58.427005+0000","last_peered":"2024-09-05T16:41:58.427005+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:41:58.427005+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9467,"log_dups_size":0,"ondisk_log_size":9467,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26909288,"num_objects":3286,"num_object_clones":0,"num_object_copies":13144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3286,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9466,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1054}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'9637","reported_seq":14366,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:42:02.676545+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:42:02.676545+0000","last_peered":"2024-09-05T16:42:02.676545+0000","last_clean":"2024-09-05T16:42:02.676545+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:42:02.676545+0000","last_undegraded":"2024-09-05T16:42:02.676545+0000","last_fullsized":"2024-09-05T16:42:02.676545+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":9637,"log_dups_size":0,"ondisk_log_size":9637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27009024,"num_objects":3297,"num_object_clones":0,"num_object_copies":13188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3297,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3937,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'9301","reported_seq":9399,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:57.998091+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:41:57.998091+0000","last_peered":"2024-09-05T16:41:57.998091+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:41:57.998091+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9301,"log_dups_size":0,"ondisk_log_size":9301,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26981536,"num_objects":3294,"num_object_clones":0,"num_object_copies":13176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3294,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9299,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1149}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'9465","reported_seq":9579,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:58.145013+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:41:58.145013+0000","last_peered":"2024-09-05T16:41:58.145013+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:41:58.145013+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9465,"log_dups_size":0,"ondisk_log_size":9465,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27090944,"num_objects":3307,"num_object_clones":0,"num_object_copies":13228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3307,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9465,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1088}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'9329","reported_seq":9436,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:58.045018+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:41:58.045018+0000","last_peered":"2024-09-05T16:41:58.045018+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:41:58.045018+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9329,"log_dups_size":0,"ondisk_log_size":9329,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27301024,"num_objects":3333,"num_object_clones":0,"num_object_copies":13332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3333,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9327,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1202}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'9487","reported_seq":9575,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:57.910799+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:41:57.910799+0000","last_peered":"2024-09-05T16:41:57.910799+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:41:57.910799+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9487,"log_dups_size":0,"ondisk_log_size":9487,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26965152,"num_objects":3292,"num_object_clones":0,"num_object_copies":13168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9485,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1060}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'9496","reported_seq":9596,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:58.139294+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:41:58.139294+0000","last_peered":"2024-09-05T16:41:58.139294+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:41:58.139294+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9496,"log_dups_size":0,"ondisk_log_size":9496,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26847552,"num_objects":3278,"num_object_clones":0,"num_object_copies":13112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3278,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9493,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1034}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'9551","reported_seq":9672,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:58.261433+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:41:58.261433+0000","last_peered":"2024-09-05T16:41:58.261433+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:41:58.261433+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9551,"log_dups_size":0,"ondisk_log_size":9551,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26694816,"num_objects":3259,"num_object_clones":0,"num_object_copies":13036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3259,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9549,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1014}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'9742","reported_seq":9868,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:58.001018+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:41:58.001018+0000","last_peered":"2024-09-05T16:41:58.001018+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:41:58.001018+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9742,"log_dups_size":0,"ondisk_log_size":9742,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27692784,"num_objects":3381,"num_object_clones":0,"num_object_copies":13524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3381,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9739,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1100}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'9483","reported_seq":9581,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:57.713112+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:41:57.713112+0000","last_peered":"2024-09-05T16:41:57.713112+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:41:57.713112+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9483,"log_dups_size":0,"ondisk_log_size":9483,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835536,"num_objects":3276,"num_object_clones":0,"num_object_copies":13104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3276,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9482,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1038}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'9564","reported_seq":14299,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:42:02.561035+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:42:02.561035+0000","last_peered":"2024-09-05T16:42:02.561035+0000","last_clean":"2024-09-05T16:42:02.561035+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:42:02.561035+0000","last_undegraded":"2024-09-05T16:42:02.561035+0000","last_fullsized":"2024-09-05T16:42:02.561035+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":9564,"log_dups_size":0,"ondisk_log_size":9564,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26574848,"num_objects":3244,"num_object_clones":0,"num_object_copies":12976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3244,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1164,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'9777","reported_seq":9885,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:58.106964+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:41:58.106964+0000","last_peered":"2024-09-05T16:41:58.106964+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:41:58.106964+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9777,"log_dups_size":0,"ondisk_log_size":9777,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27432096,"num_objects":3349,"num_object_clones":0,"num_object_copies":13396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3349,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9775,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1015}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'9585","reported_seq":9697,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:58.353420+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:41:58.353420+0000","last_peered":"2024-09-05T16:41:58.353420+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:41:58.353420+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9585,"log_dups_size":0,"ondisk_log_size":9585,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26604704,"num_objects":3248,"num_object_clones":0,"num_object_copies":12992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3248,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9583,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":940}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'9469","reported_seq":9577,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:41:57.819845+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:41:57.819845+0000","last_peered":"2024-09-05T16:41:57.819845+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:41:57.819845+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9469,"log_dups_size":0,"ondisk_log_size":9469,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27079840,"num_objects":3306,"num_object_clones":0,"num_object_copies":13224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3306,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9467,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1086}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":431886824,"num_objects":52726,"num_object_clones":0,"num_object_copies":210904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30343,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52726,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":138111,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1294,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":861802496,"data_stored":861802496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":152235,"ondisk_log_size":152235,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084384,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":730764,"kb_used_data":211756,"kb_used_omap":4,"kb_used_meta":518971,"kb_avail":93641076,"statfs":{"total":96636764160,"available":95888461824,"internally_reserved":0,"allocated":216838144,"data_stored":216636870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":531427027},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,2,4,6,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986262,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":618776,"kb_used_data":208072,"kb_used_omap":4,"kb_used_meta":410683,"kb_avail":93753064,"statfs":{"total":96636764160,"available":96003137536,"internally_reserved":0,"allocated":213065728,"data_stored":212864454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":420540114},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,3,6,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673067,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":734424,"kb_used_data":211784,"kb_used_omap":1,"kb_used_meta":522622,"kb_avail":93637416,"statfs":{"total":96636764160,"available":95884713984,"internally_reserved":0,"allocated":216866816,"data_stored":216665542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":535165381},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,0,2,1,5,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182498,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":734932,"kb_used_data":211824,"kb_used_omap":4,"kb_used_meta":523067,"kb_avail":93636908,"statfs":{"total":96636764160,"available":95884193792,"internally_reserved":0,"allocated":216907776,"data_stored":216706502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":535621325},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,0,0,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":216535040,"data_stored":216535040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":212684800,"data_stored":212684800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":216203264,"data_stored":216203264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":216379392,"data_stored":216379392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:42:04.076 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-05T16:42:04.077 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'9536", 'reported_seq': 9640, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:58.066444+0000', 'last_change': '2024-09-05T16:34:56.833497+0000', 'last_active': '2024-09-05T16:41:58.066444+0000', 'last_peered': '2024-09-05T16:41:58.066444+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.828289+0000', 'last_became_peered': '2024-09-05T16:34:56.828289+0000', 'last_unstale': '2024-09-05T16:41:58.066444+0000', 'last_undegraded': '2024-09-05T16:34:56.612160+0000', 'last_fullsized': '2024-09-05T16:34:56.557632+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9536, 'log_dups_size': 0, 'ondisk_log_size': 9536, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27590656, 'num_objects': 3368, 'num_object_clones': 0, 'num_object_copies': 13472, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2236, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3368, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9536, 'num_write_kb': 51616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1132}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:42:04.077 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-05T16:42:04.077 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "37'9346", 'reported_seq': 11447, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:58.015542+0000', 'last_change': '2024-09-05T16:40:28.919189+0000', 'last_active': '2024-09-05T16:41:58.015542+0000', 'last_peered': '2024-09-05T16:41:58.015542+0000', 'last_clean': '2024-09-05T16:33:53.200762+0000', 'last_became_active': '2024-09-05T16:34:56.820292+0000', 'last_became_peered': '2024-09-05T16:34:56.820292+0000', 'last_unstale': '2024-09-05T16:41:58.015542+0000', 'last_undegraded': '2024-09-05T16:34:56.605776+0000', 'last_fullsized': '2024-09-05T16:34:56.559383+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9346, 'log_dups_size': 0, 'ondisk_log_size': 9346, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26277024, 'num_objects': 3208, 'num_object_clones': 0, 'num_object_copies': 12832, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1256, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3208, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9344, 'num_write_kb': 50212, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1015, 'num_bytes_recovered': 2613248, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 1952}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:42:04.077 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:42:04.077 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'9467", 'reported_seq': 9589, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:58.427005+0000', 'last_change': '2024-09-05T16:34:56.867900+0000', 'last_active': '2024-09-05T16:41:58.427005+0000', 'last_peered': '2024-09-05T16:41:58.427005+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:41:58.427005+0000', 'last_undegraded': '2024-09-05T16:34:56.590972+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9467, 'log_dups_size': 0, 'ondisk_log_size': 9467, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26909288, 'num_objects': 3286, 'num_object_clones': 0, 'num_object_copies': 13144, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2233, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3286, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 9466, 'num_write_kb': 50999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1054}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:42:04.077 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:42:04.077 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'9301", 'reported_seq': 9399, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:57.998091+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:41:57.998091+0000', 'last_peered': '2024-09-05T16:41:57.998091+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:41:57.998091+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9301, 'log_dups_size': 0, 'ondisk_log_size': 9301, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26981536, 'num_objects': 3294, 'num_object_clones': 0, 'num_object_copies': 13176, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2147, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3294, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9299, 'num_write_kb': 50373, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1149}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:42:04.077 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:42:04.077 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'9465", 'reported_seq': 9579, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:58.145013+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:41:58.145013+0000', 'last_peered': '2024-09-05T16:41:58.145013+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:41:58.145013+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9465, 'log_dups_size': 0, 'ondisk_log_size': 9465, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27090944, 'num_objects': 3307, 'num_object_clones': 0, 'num_object_copies': 13228, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3307, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9465, 'num_write_kb': 51088, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1088}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:42:04.077 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:42:04.078 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'9329", 'reported_seq': 9436, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:58.045018+0000', 'last_change': '2024-09-05T16:34:56.903436+0000', 'last_active': '2024-09-05T16:41:58.045018+0000', 'last_peered': '2024-09-05T16:41:58.045018+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:41:58.045018+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9329, 'log_dups_size': 0, 'ondisk_log_size': 9329, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27301024, 'num_objects': 3333, 'num_object_clones': 0, 'num_object_copies': 13332, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2133, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3333, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9327, 'num_write_kb': 50636, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1202}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:42:04.078 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-05T16:42:04.078 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "37'9487", 'reported_seq': 9575, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:57.910799+0000', 'last_change': '2024-09-05T16:34:56.883319+0000', 'last_active': '2024-09-05T16:41:57.910799+0000', 'last_peered': '2024-09-05T16:41:57.910799+0000', 'last_clean': '2024-09-05T16:33:53.505305+0000', 'last_became_active': '2024-09-05T16:34:56.873389+0000', 'last_became_peered': '2024-09-05T16:34:56.873389+0000', 'last_unstale': '2024-09-05T16:41:57.910799+0000', 'last_undegraded': '2024-09-05T16:34:56.602567+0000', 'last_fullsized': '2024-09-05T16:34:56.559958+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9487, 'log_dups_size': 0, 'ondisk_log_size': 9487, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26965152, 'num_objects': 3292, 'num_object_clones': 0, 'num_object_copies': 13168, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2234, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3292, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9485, 'num_write_kb': 51108, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1060}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:42:04.078 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:42:04.078 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'9496", 'reported_seq': 9596, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:58.139294+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:41:58.139294+0000', 'last_peered': '2024-09-05T16:41:58.139294+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:41:58.139294+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9496, 'log_dups_size': 0, 'ondisk_log_size': 9496, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26847552, 'num_objects': 3278, 'num_object_clones': 0, 'num_object_copies': 13112, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3278, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9493, 'num_write_kb': 51081, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1034}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:42:04.078 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:42:04.078 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'9551", 'reported_seq': 9672, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:58.261433+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:41:58.261433+0000', 'last_peered': '2024-09-05T16:41:58.261433+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:41:58.261433+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9551, 'log_dups_size': 0, 'ondisk_log_size': 9551, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26694816, 'num_objects': 3259, 'num_object_clones': 0, 'num_object_copies': 13036, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3259, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9549, 'num_write_kb': 51230, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1014}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:42:04.078 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:42:04.078 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'9742", 'reported_seq': 9868, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:58.001018+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:41:58.001018+0000', 'last_peered': '2024-09-05T16:41:58.001018+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:41:58.001018+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9742, 'log_dups_size': 0, 'ondisk_log_size': 9742, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27692784, 'num_objects': 3381, 'num_object_clones': 0, 'num_object_copies': 13524, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2284, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3381, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9739, 'num_write_kb': 52476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1100}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:42:04.078 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:42:04.078 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'9483", 'reported_seq': 9581, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:57.713112+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:41:57.713112+0000', 'last_peered': '2024-09-05T16:41:57.713112+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:41:57.713112+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9483, 'log_dups_size': 0, 'ondisk_log_size': 9483, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26835536, 'num_objects': 3276, 'num_object_clones': 0, 'num_object_copies': 13104, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3276, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9482, 'num_write_kb': 51031, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1038}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:42:04.079 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:42:04.079 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'9777", 'reported_seq': 9885, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:58.106964+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:41:58.106964+0000', 'last_peered': '2024-09-05T16:41:58.106964+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:41:58.106964+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9777, 'log_dups_size': 0, 'ondisk_log_size': 9777, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27432096, 'num_objects': 3349, 'num_object_clones': 0, 'num_object_copies': 13396, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3349, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9775, 'num_write_kb': 52500, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1015}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:42:04.079 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:42:04.079 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'9585", 'reported_seq': 9697, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:58.353420+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:41:58.353420+0000', 'last_peered': '2024-09-05T16:41:58.353420+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:41:58.353420+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9585, 'log_dups_size': 0, 'ondisk_log_size': 9585, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26604704, 'num_objects': 3248, 'num_object_clones': 0, 'num_object_copies': 12992, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2310, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3248, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9583, 'num_write_kb': 51325, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 940}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:42:04.079 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:42:04.079 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'9469", 'reported_seq': 9577, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:41:57.819845+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:41:57.819845+0000', 'last_peered': '2024-09-05T16:41:57.819845+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:41:57.819845+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9469, 'log_dups_size': 0, 'ondisk_log_size': 9469, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27079840, 'num_objects': 3306, 'num_object_clones': 0, 'num_object_copies': 13224, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3306, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9467, 'num_write_kb': 51092, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1086}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:42:04.079 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:42:04.877 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:42:04.939 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 30343/210904 objects degraded (14.387%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":528,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":52726,"data_bytes":431886824,"bytes_used":2886549504,"bytes_avail":383660507136,"bytes_total":386547056640,"degraded_objects":30343,"degraded_total":210904,"degraded_ratio":0.14387114516557295,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":100,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":191},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:42:04.940 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:42:04.940 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:42:05.542 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:42:05.543 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:42:05.560 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":292,"stamp":"2024-09-05T16:42:04.981713+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":424219112,"num_objects":51790,"num_object_clones":0,"num_object_copies":207160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51790,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":139047,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1358,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":153171,"ondisk_log_size":153171,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2818728,"kb_used_data":830804,"kb_used_omap":14,"kb_used_meta":1987825,"kb_avail":374668632,"statfs":{"total":386547056640,"available":383660679168,"internally_reserved":0,"allocated":850743296,"data_stored":849938200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2035533367},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,3,2,18,15,10,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25206784,"num_objects":-3077,"num_object_clones":0,"num_object_copies":-12308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3077,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3077,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":164,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006491"},"pg_stats":[{"pgid":"1.f","version":"37'9604","reported_seq":9707,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:03.259578+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:42:03.259578+0000","last_peered":"2024-09-05T16:42:03.259578+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:42:03.259578+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9604,"log_dups_size":0,"ondisk_log_size":9604,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27033600,"num_objects":3300,"num_object_clones":0,"num_object_copies":13200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3300,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9604,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1064}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'9406","reported_seq":11631,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:03.085467+0000","last_change":"2024-09-05T16:40:28.919189+0000","last_active":"2024-09-05T16:42:03.085467+0000","last_peered":"2024-09-05T16:42:03.085467+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:42:03.085467+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9406,"log_dups_size":0,"ondisk_log_size":9406,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25785504,"num_objects":3148,"num_object_clones":0,"num_object_copies":12592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3148,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9404,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1079,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1956}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'9543","reported_seq":9665,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:03.404610+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:42:03.404610+0000","last_peered":"2024-09-05T16:42:03.404610+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:42:03.404610+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9543,"log_dups_size":0,"ondisk_log_size":9543,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26286696,"num_objects":3210,"num_object_clones":0,"num_object_copies":12840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3210,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9542,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":978}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'9637","reported_seq":14366,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:42:02.676545+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:42:02.676545+0000","last_peered":"2024-09-05T16:42:02.676545+0000","last_clean":"2024-09-05T16:42:02.676545+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:42:02.676545+0000","last_undegraded":"2024-09-05T16:42:02.676545+0000","last_fullsized":"2024-09-05T16:42:02.676545+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":9637,"log_dups_size":0,"ondisk_log_size":9637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27009024,"num_objects":3297,"num_object_clones":0,"num_object_copies":13188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3297,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3937,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'9371","reported_seq":9469,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:03.261129+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:42:03.261129+0000","last_peered":"2024-09-05T16:42:03.261129+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:42:03.261129+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9371,"log_dups_size":0,"ondisk_log_size":9371,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26408096,"num_objects":3224,"num_object_clones":0,"num_object_copies":12896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3224,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9369,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1079}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'9542","reported_seq":9656,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:03.077354+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:42:03.077354+0000","last_peered":"2024-09-05T16:42:03.077354+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:42:03.077354+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9542,"log_dups_size":0,"ondisk_log_size":9542,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26460160,"num_objects":3230,"num_object_clones":0,"num_object_copies":12920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3230,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9542,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1011}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'9387","reported_seq":9494,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:02.633718+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:42:02.633718+0000","last_peered":"2024-09-05T16:42:02.633718+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:42:02.633718+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9387,"log_dups_size":0,"ondisk_log_size":9387,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26825888,"num_objects":3275,"num_object_clones":0,"num_object_copies":13100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3275,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9385,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1144}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'9558","reported_seq":9644,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:02.966076+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:42:02.966076+0000","last_peered":"2024-09-05T16:42:02.966076+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:42:02.966076+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9558,"log_dups_size":0,"ondisk_log_size":9558,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26383520,"num_objects":3221,"num_object_clones":0,"num_object_copies":12884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3221,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9556,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":989}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'9559","reported_seq":9659,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:03.138918+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:42:03.138918+0000","last_peered":"2024-09-05T16:42:03.138918+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:42:03.138918+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9559,"log_dups_size":0,"ondisk_log_size":9559,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26331456,"num_objects":3215,"num_object_clones":0,"num_object_copies":12860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3215,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9556,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":971}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'9617","reported_seq":9738,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:03.211361+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:42:03.211361+0000","last_peered":"2024-09-05T16:42:03.211361+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:42:03.211361+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9617,"log_dups_size":0,"ondisk_log_size":9617,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26154144,"num_objects":3193,"num_object_clones":0,"num_object_copies":12772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9615,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":948}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'9810","reported_seq":9936,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:03.154920+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:42:03.154920+0000","last_peered":"2024-09-05T16:42:03.154920+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:42:03.154920+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9810,"log_dups_size":0,"ondisk_log_size":9810,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27135728,"num_objects":3313,"num_object_clones":0,"num_object_copies":13252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3313,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9807,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1032}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'9550","reported_seq":9648,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:03.108233+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:42:03.108233+0000","last_peered":"2024-09-05T16:42:03.108233+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:42:03.108233+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9550,"log_dups_size":0,"ondisk_log_size":9550,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26286672,"num_objects":3209,"num_object_clones":0,"num_object_copies":12836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3209,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9549,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":971}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'9564","reported_seq":14299,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:42:02.561035+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:42:02.561035+0000","last_peered":"2024-09-05T16:42:02.561035+0000","last_clean":"2024-09-05T16:42:02.561035+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:42:02.561035+0000","last_undegraded":"2024-09-05T16:42:02.561035+0000","last_fullsized":"2024-09-05T16:42:02.561035+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":9564,"log_dups_size":0,"ondisk_log_size":9564,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26574848,"num_objects":3244,"num_object_clones":0,"num_object_copies":12976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3244,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1164,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'9845","reported_seq":9953,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:03.220567+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:42:03.220567+0000","last_peered":"2024-09-05T16:42:03.220567+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:42:03.220567+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9845,"log_dups_size":0,"ondisk_log_size":9845,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26875040,"num_objects":3281,"num_object_clones":0,"num_object_copies":13124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3281,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9843,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":947}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'9647","reported_seq":9759,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:03.456638+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:42:03.456638+0000","last_peered":"2024-09-05T16:42:03.456638+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:42:03.456638+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9647,"log_dups_size":0,"ondisk_log_size":9647,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26096800,"num_objects":3186,"num_object_clones":0,"num_object_copies":12744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3186,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9645,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":878}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'9531","reported_seq":9639,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:03.087243+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:42:03.087243+0000","last_peered":"2024-09-05T16:42:03.087243+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:42:03.087243+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9531,"log_dups_size":0,"ondisk_log_size":9531,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26571936,"num_objects":3244,"num_object_clones":0,"num_object_copies":12976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3244,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9529,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1024}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":424219112,"num_objects":51790,"num_object_clones":0,"num_object_copies":207160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51790,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":139047,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1358,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":848789504,"data_stored":848789504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":153171,"ondisk_log_size":153171,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084385,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":730728,"kb_used_data":207556,"kb_used_omap":4,"kb_used_meta":523131,"kb_avail":93641112,"statfs":{"total":96636764160,"available":95888498688,"internally_reserved":0,"allocated":212537344,"data_stored":212336070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":535686867},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,1,8,3,2,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986262,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":618776,"kb_used_data":208072,"kb_used_omap":4,"kb_used_meta":410683,"kb_avail":93753064,"statfs":{"total":96636764160,"available":96003137536,"internally_reserved":0,"allocated":213065728,"data_stored":212864454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":420540114},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,3,6,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673068,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":734412,"kb_used_data":207612,"kb_used_omap":1,"kb_used_meta":526782,"kb_avail":93637428,"statfs":{"total":96636764160,"available":95884726272,"internally_reserved":0,"allocated":212594688,"data_stored":212393414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":539425221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182499,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":734812,"kb_used_data":207564,"kb_used_omap":4,"kb_used_meta":527227,"kb_avail":93637028,"statfs":{"total":96636764160,"available":95884316672,"internally_reserved":0,"allocated":212545536,"data_stored":212344262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":539881165},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4,4,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":212217856,"data_stored":212217856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":212684800,"data_stored":212684800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":211849216,"data_stored":211849216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":212037632,"data_stored":212037632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:42:08.562 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:42:09.190 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:42:09.191 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:42:09.213 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":294,"stamp":"2024-09-05T16:42:08.983084+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":415674856,"num_objects":50747,"num_object_clones":0,"num_object_copies":202988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50747,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":140090,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":154214,"ondisk_log_size":154214,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2857112,"kb_used_data":815064,"kb_used_omap":14,"kb_used_meta":2041969,"kb_avail":374630248,"statfs":{"total":386547056640,"available":383621373952,"internally_reserved":0,"allocated":834625536,"data_stored":833820440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2090976823},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,5,15,7,19,19,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-26173440,"num_objects":-3195,"num_object_clones":0,"num_object_copies":-12780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-179,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3195,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3195,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":179,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006133"},"pg_stats":[{"pgid":"1.f","version":"37'9668","reported_seq":9772,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:08.271940+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:42:08.271940+0000","last_peered":"2024-09-05T16:42:08.271940+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:42:08.271940+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9668,"log_dups_size":0,"ondisk_log_size":9668,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26509312,"num_objects":3236,"num_object_clones":0,"num_object_copies":12944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9668,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1000}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'9474","reported_seq":11831,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:08.164813+0000","last_change":"2024-09-05T16:40:28.919189+0000","last_active":"2024-09-05T16:42:08.164813+0000","last_peered":"2024-09-05T16:42:08.164813+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:42:08.164813+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9474,"log_dups_size":0,"ondisk_log_size":9474,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25228448,"num_objects":3080,"num_object_clones":0,"num_object_copies":12320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1128,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3080,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9472,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1143,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1952}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'9613","reported_seq":9735,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:08.421893+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:42:08.421893+0000","last_peered":"2024-09-05T16:42:08.421893+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:42:08.421893+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9613,"log_dups_size":0,"ondisk_log_size":9613,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25713256,"num_objects":3140,"num_object_clones":0,"num_object_copies":12560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3140,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9612,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":908}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'9694","reported_seq":14423,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:42:07.619351+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:42:07.619351+0000","last_peered":"2024-09-05T16:42:07.619351+0000","last_clean":"2024-09-05T16:42:07.619351+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:42:07.619351+0000","last_undegraded":"2024-09-05T16:42:07.619351+0000","last_fullsized":"2024-09-05T16:42:07.619351+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":9694,"log_dups_size":0,"ondisk_log_size":9694,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26542080,"num_objects":3240,"num_object_clones":0,"num_object_copies":12960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3240,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3994,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'9425","reported_seq":9523,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:08.342951+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:42:08.342951+0000","last_peered":"2024-09-05T16:42:08.342951+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:42:08.342951+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9425,"log_dups_size":0,"ondisk_log_size":9425,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25965728,"num_objects":3170,"num_object_clones":0,"num_object_copies":12680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3170,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9423,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1025}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'9613","reported_seq":9726,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:08.353586+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:42:08.353586+0000","last_peered":"2024-09-05T16:42:08.353586+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:42:08.353586+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9613,"log_dups_size":0,"ondisk_log_size":9613,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25878528,"num_objects":3159,"num_object_clones":0,"num_object_copies":12636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3159,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9613,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":940}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'9464","reported_seq":9571,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:07.986440+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:42:07.986440+0000","last_peered":"2024-09-05T16:42:07.986440+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:42:07.986440+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9464,"log_dups_size":0,"ondisk_log_size":9464,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26195104,"num_objects":3198,"num_object_clones":0,"num_object_copies":12792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3198,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9462,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1067}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'9636","reported_seq":9724,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:08.088269+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:42:08.088269+0000","last_peered":"2024-09-05T16:42:08.088269+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:42:08.088269+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9636,"log_dups_size":0,"ondisk_log_size":9636,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25744544,"num_objects":3143,"num_object_clones":0,"num_object_copies":12572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3143,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9634,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":911}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'9631","reported_seq":9731,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:08.471750+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:42:08.471750+0000","last_peered":"2024-09-05T16:42:08.471750+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:42:08.471750+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9631,"log_dups_size":0,"ondisk_log_size":9631,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25741632,"num_objects":3143,"num_object_clones":0,"num_object_copies":12572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3143,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9628,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":899}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'9692","reported_seq":9813,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:08.472494+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:42:08.472494+0000","last_peered":"2024-09-05T16:42:08.472494+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:42:08.472494+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9692,"log_dups_size":0,"ondisk_log_size":9692,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25539744,"num_objects":3118,"num_object_clones":0,"num_object_copies":12472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3118,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9690,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":873}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'9858","reported_seq":9984,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:08.212019+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:42:08.212019+0000","last_peered":"2024-09-05T16:42:08.212019+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:42:08.212019+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9858,"log_dups_size":0,"ondisk_log_size":9858,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26742512,"num_objects":3265,"num_object_clones":0,"num_object_copies":13060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3265,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9855,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":984}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'9614","reported_seq":9711,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:08.496734+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:42:08.496734+0000","last_peered":"2024-09-05T16:42:08.496734+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:42:08.496734+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9614,"log_dups_size":0,"ondisk_log_size":9614,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25762384,"num_objects":3145,"num_object_clones":0,"num_object_copies":12580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3145,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9613,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":907}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'9631","reported_seq":14366,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:42:07.602489+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:42:07.602489+0000","last_peered":"2024-09-05T16:42:07.602489+0000","last_clean":"2024-09-05T16:42:07.602489+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:42:07.602489+0000","last_undegraded":"2024-09-05T16:42:07.602489+0000","last_fullsized":"2024-09-05T16:42:07.602489+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":9631,"log_dups_size":0,"ondisk_log_size":9631,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26025984,"num_objects":3177,"num_object_clones":0,"num_object_copies":12708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3177,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1231,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'9899","reported_seq":10007,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:08.410654+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:42:08.410654+0000","last_peered":"2024-09-05T16:42:08.410654+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:42:08.410654+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9899,"log_dups_size":0,"ondisk_log_size":9899,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26432672,"num_objects":3227,"num_object_clones":0,"num_object_copies":12908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3227,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9897,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":893}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'9709","reported_seq":9821,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:08.442073+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:42:08.442073+0000","last_peered":"2024-09-05T16:42:08.442073+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:42:08.442073+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9709,"log_dups_size":0,"ondisk_log_size":9709,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25588896,"num_objects":3124,"num_object_clones":0,"num_object_copies":12496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3124,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9707,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":816}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'9593","reported_seq":9701,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:08.081972+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:42:08.081972+0000","last_peered":"2024-09-05T16:42:08.081972+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:42:08.081972+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9593,"log_dups_size":0,"ondisk_log_size":9593,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26064032,"num_objects":3182,"num_object_clones":0,"num_object_copies":12728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3182,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9591,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":962}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":415674856,"num_objects":50747,"num_object_clones":0,"num_object_copies":202988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50747,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":140090,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":831537152,"data_stored":831537152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":154214,"ondisk_log_size":154214,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084386,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":748764,"kb_used_data":203404,"kb_used_omap":4,"kb_used_meta":545339,"kb_avail":93623076,"statfs":{"total":96636764160,"available":95870029824,"internally_reserved":0,"allocated":208285696,"data_stored":208084422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":558427859},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1,4,4,5,6,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986263,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":618768,"kb_used_data":203904,"kb_used_omap":4,"kb_used_meta":414843,"kb_avail":93753072,"statfs":{"total":96636764160,"available":96003145728,"internally_reserved":0,"allocated":208797696,"data_stored":208596422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":424799954},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,0,5,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673069,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":735836,"kb_used_data":203468,"kb_used_omap":1,"kb_used_meta":532350,"kb_avail":93636004,"statfs":{"total":96636764160,"available":95883268096,"internally_reserved":0,"allocated":208351232,"data_stored":208149958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":545126853},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,1,0,6,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182500,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":753744,"kb_used_data":204288,"kb_used_omap":4,"kb_used_meta":549435,"kb_avail":93618096,"statfs":{"total":96636764160,"available":95864930304,"internally_reserved":0,"allocated":209190912,"data_stored":208989638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":562622157},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,3,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":207908864,"data_stored":207908864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":208441344,"data_stored":208441344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":207564800,"data_stored":207564800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":207622144,"data_stored":207622144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:42:09.215 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:42:09.796 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:42:09.796 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:42:09.817 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":294,"stamp":"2024-09-05T16:42:08.983084+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":415674856,"num_objects":50747,"num_object_clones":0,"num_object_copies":202988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50747,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":140090,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":154214,"ondisk_log_size":154214,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2857112,"kb_used_data":815064,"kb_used_omap":14,"kb_used_meta":2041969,"kb_avail":374630248,"statfs":{"total":386547056640,"available":383621373952,"internally_reserved":0,"allocated":834625536,"data_stored":833820440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2090976823},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,5,15,7,19,19,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-26173440,"num_objects":-3195,"num_object_clones":0,"num_object_copies":-12780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-179,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3195,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3195,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":179,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006133"},"pg_stats":[{"pgid":"1.f","version":"37'9668","reported_seq":9772,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:08.271940+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:42:08.271940+0000","last_peered":"2024-09-05T16:42:08.271940+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:42:08.271940+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9668,"log_dups_size":0,"ondisk_log_size":9668,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26509312,"num_objects":3236,"num_object_clones":0,"num_object_copies":12944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9668,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1000}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'9474","reported_seq":11831,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:08.164813+0000","last_change":"2024-09-05T16:40:28.919189+0000","last_active":"2024-09-05T16:42:08.164813+0000","last_peered":"2024-09-05T16:42:08.164813+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:42:08.164813+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9474,"log_dups_size":0,"ondisk_log_size":9474,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25228448,"num_objects":3080,"num_object_clones":0,"num_object_copies":12320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1128,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3080,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9472,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1143,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1952}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'9613","reported_seq":9735,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:08.421893+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:42:08.421893+0000","last_peered":"2024-09-05T16:42:08.421893+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:42:08.421893+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9613,"log_dups_size":0,"ondisk_log_size":9613,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25713256,"num_objects":3140,"num_object_clones":0,"num_object_copies":12560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3140,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9612,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":908}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'9694","reported_seq":14423,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:42:07.619351+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:42:07.619351+0000","last_peered":"2024-09-05T16:42:07.619351+0000","last_clean":"2024-09-05T16:42:07.619351+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:42:07.619351+0000","last_undegraded":"2024-09-05T16:42:07.619351+0000","last_fullsized":"2024-09-05T16:42:07.619351+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":9694,"log_dups_size":0,"ondisk_log_size":9694,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26542080,"num_objects":3240,"num_object_clones":0,"num_object_copies":12960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3240,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3994,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'9425","reported_seq":9523,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:08.342951+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:42:08.342951+0000","last_peered":"2024-09-05T16:42:08.342951+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:42:08.342951+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9425,"log_dups_size":0,"ondisk_log_size":9425,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25965728,"num_objects":3170,"num_object_clones":0,"num_object_copies":12680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3170,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9423,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1025}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'9613","reported_seq":9726,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:08.353586+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:42:08.353586+0000","last_peered":"2024-09-05T16:42:08.353586+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:42:08.353586+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9613,"log_dups_size":0,"ondisk_log_size":9613,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25878528,"num_objects":3159,"num_object_clones":0,"num_object_copies":12636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3159,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9613,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":940}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'9464","reported_seq":9571,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:07.986440+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:42:07.986440+0000","last_peered":"2024-09-05T16:42:07.986440+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:42:07.986440+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9464,"log_dups_size":0,"ondisk_log_size":9464,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26195104,"num_objects":3198,"num_object_clones":0,"num_object_copies":12792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3198,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9462,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1067}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'9636","reported_seq":9724,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:08.088269+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:42:08.088269+0000","last_peered":"2024-09-05T16:42:08.088269+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:42:08.088269+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9636,"log_dups_size":0,"ondisk_log_size":9636,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25744544,"num_objects":3143,"num_object_clones":0,"num_object_copies":12572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3143,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9634,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":911}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'9631","reported_seq":9731,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:08.471750+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:42:08.471750+0000","last_peered":"2024-09-05T16:42:08.471750+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:42:08.471750+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9631,"log_dups_size":0,"ondisk_log_size":9631,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25741632,"num_objects":3143,"num_object_clones":0,"num_object_copies":12572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3143,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9628,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":899}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'9692","reported_seq":9813,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:08.472494+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:42:08.472494+0000","last_peered":"2024-09-05T16:42:08.472494+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:42:08.472494+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9692,"log_dups_size":0,"ondisk_log_size":9692,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25539744,"num_objects":3118,"num_object_clones":0,"num_object_copies":12472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3118,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9690,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":873}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'9858","reported_seq":9984,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:08.212019+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:42:08.212019+0000","last_peered":"2024-09-05T16:42:08.212019+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:42:08.212019+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9858,"log_dups_size":0,"ondisk_log_size":9858,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26742512,"num_objects":3265,"num_object_clones":0,"num_object_copies":13060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3265,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9855,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":984}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'9614","reported_seq":9711,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:08.496734+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:42:08.496734+0000","last_peered":"2024-09-05T16:42:08.496734+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:42:08.496734+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9614,"log_dups_size":0,"ondisk_log_size":9614,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25762384,"num_objects":3145,"num_object_clones":0,"num_object_copies":12580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3145,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9613,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":907}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'9631","reported_seq":14366,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:42:07.602489+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:42:07.602489+0000","last_peered":"2024-09-05T16:42:07.602489+0000","last_clean":"2024-09-05T16:42:07.602489+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:42:07.602489+0000","last_undegraded":"2024-09-05T16:42:07.602489+0000","last_fullsized":"2024-09-05T16:42:07.602489+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":9631,"log_dups_size":0,"ondisk_log_size":9631,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26025984,"num_objects":3177,"num_object_clones":0,"num_object_copies":12708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3177,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1231,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'9899","reported_seq":10007,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:08.410654+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:42:08.410654+0000","last_peered":"2024-09-05T16:42:08.410654+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:42:08.410654+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9899,"log_dups_size":0,"ondisk_log_size":9899,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26432672,"num_objects":3227,"num_object_clones":0,"num_object_copies":12908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3227,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9897,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":893}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'9709","reported_seq":9821,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:08.442073+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:42:08.442073+0000","last_peered":"2024-09-05T16:42:08.442073+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:42:08.442073+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9709,"log_dups_size":0,"ondisk_log_size":9709,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25588896,"num_objects":3124,"num_object_clones":0,"num_object_copies":12496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3124,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9707,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":816}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'9593","reported_seq":9701,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:08.081972+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:42:08.081972+0000","last_peered":"2024-09-05T16:42:08.081972+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:42:08.081972+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9593,"log_dups_size":0,"ondisk_log_size":9593,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26064032,"num_objects":3182,"num_object_clones":0,"num_object_copies":12728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3182,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9591,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":962}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":415674856,"num_objects":50747,"num_object_clones":0,"num_object_copies":202988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50747,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":140090,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":831537152,"data_stored":831537152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":154214,"ondisk_log_size":154214,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084386,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":748764,"kb_used_data":203404,"kb_used_omap":4,"kb_used_meta":545339,"kb_avail":93623076,"statfs":{"total":96636764160,"available":95870029824,"internally_reserved":0,"allocated":208285696,"data_stored":208084422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":558427859},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1,4,4,5,6,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986263,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":618768,"kb_used_data":203904,"kb_used_omap":4,"kb_used_meta":414843,"kb_avail":93753072,"statfs":{"total":96636764160,"available":96003145728,"internally_reserved":0,"allocated":208797696,"data_stored":208596422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":424799954},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,0,5,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673069,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":735836,"kb_used_data":203468,"kb_used_omap":1,"kb_used_meta":532350,"kb_avail":93636004,"statfs":{"total":96636764160,"available":95883268096,"internally_reserved":0,"allocated":208351232,"data_stored":208149958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":545126853},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,1,0,6,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182500,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":753744,"kb_used_data":204288,"kb_used_omap":4,"kb_used_meta":549435,"kb_avail":93618096,"statfs":{"total":96636764160,"available":95864930304,"internally_reserved":0,"allocated":209190912,"data_stored":208989638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":562622157},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,3,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":207908864,"data_stored":207908864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":208441344,"data_stored":208441344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":207564800,"data_stored":207564800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":207622144,"data_stored":207622144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:42:09.818 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-05T16:42:09.818 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'9668", 'reported_seq': 9772, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:08.271940+0000', 'last_change': '2024-09-05T16:34:56.833497+0000', 'last_active': '2024-09-05T16:42:08.271940+0000', 'last_peered': '2024-09-05T16:42:08.271940+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.828289+0000', 'last_became_peered': '2024-09-05T16:34:56.828289+0000', 'last_unstale': '2024-09-05T16:42:08.271940+0000', 'last_undegraded': '2024-09-05T16:34:56.612160+0000', 'last_fullsized': '2024-09-05T16:34:56.557632+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9668, 'log_dups_size': 0, 'ondisk_log_size': 9668, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26509312, 'num_objects': 3236, 'num_object_clones': 0, 'num_object_copies': 12944, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2236, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3236, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9668, 'num_write_kb': 51616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1000}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:42:09.818 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-05T16:42:09.818 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "37'9474", 'reported_seq': 11831, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:08.164813+0000', 'last_change': '2024-09-05T16:40:28.919189+0000', 'last_active': '2024-09-05T16:42:08.164813+0000', 'last_peered': '2024-09-05T16:42:08.164813+0000', 'last_clean': '2024-09-05T16:33:53.200762+0000', 'last_became_active': '2024-09-05T16:34:56.820292+0000', 'last_became_peered': '2024-09-05T16:34:56.820292+0000', 'last_unstale': '2024-09-05T16:42:08.164813+0000', 'last_undegraded': '2024-09-05T16:34:56.605776+0000', 'last_fullsized': '2024-09-05T16:34:56.559383+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9474, 'log_dups_size': 0, 'ondisk_log_size': 9474, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25228448, 'num_objects': 3080, 'num_object_clones': 0, 'num_object_copies': 12320, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1128, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3080, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9472, 'num_write_kb': 50212, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1143, 'num_bytes_recovered': 2613248, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 1952}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:42:09.818 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:42:09.818 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'9613", 'reported_seq': 9735, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:08.421893+0000', 'last_change': '2024-09-05T16:34:56.867900+0000', 'last_active': '2024-09-05T16:42:08.421893+0000', 'last_peered': '2024-09-05T16:42:08.421893+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:42:08.421893+0000', 'last_undegraded': '2024-09-05T16:34:56.590972+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9613, 'log_dups_size': 0, 'ondisk_log_size': 9613, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25713256, 'num_objects': 3140, 'num_object_clones': 0, 'num_object_copies': 12560, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2233, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3140, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 9612, 'num_write_kb': 50999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 908}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:42:09.819 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:42:09.819 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'9425", 'reported_seq': 9523, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:08.342951+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:42:08.342951+0000', 'last_peered': '2024-09-05T16:42:08.342951+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:42:08.342951+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9425, 'log_dups_size': 0, 'ondisk_log_size': 9425, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25965728, 'num_objects': 3170, 'num_object_clones': 0, 'num_object_copies': 12680, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2147, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3170, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9423, 'num_write_kb': 50373, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1025}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:42:09.819 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:42:09.819 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'9613", 'reported_seq': 9726, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:08.353586+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:42:08.353586+0000', 'last_peered': '2024-09-05T16:42:08.353586+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:42:08.353586+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9613, 'log_dups_size': 0, 'ondisk_log_size': 9613, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25878528, 'num_objects': 3159, 'num_object_clones': 0, 'num_object_copies': 12636, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3159, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9613, 'num_write_kb': 51088, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 940}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:42:09.819 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:42:09.819 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'9464", 'reported_seq': 9571, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:07.986440+0000', 'last_change': '2024-09-05T16:34:56.903436+0000', 'last_active': '2024-09-05T16:42:07.986440+0000', 'last_peered': '2024-09-05T16:42:07.986440+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:42:07.986440+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9464, 'log_dups_size': 0, 'ondisk_log_size': 9464, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26195104, 'num_objects': 3198, 'num_object_clones': 0, 'num_object_copies': 12792, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2133, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3198, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9462, 'num_write_kb': 50636, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1067}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:42:09.819 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-05T16:42:09.819 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "37'9636", 'reported_seq': 9724, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:08.088269+0000', 'last_change': '2024-09-05T16:34:56.883319+0000', 'last_active': '2024-09-05T16:42:08.088269+0000', 'last_peered': '2024-09-05T16:42:08.088269+0000', 'last_clean': '2024-09-05T16:33:53.505305+0000', 'last_became_active': '2024-09-05T16:34:56.873389+0000', 'last_became_peered': '2024-09-05T16:34:56.873389+0000', 'last_unstale': '2024-09-05T16:42:08.088269+0000', 'last_undegraded': '2024-09-05T16:34:56.602567+0000', 'last_fullsized': '2024-09-05T16:34:56.559958+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9636, 'log_dups_size': 0, 'ondisk_log_size': 9636, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25744544, 'num_objects': 3143, 'num_object_clones': 0, 'num_object_copies': 12572, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2234, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3143, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9634, 'num_write_kb': 51108, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 911}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:42:09.819 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:42:09.819 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'9631", 'reported_seq': 9731, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:08.471750+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:42:08.471750+0000', 'last_peered': '2024-09-05T16:42:08.471750+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:42:08.471750+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9631, 'log_dups_size': 0, 'ondisk_log_size': 9631, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25741632, 'num_objects': 3143, 'num_object_clones': 0, 'num_object_copies': 12572, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3143, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9628, 'num_write_kb': 51081, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 899}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:42:09.819 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:42:09.820 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'9692", 'reported_seq': 9813, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:08.472494+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:42:08.472494+0000', 'last_peered': '2024-09-05T16:42:08.472494+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:42:08.472494+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9692, 'log_dups_size': 0, 'ondisk_log_size': 9692, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25539744, 'num_objects': 3118, 'num_object_clones': 0, 'num_object_copies': 12472, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3118, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9690, 'num_write_kb': 51230, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 873}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:42:09.820 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:42:09.820 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'9858", 'reported_seq': 9984, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:08.212019+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:42:08.212019+0000', 'last_peered': '2024-09-05T16:42:08.212019+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:42:08.212019+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9858, 'log_dups_size': 0, 'ondisk_log_size': 9858, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26742512, 'num_objects': 3265, 'num_object_clones': 0, 'num_object_copies': 13060, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2284, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3265, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9855, 'num_write_kb': 52476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 984}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:42:09.820 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:42:09.820 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'9614", 'reported_seq': 9711, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:08.496734+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:42:08.496734+0000', 'last_peered': '2024-09-05T16:42:08.496734+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:42:08.496734+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9614, 'log_dups_size': 0, 'ondisk_log_size': 9614, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25762384, 'num_objects': 3145, 'num_object_clones': 0, 'num_object_copies': 12580, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3145, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9613, 'num_write_kb': 51031, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 907}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:42:09.820 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:42:09.820 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'9899", 'reported_seq': 10007, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:08.410654+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:42:08.410654+0000', 'last_peered': '2024-09-05T16:42:08.410654+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:42:08.410654+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9899, 'log_dups_size': 0, 'ondisk_log_size': 9899, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26432672, 'num_objects': 3227, 'num_object_clones': 0, 'num_object_copies': 12908, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3227, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9897, 'num_write_kb': 52500, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 893}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:42:09.820 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:42:09.820 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'9709", 'reported_seq': 9821, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:08.442073+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:42:08.442073+0000', 'last_peered': '2024-09-05T16:42:08.442073+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:42:08.442073+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9709, 'log_dups_size': 0, 'ondisk_log_size': 9709, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25588896, 'num_objects': 3124, 'num_object_clones': 0, 'num_object_copies': 12496, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2310, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3124, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9707, 'num_write_kb': 51325, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 816}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:42:09.820 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:42:09.820 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'9593", 'reported_seq': 9701, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:08.081972+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:42:08.081972+0000', 'last_peered': '2024-09-05T16:42:08.081972+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:42:08.081972+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9593, 'log_dups_size': 0, 'ondisk_log_size': 9593, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26064032, 'num_objects': 3182, 'num_object_clones': 0, 'num_object_copies': 12728, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3182, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9591, 'num_write_kb': 51092, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 962}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:42:09.821 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:42:10.572 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:42:10.572 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 30215/202988 objects degraded (14.885%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":533,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":50747,"data_bytes":415674856,"bytes_used":2925682688,"bytes_avail":383621373952,"bytes_total":386547056640,"degraded_objects":30215,"degraded_total":202988,"degraded_ratio":0.14885116361558318,"recovering_objects_per_sec":14,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":179,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":266},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:42:10.602 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:42:10.602 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:42:11.160 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:42:11.160 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:42:11.181 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":295,"stamp":"2024-09-05T16:42:10.983770+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":415674856,"num_objects":50747,"num_object_clones":0,"num_object_copies":202988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50747,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":140090,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":154214,"ondisk_log_size":154214,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2857112,"kb_used_data":815064,"kb_used_omap":14,"kb_used_meta":2041969,"kb_avail":374630248,"statfs":{"total":386547056640,"available":383621373952,"internally_reserved":0,"allocated":834625536,"data_stored":833820440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2090976823},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,5,15,7,19,19,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17334272,"num_objects":-2116,"num_object_clones":0,"num_object_copies":-8464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-128,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2116,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2116,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":128,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005991"},"pg_stats":[{"pgid":"1.f","version":"37'9668","reported_seq":9772,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:08.271940+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:42:08.271940+0000","last_peered":"2024-09-05T16:42:08.271940+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:42:08.271940+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9668,"log_dups_size":0,"ondisk_log_size":9668,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26509312,"num_objects":3236,"num_object_clones":0,"num_object_copies":12944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9668,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1000}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'9474","reported_seq":11831,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:08.164813+0000","last_change":"2024-09-05T16:40:28.919189+0000","last_active":"2024-09-05T16:42:08.164813+0000","last_peered":"2024-09-05T16:42:08.164813+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:42:08.164813+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9474,"log_dups_size":0,"ondisk_log_size":9474,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25228448,"num_objects":3080,"num_object_clones":0,"num_object_copies":12320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1128,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3080,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9472,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1143,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1952}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'9613","reported_seq":9735,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:08.421893+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:42:08.421893+0000","last_peered":"2024-09-05T16:42:08.421893+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:42:08.421893+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9613,"log_dups_size":0,"ondisk_log_size":9613,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25713256,"num_objects":3140,"num_object_clones":0,"num_object_copies":12560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3140,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9612,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":908}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'9694","reported_seq":14423,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:42:07.619351+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:42:07.619351+0000","last_peered":"2024-09-05T16:42:07.619351+0000","last_clean":"2024-09-05T16:42:07.619351+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:42:07.619351+0000","last_undegraded":"2024-09-05T16:42:07.619351+0000","last_fullsized":"2024-09-05T16:42:07.619351+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":9694,"log_dups_size":0,"ondisk_log_size":9694,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26542080,"num_objects":3240,"num_object_clones":0,"num_object_copies":12960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3240,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3994,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'9425","reported_seq":9523,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:08.342951+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:42:08.342951+0000","last_peered":"2024-09-05T16:42:08.342951+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:42:08.342951+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9425,"log_dups_size":0,"ondisk_log_size":9425,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25965728,"num_objects":3170,"num_object_clones":0,"num_object_copies":12680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3170,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9423,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1025}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'9613","reported_seq":9726,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:08.353586+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:42:08.353586+0000","last_peered":"2024-09-05T16:42:08.353586+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:42:08.353586+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9613,"log_dups_size":0,"ondisk_log_size":9613,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25878528,"num_objects":3159,"num_object_clones":0,"num_object_copies":12636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3159,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9613,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":940}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'9464","reported_seq":9571,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:07.986440+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:42:07.986440+0000","last_peered":"2024-09-05T16:42:07.986440+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:42:07.986440+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9464,"log_dups_size":0,"ondisk_log_size":9464,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26195104,"num_objects":3198,"num_object_clones":0,"num_object_copies":12792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3198,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9462,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1067}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'9636","reported_seq":9724,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:08.088269+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:42:08.088269+0000","last_peered":"2024-09-05T16:42:08.088269+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:42:08.088269+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9636,"log_dups_size":0,"ondisk_log_size":9636,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25744544,"num_objects":3143,"num_object_clones":0,"num_object_copies":12572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3143,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9634,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":911}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'9631","reported_seq":9731,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:08.471750+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:42:08.471750+0000","last_peered":"2024-09-05T16:42:08.471750+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:42:08.471750+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9631,"log_dups_size":0,"ondisk_log_size":9631,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25741632,"num_objects":3143,"num_object_clones":0,"num_object_copies":12572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3143,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9628,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":899}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'9692","reported_seq":9813,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:08.472494+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:42:08.472494+0000","last_peered":"2024-09-05T16:42:08.472494+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:42:08.472494+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9692,"log_dups_size":0,"ondisk_log_size":9692,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25539744,"num_objects":3118,"num_object_clones":0,"num_object_copies":12472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3118,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9690,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":873}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'9858","reported_seq":9984,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:08.212019+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:42:08.212019+0000","last_peered":"2024-09-05T16:42:08.212019+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:42:08.212019+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9858,"log_dups_size":0,"ondisk_log_size":9858,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26742512,"num_objects":3265,"num_object_clones":0,"num_object_copies":13060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3265,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9855,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":984}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'9614","reported_seq":9711,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:08.496734+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:42:08.496734+0000","last_peered":"2024-09-05T16:42:08.496734+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:42:08.496734+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9614,"log_dups_size":0,"ondisk_log_size":9614,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25762384,"num_objects":3145,"num_object_clones":0,"num_object_copies":12580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3145,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9613,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":907}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'9631","reported_seq":14366,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:42:07.602489+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:42:07.602489+0000","last_peered":"2024-09-05T16:42:07.602489+0000","last_clean":"2024-09-05T16:42:07.602489+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:42:07.602489+0000","last_undegraded":"2024-09-05T16:42:07.602489+0000","last_fullsized":"2024-09-05T16:42:07.602489+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":9631,"log_dups_size":0,"ondisk_log_size":9631,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26025984,"num_objects":3177,"num_object_clones":0,"num_object_copies":12708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3177,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1231,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'9899","reported_seq":10007,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:08.410654+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:42:08.410654+0000","last_peered":"2024-09-05T16:42:08.410654+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:42:08.410654+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9899,"log_dups_size":0,"ondisk_log_size":9899,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26432672,"num_objects":3227,"num_object_clones":0,"num_object_copies":12908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3227,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9897,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":893}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'9709","reported_seq":9821,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:08.442073+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:42:08.442073+0000","last_peered":"2024-09-05T16:42:08.442073+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:42:08.442073+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9709,"log_dups_size":0,"ondisk_log_size":9709,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25588896,"num_objects":3124,"num_object_clones":0,"num_object_copies":12496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3124,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9707,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":816}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'9593","reported_seq":9701,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:08.081972+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:42:08.081972+0000","last_peered":"2024-09-05T16:42:08.081972+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:42:08.081972+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9593,"log_dups_size":0,"ondisk_log_size":9593,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26064032,"num_objects":3182,"num_object_clones":0,"num_object_copies":12728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3182,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9591,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":962}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":415674856,"num_objects":50747,"num_object_clones":0,"num_object_copies":202988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50747,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":140090,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":831537152,"data_stored":831537152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":154214,"ondisk_log_size":154214,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084386,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":748764,"kb_used_data":203404,"kb_used_omap":4,"kb_used_meta":545339,"kb_avail":93623076,"statfs":{"total":96636764160,"available":95870029824,"internally_reserved":0,"allocated":208285696,"data_stored":208084422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":558427859},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1,4,4,5,6,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986263,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":618768,"kb_used_data":203904,"kb_used_omap":4,"kb_used_meta":414843,"kb_avail":93753072,"statfs":{"total":96636764160,"available":96003145728,"internally_reserved":0,"allocated":208797696,"data_stored":208596422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":424799954},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,0,5,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673069,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":735836,"kb_used_data":203468,"kb_used_omap":1,"kb_used_meta":532350,"kb_avail":93636004,"statfs":{"total":96636764160,"available":95883268096,"internally_reserved":0,"allocated":208351232,"data_stored":208149958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":545126853},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,1,0,6,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182500,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":753744,"kb_used_data":204288,"kb_used_omap":4,"kb_used_meta":549435,"kb_avail":93618096,"statfs":{"total":96636764160,"available":95864930304,"internally_reserved":0,"allocated":209190912,"data_stored":208989638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":562622157},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,3,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":207908864,"data_stored":207908864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":208441344,"data_stored":208441344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":207564800,"data_stored":207564800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":207622144,"data_stored":207622144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:42:14.183 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:42:14.856 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:42:14.857 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:42:14.883 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":296,"stamp":"2024-09-05T16:42:12.984478+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":414544360,"num_objects":50609,"num_object_clones":0,"num_object_copies":202436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50609,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":140228,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":154352,"ondisk_log_size":154352,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2857232,"kb_used_data":811020,"kb_used_omap":14,"kb_used_meta":2046129,"kb_avail":374630128,"statfs":{"total":386547056640,"available":383621251072,"internally_reserved":0,"allocated":830484480,"data_stored":829679384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2095236663},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,7,10,10,19,18,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18464768,"num_objects":-2254,"num_object_clones":0,"num_object_copies":-9016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-128,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2254,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2254,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":128,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005947"},"pg_stats":[{"pgid":"1.f","version":"37'9668","reported_seq":9772,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:08.271940+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:42:08.271940+0000","last_peered":"2024-09-05T16:42:08.271940+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:42:08.271940+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9668,"log_dups_size":0,"ondisk_log_size":9668,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26509312,"num_objects":3236,"num_object_clones":0,"num_object_copies":12944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9668,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1000}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'9474","reported_seq":11831,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:08.164813+0000","last_change":"2024-09-05T16:40:28.919189+0000","last_active":"2024-09-05T16:42:08.164813+0000","last_peered":"2024-09-05T16:42:08.164813+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:42:08.164813+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9474,"log_dups_size":0,"ondisk_log_size":9474,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25228448,"num_objects":3080,"num_object_clones":0,"num_object_copies":12320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1128,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3080,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9472,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1143,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1952}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'9613","reported_seq":9735,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:08.421893+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:42:08.421893+0000","last_peered":"2024-09-05T16:42:08.421893+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:42:08.421893+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9613,"log_dups_size":0,"ondisk_log_size":9613,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25713256,"num_objects":3140,"num_object_clones":0,"num_object_copies":12560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3140,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9612,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":908}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'9762","reported_seq":14491,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:42:12.681610+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:42:12.681610+0000","last_peered":"2024-09-05T16:42:12.681610+0000","last_clean":"2024-09-05T16:42:12.681610+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:42:12.681610+0000","last_undegraded":"2024-09-05T16:42:12.681610+0000","last_fullsized":"2024-09-05T16:42:12.681610+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":9762,"log_dups_size":0,"ondisk_log_size":9762,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25985024,"num_objects":3172,"num_object_clones":0,"num_object_copies":12688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3172,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4062,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'9425","reported_seq":9523,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:08.342951+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:42:08.342951+0000","last_peered":"2024-09-05T16:42:08.342951+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:42:08.342951+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9425,"log_dups_size":0,"ondisk_log_size":9425,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25965728,"num_objects":3170,"num_object_clones":0,"num_object_copies":12680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3170,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9423,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1025}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'9613","reported_seq":9726,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:08.353586+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:42:08.353586+0000","last_peered":"2024-09-05T16:42:08.353586+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:42:08.353586+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9613,"log_dups_size":0,"ondisk_log_size":9613,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25878528,"num_objects":3159,"num_object_clones":0,"num_object_copies":12636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3159,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9613,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":940}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'9464","reported_seq":9571,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:07.986440+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:42:07.986440+0000","last_peered":"2024-09-05T16:42:07.986440+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:42:07.986440+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9464,"log_dups_size":0,"ondisk_log_size":9464,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26195104,"num_objects":3198,"num_object_clones":0,"num_object_copies":12792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3198,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9462,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1067}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'9636","reported_seq":9724,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:08.088269+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:42:08.088269+0000","last_peered":"2024-09-05T16:42:08.088269+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:42:08.088269+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9636,"log_dups_size":0,"ondisk_log_size":9636,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25744544,"num_objects":3143,"num_object_clones":0,"num_object_copies":12572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3143,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9634,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":911}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'9631","reported_seq":9731,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:08.471750+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:42:08.471750+0000","last_peered":"2024-09-05T16:42:08.471750+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:42:08.471750+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9631,"log_dups_size":0,"ondisk_log_size":9631,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25741632,"num_objects":3143,"num_object_clones":0,"num_object_copies":12572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3143,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9628,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":899}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'9692","reported_seq":9813,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:08.472494+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:42:08.472494+0000","last_peered":"2024-09-05T16:42:08.472494+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:42:08.472494+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9692,"log_dups_size":0,"ondisk_log_size":9692,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25539744,"num_objects":3118,"num_object_clones":0,"num_object_copies":12472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3118,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9690,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":873}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'9858","reported_seq":9984,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:08.212019+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:42:08.212019+0000","last_peered":"2024-09-05T16:42:08.212019+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:42:08.212019+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9858,"log_dups_size":0,"ondisk_log_size":9858,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26742512,"num_objects":3265,"num_object_clones":0,"num_object_copies":13060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3265,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9855,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":984}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'9614","reported_seq":9711,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:08.496734+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:42:08.496734+0000","last_peered":"2024-09-05T16:42:08.496734+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:42:08.496734+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9614,"log_dups_size":0,"ondisk_log_size":9614,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25762384,"num_objects":3145,"num_object_clones":0,"num_object_copies":12580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3145,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9613,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":907}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'9701","reported_seq":14434,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:42:12.533208+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:42:12.533208+0000","last_peered":"2024-09-05T16:42:12.533208+0000","last_clean":"2024-09-05T16:42:12.533208+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:42:12.533208+0000","last_undegraded":"2024-09-05T16:42:12.533208+0000","last_fullsized":"2024-09-05T16:42:12.533208+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":9701,"log_dups_size":0,"ondisk_log_size":9701,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25452544,"num_objects":3107,"num_object_clones":0,"num_object_copies":12428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3107,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1301,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'9899","reported_seq":10007,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:08.410654+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:42:08.410654+0000","last_peered":"2024-09-05T16:42:08.410654+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:42:08.410654+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9899,"log_dups_size":0,"ondisk_log_size":9899,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26432672,"num_objects":3227,"num_object_clones":0,"num_object_copies":12908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3227,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9897,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":893}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'9709","reported_seq":9821,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:08.442073+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:42:08.442073+0000","last_peered":"2024-09-05T16:42:08.442073+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:42:08.442073+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9709,"log_dups_size":0,"ondisk_log_size":9709,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25588896,"num_objects":3124,"num_object_clones":0,"num_object_copies":12496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3124,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9707,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":816}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'9593","reported_seq":9701,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:08.081972+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:42:08.081972+0000","last_peered":"2024-09-05T16:42:08.081972+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:42:08.081972+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9593,"log_dups_size":0,"ondisk_log_size":9593,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26064032,"num_objects":3182,"num_object_clones":0,"num_object_copies":12728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3182,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9591,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":962}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":414544360,"num_objects":50609,"num_object_clones":0,"num_object_copies":202436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50609,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":140228,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":827314176,"data_stored":827314176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":154352,"ondisk_log_size":154352,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084386,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":748764,"kb_used_data":203404,"kb_used_omap":4,"kb_used_meta":545339,"kb_avail":93623076,"statfs":{"total":96636764160,"available":95870029824,"internally_reserved":0,"allocated":208285696,"data_stored":208084422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":558427859},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1,4,4,5,6,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986264,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":618888,"kb_used_data":199860,"kb_used_omap":4,"kb_used_meta":419003,"kb_avail":93752952,"statfs":{"total":96636764160,"available":96003022848,"internally_reserved":0,"allocated":204656640,"data_stored":204455366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":429059794},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,3,5,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673069,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":735836,"kb_used_data":203468,"kb_used_omap":1,"kb_used_meta":532350,"kb_avail":93636004,"statfs":{"total":96636764160,"available":95883268096,"internally_reserved":0,"allocated":208351232,"data_stored":208149958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":545126853},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,1,0,6,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182500,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":753744,"kb_used_data":204288,"kb_used_omap":4,"kb_used_meta":549435,"kb_avail":93618096,"statfs":{"total":96636764160,"available":95864930304,"internally_reserved":0,"allocated":209190912,"data_stored":208989638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":562622157},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,3,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":207908864,"data_stored":207908864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":204218368,"data_stored":204218368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":207564800,"data_stored":207564800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":207622144,"data_stored":207622144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:42:14.885 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:42:15.532 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:42:15.532 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:42:15.553 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":297,"stamp":"2024-09-05T16:42:14.985297+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":407163368,"num_objects":49708,"num_object_clones":0,"num_object_copies":198832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30153,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49708,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":141129,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1484,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155253,"ondisk_log_size":155253,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2839880,"kb_used_data":797816,"kb_used_omap":14,"kb_used_meta":2041969,"kb_avail":374647480,"statfs":{"total":386547056640,"available":383639019520,"internally_reserved":0,"allocated":816963584,"data_stored":816158488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2090976823},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,5,13,11,13,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24723456,"num_objects":-3018,"num_object_clones":0,"num_object_copies":-12072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-190,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3018,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3018,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":190,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004180"},"pg_stats":[{"pgid":"1.f","version":"37'9741","reported_seq":9844,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:13.484362+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:42:13.484362+0000","last_peered":"2024-09-05T16:42:13.484362+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:42:13.484362+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9741,"log_dups_size":0,"ondisk_log_size":9741,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25911296,"num_objects":3163,"num_object_clones":0,"num_object_copies":12652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3163,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9741,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":927}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'9529","reported_seq":12010,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:13.222580+0000","last_change":"2024-09-05T16:40:28.919189+0000","last_active":"2024-09-05T16:42:13.222580+0000","last_peered":"2024-09-05T16:42:13.222580+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:42:13.222580+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9529,"log_dups_size":0,"ondisk_log_size":9529,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24777888,"num_objects":3025,"num_object_clones":0,"num_object_copies":12100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3025,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9527,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1205,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1959}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'9681","reported_seq":9803,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:13.425184+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:42:13.425184+0000","last_peered":"2024-09-05T16:42:13.425184+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:42:13.425184+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9681,"log_dups_size":0,"ondisk_log_size":9681,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25156200,"num_objects":3072,"num_object_clones":0,"num_object_copies":12288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3072,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9680,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":840}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'9762","reported_seq":14491,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:42:12.681610+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:42:12.681610+0000","last_peered":"2024-09-05T16:42:12.681610+0000","last_clean":"2024-09-05T16:42:12.681610+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:42:12.681610+0000","last_undegraded":"2024-09-05T16:42:12.681610+0000","last_fullsized":"2024-09-05T16:42:12.681610+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":9762,"log_dups_size":0,"ondisk_log_size":9762,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25985024,"num_objects":3172,"num_object_clones":0,"num_object_copies":12688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3172,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4062,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'9496","reported_seq":9594,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:13.387291+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:42:13.387291+0000","last_peered":"2024-09-05T16:42:13.387291+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:42:13.387291+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9496,"log_dups_size":0,"ondisk_log_size":9496,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25384096,"num_objects":3099,"num_object_clones":0,"num_object_copies":12396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3099,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9494,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":954}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'9681","reported_seq":9795,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:13.350908+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:42:13.350908+0000","last_peered":"2024-09-05T16:42:13.350908+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:42:13.350908+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9681,"log_dups_size":0,"ondisk_log_size":9681,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25321472,"num_objects":3091,"num_object_clones":0,"num_object_copies":12364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3091,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9681,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":872}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'9533","reported_seq":9640,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:13.114741+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:42:13.114741+0000","last_peered":"2024-09-05T16:42:13.114741+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:42:13.114741+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9533,"log_dups_size":0,"ondisk_log_size":9533,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25629856,"num_objects":3129,"num_object_clones":0,"num_object_copies":12516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3129,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9531,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":998}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'9692","reported_seq":9779,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:13.008495+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:42:13.008495+0000","last_peered":"2024-09-05T16:42:13.008495+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:42:13.008495+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9692,"log_dups_size":0,"ondisk_log_size":9692,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25285792,"num_objects":3087,"num_object_clones":0,"num_object_copies":12348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3087,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9690,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":855}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'9688","reported_seq":9786,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:13.469391+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:42:13.469391+0000","last_peered":"2024-09-05T16:42:13.469391+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:42:13.469391+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9688,"log_dups_size":0,"ondisk_log_size":9688,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25274688,"num_objects":3086,"num_object_clones":0,"num_object_copies":12344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3086,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9685,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":842}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'9760","reported_seq":9882,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:13.318102+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:42:13.318102+0000","last_peered":"2024-09-05T16:42:13.318102+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:42:13.318102+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9760,"log_dups_size":0,"ondisk_log_size":9760,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24982688,"num_objects":3050,"num_object_clones":0,"num_object_copies":12200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3050,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9758,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":805}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'9925","reported_seq":10051,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:13.372413+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:42:13.372413+0000","last_peered":"2024-09-05T16:42:13.372413+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:42:13.372413+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9925,"log_dups_size":0,"ondisk_log_size":9925,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26193648,"num_objects":3198,"num_object_clones":0,"num_object_copies":12792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3198,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9922,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":917}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'9680","reported_seq":9778,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:13.449786+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:42:13.449786+0000","last_peered":"2024-09-05T16:42:13.449786+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:42:13.449786+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9680,"log_dups_size":0,"ondisk_log_size":9680,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25221712,"num_objects":3079,"num_object_clones":0,"num_object_copies":12316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3079,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9679,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":841}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'9701","reported_seq":14434,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:42:12.533208+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:42:12.533208+0000","last_peered":"2024-09-05T16:42:12.533208+0000","last_clean":"2024-09-05T16:42:12.533208+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:42:12.533208+0000","last_undegraded":"2024-09-05T16:42:12.533208+0000","last_fullsized":"2024-09-05T16:42:12.533208+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":9701,"log_dups_size":0,"ondisk_log_size":9701,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25452544,"num_objects":3107,"num_object_clones":0,"num_object_copies":12428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3107,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1301,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'9960","reported_seq":10068,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:13.173571+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:42:13.173571+0000","last_peered":"2024-09-05T16:42:13.173571+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:42:13.173571+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9960,"log_dups_size":0,"ondisk_log_size":9960,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25932960,"num_objects":3166,"num_object_clones":0,"num_object_copies":12664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3166,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9958,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":832}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'9768","reported_seq":9877,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:13.541203+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:42:13.541203+0000","last_peered":"2024-09-05T16:42:13.541203+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:42:13.541203+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9768,"log_dups_size":0,"ondisk_log_size":9768,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25105568,"num_objects":3065,"num_object_clones":0,"num_object_copies":12260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3065,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9766,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":757}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'9656","reported_seq":9764,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:13.212803+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:42:13.212803+0000","last_peered":"2024-09-05T16:42:13.212803+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:42:13.212803+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9656,"log_dups_size":0,"ondisk_log_size":9656,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25547936,"num_objects":3119,"num_object_clones":0,"num_object_copies":12476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3119,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9654,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":899}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":407163368,"num_objects":49708,"num_object_clones":0,"num_object_copies":198832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30153,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49708,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":141129,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1484,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":814751744,"data_stored":814751744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155253,"ondisk_log_size":155253,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084387,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":748788,"kb_used_data":199268,"kb_used_omap":4,"kb_used_meta":549499,"kb_avail":93623052,"statfs":{"total":96636764160,"available":95870005248,"internally_reserved":0,"allocated":204050432,"data_stored":203849158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":562687699},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,4,1,4,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986264,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":618888,"kb_used_data":199860,"kb_used_omap":4,"kb_used_meta":419003,"kb_avail":93752952,"statfs":{"total":96636764160,"available":96003022848,"internally_reserved":0,"allocated":204656640,"data_stored":204455366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":429059794},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,3,5,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673070,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":735960,"kb_used_data":199432,"kb_used_omap":1,"kb_used_meta":536510,"kb_avail":93635880,"statfs":{"total":96636764160,"available":95883141120,"internally_reserved":0,"allocated":204218368,"data_stored":204017094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":549386693},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,3,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182501,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":736244,"kb_used_data":199256,"kb_used_omap":4,"kb_used_meta":536955,"kb_avail":93635596,"statfs":{"total":96636764160,"available":95882850304,"internally_reserved":0,"allocated":204038144,"data_stored":203836870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":549842637},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,3,1,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":203706368,"data_stored":203706368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":204218368,"data_stored":204218368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":203374592,"data_stored":203374592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":203452416,"data_stored":203452416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:42:15.554 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-05T16:42:15.554 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'9741", 'reported_seq': 9844, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:13.484362+0000', 'last_change': '2024-09-05T16:34:56.833497+0000', 'last_active': '2024-09-05T16:42:13.484362+0000', 'last_peered': '2024-09-05T16:42:13.484362+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.828289+0000', 'last_became_peered': '2024-09-05T16:34:56.828289+0000', 'last_unstale': '2024-09-05T16:42:13.484362+0000', 'last_undegraded': '2024-09-05T16:34:56.612160+0000', 'last_fullsized': '2024-09-05T16:34:56.557632+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9741, 'log_dups_size': 0, 'ondisk_log_size': 9741, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25911296, 'num_objects': 3163, 'num_object_clones': 0, 'num_object_copies': 12652, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2236, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3163, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9741, 'num_write_kb': 51616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 927}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:42:15.555 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-05T16:42:15.555 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "37'9529", 'reported_seq': 12010, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:13.222580+0000', 'last_change': '2024-09-05T16:40:28.919189+0000', 'last_active': '2024-09-05T16:42:13.222580+0000', 'last_peered': '2024-09-05T16:42:13.222580+0000', 'last_clean': '2024-09-05T16:33:53.200762+0000', 'last_became_active': '2024-09-05T16:34:56.820292+0000', 'last_became_peered': '2024-09-05T16:34:56.820292+0000', 'last_unstale': '2024-09-05T16:42:13.222580+0000', 'last_undegraded': '2024-09-05T16:34:56.605776+0000', 'last_fullsized': '2024-09-05T16:34:56.559383+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9529, 'log_dups_size': 0, 'ondisk_log_size': 9529, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24777888, 'num_objects': 3025, 'num_object_clones': 0, 'num_object_copies': 12100, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1066, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3025, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9527, 'num_write_kb': 50212, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1205, 'num_bytes_recovered': 2613248, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 1959}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:42:15.555 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:42:15.555 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'9681", 'reported_seq': 9803, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:13.425184+0000', 'last_change': '2024-09-05T16:34:56.867900+0000', 'last_active': '2024-09-05T16:42:13.425184+0000', 'last_peered': '2024-09-05T16:42:13.425184+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:42:13.425184+0000', 'last_undegraded': '2024-09-05T16:34:56.590972+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9681, 'log_dups_size': 0, 'ondisk_log_size': 9681, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25156200, 'num_objects': 3072, 'num_object_clones': 0, 'num_object_copies': 12288, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2233, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3072, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 9680, 'num_write_kb': 50999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 840}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:42:15.555 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:42:15.555 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'9496", 'reported_seq': 9594, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:13.387291+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:42:13.387291+0000', 'last_peered': '2024-09-05T16:42:13.387291+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:42:13.387291+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9496, 'log_dups_size': 0, 'ondisk_log_size': 9496, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25384096, 'num_objects': 3099, 'num_object_clones': 0, 'num_object_copies': 12396, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2147, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3099, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9494, 'num_write_kb': 50373, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 954}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:42:15.555 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:42:15.555 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'9681", 'reported_seq': 9795, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:13.350908+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:42:13.350908+0000', 'last_peered': '2024-09-05T16:42:13.350908+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:42:13.350908+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9681, 'log_dups_size': 0, 'ondisk_log_size': 9681, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25321472, 'num_objects': 3091, 'num_object_clones': 0, 'num_object_copies': 12364, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3091, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9681, 'num_write_kb': 51088, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 872}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:42:15.556 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:42:15.556 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'9533", 'reported_seq': 9640, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:13.114741+0000', 'last_change': '2024-09-05T16:34:56.903436+0000', 'last_active': '2024-09-05T16:42:13.114741+0000', 'last_peered': '2024-09-05T16:42:13.114741+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:42:13.114741+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9533, 'log_dups_size': 0, 'ondisk_log_size': 9533, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25629856, 'num_objects': 3129, 'num_object_clones': 0, 'num_object_copies': 12516, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2133, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3129, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9531, 'num_write_kb': 50636, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 998}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:42:15.556 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-05T16:42:15.556 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "37'9692", 'reported_seq': 9779, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:13.008495+0000', 'last_change': '2024-09-05T16:34:56.883319+0000', 'last_active': '2024-09-05T16:42:13.008495+0000', 'last_peered': '2024-09-05T16:42:13.008495+0000', 'last_clean': '2024-09-05T16:33:53.505305+0000', 'last_became_active': '2024-09-05T16:34:56.873389+0000', 'last_became_peered': '2024-09-05T16:34:56.873389+0000', 'last_unstale': '2024-09-05T16:42:13.008495+0000', 'last_undegraded': '2024-09-05T16:34:56.602567+0000', 'last_fullsized': '2024-09-05T16:34:56.559958+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9692, 'log_dups_size': 0, 'ondisk_log_size': 9692, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25285792, 'num_objects': 3087, 'num_object_clones': 0, 'num_object_copies': 12348, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2234, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3087, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9690, 'num_write_kb': 51108, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 855}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:42:15.556 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:42:15.556 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'9688", 'reported_seq': 9786, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:13.469391+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:42:13.469391+0000', 'last_peered': '2024-09-05T16:42:13.469391+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:42:13.469391+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9688, 'log_dups_size': 0, 'ondisk_log_size': 9688, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25274688, 'num_objects': 3086, 'num_object_clones': 0, 'num_object_copies': 12344, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3086, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9685, 'num_write_kb': 51081, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 842}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:42:15.556 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:42:15.556 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'9760", 'reported_seq': 9882, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:13.318102+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:42:13.318102+0000', 'last_peered': '2024-09-05T16:42:13.318102+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:42:13.318102+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9760, 'log_dups_size': 0, 'ondisk_log_size': 9760, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24982688, 'num_objects': 3050, 'num_object_clones': 0, 'num_object_copies': 12200, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3050, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9758, 'num_write_kb': 51230, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 805}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:42:15.556 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:42:15.556 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'9925", 'reported_seq': 10051, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:13.372413+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:42:13.372413+0000', 'last_peered': '2024-09-05T16:42:13.372413+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:42:13.372413+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9925, 'log_dups_size': 0, 'ondisk_log_size': 9925, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26193648, 'num_objects': 3198, 'num_object_clones': 0, 'num_object_copies': 12792, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2284, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3198, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9922, 'num_write_kb': 52476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 917}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:42:15.557 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:42:15.557 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'9680", 'reported_seq': 9778, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:13.449786+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:42:13.449786+0000', 'last_peered': '2024-09-05T16:42:13.449786+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:42:13.449786+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9680, 'log_dups_size': 0, 'ondisk_log_size': 9680, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25221712, 'num_objects': 3079, 'num_object_clones': 0, 'num_object_copies': 12316, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3079, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9679, 'num_write_kb': 51031, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 841}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:42:15.557 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:42:15.557 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'9960", 'reported_seq': 10068, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:13.173571+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:42:13.173571+0000', 'last_peered': '2024-09-05T16:42:13.173571+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:42:13.173571+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9960, 'log_dups_size': 0, 'ondisk_log_size': 9960, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25932960, 'num_objects': 3166, 'num_object_clones': 0, 'num_object_copies': 12664, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3166, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9958, 'num_write_kb': 52500, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 832}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:42:15.557 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:42:15.557 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'9768", 'reported_seq': 9877, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:13.541203+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:42:13.541203+0000', 'last_peered': '2024-09-05T16:42:13.541203+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:42:13.541203+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9768, 'log_dups_size': 0, 'ondisk_log_size': 9768, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25105568, 'num_objects': 3065, 'num_object_clones': 0, 'num_object_copies': 12260, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2310, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3065, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9766, 'num_write_kb': 51325, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 757}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:42:15.557 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:42:15.557 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'9656", 'reported_seq': 9764, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:13.212803+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:42:13.212803+0000', 'last_peered': '2024-09-05T16:42:13.212803+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:42:13.212803+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9656, 'log_dups_size': 0, 'ondisk_log_size': 9656, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25547936, 'num_objects': 3119, 'num_object_clones': 0, 'num_object_copies': 12476, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3119, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9654, 'num_write_kb': 51092, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 899}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:42:15.557 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:42:16.297 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:42:16.297 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 30153/198832 objects degraded (15.165%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":539,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":49708,"data_bytes":407163368,"bytes_used":2908037120,"bytes_avail":383639019520,"bytes_total":386547056640,"degraded_objects":30153,"degraded_total":198832,"degraded_ratio":0.15165063973605858,"recovering_objects_per_sec":15,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":190,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":251},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:42:16.326 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:42:16.326 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:42:16.926 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:42:16.927 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:42:16.954 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":297,"stamp":"2024-09-05T16:42:14.985297+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":407163368,"num_objects":49708,"num_object_clones":0,"num_object_copies":198832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30153,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49708,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":141129,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1484,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155253,"ondisk_log_size":155253,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2839880,"kb_used_data":797816,"kb_used_omap":14,"kb_used_meta":2041969,"kb_avail":374647480,"statfs":{"total":386547056640,"available":383639019520,"internally_reserved":0,"allocated":816963584,"data_stored":816158488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2090976823},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,5,13,11,13,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24723456,"num_objects":-3018,"num_object_clones":0,"num_object_copies":-12072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-190,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3018,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3018,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":190,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004180"},"pg_stats":[{"pgid":"1.f","version":"37'9741","reported_seq":9844,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:13.484362+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:42:13.484362+0000","last_peered":"2024-09-05T16:42:13.484362+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:42:13.484362+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9741,"log_dups_size":0,"ondisk_log_size":9741,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25911296,"num_objects":3163,"num_object_clones":0,"num_object_copies":12652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3163,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9741,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":927}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'9529","reported_seq":12010,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:13.222580+0000","last_change":"2024-09-05T16:40:28.919189+0000","last_active":"2024-09-05T16:42:13.222580+0000","last_peered":"2024-09-05T16:42:13.222580+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:42:13.222580+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9529,"log_dups_size":0,"ondisk_log_size":9529,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24777888,"num_objects":3025,"num_object_clones":0,"num_object_copies":12100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3025,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9527,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1205,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1959}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'9681","reported_seq":9803,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:13.425184+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:42:13.425184+0000","last_peered":"2024-09-05T16:42:13.425184+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:42:13.425184+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9681,"log_dups_size":0,"ondisk_log_size":9681,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25156200,"num_objects":3072,"num_object_clones":0,"num_object_copies":12288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3072,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9680,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":840}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'9762","reported_seq":14491,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:42:12.681610+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:42:12.681610+0000","last_peered":"2024-09-05T16:42:12.681610+0000","last_clean":"2024-09-05T16:42:12.681610+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:42:12.681610+0000","last_undegraded":"2024-09-05T16:42:12.681610+0000","last_fullsized":"2024-09-05T16:42:12.681610+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":9762,"log_dups_size":0,"ondisk_log_size":9762,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25985024,"num_objects":3172,"num_object_clones":0,"num_object_copies":12688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3172,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4062,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'9496","reported_seq":9594,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:13.387291+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:42:13.387291+0000","last_peered":"2024-09-05T16:42:13.387291+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:42:13.387291+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9496,"log_dups_size":0,"ondisk_log_size":9496,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25384096,"num_objects":3099,"num_object_clones":0,"num_object_copies":12396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3099,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9494,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":954}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'9681","reported_seq":9795,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:13.350908+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:42:13.350908+0000","last_peered":"2024-09-05T16:42:13.350908+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:42:13.350908+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9681,"log_dups_size":0,"ondisk_log_size":9681,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25321472,"num_objects":3091,"num_object_clones":0,"num_object_copies":12364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3091,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9681,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":872}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'9533","reported_seq":9640,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:13.114741+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:42:13.114741+0000","last_peered":"2024-09-05T16:42:13.114741+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:42:13.114741+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9533,"log_dups_size":0,"ondisk_log_size":9533,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25629856,"num_objects":3129,"num_object_clones":0,"num_object_copies":12516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3129,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9531,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":998}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'9692","reported_seq":9779,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:13.008495+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:42:13.008495+0000","last_peered":"2024-09-05T16:42:13.008495+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:42:13.008495+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9692,"log_dups_size":0,"ondisk_log_size":9692,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25285792,"num_objects":3087,"num_object_clones":0,"num_object_copies":12348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3087,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9690,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":855}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'9688","reported_seq":9786,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:13.469391+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:42:13.469391+0000","last_peered":"2024-09-05T16:42:13.469391+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:42:13.469391+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9688,"log_dups_size":0,"ondisk_log_size":9688,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25274688,"num_objects":3086,"num_object_clones":0,"num_object_copies":12344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3086,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9685,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":842}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'9760","reported_seq":9882,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:13.318102+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:42:13.318102+0000","last_peered":"2024-09-05T16:42:13.318102+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:42:13.318102+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9760,"log_dups_size":0,"ondisk_log_size":9760,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24982688,"num_objects":3050,"num_object_clones":0,"num_object_copies":12200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3050,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9758,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":805}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'9925","reported_seq":10051,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:13.372413+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:42:13.372413+0000","last_peered":"2024-09-05T16:42:13.372413+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:42:13.372413+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9925,"log_dups_size":0,"ondisk_log_size":9925,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26193648,"num_objects":3198,"num_object_clones":0,"num_object_copies":12792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3198,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9922,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":917}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'9680","reported_seq":9778,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:13.449786+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:42:13.449786+0000","last_peered":"2024-09-05T16:42:13.449786+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:42:13.449786+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9680,"log_dups_size":0,"ondisk_log_size":9680,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25221712,"num_objects":3079,"num_object_clones":0,"num_object_copies":12316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3079,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9679,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":841}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'9701","reported_seq":14434,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:42:12.533208+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:42:12.533208+0000","last_peered":"2024-09-05T16:42:12.533208+0000","last_clean":"2024-09-05T16:42:12.533208+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:42:12.533208+0000","last_undegraded":"2024-09-05T16:42:12.533208+0000","last_fullsized":"2024-09-05T16:42:12.533208+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":9701,"log_dups_size":0,"ondisk_log_size":9701,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25452544,"num_objects":3107,"num_object_clones":0,"num_object_copies":12428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3107,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1301,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'9960","reported_seq":10068,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:13.173571+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:42:13.173571+0000","last_peered":"2024-09-05T16:42:13.173571+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:42:13.173571+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9960,"log_dups_size":0,"ondisk_log_size":9960,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25932960,"num_objects":3166,"num_object_clones":0,"num_object_copies":12664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3166,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9958,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":832}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'9768","reported_seq":9877,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:13.541203+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:42:13.541203+0000","last_peered":"2024-09-05T16:42:13.541203+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:42:13.541203+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9768,"log_dups_size":0,"ondisk_log_size":9768,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25105568,"num_objects":3065,"num_object_clones":0,"num_object_copies":12260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3065,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9766,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":757}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'9656","reported_seq":9764,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:13.212803+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:42:13.212803+0000","last_peered":"2024-09-05T16:42:13.212803+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:42:13.212803+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9656,"log_dups_size":0,"ondisk_log_size":9656,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25547936,"num_objects":3119,"num_object_clones":0,"num_object_copies":12476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3119,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9654,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":899}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":407163368,"num_objects":49708,"num_object_clones":0,"num_object_copies":198832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30153,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49708,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":141129,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1484,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":814751744,"data_stored":814751744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155253,"ondisk_log_size":155253,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084387,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":748788,"kb_used_data":199268,"kb_used_omap":4,"kb_used_meta":549499,"kb_avail":93623052,"statfs":{"total":96636764160,"available":95870005248,"internally_reserved":0,"allocated":204050432,"data_stored":203849158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":562687699},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,4,1,4,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986264,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":618888,"kb_used_data":199860,"kb_used_omap":4,"kb_used_meta":419003,"kb_avail":93752952,"statfs":{"total":96636764160,"available":96003022848,"internally_reserved":0,"allocated":204656640,"data_stored":204455366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":429059794},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,3,5,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673070,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":735960,"kb_used_data":199432,"kb_used_omap":1,"kb_used_meta":536510,"kb_avail":93635880,"statfs":{"total":96636764160,"available":95883141120,"internally_reserved":0,"allocated":204218368,"data_stored":204017094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":549386693},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,3,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182501,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":736244,"kb_used_data":199256,"kb_used_omap":4,"kb_used_meta":536955,"kb_avail":93635596,"statfs":{"total":96636764160,"available":95882850304,"internally_reserved":0,"allocated":204038144,"data_stored":203836870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":549842637},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,3,1,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":203706368,"data_stored":203706368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":204218368,"data_stored":204218368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":203374592,"data_stored":203374592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":203452416,"data_stored":203452416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:42:19.956 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:42:20.535 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:42:20.535 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:42:20.556 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":299,"stamp":"2024-09-05T16:42:18.986731+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":398684648,"num_objects":48673,"num_object_clones":0,"num_object_copies":194692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30101,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48673,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":142164,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1536,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156288,"ondisk_log_size":156288,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2806924,"kb_used_data":781472,"kb_used_omap":14,"kb_used_meta":2025329,"kb_avail":374680436,"statfs":{"total":386547056640,"available":383672766464,"internally_reserved":0,"allocated":800227328,"data_stored":799422232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2073937463},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,2,2,4,6,25,15,13],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25534464,"num_objects":-3117,"num_object_clones":0,"num_object_copies":-12468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3117,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3117,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":178,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004491"},"pg_stats":[{"pgid":"1.f","version":"37'9816","reported_seq":9920,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:18.420833+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:42:18.420833+0000","last_peered":"2024-09-05T16:42:18.420833+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:42:18.420833+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9816,"log_dups_size":0,"ondisk_log_size":9816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25296896,"num_objects":3088,"num_object_clones":0,"num_object_copies":12352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3088,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9816,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":852}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'9592","reported_seq":12177,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:18.157975+0000","last_change":"2024-09-05T16:40:28.919189+0000","last_active":"2024-09-05T16:42:18.157975+0000","last_peered":"2024-09-05T16:42:18.157975+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:42:18.157975+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9592,"log_dups_size":0,"ondisk_log_size":9592,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24261792,"num_objects":2962,"num_object_clones":0,"num_object_copies":11848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1014,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2962,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9590,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1257,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1948}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'9746","reported_seq":9868,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:18.523135+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:42:18.523135+0000","last_peered":"2024-09-05T16:42:18.523135+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:42:18.523135+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9746,"log_dups_size":0,"ondisk_log_size":9746,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24623720,"num_objects":3007,"num_object_clones":0,"num_object_copies":12028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3007,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9745,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":775}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'9840","reported_seq":14569,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:42:17.732829+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:42:17.732829+0000","last_peered":"2024-09-05T16:42:17.732829+0000","last_clean":"2024-09-05T16:42:17.732829+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:42:17.732829+0000","last_undegraded":"2024-09-05T16:42:17.732829+0000","last_fullsized":"2024-09-05T16:42:17.732829+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":9840,"log_dups_size":0,"ondisk_log_size":9840,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25346048,"num_objects":3094,"num_object_clones":0,"num_object_copies":12376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3094,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4140,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'9563","reported_seq":9659,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:18.454437+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:42:18.454437+0000","last_peered":"2024-09-05T16:42:18.454437+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:42:18.454437+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9563,"log_dups_size":0,"ondisk_log_size":9563,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24835232,"num_objects":3032,"num_object_clones":0,"num_object_copies":12128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3032,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9561,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":887}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'9754","reported_seq":9867,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:18.481053+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:42:18.481053+0000","last_peered":"2024-09-05T16:42:18.481053+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:42:18.481053+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9754,"log_dups_size":0,"ondisk_log_size":9754,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24723456,"num_objects":3018,"num_object_clones":0,"num_object_copies":12072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3018,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9754,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":799}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'9603","reported_seq":9710,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:18.115988+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:42:18.115988+0000","last_peered":"2024-09-05T16:42:18.115988+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:42:18.115988+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9603,"log_dups_size":0,"ondisk_log_size":9603,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25056416,"num_objects":3059,"num_object_clones":0,"num_object_copies":12236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3059,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9601,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":928}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'9749","reported_seq":9837,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:18.028382+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:42:18.028382+0000","last_peered":"2024-09-05T16:42:18.028382+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:42:18.028382+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9749,"log_dups_size":0,"ondisk_log_size":9749,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24818848,"num_objects":3030,"num_object_clones":0,"num_object_copies":12120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3030,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9747,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":798}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'9748","reported_seq":9848,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:18.334006+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:42:18.334006+0000","last_peered":"2024-09-05T16:42:18.334006+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:42:18.334006+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9748,"log_dups_size":0,"ondisk_log_size":9748,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24783168,"num_objects":3026,"num_object_clones":0,"num_object_copies":12104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3026,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9745,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":782}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'9820","reported_seq":9941,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:18.456311+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:42:18.456311+0000","last_peered":"2024-09-05T16:42:18.456311+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:42:18.456311+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9820,"log_dups_size":0,"ondisk_log_size":9820,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24491168,"num_objects":2990,"num_object_clones":0,"num_object_copies":11960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2990,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9818,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":745}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'9990","reported_seq":10116,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:18.476681+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:42:18.476681+0000","last_peered":"2024-09-05T16:42:18.476681+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:42:18.476681+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9990,"log_dups_size":0,"ondisk_log_size":9990,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25661168,"num_objects":3133,"num_object_clones":0,"num_object_copies":12532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3133,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9987,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":852}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'9733","reported_seq":9831,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:18.466016+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:42:18.466016+0000","last_peered":"2024-09-05T16:42:18.466016+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:42:18.466016+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9733,"log_dups_size":0,"ondisk_log_size":9733,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24787536,"num_objects":3026,"num_object_clones":0,"num_object_copies":12104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3026,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9732,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":788}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'9759","reported_seq":14494,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:42:17.844961+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:42:17.844961+0000","last_peered":"2024-09-05T16:42:17.844961+0000","last_clean":"2024-09-05T16:42:17.844961+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:42:17.844961+0000","last_undegraded":"2024-09-05T16:42:17.844961+0000","last_fullsized":"2024-09-05T16:42:17.844961+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":9759,"log_dups_size":0,"ondisk_log_size":9759,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24977408,"num_objects":3049,"num_object_clones":0,"num_object_copies":12196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3049,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1359,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'10027","reported_seq":10134,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:18.453010+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:42:18.453010+0000","last_peered":"2024-09-05T16:42:18.453010+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:42:18.453010+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25384096,"num_objects":3099,"num_object_clones":0,"num_object_copies":12396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3099,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10025,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":765}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'9823","reported_seq":9935,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:18.271256+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:42:18.271256+0000","last_peered":"2024-09-05T16:42:18.271256+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:42:18.271256+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9823,"log_dups_size":0,"ondisk_log_size":9823,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24655008,"num_objects":3010,"num_object_clones":0,"num_object_copies":12040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3010,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9821,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":702}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'9725","reported_seq":9831,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:18.130140+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:42:18.130140+0000","last_peered":"2024-09-05T16:42:18.130140+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:42:18.130140+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9725,"log_dups_size":0,"ondisk_log_size":9725,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24982688,"num_objects":3050,"num_object_clones":0,"num_object_copies":12200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3050,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9723,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":830}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":398684648,"num_objects":48673,"num_object_clones":0,"num_object_copies":194692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30101,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48673,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":142164,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1536,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":797663232,"data_stored":797663232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156288,"ondisk_log_size":156288,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084388,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":732192,"kb_used_data":195144,"kb_used_omap":4,"kb_used_meta":537019,"kb_avail":93639648,"statfs":{"total":96636764160,"available":95886999552,"internally_reserved":0,"allocated":199827456,"data_stored":199626182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":549908179},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,1,3,9,4,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986265,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":602336,"kb_used_data":195756,"kb_used_omap":4,"kb_used_meta":406523,"kb_avail":93769504,"statfs":{"total":96636764160,"available":96019972096,"internally_reserved":0,"allocated":200454144,"data_stored":200252870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":416280274},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,1,4,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673071,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":736140,"kb_used_data":195452,"kb_used_omap":1,"kb_used_meta":540670,"kb_avail":93635700,"statfs":{"total":96636764160,"available":95882956800,"internally_reserved":0,"allocated":200142848,"data_stored":199941574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":553646533},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,7,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182502,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":736256,"kb_used_data":195120,"kb_used_omap":4,"kb_used_meta":541115,"kb_avail":93635584,"statfs":{"total":96636764160,"available":95882838016,"internally_reserved":0,"allocated":199802880,"data_stored":199601606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":554102477},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,2,0,0,8,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":199462912,"data_stored":199462912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":199929856,"data_stored":199929856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":199098368,"data_stored":199098368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":199172096,"data_stored":199172096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:42:20.557 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:42:21.179 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:42:21.180 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:42:21.202 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":300,"stamp":"2024-09-05T16:42:20.987660+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":398684648,"num_objects":48673,"num_object_clones":0,"num_object_copies":194692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30101,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48673,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":142164,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1536,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156288,"ondisk_log_size":156288,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2806924,"kb_used_data":781472,"kb_used_omap":14,"kb_used_meta":2025329,"kb_avail":374680436,"statfs":{"total":386547056640,"available":383672766464,"internally_reserved":0,"allocated":800227328,"data_stored":799422232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2073937463},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,2,2,4,6,25,15,13],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16990208,"num_objects":-2074,"num_object_clones":0,"num_object_copies":-8296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-114,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2074,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2074,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":114,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004576"},"pg_stats":[{"pgid":"1.f","version":"37'9816","reported_seq":9920,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:18.420833+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:42:18.420833+0000","last_peered":"2024-09-05T16:42:18.420833+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:42:18.420833+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9816,"log_dups_size":0,"ondisk_log_size":9816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25296896,"num_objects":3088,"num_object_clones":0,"num_object_copies":12352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3088,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9816,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":852}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'9592","reported_seq":12177,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:18.157975+0000","last_change":"2024-09-05T16:40:28.919189+0000","last_active":"2024-09-05T16:42:18.157975+0000","last_peered":"2024-09-05T16:42:18.157975+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:42:18.157975+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9592,"log_dups_size":0,"ondisk_log_size":9592,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24261792,"num_objects":2962,"num_object_clones":0,"num_object_copies":11848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1014,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2962,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9590,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1257,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1948}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'9746","reported_seq":9868,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:18.523135+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:42:18.523135+0000","last_peered":"2024-09-05T16:42:18.523135+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:42:18.523135+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9746,"log_dups_size":0,"ondisk_log_size":9746,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24623720,"num_objects":3007,"num_object_clones":0,"num_object_copies":12028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3007,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9745,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":775}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'9840","reported_seq":14569,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:42:17.732829+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:42:17.732829+0000","last_peered":"2024-09-05T16:42:17.732829+0000","last_clean":"2024-09-05T16:42:17.732829+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:42:17.732829+0000","last_undegraded":"2024-09-05T16:42:17.732829+0000","last_fullsized":"2024-09-05T16:42:17.732829+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":9840,"log_dups_size":0,"ondisk_log_size":9840,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25346048,"num_objects":3094,"num_object_clones":0,"num_object_copies":12376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3094,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4140,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'9563","reported_seq":9659,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:18.454437+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:42:18.454437+0000","last_peered":"2024-09-05T16:42:18.454437+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:42:18.454437+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9563,"log_dups_size":0,"ondisk_log_size":9563,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24835232,"num_objects":3032,"num_object_clones":0,"num_object_copies":12128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3032,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9561,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":887}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'9754","reported_seq":9867,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:18.481053+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:42:18.481053+0000","last_peered":"2024-09-05T16:42:18.481053+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:42:18.481053+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9754,"log_dups_size":0,"ondisk_log_size":9754,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24723456,"num_objects":3018,"num_object_clones":0,"num_object_copies":12072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3018,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9754,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":799}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'9603","reported_seq":9710,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:18.115988+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:42:18.115988+0000","last_peered":"2024-09-05T16:42:18.115988+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:42:18.115988+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9603,"log_dups_size":0,"ondisk_log_size":9603,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25056416,"num_objects":3059,"num_object_clones":0,"num_object_copies":12236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3059,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9601,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":928}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'9749","reported_seq":9837,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:18.028382+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:42:18.028382+0000","last_peered":"2024-09-05T16:42:18.028382+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:42:18.028382+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9749,"log_dups_size":0,"ondisk_log_size":9749,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24818848,"num_objects":3030,"num_object_clones":0,"num_object_copies":12120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3030,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9747,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":798}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'9748","reported_seq":9848,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:18.334006+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:42:18.334006+0000","last_peered":"2024-09-05T16:42:18.334006+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:42:18.334006+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9748,"log_dups_size":0,"ondisk_log_size":9748,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24783168,"num_objects":3026,"num_object_clones":0,"num_object_copies":12104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3026,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9745,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":782}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'9820","reported_seq":9941,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:18.456311+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:42:18.456311+0000","last_peered":"2024-09-05T16:42:18.456311+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:42:18.456311+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9820,"log_dups_size":0,"ondisk_log_size":9820,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24491168,"num_objects":2990,"num_object_clones":0,"num_object_copies":11960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2990,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9818,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":745}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'9990","reported_seq":10116,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:18.476681+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:42:18.476681+0000","last_peered":"2024-09-05T16:42:18.476681+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:42:18.476681+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9990,"log_dups_size":0,"ondisk_log_size":9990,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25661168,"num_objects":3133,"num_object_clones":0,"num_object_copies":12532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3133,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9987,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":852}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'9733","reported_seq":9831,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:18.466016+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:42:18.466016+0000","last_peered":"2024-09-05T16:42:18.466016+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:42:18.466016+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9733,"log_dups_size":0,"ondisk_log_size":9733,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24787536,"num_objects":3026,"num_object_clones":0,"num_object_copies":12104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3026,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9732,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":788}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'9759","reported_seq":14494,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:42:17.844961+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:42:17.844961+0000","last_peered":"2024-09-05T16:42:17.844961+0000","last_clean":"2024-09-05T16:42:17.844961+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:42:17.844961+0000","last_undegraded":"2024-09-05T16:42:17.844961+0000","last_fullsized":"2024-09-05T16:42:17.844961+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":9759,"log_dups_size":0,"ondisk_log_size":9759,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24977408,"num_objects":3049,"num_object_clones":0,"num_object_copies":12196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3049,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1359,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'10027","reported_seq":10134,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:18.453010+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:42:18.453010+0000","last_peered":"2024-09-05T16:42:18.453010+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:42:18.453010+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25384096,"num_objects":3099,"num_object_clones":0,"num_object_copies":12396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3099,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10025,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":765}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'9823","reported_seq":9935,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:18.271256+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:42:18.271256+0000","last_peered":"2024-09-05T16:42:18.271256+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:42:18.271256+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9823,"log_dups_size":0,"ondisk_log_size":9823,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24655008,"num_objects":3010,"num_object_clones":0,"num_object_copies":12040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3010,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9821,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":702}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'9725","reported_seq":9831,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:18.130140+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:42:18.130140+0000","last_peered":"2024-09-05T16:42:18.130140+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:42:18.130140+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9725,"log_dups_size":0,"ondisk_log_size":9725,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24982688,"num_objects":3050,"num_object_clones":0,"num_object_copies":12200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3050,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9723,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":830}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":398684648,"num_objects":48673,"num_object_clones":0,"num_object_copies":194692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30101,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48673,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":142164,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1536,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":797663232,"data_stored":797663232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156288,"ondisk_log_size":156288,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084388,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":732192,"kb_used_data":195144,"kb_used_omap":4,"kb_used_meta":537019,"kb_avail":93639648,"statfs":{"total":96636764160,"available":95886999552,"internally_reserved":0,"allocated":199827456,"data_stored":199626182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":549908179},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,1,3,9,4,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986265,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":602336,"kb_used_data":195756,"kb_used_omap":4,"kb_used_meta":406523,"kb_avail":93769504,"statfs":{"total":96636764160,"available":96019972096,"internally_reserved":0,"allocated":200454144,"data_stored":200252870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":416280274},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,1,4,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673071,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":736140,"kb_used_data":195452,"kb_used_omap":1,"kb_used_meta":540670,"kb_avail":93635700,"statfs":{"total":96636764160,"available":95882956800,"internally_reserved":0,"allocated":200142848,"data_stored":199941574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":553646533},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,7,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182502,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":736256,"kb_used_data":195120,"kb_used_omap":4,"kb_used_meta":541115,"kb_avail":93635584,"statfs":{"total":96636764160,"available":95882838016,"internally_reserved":0,"allocated":199802880,"data_stored":199601606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":554102477},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,2,0,0,8,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":199462912,"data_stored":199462912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":199929856,"data_stored":199929856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":199098368,"data_stored":199098368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":199172096,"data_stored":199172096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:42:21.203 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-05T16:42:21.203 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'9816", 'reported_seq': 9920, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:18.420833+0000', 'last_change': '2024-09-05T16:34:56.833497+0000', 'last_active': '2024-09-05T16:42:18.420833+0000', 'last_peered': '2024-09-05T16:42:18.420833+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.828289+0000', 'last_became_peered': '2024-09-05T16:34:56.828289+0000', 'last_unstale': '2024-09-05T16:42:18.420833+0000', 'last_undegraded': '2024-09-05T16:34:56.612160+0000', 'last_fullsized': '2024-09-05T16:34:56.557632+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9816, 'log_dups_size': 0, 'ondisk_log_size': 9816, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25296896, 'num_objects': 3088, 'num_object_clones': 0, 'num_object_copies': 12352, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2236, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3088, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9816, 'num_write_kb': 51616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 852}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:42:21.203 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-05T16:42:21.203 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "37'9592", 'reported_seq': 12177, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:18.157975+0000', 'last_change': '2024-09-05T16:40:28.919189+0000', 'last_active': '2024-09-05T16:42:18.157975+0000', 'last_peered': '2024-09-05T16:42:18.157975+0000', 'last_clean': '2024-09-05T16:33:53.200762+0000', 'last_became_active': '2024-09-05T16:34:56.820292+0000', 'last_became_peered': '2024-09-05T16:34:56.820292+0000', 'last_unstale': '2024-09-05T16:42:18.157975+0000', 'last_undegraded': '2024-09-05T16:34:56.605776+0000', 'last_fullsized': '2024-09-05T16:34:56.559383+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9592, 'log_dups_size': 0, 'ondisk_log_size': 9592, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24261792, 'num_objects': 2962, 'num_object_clones': 0, 'num_object_copies': 11848, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1014, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2962, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9590, 'num_write_kb': 50212, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1257, 'num_bytes_recovered': 2613248, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 1948}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:42:21.203 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:42:21.203 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'9746", 'reported_seq': 9868, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:18.523135+0000', 'last_change': '2024-09-05T16:34:56.867900+0000', 'last_active': '2024-09-05T16:42:18.523135+0000', 'last_peered': '2024-09-05T16:42:18.523135+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:42:18.523135+0000', 'last_undegraded': '2024-09-05T16:34:56.590972+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9746, 'log_dups_size': 0, 'ondisk_log_size': 9746, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24623720, 'num_objects': 3007, 'num_object_clones': 0, 'num_object_copies': 12028, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2233, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3007, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 9745, 'num_write_kb': 50999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 775}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:42:21.203 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:42:21.204 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'9563", 'reported_seq': 9659, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:18.454437+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:42:18.454437+0000', 'last_peered': '2024-09-05T16:42:18.454437+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:42:18.454437+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9563, 'log_dups_size': 0, 'ondisk_log_size': 9563, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24835232, 'num_objects': 3032, 'num_object_clones': 0, 'num_object_copies': 12128, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2147, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3032, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9561, 'num_write_kb': 50373, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 887}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:42:21.204 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:42:21.204 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'9754", 'reported_seq': 9867, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:18.481053+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:42:18.481053+0000', 'last_peered': '2024-09-05T16:42:18.481053+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:42:18.481053+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9754, 'log_dups_size': 0, 'ondisk_log_size': 9754, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24723456, 'num_objects': 3018, 'num_object_clones': 0, 'num_object_copies': 12072, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3018, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9754, 'num_write_kb': 51088, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 799}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:42:21.204 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:42:21.204 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'9603", 'reported_seq': 9710, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:18.115988+0000', 'last_change': '2024-09-05T16:34:56.903436+0000', 'last_active': '2024-09-05T16:42:18.115988+0000', 'last_peered': '2024-09-05T16:42:18.115988+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:42:18.115988+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9603, 'log_dups_size': 0, 'ondisk_log_size': 9603, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25056416, 'num_objects': 3059, 'num_object_clones': 0, 'num_object_copies': 12236, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2133, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3059, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9601, 'num_write_kb': 50636, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 928}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:42:21.204 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-05T16:42:21.204 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "37'9749", 'reported_seq': 9837, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:18.028382+0000', 'last_change': '2024-09-05T16:34:56.883319+0000', 'last_active': '2024-09-05T16:42:18.028382+0000', 'last_peered': '2024-09-05T16:42:18.028382+0000', 'last_clean': '2024-09-05T16:33:53.505305+0000', 'last_became_active': '2024-09-05T16:34:56.873389+0000', 'last_became_peered': '2024-09-05T16:34:56.873389+0000', 'last_unstale': '2024-09-05T16:42:18.028382+0000', 'last_undegraded': '2024-09-05T16:34:56.602567+0000', 'last_fullsized': '2024-09-05T16:34:56.559958+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9749, 'log_dups_size': 0, 'ondisk_log_size': 9749, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24818848, 'num_objects': 3030, 'num_object_clones': 0, 'num_object_copies': 12120, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2234, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3030, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9747, 'num_write_kb': 51108, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 798}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:42:21.204 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:42:21.204 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'9748", 'reported_seq': 9848, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:18.334006+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:42:18.334006+0000', 'last_peered': '2024-09-05T16:42:18.334006+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:42:18.334006+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9748, 'log_dups_size': 0, 'ondisk_log_size': 9748, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24783168, 'num_objects': 3026, 'num_object_clones': 0, 'num_object_copies': 12104, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3026, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9745, 'num_write_kb': 51081, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 782}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:42:21.204 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:42:21.205 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'9820", 'reported_seq': 9941, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:18.456311+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:42:18.456311+0000', 'last_peered': '2024-09-05T16:42:18.456311+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:42:18.456311+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9820, 'log_dups_size': 0, 'ondisk_log_size': 9820, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24491168, 'num_objects': 2990, 'num_object_clones': 0, 'num_object_copies': 11960, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2990, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9818, 'num_write_kb': 51230, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 745}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:42:21.205 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:42:21.205 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'9990", 'reported_seq': 10116, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:18.476681+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:42:18.476681+0000', 'last_peered': '2024-09-05T16:42:18.476681+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:42:18.476681+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9990, 'log_dups_size': 0, 'ondisk_log_size': 9990, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25661168, 'num_objects': 3133, 'num_object_clones': 0, 'num_object_copies': 12532, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2284, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3133, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9987, 'num_write_kb': 52476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 852}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:42:21.205 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:42:21.205 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'9733", 'reported_seq': 9831, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:18.466016+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:42:18.466016+0000', 'last_peered': '2024-09-05T16:42:18.466016+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:42:18.466016+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9733, 'log_dups_size': 0, 'ondisk_log_size': 9733, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24787536, 'num_objects': 3026, 'num_object_clones': 0, 'num_object_copies': 12104, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3026, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9732, 'num_write_kb': 51031, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 788}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:42:21.205 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:42:21.205 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'10027", 'reported_seq': 10134, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:18.453010+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:42:18.453010+0000', 'last_peered': '2024-09-05T16:42:18.453010+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:42:18.453010+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10027, 'log_dups_size': 0, 'ondisk_log_size': 10027, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25384096, 'num_objects': 3099, 'num_object_clones': 0, 'num_object_copies': 12396, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3099, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10025, 'num_write_kb': 52500, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 765}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:42:21.205 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:42:21.205 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'9823", 'reported_seq': 9935, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:18.271256+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:42:18.271256+0000', 'last_peered': '2024-09-05T16:42:18.271256+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:42:18.271256+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9823, 'log_dups_size': 0, 'ondisk_log_size': 9823, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24655008, 'num_objects': 3010, 'num_object_clones': 0, 'num_object_copies': 12040, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2310, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3010, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9821, 'num_write_kb': 51325, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 702}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:42:21.205 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:42:21.205 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'9725", 'reported_seq': 9831, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:18.130140+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:42:18.130140+0000', 'last_peered': '2024-09-05T16:42:18.130140+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:42:18.130140+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9725, 'log_dups_size': 0, 'ondisk_log_size': 9725, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24982688, 'num_objects': 3050, 'num_object_clones': 0, 'num_object_copies': 12200, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3050, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9723, 'num_write_kb': 51092, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 830}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:42:21.206 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:42:21.946 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:42:21.946 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 30101/194692 objects degraded (15.461%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":545,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":48673,"data_bytes":398684648,"bytes_used":2874290176,"bytes_avail":383672766464,"bytes_total":386547056640,"degraded_objects":30101,"degraded_total":194692,"degraded_ratio":0.15460830439874262,"recovering_objects_per_sec":9,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":114,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":172},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:42:21.968 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:42:21.969 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:42:22.606 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:42:22.607 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:42:22.629 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":300,"stamp":"2024-09-05T16:42:20.987660+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":398684648,"num_objects":48673,"num_object_clones":0,"num_object_copies":194692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30101,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48673,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":142164,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1536,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156288,"ondisk_log_size":156288,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2806924,"kb_used_data":781472,"kb_used_omap":14,"kb_used_meta":2025329,"kb_avail":374680436,"statfs":{"total":386547056640,"available":383672766464,"internally_reserved":0,"allocated":800227328,"data_stored":799422232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2073937463},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,2,2,4,6,25,15,13],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16990208,"num_objects":-2074,"num_object_clones":0,"num_object_copies":-8296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-114,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2074,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2074,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":114,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004576"},"pg_stats":[{"pgid":"1.f","version":"37'9816","reported_seq":9920,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:18.420833+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:42:18.420833+0000","last_peered":"2024-09-05T16:42:18.420833+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:42:18.420833+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9816,"log_dups_size":0,"ondisk_log_size":9816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25296896,"num_objects":3088,"num_object_clones":0,"num_object_copies":12352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3088,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9816,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":852}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'9592","reported_seq":12177,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:18.157975+0000","last_change":"2024-09-05T16:40:28.919189+0000","last_active":"2024-09-05T16:42:18.157975+0000","last_peered":"2024-09-05T16:42:18.157975+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:42:18.157975+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9592,"log_dups_size":0,"ondisk_log_size":9592,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24261792,"num_objects":2962,"num_object_clones":0,"num_object_copies":11848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1014,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2962,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9590,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1257,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1948}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'9746","reported_seq":9868,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:18.523135+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:42:18.523135+0000","last_peered":"2024-09-05T16:42:18.523135+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:42:18.523135+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9746,"log_dups_size":0,"ondisk_log_size":9746,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24623720,"num_objects":3007,"num_object_clones":0,"num_object_copies":12028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3007,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9745,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":775}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'9840","reported_seq":14569,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:42:17.732829+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:42:17.732829+0000","last_peered":"2024-09-05T16:42:17.732829+0000","last_clean":"2024-09-05T16:42:17.732829+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:42:17.732829+0000","last_undegraded":"2024-09-05T16:42:17.732829+0000","last_fullsized":"2024-09-05T16:42:17.732829+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":9840,"log_dups_size":0,"ondisk_log_size":9840,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25346048,"num_objects":3094,"num_object_clones":0,"num_object_copies":12376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3094,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4140,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'9563","reported_seq":9659,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:18.454437+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:42:18.454437+0000","last_peered":"2024-09-05T16:42:18.454437+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:42:18.454437+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9563,"log_dups_size":0,"ondisk_log_size":9563,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24835232,"num_objects":3032,"num_object_clones":0,"num_object_copies":12128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3032,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9561,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":887}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'9754","reported_seq":9867,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:18.481053+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:42:18.481053+0000","last_peered":"2024-09-05T16:42:18.481053+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:42:18.481053+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9754,"log_dups_size":0,"ondisk_log_size":9754,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24723456,"num_objects":3018,"num_object_clones":0,"num_object_copies":12072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3018,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9754,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":799}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'9603","reported_seq":9710,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:18.115988+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:42:18.115988+0000","last_peered":"2024-09-05T16:42:18.115988+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:42:18.115988+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9603,"log_dups_size":0,"ondisk_log_size":9603,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25056416,"num_objects":3059,"num_object_clones":0,"num_object_copies":12236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3059,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9601,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":928}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'9749","reported_seq":9837,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:18.028382+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:42:18.028382+0000","last_peered":"2024-09-05T16:42:18.028382+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:42:18.028382+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9749,"log_dups_size":0,"ondisk_log_size":9749,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24818848,"num_objects":3030,"num_object_clones":0,"num_object_copies":12120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3030,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9747,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":798}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'9748","reported_seq":9848,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:18.334006+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:42:18.334006+0000","last_peered":"2024-09-05T16:42:18.334006+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:42:18.334006+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9748,"log_dups_size":0,"ondisk_log_size":9748,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24783168,"num_objects":3026,"num_object_clones":0,"num_object_copies":12104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3026,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9745,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":782}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'9820","reported_seq":9941,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:18.456311+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:42:18.456311+0000","last_peered":"2024-09-05T16:42:18.456311+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:42:18.456311+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9820,"log_dups_size":0,"ondisk_log_size":9820,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24491168,"num_objects":2990,"num_object_clones":0,"num_object_copies":11960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2990,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9818,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":745}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'9990","reported_seq":10116,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:18.476681+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:42:18.476681+0000","last_peered":"2024-09-05T16:42:18.476681+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:42:18.476681+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9990,"log_dups_size":0,"ondisk_log_size":9990,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25661168,"num_objects":3133,"num_object_clones":0,"num_object_copies":12532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3133,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9987,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":852}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'9733","reported_seq":9831,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:18.466016+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:42:18.466016+0000","last_peered":"2024-09-05T16:42:18.466016+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:42:18.466016+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9733,"log_dups_size":0,"ondisk_log_size":9733,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24787536,"num_objects":3026,"num_object_clones":0,"num_object_copies":12104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3026,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9732,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":788}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'9759","reported_seq":14494,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:42:17.844961+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:42:17.844961+0000","last_peered":"2024-09-05T16:42:17.844961+0000","last_clean":"2024-09-05T16:42:17.844961+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:42:17.844961+0000","last_undegraded":"2024-09-05T16:42:17.844961+0000","last_fullsized":"2024-09-05T16:42:17.844961+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":9759,"log_dups_size":0,"ondisk_log_size":9759,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24977408,"num_objects":3049,"num_object_clones":0,"num_object_copies":12196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3049,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1359,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'10027","reported_seq":10134,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:18.453010+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:42:18.453010+0000","last_peered":"2024-09-05T16:42:18.453010+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:42:18.453010+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25384096,"num_objects":3099,"num_object_clones":0,"num_object_copies":12396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3099,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10025,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":765}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'9823","reported_seq":9935,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:18.271256+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:42:18.271256+0000","last_peered":"2024-09-05T16:42:18.271256+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:42:18.271256+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9823,"log_dups_size":0,"ondisk_log_size":9823,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24655008,"num_objects":3010,"num_object_clones":0,"num_object_copies":12040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3010,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9821,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":702}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'9725","reported_seq":9831,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:18.130140+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:42:18.130140+0000","last_peered":"2024-09-05T16:42:18.130140+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:42:18.130140+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9725,"log_dups_size":0,"ondisk_log_size":9725,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24982688,"num_objects":3050,"num_object_clones":0,"num_object_copies":12200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3050,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9723,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":830}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":398684648,"num_objects":48673,"num_object_clones":0,"num_object_copies":194692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30101,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48673,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":142164,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1536,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":797663232,"data_stored":797663232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156288,"ondisk_log_size":156288,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084388,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":732192,"kb_used_data":195144,"kb_used_omap":4,"kb_used_meta":537019,"kb_avail":93639648,"statfs":{"total":96636764160,"available":95886999552,"internally_reserved":0,"allocated":199827456,"data_stored":199626182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":549908179},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,1,3,9,4,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986265,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":602336,"kb_used_data":195756,"kb_used_omap":4,"kb_used_meta":406523,"kb_avail":93769504,"statfs":{"total":96636764160,"available":96019972096,"internally_reserved":0,"allocated":200454144,"data_stored":200252870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":416280274},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,1,4,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673071,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":736140,"kb_used_data":195452,"kb_used_omap":1,"kb_used_meta":540670,"kb_avail":93635700,"statfs":{"total":96636764160,"available":95882956800,"internally_reserved":0,"allocated":200142848,"data_stored":199941574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":553646533},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,7,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182502,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":736256,"kb_used_data":195120,"kb_used_omap":4,"kb_used_meta":541115,"kb_avail":93635584,"statfs":{"total":96636764160,"available":95882838016,"internally_reserved":0,"allocated":199802880,"data_stored":199601606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":554102477},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,2,0,0,8,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":199462912,"data_stored":199462912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":199929856,"data_stored":199929856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":199098368,"data_stored":199098368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":199172096,"data_stored":199172096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:42:25.631 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:42:26.292 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:42:26.292 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:42:26.314 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":302,"stamp":"2024-09-05T16:42:24.989297+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":390222312,"num_objects":47640,"num_object_clones":0,"num_object_copies":190560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47640,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":143197,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1596,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157321,"ondisk_log_size":157321,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2911452,"kb_used_data":765260,"kb_used_omap":14,"kb_used_meta":2146097,"kb_avail":374575908,"statfs":{"total":386547056640,"available":383565729792,"internally_reserved":0,"allocated":783626240,"data_stored":782821144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2197603895},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,2,8,11,13,10,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24322048,"num_objects":-2969,"num_object_clones":0,"num_object_copies":-11876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-174,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2969,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2969,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":174,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004819"},"pg_stats":[{"pgid":"1.f","version":"37'9886","reported_seq":9988,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:23.407097+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:42:23.407097+0000","last_peered":"2024-09-05T16:42:23.407097+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:42:23.407097+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9886,"log_dups_size":0,"ondisk_log_size":9886,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24723456,"num_objects":3018,"num_object_clones":0,"num_object_copies":12072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3018,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9886,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":782}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'9652","reported_seq":12357,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:23.282391+0000","last_change":"2024-09-05T16:40:28.919189+0000","last_active":"2024-09-05T16:42:23.282391+0000","last_peered":"2024-09-05T16:42:23.282391+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:42:23.282391+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9652,"log_dups_size":0,"ondisk_log_size":9652,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23770272,"num_objects":2902,"num_object_clones":0,"num_object_copies":11608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":954,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2902,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9650,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1317,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1948}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'9819","reported_seq":9941,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:23.554566+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:42:23.554566+0000","last_peered":"2024-09-05T16:42:23.554566+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:42:23.554566+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9819,"log_dups_size":0,"ondisk_log_size":9819,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24025704,"num_objects":2934,"num_object_clones":0,"num_object_copies":11736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2934,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9818,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":702}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'9909","reported_seq":14638,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:42:22.957917+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:42:22.957917+0000","last_peered":"2024-09-05T16:42:22.957917+0000","last_clean":"2024-09-05T16:42:22.957917+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:42:22.957917+0000","last_undegraded":"2024-09-05T16:42:22.957917+0000","last_fullsized":"2024-09-05T16:42:22.957917+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":9909,"log_dups_size":0,"ondisk_log_size":9909,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24780800,"num_objects":3025,"num_object_clones":0,"num_object_copies":12100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3025,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4209,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'9617","reported_seq":9716,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:23.460393+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:42:23.460393+0000","last_peered":"2024-09-05T16:42:23.460393+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:42:23.460393+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9617,"log_dups_size":0,"ondisk_log_size":9617,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24392864,"num_objects":2978,"num_object_clones":0,"num_object_copies":11912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2978,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9615,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":833}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'9815","reported_seq":9929,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:23.226892+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:42:23.226892+0000","last_peered":"2024-09-05T16:42:23.226892+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:42:23.226892+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9815,"log_dups_size":0,"ondisk_log_size":9815,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24223744,"num_objects":2957,"num_object_clones":0,"num_object_copies":11828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2957,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9815,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":738}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'9664","reported_seq":9770,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:23.240564+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:42:23.240564+0000","last_peered":"2024-09-05T16:42:23.240564+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:42:23.240564+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9664,"log_dups_size":0,"ondisk_log_size":9664,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24556704,"num_objects":2998,"num_object_clones":0,"num_object_copies":11992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2998,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9662,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":867}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'9812","reported_seq":9900,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:23.084814+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:42:23.084814+0000","last_peered":"2024-09-05T16:42:23.084814+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:42:23.084814+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9812,"log_dups_size":0,"ondisk_log_size":9812,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24302752,"num_objects":2967,"num_object_clones":0,"num_object_copies":11868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2967,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9810,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":735}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'9818","reported_seq":9918,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:23.489225+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:42:23.489225+0000","last_peered":"2024-09-05T16:42:23.489225+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:42:23.489225+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9818,"log_dups_size":0,"ondisk_log_size":9818,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24209728,"num_objects":2956,"num_object_clones":0,"num_object_copies":11824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9815,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":712}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'9885","reported_seq":10007,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:23.568680+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:42:23.568680+0000","last_peered":"2024-09-05T16:42:23.568680+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:42:23.568680+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9885,"log_dups_size":0,"ondisk_log_size":9885,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23958688,"num_objects":2925,"num_object_clones":0,"num_object_copies":11700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2925,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9883,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":680}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'10057","reported_seq":10181,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:23.552655+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:42:23.552655+0000","last_peered":"2024-09-05T16:42:23.552655+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:42:23.552655+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25112304,"num_objects":3066,"num_object_clones":0,"num_object_copies":12264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3066,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10054,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":785}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'9790","reported_seq":9888,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:23.506699+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:42:23.506699+0000","last_peered":"2024-09-05T16:42:23.506699+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:42:23.506699+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9790,"log_dups_size":0,"ondisk_log_size":9790,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24320592,"num_objects":2969,"num_object_clones":0,"num_object_copies":11876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2969,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9789,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":731}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'9819","reported_seq":14554,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:42:22.964887+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:42:22.964887+0000","last_peered":"2024-09-05T16:42:22.964887+0000","last_clean":"2024-09-05T16:42:22.964887+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:42:22.964887+0000","last_undegraded":"2024-09-05T16:42:22.964887+0000","last_fullsized":"2024-09-05T16:42:22.964887+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":9819,"log_dups_size":0,"ondisk_log_size":9819,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24485888,"num_objects":2989,"num_object_clones":0,"num_object_copies":11956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2989,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1419,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'10095","reported_seq":10203,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:23.497446+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:42:23.497446+0000","last_peered":"2024-09-05T16:42:23.497446+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:42:23.497446+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24827040,"num_objects":3031,"num_object_clones":0,"num_object_copies":12124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3031,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10093,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":697}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'9901","reported_seq":10012,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:23.676203+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:42:23.676203+0000","last_peered":"2024-09-05T16:42:23.676203+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:42:23.676203+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9901,"log_dups_size":0,"ondisk_log_size":9901,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24016032,"num_objects":2932,"num_object_clones":0,"num_object_copies":11728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2932,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9899,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":624}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'9782","reported_seq":9890,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:23.083059+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:42:23.083059+0000","last_peered":"2024-09-05T16:42:23.083059+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:42:23.083059+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9782,"log_dups_size":0,"ondisk_log_size":9782,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24515744,"num_objects":2993,"num_object_clones":0,"num_object_copies":11972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2993,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9780,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":773}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":390222312,"num_objects":47640,"num_object_clones":0,"num_object_copies":190560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47640,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":143197,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1596,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":780726272,"data_stored":780726272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157321,"ondisk_log_size":157321,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084389,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":764296,"kb_used_data":191156,"kb_used_omap":4,"kb_used_meta":573115,"kb_avail":93607544,"statfs":{"total":96636764160,"available":95854125056,"internally_reserved":0,"allocated":195743744,"data_stored":195542470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":586870483},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,2,3,4,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986266,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":602300,"kb_used_data":191596,"kb_used_omap":4,"kb_used_meta":410683,"kb_avail":93769540,"statfs":{"total":96636764160,"available":96020008960,"internally_reserved":0,"allocated":196194304,"data_stored":195993030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":420540114},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,4,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673072,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":772416,"kb_used_data":191472,"kb_used_omap":1,"kb_used_meta":580926,"kb_avail":93599424,"statfs":{"total":96636764160,"available":95845810176,"internally_reserved":0,"allocated":196067328,"data_stored":195866054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":594868677},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,2,0,2,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182503,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":772440,"kb_used_data":191036,"kb_used_omap":4,"kb_used_meta":581371,"kb_avail":93599400,"statfs":{"total":96636764160,"available":95845785600,"internally_reserved":0,"allocated":195620864,"data_stored":195419590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":595324621},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,3,3,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":195244032,"data_stored":195244032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":195588096,"data_stored":195588096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":194904064,"data_stored":194904064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":194990080,"data_stored":194990080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:42:26.315 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:42:26.975 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:42:26.975 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:42:26.995 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":302,"stamp":"2024-09-05T16:42:24.989297+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":390222312,"num_objects":47640,"num_object_clones":0,"num_object_copies":190560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47640,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":143197,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1596,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157321,"ondisk_log_size":157321,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2911452,"kb_used_data":765260,"kb_used_omap":14,"kb_used_meta":2146097,"kb_avail":374575908,"statfs":{"total":386547056640,"available":383565729792,"internally_reserved":0,"allocated":783626240,"data_stored":782821144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2197603895},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,2,8,11,13,10,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24322048,"num_objects":-2969,"num_object_clones":0,"num_object_copies":-11876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-174,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2969,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2969,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":174,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004819"},"pg_stats":[{"pgid":"1.f","version":"37'9886","reported_seq":9988,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:23.407097+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:42:23.407097+0000","last_peered":"2024-09-05T16:42:23.407097+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:42:23.407097+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9886,"log_dups_size":0,"ondisk_log_size":9886,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24723456,"num_objects":3018,"num_object_clones":0,"num_object_copies":12072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3018,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9886,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":782}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'9652","reported_seq":12357,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:23.282391+0000","last_change":"2024-09-05T16:40:28.919189+0000","last_active":"2024-09-05T16:42:23.282391+0000","last_peered":"2024-09-05T16:42:23.282391+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:42:23.282391+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9652,"log_dups_size":0,"ondisk_log_size":9652,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23770272,"num_objects":2902,"num_object_clones":0,"num_object_copies":11608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":954,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2902,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9650,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1317,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1948}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'9819","reported_seq":9941,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:23.554566+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:42:23.554566+0000","last_peered":"2024-09-05T16:42:23.554566+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:42:23.554566+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9819,"log_dups_size":0,"ondisk_log_size":9819,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24025704,"num_objects":2934,"num_object_clones":0,"num_object_copies":11736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2934,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9818,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":702}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'9909","reported_seq":14638,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:42:22.957917+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:42:22.957917+0000","last_peered":"2024-09-05T16:42:22.957917+0000","last_clean":"2024-09-05T16:42:22.957917+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:42:22.957917+0000","last_undegraded":"2024-09-05T16:42:22.957917+0000","last_fullsized":"2024-09-05T16:42:22.957917+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":9909,"log_dups_size":0,"ondisk_log_size":9909,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24780800,"num_objects":3025,"num_object_clones":0,"num_object_copies":12100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3025,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4209,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'9617","reported_seq":9716,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:23.460393+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:42:23.460393+0000","last_peered":"2024-09-05T16:42:23.460393+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:42:23.460393+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9617,"log_dups_size":0,"ondisk_log_size":9617,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24392864,"num_objects":2978,"num_object_clones":0,"num_object_copies":11912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2978,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9615,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":833}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'9815","reported_seq":9929,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:23.226892+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:42:23.226892+0000","last_peered":"2024-09-05T16:42:23.226892+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:42:23.226892+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9815,"log_dups_size":0,"ondisk_log_size":9815,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24223744,"num_objects":2957,"num_object_clones":0,"num_object_copies":11828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2957,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9815,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":738}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'9664","reported_seq":9770,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:23.240564+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:42:23.240564+0000","last_peered":"2024-09-05T16:42:23.240564+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:42:23.240564+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9664,"log_dups_size":0,"ondisk_log_size":9664,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24556704,"num_objects":2998,"num_object_clones":0,"num_object_copies":11992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2998,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9662,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":867}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'9812","reported_seq":9900,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:23.084814+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:42:23.084814+0000","last_peered":"2024-09-05T16:42:23.084814+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:42:23.084814+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9812,"log_dups_size":0,"ondisk_log_size":9812,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24302752,"num_objects":2967,"num_object_clones":0,"num_object_copies":11868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2967,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9810,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":735}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'9818","reported_seq":9918,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:23.489225+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:42:23.489225+0000","last_peered":"2024-09-05T16:42:23.489225+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:42:23.489225+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9818,"log_dups_size":0,"ondisk_log_size":9818,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24209728,"num_objects":2956,"num_object_clones":0,"num_object_copies":11824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9815,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":712}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'9885","reported_seq":10007,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:23.568680+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:42:23.568680+0000","last_peered":"2024-09-05T16:42:23.568680+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:42:23.568680+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9885,"log_dups_size":0,"ondisk_log_size":9885,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23958688,"num_objects":2925,"num_object_clones":0,"num_object_copies":11700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2925,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9883,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":680}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'10057","reported_seq":10181,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:23.552655+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:42:23.552655+0000","last_peered":"2024-09-05T16:42:23.552655+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:42:23.552655+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25112304,"num_objects":3066,"num_object_clones":0,"num_object_copies":12264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3066,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10054,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":785}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'9790","reported_seq":9888,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:23.506699+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:42:23.506699+0000","last_peered":"2024-09-05T16:42:23.506699+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:42:23.506699+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9790,"log_dups_size":0,"ondisk_log_size":9790,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24320592,"num_objects":2969,"num_object_clones":0,"num_object_copies":11876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2969,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9789,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":731}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'9819","reported_seq":14554,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:42:22.964887+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:42:22.964887+0000","last_peered":"2024-09-05T16:42:22.964887+0000","last_clean":"2024-09-05T16:42:22.964887+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:42:22.964887+0000","last_undegraded":"2024-09-05T16:42:22.964887+0000","last_fullsized":"2024-09-05T16:42:22.964887+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":9819,"log_dups_size":0,"ondisk_log_size":9819,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24485888,"num_objects":2989,"num_object_clones":0,"num_object_copies":11956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2989,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1419,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'10095","reported_seq":10203,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:23.497446+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:42:23.497446+0000","last_peered":"2024-09-05T16:42:23.497446+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:42:23.497446+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24827040,"num_objects":3031,"num_object_clones":0,"num_object_copies":12124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3031,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10093,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":697}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'9901","reported_seq":10012,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:23.676203+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:42:23.676203+0000","last_peered":"2024-09-05T16:42:23.676203+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:42:23.676203+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9901,"log_dups_size":0,"ondisk_log_size":9901,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24016032,"num_objects":2932,"num_object_clones":0,"num_object_copies":11728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2932,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9899,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":624}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'9782","reported_seq":9890,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:23.083059+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:42:23.083059+0000","last_peered":"2024-09-05T16:42:23.083059+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:42:23.083059+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9782,"log_dups_size":0,"ondisk_log_size":9782,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24515744,"num_objects":2993,"num_object_clones":0,"num_object_copies":11972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2993,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9780,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":773}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":390222312,"num_objects":47640,"num_object_clones":0,"num_object_copies":190560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47640,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":143197,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1596,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":780726272,"data_stored":780726272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157321,"ondisk_log_size":157321,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084389,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":764296,"kb_used_data":191156,"kb_used_omap":4,"kb_used_meta":573115,"kb_avail":93607544,"statfs":{"total":96636764160,"available":95854125056,"internally_reserved":0,"allocated":195743744,"data_stored":195542470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":586870483},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,2,3,4,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986266,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":602300,"kb_used_data":191596,"kb_used_omap":4,"kb_used_meta":410683,"kb_avail":93769540,"statfs":{"total":96636764160,"available":96020008960,"internally_reserved":0,"allocated":196194304,"data_stored":195993030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":420540114},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,4,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673072,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":772416,"kb_used_data":191472,"kb_used_omap":1,"kb_used_meta":580926,"kb_avail":93599424,"statfs":{"total":96636764160,"available":95845810176,"internally_reserved":0,"allocated":196067328,"data_stored":195866054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":594868677},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,2,0,2,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182503,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":772440,"kb_used_data":191036,"kb_used_omap":4,"kb_used_meta":581371,"kb_avail":93599400,"statfs":{"total":96636764160,"available":95845785600,"internally_reserved":0,"allocated":195620864,"data_stored":195419590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":595324621},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,3,3,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":195244032,"data_stored":195244032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":195588096,"data_stored":195588096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":194904064,"data_stored":194904064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":194990080,"data_stored":194990080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:42:26.996 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-05T16:42:26.996 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'9886", 'reported_seq': 9988, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:23.407097+0000', 'last_change': '2024-09-05T16:34:56.833497+0000', 'last_active': '2024-09-05T16:42:23.407097+0000', 'last_peered': '2024-09-05T16:42:23.407097+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.828289+0000', 'last_became_peered': '2024-09-05T16:34:56.828289+0000', 'last_unstale': '2024-09-05T16:42:23.407097+0000', 'last_undegraded': '2024-09-05T16:34:56.612160+0000', 'last_fullsized': '2024-09-05T16:34:56.557632+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9886, 'log_dups_size': 0, 'ondisk_log_size': 9886, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24723456, 'num_objects': 3018, 'num_object_clones': 0, 'num_object_copies': 12072, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2236, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3018, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9886, 'num_write_kb': 51616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 782}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:42:26.996 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-05T16:42:26.996 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "37'9652", 'reported_seq': 12357, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:23.282391+0000', 'last_change': '2024-09-05T16:40:28.919189+0000', 'last_active': '2024-09-05T16:42:23.282391+0000', 'last_peered': '2024-09-05T16:42:23.282391+0000', 'last_clean': '2024-09-05T16:33:53.200762+0000', 'last_became_active': '2024-09-05T16:34:56.820292+0000', 'last_became_peered': '2024-09-05T16:34:56.820292+0000', 'last_unstale': '2024-09-05T16:42:23.282391+0000', 'last_undegraded': '2024-09-05T16:34:56.605776+0000', 'last_fullsized': '2024-09-05T16:34:56.559383+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9652, 'log_dups_size': 0, 'ondisk_log_size': 9652, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23770272, 'num_objects': 2902, 'num_object_clones': 0, 'num_object_copies': 11608, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 954, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2902, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9650, 'num_write_kb': 50212, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1317, 'num_bytes_recovered': 2613248, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 1948}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:42:26.996 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:42:26.996 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'9819", 'reported_seq': 9941, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:23.554566+0000', 'last_change': '2024-09-05T16:34:56.867900+0000', 'last_active': '2024-09-05T16:42:23.554566+0000', 'last_peered': '2024-09-05T16:42:23.554566+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:42:23.554566+0000', 'last_undegraded': '2024-09-05T16:34:56.590972+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9819, 'log_dups_size': 0, 'ondisk_log_size': 9819, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24025704, 'num_objects': 2934, 'num_object_clones': 0, 'num_object_copies': 11736, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2233, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2934, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 9818, 'num_write_kb': 50999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 702}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:42:26.996 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:42:26.996 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'9617", 'reported_seq': 9716, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:23.460393+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:42:23.460393+0000', 'last_peered': '2024-09-05T16:42:23.460393+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:42:23.460393+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9617, 'log_dups_size': 0, 'ondisk_log_size': 9617, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24392864, 'num_objects': 2978, 'num_object_clones': 0, 'num_object_copies': 11912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2147, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2978, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9615, 'num_write_kb': 50373, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 833}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:42:26.996 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:42:26.997 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'9815", 'reported_seq': 9929, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:23.226892+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:42:23.226892+0000', 'last_peered': '2024-09-05T16:42:23.226892+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:42:23.226892+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9815, 'log_dups_size': 0, 'ondisk_log_size': 9815, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24223744, 'num_objects': 2957, 'num_object_clones': 0, 'num_object_copies': 11828, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2957, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9815, 'num_write_kb': 51088, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 738}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:42:26.997 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:42:26.997 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'9664", 'reported_seq': 9770, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:23.240564+0000', 'last_change': '2024-09-05T16:34:56.903436+0000', 'last_active': '2024-09-05T16:42:23.240564+0000', 'last_peered': '2024-09-05T16:42:23.240564+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:42:23.240564+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9664, 'log_dups_size': 0, 'ondisk_log_size': 9664, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24556704, 'num_objects': 2998, 'num_object_clones': 0, 'num_object_copies': 11992, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2133, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2998, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9662, 'num_write_kb': 50636, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 867}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:42:26.997 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-05T16:42:26.997 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "37'9812", 'reported_seq': 9900, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:23.084814+0000', 'last_change': '2024-09-05T16:34:56.883319+0000', 'last_active': '2024-09-05T16:42:23.084814+0000', 'last_peered': '2024-09-05T16:42:23.084814+0000', 'last_clean': '2024-09-05T16:33:53.505305+0000', 'last_became_active': '2024-09-05T16:34:56.873389+0000', 'last_became_peered': '2024-09-05T16:34:56.873389+0000', 'last_unstale': '2024-09-05T16:42:23.084814+0000', 'last_undegraded': '2024-09-05T16:34:56.602567+0000', 'last_fullsized': '2024-09-05T16:34:56.559958+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9812, 'log_dups_size': 0, 'ondisk_log_size': 9812, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24302752, 'num_objects': 2967, 'num_object_clones': 0, 'num_object_copies': 11868, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2234, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2967, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9810, 'num_write_kb': 51108, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 735}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:42:26.997 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:42:26.997 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'9818", 'reported_seq': 9918, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:23.489225+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:42:23.489225+0000', 'last_peered': '2024-09-05T16:42:23.489225+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:42:23.489225+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9818, 'log_dups_size': 0, 'ondisk_log_size': 9818, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24209728, 'num_objects': 2956, 'num_object_clones': 0, 'num_object_copies': 11824, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2956, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9815, 'num_write_kb': 51081, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 712}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:42:26.997 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:42:26.997 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'9885", 'reported_seq': 10007, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:23.568680+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:42:23.568680+0000', 'last_peered': '2024-09-05T16:42:23.568680+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:42:23.568680+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9885, 'log_dups_size': 0, 'ondisk_log_size': 9885, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23958688, 'num_objects': 2925, 'num_object_clones': 0, 'num_object_copies': 11700, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2925, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9883, 'num_write_kb': 51230, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 680}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:42:26.997 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:42:26.997 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'10057", 'reported_seq': 10181, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:23.552655+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:42:23.552655+0000', 'last_peered': '2024-09-05T16:42:23.552655+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:42:23.552655+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10057, 'log_dups_size': 0, 'ondisk_log_size': 10057, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25112304, 'num_objects': 3066, 'num_object_clones': 0, 'num_object_copies': 12264, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2284, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3066, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10054, 'num_write_kb': 52476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 785}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:42:26.998 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:42:26.998 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'9790", 'reported_seq': 9888, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:23.506699+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:42:23.506699+0000', 'last_peered': '2024-09-05T16:42:23.506699+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:42:23.506699+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9790, 'log_dups_size': 0, 'ondisk_log_size': 9790, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24320592, 'num_objects': 2969, 'num_object_clones': 0, 'num_object_copies': 11876, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2969, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9789, 'num_write_kb': 51031, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 731}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:42:26.998 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:42:26.998 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'10095", 'reported_seq': 10203, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:23.497446+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:42:23.497446+0000', 'last_peered': '2024-09-05T16:42:23.497446+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:42:23.497446+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10095, 'log_dups_size': 0, 'ondisk_log_size': 10095, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24827040, 'num_objects': 3031, 'num_object_clones': 0, 'num_object_copies': 12124, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3031, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10093, 'num_write_kb': 52500, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 697}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:42:26.998 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:42:26.998 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'9901", 'reported_seq': 10012, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:23.676203+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:42:23.676203+0000', 'last_peered': '2024-09-05T16:42:23.676203+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:42:23.676203+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9901, 'log_dups_size': 0, 'ondisk_log_size': 9901, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24016032, 'num_objects': 2932, 'num_object_clones': 0, 'num_object_copies': 11728, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2310, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2932, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9899, 'num_write_kb': 51325, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 624}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:42:26.998 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:42:26.998 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'9782", 'reported_seq': 9890, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:23.083059+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:42:23.083059+0000', 'last_peered': '2024-09-05T16:42:23.083059+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:42:23.083059+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9782, 'log_dups_size': 0, 'ondisk_log_size': 9782, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24515744, 'num_objects': 2993, 'num_object_clones': 0, 'num_object_copies': 11972, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2993, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9780, 'num_write_kb': 51092, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 773}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:42:26.998 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:42:27.744 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:42:27.744 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 30041/190560 objects degraded (15.765%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":550,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":47640,"data_bytes":390222312,"bytes_used":2981326848,"bytes_avail":383565729792,"bytes_total":386547056640,"degraded_objects":30041,"degraded_total":190560,"degraded_ratio":0.1576458858102435,"recovering_objects_per_sec":9,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":112,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":172},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:42:27.766 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:42:27.766 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:42:28.339 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:42:28.339 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:42:28.369 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":303,"stamp":"2024-09-05T16:42:26.990190+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":390222312,"num_objects":47640,"num_object_clones":0,"num_object_copies":190560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47640,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":143197,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1596,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157321,"ondisk_log_size":157321,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2911452,"kb_used_data":765260,"kb_used_omap":14,"kb_used_meta":2146097,"kb_avail":374575908,"statfs":{"total":386547056640,"available":383565729792,"internally_reserved":0,"allocated":783626240,"data_stored":782821144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2197603895},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,2,8,11,13,10,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16941056,"num_objects":-2068,"num_object_clones":0,"num_object_copies":-8272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2068,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2068,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":112,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004892"},"pg_stats":[{"pgid":"1.f","version":"37'9886","reported_seq":9988,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:23.407097+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:42:23.407097+0000","last_peered":"2024-09-05T16:42:23.407097+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:42:23.407097+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9886,"log_dups_size":0,"ondisk_log_size":9886,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24723456,"num_objects":3018,"num_object_clones":0,"num_object_copies":12072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3018,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9886,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":782}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'9652","reported_seq":12357,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:23.282391+0000","last_change":"2024-09-05T16:40:28.919189+0000","last_active":"2024-09-05T16:42:23.282391+0000","last_peered":"2024-09-05T16:42:23.282391+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:42:23.282391+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9652,"log_dups_size":0,"ondisk_log_size":9652,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23770272,"num_objects":2902,"num_object_clones":0,"num_object_copies":11608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":954,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2902,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9650,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1317,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1948}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'9819","reported_seq":9941,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:23.554566+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:42:23.554566+0000","last_peered":"2024-09-05T16:42:23.554566+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:42:23.554566+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9819,"log_dups_size":0,"ondisk_log_size":9819,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24025704,"num_objects":2934,"num_object_clones":0,"num_object_copies":11736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2934,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9818,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":702}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'9909","reported_seq":14638,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:42:22.957917+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:42:22.957917+0000","last_peered":"2024-09-05T16:42:22.957917+0000","last_clean":"2024-09-05T16:42:22.957917+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:42:22.957917+0000","last_undegraded":"2024-09-05T16:42:22.957917+0000","last_fullsized":"2024-09-05T16:42:22.957917+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":9909,"log_dups_size":0,"ondisk_log_size":9909,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24780800,"num_objects":3025,"num_object_clones":0,"num_object_copies":12100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3025,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4209,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'9617","reported_seq":9716,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:23.460393+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:42:23.460393+0000","last_peered":"2024-09-05T16:42:23.460393+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:42:23.460393+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9617,"log_dups_size":0,"ondisk_log_size":9617,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24392864,"num_objects":2978,"num_object_clones":0,"num_object_copies":11912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2978,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9615,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":833}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'9815","reported_seq":9929,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:23.226892+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:42:23.226892+0000","last_peered":"2024-09-05T16:42:23.226892+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:42:23.226892+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9815,"log_dups_size":0,"ondisk_log_size":9815,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24223744,"num_objects":2957,"num_object_clones":0,"num_object_copies":11828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2957,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9815,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":738}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'9664","reported_seq":9770,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:23.240564+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:42:23.240564+0000","last_peered":"2024-09-05T16:42:23.240564+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:42:23.240564+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9664,"log_dups_size":0,"ondisk_log_size":9664,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24556704,"num_objects":2998,"num_object_clones":0,"num_object_copies":11992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2998,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9662,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":867}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'9812","reported_seq":9900,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:23.084814+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:42:23.084814+0000","last_peered":"2024-09-05T16:42:23.084814+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:42:23.084814+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9812,"log_dups_size":0,"ondisk_log_size":9812,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24302752,"num_objects":2967,"num_object_clones":0,"num_object_copies":11868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2967,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9810,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":735}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'9818","reported_seq":9918,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:23.489225+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:42:23.489225+0000","last_peered":"2024-09-05T16:42:23.489225+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:42:23.489225+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9818,"log_dups_size":0,"ondisk_log_size":9818,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24209728,"num_objects":2956,"num_object_clones":0,"num_object_copies":11824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9815,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":712}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'9885","reported_seq":10007,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:23.568680+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:42:23.568680+0000","last_peered":"2024-09-05T16:42:23.568680+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:42:23.568680+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9885,"log_dups_size":0,"ondisk_log_size":9885,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23958688,"num_objects":2925,"num_object_clones":0,"num_object_copies":11700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2925,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9883,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":680}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'10057","reported_seq":10181,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:23.552655+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:42:23.552655+0000","last_peered":"2024-09-05T16:42:23.552655+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:42:23.552655+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25112304,"num_objects":3066,"num_object_clones":0,"num_object_copies":12264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3066,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10054,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":785}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'9790","reported_seq":9888,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:23.506699+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:42:23.506699+0000","last_peered":"2024-09-05T16:42:23.506699+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:42:23.506699+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9790,"log_dups_size":0,"ondisk_log_size":9790,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24320592,"num_objects":2969,"num_object_clones":0,"num_object_copies":11876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2969,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9789,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":731}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'9819","reported_seq":14554,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:42:22.964887+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:42:22.964887+0000","last_peered":"2024-09-05T16:42:22.964887+0000","last_clean":"2024-09-05T16:42:22.964887+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:42:22.964887+0000","last_undegraded":"2024-09-05T16:42:22.964887+0000","last_fullsized":"2024-09-05T16:42:22.964887+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":9819,"log_dups_size":0,"ondisk_log_size":9819,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24485888,"num_objects":2989,"num_object_clones":0,"num_object_copies":11956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2989,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1419,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'10095","reported_seq":10203,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:23.497446+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:42:23.497446+0000","last_peered":"2024-09-05T16:42:23.497446+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:42:23.497446+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24827040,"num_objects":3031,"num_object_clones":0,"num_object_copies":12124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3031,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10093,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":697}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'9901","reported_seq":10012,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:23.676203+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:42:23.676203+0000","last_peered":"2024-09-05T16:42:23.676203+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:42:23.676203+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9901,"log_dups_size":0,"ondisk_log_size":9901,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24016032,"num_objects":2932,"num_object_clones":0,"num_object_copies":11728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2932,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9899,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":624}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'9782","reported_seq":9890,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:23.083059+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:42:23.083059+0000","last_peered":"2024-09-05T16:42:23.083059+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:42:23.083059+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9782,"log_dups_size":0,"ondisk_log_size":9782,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24515744,"num_objects":2993,"num_object_clones":0,"num_object_copies":11972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2993,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9780,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":773}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":390222312,"num_objects":47640,"num_object_clones":0,"num_object_copies":190560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47640,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":143197,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1596,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":780726272,"data_stored":780726272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157321,"ondisk_log_size":157321,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084389,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":764296,"kb_used_data":191156,"kb_used_omap":4,"kb_used_meta":573115,"kb_avail":93607544,"statfs":{"total":96636764160,"available":95854125056,"internally_reserved":0,"allocated":195743744,"data_stored":195542470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":586870483},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,2,3,4,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986266,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":602300,"kb_used_data":191596,"kb_used_omap":4,"kb_used_meta":410683,"kb_avail":93769540,"statfs":{"total":96636764160,"available":96020008960,"internally_reserved":0,"allocated":196194304,"data_stored":195993030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":420540114},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,4,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673072,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":772416,"kb_used_data":191472,"kb_used_omap":1,"kb_used_meta":580926,"kb_avail":93599424,"statfs":{"total":96636764160,"available":95845810176,"internally_reserved":0,"allocated":196067328,"data_stored":195866054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":594868677},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,2,0,2,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182503,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":772440,"kb_used_data":191036,"kb_used_omap":4,"kb_used_meta":581371,"kb_avail":93599400,"statfs":{"total":96636764160,"available":95845785600,"internally_reserved":0,"allocated":195620864,"data_stored":195419590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":595324621},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,3,3,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":195244032,"data_stored":195244032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":195588096,"data_stored":195588096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":194904064,"data_stored":194904064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":194990080,"data_stored":194990080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:42:31.371 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:42:31.988 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:42:31.988 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:42:32.009 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":305,"stamp":"2024-09-05T16:42:30.991698+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":381800936,"num_objects":46612,"num_object_clones":0,"num_object_copies":186448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29984,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46612,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":144225,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1653,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":158149,"ondisk_log_size":158149,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2962360,"kb_used_data":749540,"kb_used_omap":14,"kb_used_meta":2212721,"kb_avail":374525000,"statfs":{"total":386547056640,"available":383513600000,"internally_reserved":0,"allocated":767528960,"data_stored":766723864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2265826871},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[13,11,5,6,3,12,10,7,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16883712,"num_objects":-2061,"num_object_clones":0,"num_object_copies":-8244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-117,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2061,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2061,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":117,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004967"},"pg_stats":[{"pgid":"1.f","version":"37'9943","reported_seq":10047,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:28.433078+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:42:28.433078+0000","last_peered":"2024-09-05T16:42:28.433078+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:42:28.433078+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9943,"log_dups_size":0,"ondisk_log_size":9943,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24256512,"num_objects":2961,"num_object_clones":0,"num_object_copies":11844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2961,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9943,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":725}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'9711","reported_seq":12530,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:28.250645+0000","last_change":"2024-09-05T16:40:28.919189+0000","last_active":"2024-09-05T16:42:28.250645+0000","last_peered":"2024-09-05T16:42:28.250645+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:42:28.250645+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9711,"log_dups_size":0,"ondisk_log_size":9711,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23286944,"num_objects":2843,"num_object_clones":0,"num_object_copies":11372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":897,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2843,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9709,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1374,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1946}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'9887","reported_seq":10009,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:28.511454+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:42:28.511454+0000","last_peered":"2024-09-05T16:42:28.511454+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:42:28.511454+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9887,"log_dups_size":0,"ondisk_log_size":9887,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23468648,"num_objects":2866,"num_object_clones":0,"num_object_copies":11464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2866,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9886,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":634}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'9978","reported_seq":14707,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:42:27.734806+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:42:27.734806+0000","last_peered":"2024-09-05T16:42:27.734806+0000","last_clean":"2024-09-05T16:42:27.734806+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:42:27.734806+0000","last_undegraded":"2024-09-05T16:42:27.734806+0000","last_fullsized":"2024-09-05T16:42:27.734806+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":9978,"log_dups_size":0,"ondisk_log_size":9978,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24215552,"num_objects":2956,"num_object_clones":0,"num_object_copies":11824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4278,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'9689","reported_seq":9788,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:28.400450+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:42:28.400450+0000","last_peered":"2024-09-05T16:42:28.400450+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:42:28.400450+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9689,"log_dups_size":0,"ondisk_log_size":9689,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23803040,"num_objects":2906,"num_object_clones":0,"num_object_copies":11624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2906,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9687,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":761}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'9881","reported_seq":9994,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:28.497915+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:42:28.497915+0000","last_peered":"2024-09-05T16:42:28.497915+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:42:28.497915+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9881,"log_dups_size":0,"ondisk_log_size":9881,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23683072,"num_objects":2891,"num_object_clones":0,"num_object_copies":11564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2891,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9881,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":672}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'9718","reported_seq":9823,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:27.977157+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:42:27.977157+0000","last_peered":"2024-09-05T16:42:27.977157+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:42:27.977157+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9718,"log_dups_size":0,"ondisk_log_size":9718,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24114336,"num_objects":2944,"num_object_clones":0,"num_object_copies":11776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2944,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9716,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":813}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'9858","reported_seq":9946,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:27.797790+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:42:27.797790+0000","last_peered":"2024-09-05T16:42:27.797790+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:42:27.797790+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9858,"log_dups_size":0,"ondisk_log_size":9858,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23925920,"num_objects":2921,"num_object_clones":0,"num_object_copies":11684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2921,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9856,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":689}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'9883","reported_seq":9981,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:28.540958+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:42:28.540958+0000","last_peered":"2024-09-05T16:42:28.540958+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:42:28.540958+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9883,"log_dups_size":0,"ondisk_log_size":9883,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23677248,"num_objects":2891,"num_object_clones":0,"num_object_copies":11564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2891,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9880,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":647}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'9952","reported_seq":10073,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:28.667467+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:42:28.667467+0000","last_peered":"2024-09-05T16:42:28.667467+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:42:28.667467+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9952,"log_dups_size":0,"ondisk_log_size":9952,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23409824,"num_objects":2858,"num_object_clones":0,"num_object_copies":11432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2858,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9950,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":613}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'10115","reported_seq":10241,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:28.423855+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:42:28.423855+0000","last_peered":"2024-09-05T16:42:28.423855+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:42:28.423855+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"29'100","ondisk_log_start":"29'100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24637168,"num_objects":3008,"num_object_clones":0,"num_object_copies":12032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3008,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10112,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":727}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'9836","reported_seq":9934,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:28.574285+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:42:28.574285+0000","last_peered":"2024-09-05T16:42:28.574285+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:42:28.574285+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9836,"log_dups_size":0,"ondisk_log_size":9836,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23943760,"num_objects":2923,"num_object_clones":0,"num_object_copies":11692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9835,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":685}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'9892","reported_seq":14627,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:42:28.074709+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:42:28.074709+0000","last_peered":"2024-09-05T16:42:28.074709+0000","last_clean":"2024-09-05T16:42:28.074709+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:42:28.074709+0000","last_undegraded":"2024-09-05T16:42:28.074709+0000","last_fullsized":"2024-09-05T16:42:28.074709+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":9892,"log_dups_size":0,"ondisk_log_size":9892,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23887872,"num_objects":2916,"num_object_clones":0,"num_object_copies":11664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2916,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1492,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'10171","reported_seq":10279,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:28.420759+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:42:28.420759+0000","last_peered":"2024-09-05T16:42:28.420759+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:42:28.420759+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"29'100","ondisk_log_start":"29'100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24204448,"num_objects":2955,"num_object_clones":0,"num_object_copies":11820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10169,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":621}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'9969","reported_seq":10081,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:28.653255+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:42:28.653255+0000","last_peered":"2024-09-05T16:42:28.653255+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:42:28.653255+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9969,"log_dups_size":0,"ondisk_log_size":9969,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23458976,"num_objects":2864,"num_object_clones":0,"num_object_copies":11456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2864,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9967,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":556}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'9866","reported_seq":9974,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:28.102513+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:42:28.102513+0000","last_peered":"2024-09-05T16:42:28.102513+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:42:28.102513+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9866,"log_dups_size":0,"ondisk_log_size":9866,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23827616,"num_objects":2909,"num_object_clones":0,"num_object_copies":11636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2909,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9864,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":689}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":381800936,"num_objects":46612,"num_object_clones":0,"num_object_copies":186448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29984,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46612,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":144225,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1653,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":763965440,"data_stored":763965440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":158149,"ondisk_log_size":158149,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084390,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":782356,"kb_used_data":187012,"kb_used_omap":4,"kb_used_meta":595323,"kb_avail":93589484,"statfs":{"total":96636764160,"available":95835631616,"internally_reserved":0,"allocated":191500288,"data_stored":191299014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":609611475},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,9,0,1,0,5,6,2,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986267,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":602380,"kb_used_data":187516,"kb_used_omap":4,"kb_used_meta":414843,"kb_avail":93769460,"statfs":{"total":96636764160,"available":96019927040,"internally_reserved":0,"allocated":192016384,"data_stored":191815110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":424799954},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,1,1,2,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673073,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":790524,"kb_used_data":187364,"kb_used_omap":1,"kb_used_meta":603134,"kb_avail":93581316,"statfs":{"total":96636764160,"available":95827267584,"internally_reserved":0,"allocated":191860736,"data_stored":191659462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":617609669},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,2,0,2,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182504,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":787100,"kb_used_data":187648,"kb_used_omap":4,"kb_used_meta":599419,"kb_avail":93584740,"statfs":{"total":96636764160,"available":95830773760,"internally_reserved":0,"allocated":192151552,"data_stored":191950278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":613805773},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[9,2,0,2,2,3,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":191078400,"data_stored":191078400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":191356928,"data_stored":191356928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":190738432,"data_stored":190738432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":190791680,"data_stored":190791680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:42:32.010 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:42:32.580 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:42:32.580 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:42:32.622 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":305,"stamp":"2024-09-05T16:42:30.991698+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":381800936,"num_objects":46612,"num_object_clones":0,"num_object_copies":186448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29984,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46612,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":144225,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1653,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":158149,"ondisk_log_size":158149,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2962360,"kb_used_data":749540,"kb_used_omap":14,"kb_used_meta":2212721,"kb_avail":374525000,"statfs":{"total":386547056640,"available":383513600000,"internally_reserved":0,"allocated":767528960,"data_stored":766723864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2265826871},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[13,11,5,6,3,12,10,7,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16883712,"num_objects":-2061,"num_object_clones":0,"num_object_copies":-8244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-117,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2061,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2061,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":117,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004967"},"pg_stats":[{"pgid":"1.f","version":"37'9943","reported_seq":10047,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:28.433078+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:42:28.433078+0000","last_peered":"2024-09-05T16:42:28.433078+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:42:28.433078+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9943,"log_dups_size":0,"ondisk_log_size":9943,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24256512,"num_objects":2961,"num_object_clones":0,"num_object_copies":11844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2961,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9943,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":725}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'9711","reported_seq":12530,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:28.250645+0000","last_change":"2024-09-05T16:40:28.919189+0000","last_active":"2024-09-05T16:42:28.250645+0000","last_peered":"2024-09-05T16:42:28.250645+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:42:28.250645+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9711,"log_dups_size":0,"ondisk_log_size":9711,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23286944,"num_objects":2843,"num_object_clones":0,"num_object_copies":11372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":897,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2843,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9709,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1374,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1946}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'9887","reported_seq":10009,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:28.511454+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:42:28.511454+0000","last_peered":"2024-09-05T16:42:28.511454+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:42:28.511454+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9887,"log_dups_size":0,"ondisk_log_size":9887,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23468648,"num_objects":2866,"num_object_clones":0,"num_object_copies":11464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2866,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9886,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":634}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'9978","reported_seq":14707,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:42:27.734806+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:42:27.734806+0000","last_peered":"2024-09-05T16:42:27.734806+0000","last_clean":"2024-09-05T16:42:27.734806+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:42:27.734806+0000","last_undegraded":"2024-09-05T16:42:27.734806+0000","last_fullsized":"2024-09-05T16:42:27.734806+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":9978,"log_dups_size":0,"ondisk_log_size":9978,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24215552,"num_objects":2956,"num_object_clones":0,"num_object_copies":11824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4278,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'9689","reported_seq":9788,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:28.400450+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:42:28.400450+0000","last_peered":"2024-09-05T16:42:28.400450+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:42:28.400450+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9689,"log_dups_size":0,"ondisk_log_size":9689,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23803040,"num_objects":2906,"num_object_clones":0,"num_object_copies":11624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2906,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9687,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":761}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'9881","reported_seq":9994,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:28.497915+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:42:28.497915+0000","last_peered":"2024-09-05T16:42:28.497915+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:42:28.497915+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9881,"log_dups_size":0,"ondisk_log_size":9881,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23683072,"num_objects":2891,"num_object_clones":0,"num_object_copies":11564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2891,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9881,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":672}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'9718","reported_seq":9823,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:27.977157+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:42:27.977157+0000","last_peered":"2024-09-05T16:42:27.977157+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:42:27.977157+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9718,"log_dups_size":0,"ondisk_log_size":9718,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24114336,"num_objects":2944,"num_object_clones":0,"num_object_copies":11776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2944,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9716,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":813}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'9858","reported_seq":9946,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:27.797790+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:42:27.797790+0000","last_peered":"2024-09-05T16:42:27.797790+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:42:27.797790+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9858,"log_dups_size":0,"ondisk_log_size":9858,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23925920,"num_objects":2921,"num_object_clones":0,"num_object_copies":11684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2921,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9856,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":689}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'9883","reported_seq":9981,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:28.540958+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:42:28.540958+0000","last_peered":"2024-09-05T16:42:28.540958+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:42:28.540958+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9883,"log_dups_size":0,"ondisk_log_size":9883,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23677248,"num_objects":2891,"num_object_clones":0,"num_object_copies":11564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2891,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9880,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":647}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'9952","reported_seq":10073,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:28.667467+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:42:28.667467+0000","last_peered":"2024-09-05T16:42:28.667467+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:42:28.667467+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9952,"log_dups_size":0,"ondisk_log_size":9952,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23409824,"num_objects":2858,"num_object_clones":0,"num_object_copies":11432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2858,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9950,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":613}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'10115","reported_seq":10241,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:28.423855+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:42:28.423855+0000","last_peered":"2024-09-05T16:42:28.423855+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:42:28.423855+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"29'100","ondisk_log_start":"29'100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24637168,"num_objects":3008,"num_object_clones":0,"num_object_copies":12032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3008,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10112,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":727}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'9836","reported_seq":9934,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:28.574285+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:42:28.574285+0000","last_peered":"2024-09-05T16:42:28.574285+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:42:28.574285+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9836,"log_dups_size":0,"ondisk_log_size":9836,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23943760,"num_objects":2923,"num_object_clones":0,"num_object_copies":11692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9835,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":685}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'9892","reported_seq":14627,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:42:28.074709+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:42:28.074709+0000","last_peered":"2024-09-05T16:42:28.074709+0000","last_clean":"2024-09-05T16:42:28.074709+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:42:28.074709+0000","last_undegraded":"2024-09-05T16:42:28.074709+0000","last_fullsized":"2024-09-05T16:42:28.074709+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":9892,"log_dups_size":0,"ondisk_log_size":9892,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23887872,"num_objects":2916,"num_object_clones":0,"num_object_copies":11664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2916,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1492,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'10171","reported_seq":10279,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:28.420759+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:42:28.420759+0000","last_peered":"2024-09-05T16:42:28.420759+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:42:28.420759+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"29'100","ondisk_log_start":"29'100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24204448,"num_objects":2955,"num_object_clones":0,"num_object_copies":11820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10169,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":621}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'9969","reported_seq":10081,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:28.653255+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:42:28.653255+0000","last_peered":"2024-09-05T16:42:28.653255+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:42:28.653255+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9969,"log_dups_size":0,"ondisk_log_size":9969,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23458976,"num_objects":2864,"num_object_clones":0,"num_object_copies":11456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2864,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9967,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":556}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'9866","reported_seq":9974,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:28.102513+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:42:28.102513+0000","last_peered":"2024-09-05T16:42:28.102513+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:42:28.102513+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9866,"log_dups_size":0,"ondisk_log_size":9866,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23827616,"num_objects":2909,"num_object_clones":0,"num_object_copies":11636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2909,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9864,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":689}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":381800936,"num_objects":46612,"num_object_clones":0,"num_object_copies":186448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29984,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46612,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":144225,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1653,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":763965440,"data_stored":763965440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":158149,"ondisk_log_size":158149,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084390,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":782356,"kb_used_data":187012,"kb_used_omap":4,"kb_used_meta":595323,"kb_avail":93589484,"statfs":{"total":96636764160,"available":95835631616,"internally_reserved":0,"allocated":191500288,"data_stored":191299014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":609611475},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,9,0,1,0,5,6,2,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986267,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":602380,"kb_used_data":187516,"kb_used_omap":4,"kb_used_meta":414843,"kb_avail":93769460,"statfs":{"total":96636764160,"available":96019927040,"internally_reserved":0,"allocated":192016384,"data_stored":191815110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":424799954},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,1,1,2,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673073,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":790524,"kb_used_data":187364,"kb_used_omap":1,"kb_used_meta":603134,"kb_avail":93581316,"statfs":{"total":96636764160,"available":95827267584,"internally_reserved":0,"allocated":191860736,"data_stored":191659462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":617609669},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,2,0,2,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182504,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":787100,"kb_used_data":187648,"kb_used_omap":4,"kb_used_meta":599419,"kb_avail":93584740,"statfs":{"total":96636764160,"available":95830773760,"internally_reserved":0,"allocated":192151552,"data_stored":191950278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":613805773},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[9,2,0,2,2,3,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":191078400,"data_stored":191078400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":191356928,"data_stored":191356928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":190738432,"data_stored":190738432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":190791680,"data_stored":190791680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:42:32.623 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-05T16:42:32.623 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'9943", 'reported_seq': 10047, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:28.433078+0000', 'last_change': '2024-09-05T16:34:56.833497+0000', 'last_active': '2024-09-05T16:42:28.433078+0000', 'last_peered': '2024-09-05T16:42:28.433078+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.828289+0000', 'last_became_peered': '2024-09-05T16:34:56.828289+0000', 'last_unstale': '2024-09-05T16:42:28.433078+0000', 'last_undegraded': '2024-09-05T16:34:56.612160+0000', 'last_fullsized': '2024-09-05T16:34:56.557632+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9943, 'log_dups_size': 0, 'ondisk_log_size': 9943, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24256512, 'num_objects': 2961, 'num_object_clones': 0, 'num_object_copies': 11844, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2236, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2961, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9943, 'num_write_kb': 51616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 725}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:42:32.623 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-05T16:42:32.624 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "37'9711", 'reported_seq': 12530, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:28.250645+0000', 'last_change': '2024-09-05T16:40:28.919189+0000', 'last_active': '2024-09-05T16:42:28.250645+0000', 'last_peered': '2024-09-05T16:42:28.250645+0000', 'last_clean': '2024-09-05T16:33:53.200762+0000', 'last_became_active': '2024-09-05T16:34:56.820292+0000', 'last_became_peered': '2024-09-05T16:34:56.820292+0000', 'last_unstale': '2024-09-05T16:42:28.250645+0000', 'last_undegraded': '2024-09-05T16:34:56.605776+0000', 'last_fullsized': '2024-09-05T16:34:56.559383+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9711, 'log_dups_size': 0, 'ondisk_log_size': 9711, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23286944, 'num_objects': 2843, 'num_object_clones': 0, 'num_object_copies': 11372, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 897, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2843, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9709, 'num_write_kb': 50212, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1374, 'num_bytes_recovered': 2613248, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 1946}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:42:32.624 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:42:32.624 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'9887", 'reported_seq': 10009, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:28.511454+0000', 'last_change': '2024-09-05T16:34:56.867900+0000', 'last_active': '2024-09-05T16:42:28.511454+0000', 'last_peered': '2024-09-05T16:42:28.511454+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:42:28.511454+0000', 'last_undegraded': '2024-09-05T16:34:56.590972+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9887, 'log_dups_size': 0, 'ondisk_log_size': 9887, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23468648, 'num_objects': 2866, 'num_object_clones': 0, 'num_object_copies': 11464, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2233, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2866, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 9886, 'num_write_kb': 50999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 634}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:42:32.624 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:42:32.624 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'9689", 'reported_seq': 9788, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:28.400450+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:42:28.400450+0000', 'last_peered': '2024-09-05T16:42:28.400450+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:42:28.400450+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9689, 'log_dups_size': 0, 'ondisk_log_size': 9689, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23803040, 'num_objects': 2906, 'num_object_clones': 0, 'num_object_copies': 11624, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2147, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2906, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9687, 'num_write_kb': 50373, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 761}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:42:32.624 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:42:32.624 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'9881", 'reported_seq': 9994, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:28.497915+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:42:28.497915+0000', 'last_peered': '2024-09-05T16:42:28.497915+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:42:28.497915+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9881, 'log_dups_size': 0, 'ondisk_log_size': 9881, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23683072, 'num_objects': 2891, 'num_object_clones': 0, 'num_object_copies': 11564, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2891, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9881, 'num_write_kb': 51088, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 672}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:42:32.624 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:42:32.624 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'9718", 'reported_seq': 9823, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:27.977157+0000', 'last_change': '2024-09-05T16:34:56.903436+0000', 'last_active': '2024-09-05T16:42:27.977157+0000', 'last_peered': '2024-09-05T16:42:27.977157+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:42:27.977157+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9718, 'log_dups_size': 0, 'ondisk_log_size': 9718, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24114336, 'num_objects': 2944, 'num_object_clones': 0, 'num_object_copies': 11776, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2133, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2944, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9716, 'num_write_kb': 50636, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 813}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:42:32.624 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-05T16:42:32.625 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "37'9858", 'reported_seq': 9946, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:27.797790+0000', 'last_change': '2024-09-05T16:34:56.883319+0000', 'last_active': '2024-09-05T16:42:27.797790+0000', 'last_peered': '2024-09-05T16:42:27.797790+0000', 'last_clean': '2024-09-05T16:33:53.505305+0000', 'last_became_active': '2024-09-05T16:34:56.873389+0000', 'last_became_peered': '2024-09-05T16:34:56.873389+0000', 'last_unstale': '2024-09-05T16:42:27.797790+0000', 'last_undegraded': '2024-09-05T16:34:56.602567+0000', 'last_fullsized': '2024-09-05T16:34:56.559958+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9858, 'log_dups_size': 0, 'ondisk_log_size': 9858, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23925920, 'num_objects': 2921, 'num_object_clones': 0, 'num_object_copies': 11684, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2234, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2921, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9856, 'num_write_kb': 51108, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 689}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:42:32.625 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:42:32.625 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'9883", 'reported_seq': 9981, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:28.540958+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:42:28.540958+0000', 'last_peered': '2024-09-05T16:42:28.540958+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:42:28.540958+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9883, 'log_dups_size': 0, 'ondisk_log_size': 9883, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23677248, 'num_objects': 2891, 'num_object_clones': 0, 'num_object_copies': 11564, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2891, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9880, 'num_write_kb': 51081, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 647}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:42:32.625 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:42:32.625 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'9952", 'reported_seq': 10073, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:28.667467+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:42:28.667467+0000', 'last_peered': '2024-09-05T16:42:28.667467+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:42:28.667467+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9952, 'log_dups_size': 0, 'ondisk_log_size': 9952, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23409824, 'num_objects': 2858, 'num_object_clones': 0, 'num_object_copies': 11432, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2858, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9950, 'num_write_kb': 51230, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 613}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:42:32.625 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:42:32.625 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'10115", 'reported_seq': 10241, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:28.423855+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:42:28.423855+0000', 'last_peered': '2024-09-05T16:42:28.423855+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:42:28.423855+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "29'100", 'ondisk_log_start': "29'100", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10015, 'log_dups_size': 0, 'ondisk_log_size': 10015, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24637168, 'num_objects': 3008, 'num_object_clones': 0, 'num_object_copies': 12032, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2284, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3008, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10112, 'num_write_kb': 52476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 727}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:42:32.625 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:42:32.625 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'9836", 'reported_seq': 9934, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:28.574285+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:42:28.574285+0000', 'last_peered': '2024-09-05T16:42:28.574285+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:42:28.574285+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9836, 'log_dups_size': 0, 'ondisk_log_size': 9836, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23943760, 'num_objects': 2923, 'num_object_clones': 0, 'num_object_copies': 11692, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2923, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9835, 'num_write_kb': 51031, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 685}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:42:32.625 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:42:32.626 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'10171", 'reported_seq': 10279, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:28.420759+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:42:28.420759+0000', 'last_peered': '2024-09-05T16:42:28.420759+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:42:28.420759+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "29'100", 'ondisk_log_start': "29'100", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10071, 'log_dups_size': 0, 'ondisk_log_size': 10071, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24204448, 'num_objects': 2955, 'num_object_clones': 0, 'num_object_copies': 11820, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2955, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10169, 'num_write_kb': 52500, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 621}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:42:32.626 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:42:32.626 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'9969", 'reported_seq': 10081, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:28.653255+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:42:28.653255+0000', 'last_peered': '2024-09-05T16:42:28.653255+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:42:28.653255+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9969, 'log_dups_size': 0, 'ondisk_log_size': 9969, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23458976, 'num_objects': 2864, 'num_object_clones': 0, 'num_object_copies': 11456, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2310, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2864, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9967, 'num_write_kb': 51325, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 556}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:42:32.626 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:42:32.626 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'9866", 'reported_seq': 9974, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:28.102513+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:42:28.102513+0000', 'last_peered': '2024-09-05T16:42:28.102513+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:42:28.102513+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9866, 'log_dups_size': 0, 'ondisk_log_size': 9866, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23827616, 'num_objects': 2909, 'num_object_clones': 0, 'num_object_copies': 11636, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2909, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9864, 'num_write_kb': 51092, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 689}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:42:32.626 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:42:33.392 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:42:33.393 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 29984/186448 objects degraded (16.082%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":556,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":46612,"data_bytes":381800936,"bytes_used":3033456640,"bytes_avail":383513600000,"bytes_total":386547056640,"degraded_objects":29984,"degraded_total":186448,"degraded_ratio":0.16081695700677937,"recovering_objects_per_sec":9,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":117,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":171},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:42:33.420 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:42:33.420 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:42:34.030 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:42:34.031 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:42:34.049 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":306,"stamp":"2024-09-05T16:42:32.994175+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":381800936,"num_objects":46612,"num_object_clones":0,"num_object_copies":186448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29984,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46612,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":144225,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1653,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":158149,"ondisk_log_size":158149,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2962360,"kb_used_data":749540,"kb_used_omap":14,"kb_used_meta":2212721,"kb_avail":374525000,"statfs":{"total":386547056640,"available":383513600000,"internally_reserved":0,"allocated":767528960,"data_stored":766723864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2265826871},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[13,11,5,6,3,12,10,7,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16883712,"num_objects":-2061,"num_object_clones":0,"num_object_copies":-8244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-117,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2061,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2061,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":117,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006514"},"pg_stats":[{"pgid":"1.f","version":"37'9943","reported_seq":10047,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:28.433078+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:42:28.433078+0000","last_peered":"2024-09-05T16:42:28.433078+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:42:28.433078+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9943,"log_dups_size":0,"ondisk_log_size":9943,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24256512,"num_objects":2961,"num_object_clones":0,"num_object_copies":11844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2961,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9943,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":725}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'9711","reported_seq":12530,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:28.250645+0000","last_change":"2024-09-05T16:40:28.919189+0000","last_active":"2024-09-05T16:42:28.250645+0000","last_peered":"2024-09-05T16:42:28.250645+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:42:28.250645+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9711,"log_dups_size":0,"ondisk_log_size":9711,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23286944,"num_objects":2843,"num_object_clones":0,"num_object_copies":11372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":897,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2843,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9709,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1374,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1946}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'9887","reported_seq":10009,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:28.511454+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:42:28.511454+0000","last_peered":"2024-09-05T16:42:28.511454+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:42:28.511454+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9887,"log_dups_size":0,"ondisk_log_size":9887,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23468648,"num_objects":2866,"num_object_clones":0,"num_object_copies":11464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2866,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9886,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":634}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'9978","reported_seq":14707,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:42:27.734806+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:42:27.734806+0000","last_peered":"2024-09-05T16:42:27.734806+0000","last_clean":"2024-09-05T16:42:27.734806+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:42:27.734806+0000","last_undegraded":"2024-09-05T16:42:27.734806+0000","last_fullsized":"2024-09-05T16:42:27.734806+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":9978,"log_dups_size":0,"ondisk_log_size":9978,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24215552,"num_objects":2956,"num_object_clones":0,"num_object_copies":11824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4278,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'9689","reported_seq":9788,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:28.400450+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:42:28.400450+0000","last_peered":"2024-09-05T16:42:28.400450+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:42:28.400450+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9689,"log_dups_size":0,"ondisk_log_size":9689,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23803040,"num_objects":2906,"num_object_clones":0,"num_object_copies":11624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2906,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9687,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":761}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'9881","reported_seq":9994,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:28.497915+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:42:28.497915+0000","last_peered":"2024-09-05T16:42:28.497915+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:42:28.497915+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9881,"log_dups_size":0,"ondisk_log_size":9881,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23683072,"num_objects":2891,"num_object_clones":0,"num_object_copies":11564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2891,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9881,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":672}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'9718","reported_seq":9823,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:27.977157+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:42:27.977157+0000","last_peered":"2024-09-05T16:42:27.977157+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:42:27.977157+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9718,"log_dups_size":0,"ondisk_log_size":9718,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24114336,"num_objects":2944,"num_object_clones":0,"num_object_copies":11776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2944,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9716,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":813}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'9858","reported_seq":9946,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:27.797790+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:42:27.797790+0000","last_peered":"2024-09-05T16:42:27.797790+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:42:27.797790+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9858,"log_dups_size":0,"ondisk_log_size":9858,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23925920,"num_objects":2921,"num_object_clones":0,"num_object_copies":11684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2921,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9856,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":689}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'9883","reported_seq":9981,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:28.540958+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:42:28.540958+0000","last_peered":"2024-09-05T16:42:28.540958+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:42:28.540958+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9883,"log_dups_size":0,"ondisk_log_size":9883,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23677248,"num_objects":2891,"num_object_clones":0,"num_object_copies":11564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2891,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9880,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":647}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'9952","reported_seq":10073,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:28.667467+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:42:28.667467+0000","last_peered":"2024-09-05T16:42:28.667467+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:42:28.667467+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9952,"log_dups_size":0,"ondisk_log_size":9952,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23409824,"num_objects":2858,"num_object_clones":0,"num_object_copies":11432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2858,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9950,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":613}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'10115","reported_seq":10241,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:28.423855+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:42:28.423855+0000","last_peered":"2024-09-05T16:42:28.423855+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:42:28.423855+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"29'100","ondisk_log_start":"29'100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24637168,"num_objects":3008,"num_object_clones":0,"num_object_copies":12032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3008,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10112,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":727}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'9836","reported_seq":9934,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:28.574285+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:42:28.574285+0000","last_peered":"2024-09-05T16:42:28.574285+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:42:28.574285+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9836,"log_dups_size":0,"ondisk_log_size":9836,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23943760,"num_objects":2923,"num_object_clones":0,"num_object_copies":11692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9835,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":685}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'9892","reported_seq":14627,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:42:28.074709+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:42:28.074709+0000","last_peered":"2024-09-05T16:42:28.074709+0000","last_clean":"2024-09-05T16:42:28.074709+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:42:28.074709+0000","last_undegraded":"2024-09-05T16:42:28.074709+0000","last_fullsized":"2024-09-05T16:42:28.074709+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":9892,"log_dups_size":0,"ondisk_log_size":9892,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23887872,"num_objects":2916,"num_object_clones":0,"num_object_copies":11664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2916,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1492,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'10171","reported_seq":10279,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:28.420759+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:42:28.420759+0000","last_peered":"2024-09-05T16:42:28.420759+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:42:28.420759+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"29'100","ondisk_log_start":"29'100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24204448,"num_objects":2955,"num_object_clones":0,"num_object_copies":11820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10169,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":621}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'9969","reported_seq":10081,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:28.653255+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:42:28.653255+0000","last_peered":"2024-09-05T16:42:28.653255+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:42:28.653255+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9969,"log_dups_size":0,"ondisk_log_size":9969,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23458976,"num_objects":2864,"num_object_clones":0,"num_object_copies":11456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2864,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9967,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":556}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'9866","reported_seq":9974,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:28.102513+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:42:28.102513+0000","last_peered":"2024-09-05T16:42:28.102513+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:42:28.102513+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9866,"log_dups_size":0,"ondisk_log_size":9866,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23827616,"num_objects":2909,"num_object_clones":0,"num_object_copies":11636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2909,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9864,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":689}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":381800936,"num_objects":46612,"num_object_clones":0,"num_object_copies":186448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29984,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46612,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":144225,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1653,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":763965440,"data_stored":763965440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":158149,"ondisk_log_size":158149,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084390,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":782356,"kb_used_data":187012,"kb_used_omap":4,"kb_used_meta":595323,"kb_avail":93589484,"statfs":{"total":96636764160,"available":95835631616,"internally_reserved":0,"allocated":191500288,"data_stored":191299014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":609611475},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,9,0,1,0,5,6,2,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986267,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":602380,"kb_used_data":187516,"kb_used_omap":4,"kb_used_meta":414843,"kb_avail":93769460,"statfs":{"total":96636764160,"available":96019927040,"internally_reserved":0,"allocated":192016384,"data_stored":191815110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":424799954},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,1,1,2,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673073,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":790524,"kb_used_data":187364,"kb_used_omap":1,"kb_used_meta":603134,"kb_avail":93581316,"statfs":{"total":96636764160,"available":95827267584,"internally_reserved":0,"allocated":191860736,"data_stored":191659462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":617609669},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,2,0,2,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182504,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":787100,"kb_used_data":187648,"kb_used_omap":4,"kb_used_meta":599419,"kb_avail":93584740,"statfs":{"total":96636764160,"available":95830773760,"internally_reserved":0,"allocated":192151552,"data_stored":191950278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":613805773},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[9,2,0,2,2,3,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":191078400,"data_stored":191078400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":191356928,"data_stored":191356928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":190738432,"data_stored":190738432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":190791680,"data_stored":190791680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:42:37.051 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:42:37.674 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:42:37.674 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:42:37.697 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":308,"stamp":"2024-09-05T16:42:36.995738+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":373535208,"num_objects":45603,"num_object_clones":0,"num_object_copies":182412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45603,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":145234,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1706,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159058,"ondisk_log_size":159058,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2929412,"kb_used_data":733212,"kb_used_omap":14,"kb_used_meta":2196081,"kb_avail":374557948,"statfs":{"total":386547056640,"available":383547338752,"internally_reserved":0,"allocated":750809088,"data_stored":750003992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2248787511},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,9,2,12,15,11,24,5,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16687104,"num_objects":-2037,"num_object_clones":0,"num_object_copies":-8148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2037,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":110,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006440"},"pg_stats":[{"pgid":"1.f","version":"37'10004","reported_seq":10108,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:33.571091+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:42:33.571091+0000","last_peered":"2024-09-05T16:42:33.571091+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:42:33.571091+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23756800,"num_objects":2900,"num_object_clones":0,"num_object_copies":11600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2900,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10004,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":664}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'9760","reported_seq":12685,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:33.291486+0000","last_change":"2024-09-05T16:40:28.919189+0000","last_active":"2024-09-05T16:42:33.291486+0000","last_peered":"2024-09-05T16:42:33.291486+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:42:33.291486+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9760,"log_dups_size":0,"ondisk_log_size":9760,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22885536,"num_objects":2794,"num_object_clones":0,"num_object_copies":11176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":844,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2794,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9758,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1427,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1950}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'9945","reported_seq":10067,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:33.674017+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:42:33.674017+0000","last_peered":"2024-09-05T16:42:33.674017+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:42:33.674017+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9945,"log_dups_size":0,"ondisk_log_size":9945,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22993512,"num_objects":2808,"num_object_clones":0,"num_object_copies":11232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2808,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9944,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":576}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'10036","reported_seq":14765,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:42:33.201814+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:42:33.201814+0000","last_peered":"2024-09-05T16:42:33.201814+0000","last_clean":"2024-09-05T16:42:33.201814+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:42:33.201814+0000","last_undegraded":"2024-09-05T16:42:33.201814+0000","last_fullsized":"2024-09-05T16:42:33.201814+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23740416,"num_objects":2898,"num_object_clones":0,"num_object_copies":11592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2898,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4336,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'9754","reported_seq":9851,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:33.546660+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:42:33.546660+0000","last_peered":"2024-09-05T16:42:33.546660+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:42:33.546660+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9754,"log_dups_size":0,"ondisk_log_size":9754,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23270560,"num_objects":2841,"num_object_clones":0,"num_object_copies":11364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2841,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9752,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":696}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'9948","reported_seq":10062,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:33.643386+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:42:33.643386+0000","last_peered":"2024-09-05T16:42:33.643386+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:42:33.643386+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9948,"log_dups_size":0,"ondisk_log_size":9948,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23134208,"num_objects":2824,"num_object_clones":0,"num_object_copies":11296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2824,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9948,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":605}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'9811","reported_seq":9919,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:33.290226+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:42:33.290226+0000","last_peered":"2024-09-05T16:42:33.290226+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:42:33.290226+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9811,"log_dups_size":0,"ondisk_log_size":9811,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23352480,"num_objects":2851,"num_object_clones":0,"num_object_copies":11404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2851,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9809,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":720}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'9917","reported_seq":10006,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:33.333214+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:42:33.333214+0000","last_peered":"2024-09-05T16:42:33.333214+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:42:33.333214+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9917,"log_dups_size":0,"ondisk_log_size":9917,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23442592,"num_objects":2862,"num_object_clones":0,"num_object_copies":11448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2862,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9915,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":630}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'9941","reported_seq":10041,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:33.642213+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:42:33.642213+0000","last_peered":"2024-09-05T16:42:33.642213+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:42:33.642213+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9941,"log_dups_size":0,"ondisk_log_size":9941,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23202112,"num_objects":2833,"num_object_clones":0,"num_object_copies":11332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2833,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9938,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":589}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'10021","reported_seq":10142,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:33.695179+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:42:33.695179+0000","last_peered":"2024-09-05T16:42:33.695179+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:42:33.695179+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22844576,"num_objects":2789,"num_object_clones":0,"num_object_copies":11156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2789,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10019,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":544}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'10190","reported_seq":10314,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:33.603806+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:42:33.603806+0000","last_peered":"2024-09-05T16:42:33.603806+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:42:33.603806+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"29'100","ondisk_log_start":"29'100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24022768,"num_objects":2933,"num_object_clones":0,"num_object_copies":11732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2933,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10187,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":652}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'9892","reported_seq":9989,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:33.383916+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:42:33.383916+0000","last_peered":"2024-09-05T16:42:33.383916+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:42:33.383916+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9892,"log_dups_size":0,"ondisk_log_size":9892,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23485008,"num_objects":2867,"num_object_clones":0,"num_object_copies":11468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2867,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9891,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":629}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'9963","reported_seq":14698,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:42:33.076129+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:42:33.076129+0000","last_peered":"2024-09-05T16:42:33.076129+0000","last_clean":"2024-09-05T16:42:33.076129+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:42:33.076129+0000","last_undegraded":"2024-09-05T16:42:33.076129+0000","last_fullsized":"2024-09-05T16:42:33.076129+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":9963,"log_dups_size":0,"ondisk_log_size":9963,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23306240,"num_objects":2845,"num_object_clones":0,"num_object_copies":11380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2845,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1563,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'10221","reported_seq":10329,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:33.472186+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:42:33.472186+0000","last_peered":"2024-09-05T16:42:33.472186+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:42:33.472186+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"29'200","ondisk_log_start":"29'200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23794848,"num_objects":2905,"num_object_clones":0,"num_object_copies":11620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2905,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10219,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":571}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'10031","reported_seq":10143,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:33.553890+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:42:33.553890+0000","last_peered":"2024-09-05T16:42:33.553890+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:42:33.553890+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22951072,"num_objects":2802,"num_object_clones":0,"num_object_copies":11208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2802,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10029,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":494}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'9924","reported_seq":10032,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:33.175880+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:42:33.175880+0000","last_peered":"2024-09-05T16:42:33.175880+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:42:33.175880+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9924,"log_dups_size":0,"ondisk_log_size":9924,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23352480,"num_objects":2851,"num_object_clones":0,"num_object_copies":11404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2851,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9922,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":631}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":373535208,"num_objects":45603,"num_object_clones":0,"num_object_copies":182412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45603,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":145234,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1706,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":747274240,"data_stored":747274240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159058,"ondisk_log_size":159058,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084391,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":782508,"kb_used_data":183004,"kb_used_omap":4,"kb_used_meta":599483,"kb_avail":93589332,"statfs":{"total":96636764160,"available":95835475968,"internally_reserved":0,"allocated":187396096,"data_stored":187194822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":613871315},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,9,0,1,0,5,6,2,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986268,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":602528,"kb_used_data":183496,"kb_used_omap":4,"kb_used_meta":419003,"kb_avail":93769312,"statfs":{"total":96636764160,"available":96019775488,"internally_reserved":0,"allocated":187899904,"data_stored":187698630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":429059794},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,11,0,1,7,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673074,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":773872,"kb_used_data":183168,"kb_used_omap":1,"kb_used_meta":590654,"kb_avail":93597968,"statfs":{"total":96636764160,"available":95844319232,"internally_reserved":0,"allocated":187564032,"data_stored":187362758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":604830149},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,12,1,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182505,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":770504,"kb_used_data":183544,"kb_used_omap":4,"kb_used_meta":586939,"kb_avail":93601336,"statfs":{"total":96636764160,"available":95847768064,"internally_reserved":0,"allocated":187949056,"data_stored":187747782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":601026253},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,3,4,6,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":186896384,"data_stored":186896384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":187121664,"data_stored":187121664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":186613760,"data_stored":186613760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":186642432,"data_stored":186642432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:42:37.698 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:42:38.307 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:42:38.308 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:42:38.334 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":308,"stamp":"2024-09-05T16:42:36.995738+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":373535208,"num_objects":45603,"num_object_clones":0,"num_object_copies":182412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45603,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":145234,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1706,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159058,"ondisk_log_size":159058,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2929412,"kb_used_data":733212,"kb_used_omap":14,"kb_used_meta":2196081,"kb_avail":374557948,"statfs":{"total":386547056640,"available":383547338752,"internally_reserved":0,"allocated":750809088,"data_stored":750003992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2248787511},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,9,2,12,15,11,24,5,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16687104,"num_objects":-2037,"num_object_clones":0,"num_object_copies":-8148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2037,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":110,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006440"},"pg_stats":[{"pgid":"1.f","version":"37'10004","reported_seq":10108,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:33.571091+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:42:33.571091+0000","last_peered":"2024-09-05T16:42:33.571091+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:42:33.571091+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23756800,"num_objects":2900,"num_object_clones":0,"num_object_copies":11600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2900,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10004,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":664}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'9760","reported_seq":12685,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:33.291486+0000","last_change":"2024-09-05T16:40:28.919189+0000","last_active":"2024-09-05T16:42:33.291486+0000","last_peered":"2024-09-05T16:42:33.291486+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:42:33.291486+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9760,"log_dups_size":0,"ondisk_log_size":9760,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22885536,"num_objects":2794,"num_object_clones":0,"num_object_copies":11176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":844,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2794,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9758,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1427,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1950}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'9945","reported_seq":10067,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:33.674017+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:42:33.674017+0000","last_peered":"2024-09-05T16:42:33.674017+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:42:33.674017+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9945,"log_dups_size":0,"ondisk_log_size":9945,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22993512,"num_objects":2808,"num_object_clones":0,"num_object_copies":11232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2808,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9944,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":576}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'10036","reported_seq":14765,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:42:33.201814+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:42:33.201814+0000","last_peered":"2024-09-05T16:42:33.201814+0000","last_clean":"2024-09-05T16:42:33.201814+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:42:33.201814+0000","last_undegraded":"2024-09-05T16:42:33.201814+0000","last_fullsized":"2024-09-05T16:42:33.201814+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23740416,"num_objects":2898,"num_object_clones":0,"num_object_copies":11592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2898,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4336,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'9754","reported_seq":9851,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:33.546660+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:42:33.546660+0000","last_peered":"2024-09-05T16:42:33.546660+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:42:33.546660+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9754,"log_dups_size":0,"ondisk_log_size":9754,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23270560,"num_objects":2841,"num_object_clones":0,"num_object_copies":11364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2841,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9752,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":696}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'9948","reported_seq":10062,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:33.643386+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:42:33.643386+0000","last_peered":"2024-09-05T16:42:33.643386+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:42:33.643386+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9948,"log_dups_size":0,"ondisk_log_size":9948,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23134208,"num_objects":2824,"num_object_clones":0,"num_object_copies":11296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2824,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9948,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":605}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'9811","reported_seq":9919,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:33.290226+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:42:33.290226+0000","last_peered":"2024-09-05T16:42:33.290226+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:42:33.290226+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9811,"log_dups_size":0,"ondisk_log_size":9811,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23352480,"num_objects":2851,"num_object_clones":0,"num_object_copies":11404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2851,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9809,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":720}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'9917","reported_seq":10006,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:33.333214+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:42:33.333214+0000","last_peered":"2024-09-05T16:42:33.333214+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:42:33.333214+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9917,"log_dups_size":0,"ondisk_log_size":9917,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23442592,"num_objects":2862,"num_object_clones":0,"num_object_copies":11448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2862,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9915,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":630}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'9941","reported_seq":10041,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:33.642213+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:42:33.642213+0000","last_peered":"2024-09-05T16:42:33.642213+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:42:33.642213+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9941,"log_dups_size":0,"ondisk_log_size":9941,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23202112,"num_objects":2833,"num_object_clones":0,"num_object_copies":11332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2833,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9938,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":589}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'10021","reported_seq":10142,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:33.695179+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:42:33.695179+0000","last_peered":"2024-09-05T16:42:33.695179+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:42:33.695179+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22844576,"num_objects":2789,"num_object_clones":0,"num_object_copies":11156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2789,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10019,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":544}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'10190","reported_seq":10314,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:33.603806+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:42:33.603806+0000","last_peered":"2024-09-05T16:42:33.603806+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:42:33.603806+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"29'100","ondisk_log_start":"29'100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24022768,"num_objects":2933,"num_object_clones":0,"num_object_copies":11732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2933,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10187,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":652}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'9892","reported_seq":9989,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:33.383916+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:42:33.383916+0000","last_peered":"2024-09-05T16:42:33.383916+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:42:33.383916+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9892,"log_dups_size":0,"ondisk_log_size":9892,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23485008,"num_objects":2867,"num_object_clones":0,"num_object_copies":11468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2867,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9891,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":629}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'9963","reported_seq":14698,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:42:33.076129+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:42:33.076129+0000","last_peered":"2024-09-05T16:42:33.076129+0000","last_clean":"2024-09-05T16:42:33.076129+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:42:33.076129+0000","last_undegraded":"2024-09-05T16:42:33.076129+0000","last_fullsized":"2024-09-05T16:42:33.076129+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":9963,"log_dups_size":0,"ondisk_log_size":9963,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23306240,"num_objects":2845,"num_object_clones":0,"num_object_copies":11380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2845,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1563,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'10221","reported_seq":10329,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:33.472186+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:42:33.472186+0000","last_peered":"2024-09-05T16:42:33.472186+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:42:33.472186+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"29'200","ondisk_log_start":"29'200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23794848,"num_objects":2905,"num_object_clones":0,"num_object_copies":11620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2905,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10219,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":571}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'10031","reported_seq":10143,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:33.553890+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:42:33.553890+0000","last_peered":"2024-09-05T16:42:33.553890+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:42:33.553890+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22951072,"num_objects":2802,"num_object_clones":0,"num_object_copies":11208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2802,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10029,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":494}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'9924","reported_seq":10032,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:33.175880+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:42:33.175880+0000","last_peered":"2024-09-05T16:42:33.175880+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:42:33.175880+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9924,"log_dups_size":0,"ondisk_log_size":9924,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23352480,"num_objects":2851,"num_object_clones":0,"num_object_copies":11404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2851,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9922,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":631}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":373535208,"num_objects":45603,"num_object_clones":0,"num_object_copies":182412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45603,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":145234,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1706,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":747274240,"data_stored":747274240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159058,"ondisk_log_size":159058,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084391,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":782508,"kb_used_data":183004,"kb_used_omap":4,"kb_used_meta":599483,"kb_avail":93589332,"statfs":{"total":96636764160,"available":95835475968,"internally_reserved":0,"allocated":187396096,"data_stored":187194822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":613871315},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,9,0,1,0,5,6,2,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986268,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":602528,"kb_used_data":183496,"kb_used_omap":4,"kb_used_meta":419003,"kb_avail":93769312,"statfs":{"total":96636764160,"available":96019775488,"internally_reserved":0,"allocated":187899904,"data_stored":187698630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":429059794},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,11,0,1,7,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673074,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":773872,"kb_used_data":183168,"kb_used_omap":1,"kb_used_meta":590654,"kb_avail":93597968,"statfs":{"total":96636764160,"available":95844319232,"internally_reserved":0,"allocated":187564032,"data_stored":187362758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":604830149},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,12,1,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182505,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":770504,"kb_used_data":183544,"kb_used_omap":4,"kb_used_meta":586939,"kb_avail":93601336,"statfs":{"total":96636764160,"available":95847768064,"internally_reserved":0,"allocated":187949056,"data_stored":187747782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":601026253},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,3,4,6,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":186896384,"data_stored":186896384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":187121664,"data_stored":187121664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":186613760,"data_stored":186613760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":186642432,"data_stored":186642432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:42:38.335 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-05T16:42:38.335 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'10004", 'reported_seq': 10108, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:33.571091+0000', 'last_change': '2024-09-05T16:34:56.833497+0000', 'last_active': '2024-09-05T16:42:33.571091+0000', 'last_peered': '2024-09-05T16:42:33.571091+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.828289+0000', 'last_became_peered': '2024-09-05T16:34:56.828289+0000', 'last_unstale': '2024-09-05T16:42:33.571091+0000', 'last_undegraded': '2024-09-05T16:34:56.612160+0000', 'last_fullsized': '2024-09-05T16:34:56.557632+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10004, 'log_dups_size': 0, 'ondisk_log_size': 10004, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23756800, 'num_objects': 2900, 'num_object_clones': 0, 'num_object_copies': 11600, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2236, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2900, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10004, 'num_write_kb': 51616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 664}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:42:38.335 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-05T16:42:38.335 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "37'9760", 'reported_seq': 12685, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:33.291486+0000', 'last_change': '2024-09-05T16:40:28.919189+0000', 'last_active': '2024-09-05T16:42:33.291486+0000', 'last_peered': '2024-09-05T16:42:33.291486+0000', 'last_clean': '2024-09-05T16:33:53.200762+0000', 'last_became_active': '2024-09-05T16:34:56.820292+0000', 'last_became_peered': '2024-09-05T16:34:56.820292+0000', 'last_unstale': '2024-09-05T16:42:33.291486+0000', 'last_undegraded': '2024-09-05T16:34:56.605776+0000', 'last_fullsized': '2024-09-05T16:34:56.559383+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9760, 'log_dups_size': 0, 'ondisk_log_size': 9760, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22885536, 'num_objects': 2794, 'num_object_clones': 0, 'num_object_copies': 11176, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 844, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2794, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9758, 'num_write_kb': 50212, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1427, 'num_bytes_recovered': 2613248, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 1950}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:42:38.335 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:42:38.335 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'9945", 'reported_seq': 10067, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:33.674017+0000', 'last_change': '2024-09-05T16:34:56.867900+0000', 'last_active': '2024-09-05T16:42:33.674017+0000', 'last_peered': '2024-09-05T16:42:33.674017+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:42:33.674017+0000', 'last_undegraded': '2024-09-05T16:34:56.590972+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9945, 'log_dups_size': 0, 'ondisk_log_size': 9945, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22993512, 'num_objects': 2808, 'num_object_clones': 0, 'num_object_copies': 11232, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2233, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2808, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 9944, 'num_write_kb': 50999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 576}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:42:38.335 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:42:38.335 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'9754", 'reported_seq': 9851, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:33.546660+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:42:33.546660+0000', 'last_peered': '2024-09-05T16:42:33.546660+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:42:33.546660+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9754, 'log_dups_size': 0, 'ondisk_log_size': 9754, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23270560, 'num_objects': 2841, 'num_object_clones': 0, 'num_object_copies': 11364, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2147, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2841, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9752, 'num_write_kb': 50373, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 696}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:42:38.335 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:42:38.335 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'9948", 'reported_seq': 10062, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:33.643386+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:42:33.643386+0000', 'last_peered': '2024-09-05T16:42:33.643386+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:42:33.643386+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9948, 'log_dups_size': 0, 'ondisk_log_size': 9948, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23134208, 'num_objects': 2824, 'num_object_clones': 0, 'num_object_copies': 11296, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2824, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9948, 'num_write_kb': 51088, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 605}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:42:38.336 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:42:38.336 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'9811", 'reported_seq': 9919, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:33.290226+0000', 'last_change': '2024-09-05T16:34:56.903436+0000', 'last_active': '2024-09-05T16:42:33.290226+0000', 'last_peered': '2024-09-05T16:42:33.290226+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:42:33.290226+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9811, 'log_dups_size': 0, 'ondisk_log_size': 9811, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23352480, 'num_objects': 2851, 'num_object_clones': 0, 'num_object_copies': 11404, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2133, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2851, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9809, 'num_write_kb': 50636, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 720}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:42:38.336 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-05T16:42:38.336 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "37'9917", 'reported_seq': 10006, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:33.333214+0000', 'last_change': '2024-09-05T16:34:56.883319+0000', 'last_active': '2024-09-05T16:42:33.333214+0000', 'last_peered': '2024-09-05T16:42:33.333214+0000', 'last_clean': '2024-09-05T16:33:53.505305+0000', 'last_became_active': '2024-09-05T16:34:56.873389+0000', 'last_became_peered': '2024-09-05T16:34:56.873389+0000', 'last_unstale': '2024-09-05T16:42:33.333214+0000', 'last_undegraded': '2024-09-05T16:34:56.602567+0000', 'last_fullsized': '2024-09-05T16:34:56.559958+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9917, 'log_dups_size': 0, 'ondisk_log_size': 9917, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23442592, 'num_objects': 2862, 'num_object_clones': 0, 'num_object_copies': 11448, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2234, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2862, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9915, 'num_write_kb': 51108, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 630}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:42:38.336 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:42:38.336 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'9941", 'reported_seq': 10041, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:33.642213+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:42:33.642213+0000', 'last_peered': '2024-09-05T16:42:33.642213+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:42:33.642213+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9941, 'log_dups_size': 0, 'ondisk_log_size': 9941, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23202112, 'num_objects': 2833, 'num_object_clones': 0, 'num_object_copies': 11332, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2833, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9938, 'num_write_kb': 51081, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 589}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:42:38.336 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:42:38.336 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'10021", 'reported_seq': 10142, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:33.695179+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:42:33.695179+0000', 'last_peered': '2024-09-05T16:42:33.695179+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:42:33.695179+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10021, 'log_dups_size': 0, 'ondisk_log_size': 10021, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22844576, 'num_objects': 2789, 'num_object_clones': 0, 'num_object_copies': 11156, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2789, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10019, 'num_write_kb': 51230, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 544}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:42:38.336 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:42:38.336 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'10190", 'reported_seq': 10314, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:33.603806+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:42:33.603806+0000', 'last_peered': '2024-09-05T16:42:33.603806+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:42:33.603806+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "29'100", 'ondisk_log_start': "29'100", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10090, 'log_dups_size': 0, 'ondisk_log_size': 10090, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24022768, 'num_objects': 2933, 'num_object_clones': 0, 'num_object_copies': 11732, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2284, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2933, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10187, 'num_write_kb': 52476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 652}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:42:38.337 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:42:38.337 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'9892", 'reported_seq': 9989, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:33.383916+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:42:33.383916+0000', 'last_peered': '2024-09-05T16:42:33.383916+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:42:33.383916+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9892, 'log_dups_size': 0, 'ondisk_log_size': 9892, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23485008, 'num_objects': 2867, 'num_object_clones': 0, 'num_object_copies': 11468, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2867, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9891, 'num_write_kb': 51031, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 629}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:42:38.337 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:42:38.337 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'10221", 'reported_seq': 10329, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:33.472186+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:42:33.472186+0000', 'last_peered': '2024-09-05T16:42:33.472186+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:42:33.472186+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "29'200", 'ondisk_log_start': "29'200", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10021, 'log_dups_size': 0, 'ondisk_log_size': 10021, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23794848, 'num_objects': 2905, 'num_object_clones': 0, 'num_object_copies': 11620, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2905, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10219, 'num_write_kb': 52500, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 571}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:42:38.337 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:42:38.337 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'10031", 'reported_seq': 10143, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:33.553890+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:42:33.553890+0000', 'last_peered': '2024-09-05T16:42:33.553890+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:42:33.553890+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10031, 'log_dups_size': 0, 'ondisk_log_size': 10031, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22951072, 'num_objects': 2802, 'num_object_clones': 0, 'num_object_copies': 11208, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2310, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2802, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10029, 'num_write_kb': 51325, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 494}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:42:38.337 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:42:38.337 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'9924", 'reported_seq': 10032, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:33.175880+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:42:33.175880+0000', 'last_peered': '2024-09-05T16:42:33.175880+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:42:33.175880+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9924, 'log_dups_size': 0, 'ondisk_log_size': 9924, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23352480, 'num_objects': 2851, 'num_object_clones': 0, 'num_object_copies': 11404, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2851, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9922, 'num_write_kb': 51092, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 631}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:42:38.337 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:42:39.081 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:42:39.081 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 29931/182412 objects degraded (16.408%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":562,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":45603,"data_bytes":373535208,"bytes_used":2999717888,"bytes_avail":383547338752,"bytes_total":386547056640,"degraded_objects":29931,"degraded_total":182412,"degraded_ratio":0.16408459969738837,"recovering_objects_per_sec":9,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":110,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":169},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:42:39.115 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:42:39.115 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:42:39.761 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:42:39.762 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:42:39.782 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":309,"stamp":"2024-09-05T16:42:38.996529+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":365130216,"num_objects":44577,"num_object_clones":0,"num_object_copies":178308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29869,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44577,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":146260,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1768,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159984,"ondisk_log_size":159984,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2899864,"kb_used_data":716160,"kb_used_omap":14,"kb_used_meta":2183601,"kb_avail":374587496,"statfs":{"total":386547056640,"available":383577595904,"internally_reserved":0,"allocated":733347840,"data_stored":732542744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2236007991},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,6,6,24,8,15,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25092096,"num_objects":-3063,"num_object_clones":0,"num_object_copies":-12252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-172,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3063,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3063,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":172,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006339"},"pg_stats":[{"pgid":"1.f","version":"37'10070","reported_seq":10174,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:38.708140+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:42:38.708140+0000","last_peered":"2024-09-05T16:42:38.708140+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:42:38.708140+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23216128,"num_objects":2834,"num_object_clones":0,"num_object_copies":11336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2834,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10070,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":598}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'9828","reported_seq":12874,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:38.442104+0000","last_change":"2024-09-05T16:40:28.919189+0000","last_active":"2024-09-05T16:42:38.442104+0000","last_peered":"2024-09-05T16:42:38.442104+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:42:38.442104+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9828,"log_dups_size":0,"ondisk_log_size":9828,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22328480,"num_objects":2726,"num_object_clones":0,"num_object_copies":10904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":782,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2726,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9826,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1489,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1944}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'10011","reported_seq":10133,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:38.574967+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:42:38.574967+0000","last_peered":"2024-09-05T16:42:38.574967+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:42:38.574967+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22452840,"num_objects":2742,"num_object_clones":0,"num_object_copies":10968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2742,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10010,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":510}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'10094","reported_seq":14823,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:42:38.232050+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:42:38.232050+0000","last_peered":"2024-09-05T16:42:38.232050+0000","last_clean":"2024-09-05T16:42:38.232050+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:42:38.232050+0000","last_undegraded":"2024-09-05T16:42:38.232050+0000","last_fullsized":"2024-09-05T16:42:38.232050+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23265280,"num_objects":2840,"num_object_clones":0,"num_object_copies":11360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2840,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4394,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'9813","reported_seq":9912,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:38.162657+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:42:38.162657+0000","last_peered":"2024-09-05T16:42:38.162657+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:42:38.162657+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9813,"log_dups_size":0,"ondisk_log_size":9813,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22787232,"num_objects":2782,"num_object_clones":0,"num_object_copies":11128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2782,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9811,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":637}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'10021","reported_seq":10135,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:38.637166+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:42:38.637166+0000","last_peered":"2024-09-05T16:42:38.637166+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:42:38.637166+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22536192,"num_objects":2751,"num_object_clones":0,"num_object_copies":11004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2751,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10021,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":532}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'9864","reported_seq":9972,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:38.079914+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:42:38.079914+0000","last_peered":"2024-09-05T16:42:38.079914+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:42:38.079914+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9864,"log_dups_size":0,"ondisk_log_size":9864,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22918304,"num_objects":2798,"num_object_clones":0,"num_object_copies":11192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9862,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":667}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'9981","reported_seq":10070,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:38.444945+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:42:38.444945+0000","last_peered":"2024-09-05T16:42:38.444945+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:42:38.444945+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9981,"log_dups_size":0,"ondisk_log_size":9981,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22918304,"num_objects":2798,"num_object_clones":0,"num_object_copies":11192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9979,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":566}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'10014","reported_seq":10115,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:38.779138+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:42:38.779138+0000","last_peered":"2024-09-05T16:42:38.779138+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:42:38.779138+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22604096,"num_objects":2760,"num_object_clones":0,"num_object_copies":11040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2760,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10011,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":516}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'10083","reported_seq":10205,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:38.372169+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:42:38.372169+0000","last_peered":"2024-09-05T16:42:38.372169+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:42:38.372169+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22336672,"num_objects":2727,"num_object_clones":0,"num_object_copies":10908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2727,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10081,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":482}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'10259","reported_seq":10385,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:38.794695+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:42:38.794695+0000","last_peered":"2024-09-05T16:42:38.794695+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:42:38.794695+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"29'200","ondisk_log_start":"29'200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23457520,"num_objects":2864,"num_object_clones":0,"num_object_copies":11456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2864,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10256,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":583}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'9953","reported_seq":10051,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:38.539430+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:42:38.539430+0000","last_peered":"2024-09-05T16:42:38.539430+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:42:38.539430+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9953,"log_dups_size":0,"ondisk_log_size":9953,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22985296,"num_objects":2806,"num_object_clones":0,"num_object_copies":11224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2806,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9952,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":568}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'10019","reported_seq":14754,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:42:38.278402+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:42:38.278402+0000","last_peered":"2024-09-05T16:42:38.278402+0000","last_clean":"2024-09-05T16:42:38.278402+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:42:38.278402+0000","last_undegraded":"2024-09-05T16:42:38.278402+0000","last_fullsized":"2024-09-05T16:42:38.278402+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22847488,"num_objects":2789,"num_object_clones":0,"num_object_copies":11156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2789,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1619,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'10290","reported_seq":10398,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:38.680200+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:42:38.680200+0000","last_peered":"2024-09-05T16:42:38.680200+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:42:38.680200+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"29'200","ondisk_log_start":"29'200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23229600,"num_objects":2836,"num_object_clones":0,"num_object_copies":11344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2836,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10288,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":502}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'10091","reported_seq":10203,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:38.680092+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:42:38.680092+0000","last_peered":"2024-09-05T16:42:38.680092+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:42:38.680092+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22459552,"num_objects":2742,"num_object_clones":0,"num_object_copies":10968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2742,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10089,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":434}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'9993","reported_seq":10101,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:38.446163+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:42:38.446163+0000","last_peered":"2024-09-05T16:42:38.446163+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:42:38.446163+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9993,"log_dups_size":0,"ondisk_log_size":9993,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22787232,"num_objects":2782,"num_object_clones":0,"num_object_copies":11128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2782,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9991,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":562}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":365130216,"num_objects":44577,"num_object_clones":0,"num_object_copies":178308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29869,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44577,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":146260,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1768,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":730333184,"data_stored":730333184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159984,"ondisk_log_size":159984,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084392,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":765876,"kb_used_data":178848,"kb_used_omap":4,"kb_used_meta":587003,"kb_avail":93605964,"statfs":{"total":96636764160,"available":95852507136,"internally_reserved":0,"allocated":183140352,"data_stored":182939078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":601091795},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,1,4,3,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986269,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":585984,"kb_used_data":179440,"kb_used_omap":4,"kb_used_meta":406523,"kb_avail":93785856,"statfs":{"total":96636764160,"available":96036716544,"internally_reserved":0,"allocated":183746560,"data_stored":183545286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":416280274},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,5,6,4,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673075,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":773900,"kb_used_data":179068,"kb_used_omap":1,"kb_used_meta":594814,"kb_avail":93597940,"statfs":{"total":96636764160,"available":95844290560,"internally_reserved":0,"allocated":183365632,"data_stored":183164358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":609089989},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182506,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":774104,"kb_used_data":178804,"kb_used_omap":4,"kb_used_meta":595259,"kb_avail":93597736,"statfs":{"total":96636764160,"available":95844081664,"internally_reserved":0,"allocated":183095296,"data_stored":182894022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":609545933},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,3,0,13,1,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":182640640,"data_stored":182640640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":182919168,"data_stored":182919168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":182423552,"data_stored":182423552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":182349824,"data_stored":182349824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:42:42.784 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:42:43.435 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:42:43.436 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:42:43.465 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":311,"stamp":"2024-09-05T16:42:42.998196+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":365130216,"num_objects":44577,"num_object_clones":0,"num_object_copies":178308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29869,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44577,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":146260,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1768,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159984,"ondisk_log_size":159984,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2899864,"kb_used_data":716160,"kb_used_omap":14,"kb_used_meta":2183601,"kb_avail":374587496,"statfs":{"total":386547056640,"available":383577595904,"internally_reserved":0,"allocated":733347840,"data_stored":732542744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2236007991},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,6,6,24,8,15,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16670720,"num_objects":-2035,"num_object_clones":0,"num_object_copies":-8140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2035,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2035,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":115,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006498"},"pg_stats":[{"pgid":"1.f","version":"37'10070","reported_seq":10174,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:38.708140+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:42:38.708140+0000","last_peered":"2024-09-05T16:42:38.708140+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:42:38.708140+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23216128,"num_objects":2834,"num_object_clones":0,"num_object_copies":11336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2834,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10070,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":598}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'9828","reported_seq":12874,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:38.442104+0000","last_change":"2024-09-05T16:40:28.919189+0000","last_active":"2024-09-05T16:42:38.442104+0000","last_peered":"2024-09-05T16:42:38.442104+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:42:38.442104+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9828,"log_dups_size":0,"ondisk_log_size":9828,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22328480,"num_objects":2726,"num_object_clones":0,"num_object_copies":10904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":782,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2726,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9826,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1489,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1944}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'10011","reported_seq":10133,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:38.574967+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:42:38.574967+0000","last_peered":"2024-09-05T16:42:38.574967+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:42:38.574967+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22452840,"num_objects":2742,"num_object_clones":0,"num_object_copies":10968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2742,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10010,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":510}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'10094","reported_seq":14823,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:42:38.232050+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:42:38.232050+0000","last_peered":"2024-09-05T16:42:38.232050+0000","last_clean":"2024-09-05T16:42:38.232050+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:42:38.232050+0000","last_undegraded":"2024-09-05T16:42:38.232050+0000","last_fullsized":"2024-09-05T16:42:38.232050+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23265280,"num_objects":2840,"num_object_clones":0,"num_object_copies":11360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2840,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4394,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'9813","reported_seq":9912,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:38.162657+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:42:38.162657+0000","last_peered":"2024-09-05T16:42:38.162657+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:42:38.162657+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9813,"log_dups_size":0,"ondisk_log_size":9813,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22787232,"num_objects":2782,"num_object_clones":0,"num_object_copies":11128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2782,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9811,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":637}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'10021","reported_seq":10135,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:38.637166+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:42:38.637166+0000","last_peered":"2024-09-05T16:42:38.637166+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:42:38.637166+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22536192,"num_objects":2751,"num_object_clones":0,"num_object_copies":11004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2751,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10021,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":532}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'9864","reported_seq":9972,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:38.079914+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:42:38.079914+0000","last_peered":"2024-09-05T16:42:38.079914+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:42:38.079914+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9864,"log_dups_size":0,"ondisk_log_size":9864,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22918304,"num_objects":2798,"num_object_clones":0,"num_object_copies":11192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9862,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":667}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'9981","reported_seq":10070,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:38.444945+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:42:38.444945+0000","last_peered":"2024-09-05T16:42:38.444945+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:42:38.444945+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9981,"log_dups_size":0,"ondisk_log_size":9981,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22918304,"num_objects":2798,"num_object_clones":0,"num_object_copies":11192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9979,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":566}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'10014","reported_seq":10115,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:38.779138+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:42:38.779138+0000","last_peered":"2024-09-05T16:42:38.779138+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:42:38.779138+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22604096,"num_objects":2760,"num_object_clones":0,"num_object_copies":11040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2760,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10011,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":516}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'10083","reported_seq":10205,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:38.372169+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:42:38.372169+0000","last_peered":"2024-09-05T16:42:38.372169+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:42:38.372169+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22336672,"num_objects":2727,"num_object_clones":0,"num_object_copies":10908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2727,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10081,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":482}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'10259","reported_seq":10385,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:38.794695+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:42:38.794695+0000","last_peered":"2024-09-05T16:42:38.794695+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:42:38.794695+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"29'200","ondisk_log_start":"29'200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23457520,"num_objects":2864,"num_object_clones":0,"num_object_copies":11456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2864,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10256,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":583}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'9953","reported_seq":10051,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:38.539430+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:42:38.539430+0000","last_peered":"2024-09-05T16:42:38.539430+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:42:38.539430+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9953,"log_dups_size":0,"ondisk_log_size":9953,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22985296,"num_objects":2806,"num_object_clones":0,"num_object_copies":11224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2806,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9952,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":568}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'10019","reported_seq":14754,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:42:38.278402+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:42:38.278402+0000","last_peered":"2024-09-05T16:42:38.278402+0000","last_clean":"2024-09-05T16:42:38.278402+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:42:38.278402+0000","last_undegraded":"2024-09-05T16:42:38.278402+0000","last_fullsized":"2024-09-05T16:42:38.278402+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22847488,"num_objects":2789,"num_object_clones":0,"num_object_copies":11156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2789,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1619,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'10290","reported_seq":10398,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:38.680200+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:42:38.680200+0000","last_peered":"2024-09-05T16:42:38.680200+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:42:38.680200+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"29'200","ondisk_log_start":"29'200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23229600,"num_objects":2836,"num_object_clones":0,"num_object_copies":11344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2836,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10288,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":502}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'10091","reported_seq":10203,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:38.680092+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:42:38.680092+0000","last_peered":"2024-09-05T16:42:38.680092+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:42:38.680092+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22459552,"num_objects":2742,"num_object_clones":0,"num_object_copies":10968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2742,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10089,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":434}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'9993","reported_seq":10101,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:38.446163+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:42:38.446163+0000","last_peered":"2024-09-05T16:42:38.446163+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:42:38.446163+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9993,"log_dups_size":0,"ondisk_log_size":9993,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22787232,"num_objects":2782,"num_object_clones":0,"num_object_copies":11128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2782,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9991,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":562}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":365130216,"num_objects":44577,"num_object_clones":0,"num_object_copies":178308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29869,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44577,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":146260,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1768,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":730333184,"data_stored":730333184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159984,"ondisk_log_size":159984,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084392,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":765876,"kb_used_data":178848,"kb_used_omap":4,"kb_used_meta":587003,"kb_avail":93605964,"statfs":{"total":96636764160,"available":95852507136,"internally_reserved":0,"allocated":183140352,"data_stored":182939078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":601091795},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,1,4,3,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986269,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":585984,"kb_used_data":179440,"kb_used_omap":4,"kb_used_meta":406523,"kb_avail":93785856,"statfs":{"total":96636764160,"available":96036716544,"internally_reserved":0,"allocated":183746560,"data_stored":183545286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":416280274},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,5,6,4,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673075,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":773900,"kb_used_data":179068,"kb_used_omap":1,"kb_used_meta":594814,"kb_avail":93597940,"statfs":{"total":96636764160,"available":95844290560,"internally_reserved":0,"allocated":183365632,"data_stored":183164358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":609089989},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182506,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":774104,"kb_used_data":178804,"kb_used_omap":4,"kb_used_meta":595259,"kb_avail":93597736,"statfs":{"total":96636764160,"available":95844081664,"internally_reserved":0,"allocated":183095296,"data_stored":182894022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":609545933},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,3,0,13,1,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":182640640,"data_stored":182640640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":182919168,"data_stored":182919168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":182423552,"data_stored":182423552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":182349824,"data_stored":182349824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:42:43.466 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:42:44.108 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:42:44.113 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:42:44.138 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":311,"stamp":"2024-09-05T16:42:42.998196+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":365130216,"num_objects":44577,"num_object_clones":0,"num_object_copies":178308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29869,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44577,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":146260,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1768,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159984,"ondisk_log_size":159984,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2899864,"kb_used_data":716160,"kb_used_omap":14,"kb_used_meta":2183601,"kb_avail":374587496,"statfs":{"total":386547056640,"available":383577595904,"internally_reserved":0,"allocated":733347840,"data_stored":732542744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2236007991},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,6,6,24,8,15,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16670720,"num_objects":-2035,"num_object_clones":0,"num_object_copies":-8140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2035,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2035,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":115,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006498"},"pg_stats":[{"pgid":"1.f","version":"37'10070","reported_seq":10174,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:38.708140+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:42:38.708140+0000","last_peered":"2024-09-05T16:42:38.708140+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:42:38.708140+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23216128,"num_objects":2834,"num_object_clones":0,"num_object_copies":11336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2834,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10070,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":598}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'9828","reported_seq":12874,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:38.442104+0000","last_change":"2024-09-05T16:40:28.919189+0000","last_active":"2024-09-05T16:42:38.442104+0000","last_peered":"2024-09-05T16:42:38.442104+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:42:38.442104+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9828,"log_dups_size":0,"ondisk_log_size":9828,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22328480,"num_objects":2726,"num_object_clones":0,"num_object_copies":10904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":782,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2726,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9826,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1489,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1944}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'10011","reported_seq":10133,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:38.574967+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:42:38.574967+0000","last_peered":"2024-09-05T16:42:38.574967+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:42:38.574967+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22452840,"num_objects":2742,"num_object_clones":0,"num_object_copies":10968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2742,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10010,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":510}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'10094","reported_seq":14823,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:42:38.232050+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:42:38.232050+0000","last_peered":"2024-09-05T16:42:38.232050+0000","last_clean":"2024-09-05T16:42:38.232050+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:42:38.232050+0000","last_undegraded":"2024-09-05T16:42:38.232050+0000","last_fullsized":"2024-09-05T16:42:38.232050+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23265280,"num_objects":2840,"num_object_clones":0,"num_object_copies":11360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2840,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4394,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'9813","reported_seq":9912,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:38.162657+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:42:38.162657+0000","last_peered":"2024-09-05T16:42:38.162657+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:42:38.162657+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9813,"log_dups_size":0,"ondisk_log_size":9813,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22787232,"num_objects":2782,"num_object_clones":0,"num_object_copies":11128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2782,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9811,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":637}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'10021","reported_seq":10135,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:38.637166+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:42:38.637166+0000","last_peered":"2024-09-05T16:42:38.637166+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:42:38.637166+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22536192,"num_objects":2751,"num_object_clones":0,"num_object_copies":11004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2751,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10021,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":532}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'9864","reported_seq":9972,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:38.079914+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:42:38.079914+0000","last_peered":"2024-09-05T16:42:38.079914+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:42:38.079914+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9864,"log_dups_size":0,"ondisk_log_size":9864,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22918304,"num_objects":2798,"num_object_clones":0,"num_object_copies":11192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9862,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":667}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'9981","reported_seq":10070,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:38.444945+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:42:38.444945+0000","last_peered":"2024-09-05T16:42:38.444945+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:42:38.444945+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9981,"log_dups_size":0,"ondisk_log_size":9981,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22918304,"num_objects":2798,"num_object_clones":0,"num_object_copies":11192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9979,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":566}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'10014","reported_seq":10115,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:38.779138+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:42:38.779138+0000","last_peered":"2024-09-05T16:42:38.779138+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:42:38.779138+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22604096,"num_objects":2760,"num_object_clones":0,"num_object_copies":11040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2760,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10011,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":516}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'10083","reported_seq":10205,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:38.372169+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:42:38.372169+0000","last_peered":"2024-09-05T16:42:38.372169+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:42:38.372169+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22336672,"num_objects":2727,"num_object_clones":0,"num_object_copies":10908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2727,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10081,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":482}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'10259","reported_seq":10385,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:38.794695+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:42:38.794695+0000","last_peered":"2024-09-05T16:42:38.794695+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:42:38.794695+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"29'200","ondisk_log_start":"29'200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23457520,"num_objects":2864,"num_object_clones":0,"num_object_copies":11456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2864,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10256,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":583}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'9953","reported_seq":10051,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:38.539430+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:42:38.539430+0000","last_peered":"2024-09-05T16:42:38.539430+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:42:38.539430+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9953,"log_dups_size":0,"ondisk_log_size":9953,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22985296,"num_objects":2806,"num_object_clones":0,"num_object_copies":11224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2806,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9952,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":568}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'10019","reported_seq":14754,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:42:38.278402+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:42:38.278402+0000","last_peered":"2024-09-05T16:42:38.278402+0000","last_clean":"2024-09-05T16:42:38.278402+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:42:38.278402+0000","last_undegraded":"2024-09-05T16:42:38.278402+0000","last_fullsized":"2024-09-05T16:42:38.278402+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22847488,"num_objects":2789,"num_object_clones":0,"num_object_copies":11156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2789,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1619,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'10290","reported_seq":10398,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:38.680200+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:42:38.680200+0000","last_peered":"2024-09-05T16:42:38.680200+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:42:38.680200+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"29'200","ondisk_log_start":"29'200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23229600,"num_objects":2836,"num_object_clones":0,"num_object_copies":11344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2836,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10288,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":502}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'10091","reported_seq":10203,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:38.680092+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:42:38.680092+0000","last_peered":"2024-09-05T16:42:38.680092+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:42:38.680092+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22459552,"num_objects":2742,"num_object_clones":0,"num_object_copies":10968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2742,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10089,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":434}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'9993","reported_seq":10101,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:38.446163+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:42:38.446163+0000","last_peered":"2024-09-05T16:42:38.446163+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:42:38.446163+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9993,"log_dups_size":0,"ondisk_log_size":9993,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22787232,"num_objects":2782,"num_object_clones":0,"num_object_copies":11128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2782,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9991,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":562}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":365130216,"num_objects":44577,"num_object_clones":0,"num_object_copies":178308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29869,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44577,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":146260,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1768,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":730333184,"data_stored":730333184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159984,"ondisk_log_size":159984,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084392,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":765876,"kb_used_data":178848,"kb_used_omap":4,"kb_used_meta":587003,"kb_avail":93605964,"statfs":{"total":96636764160,"available":95852507136,"internally_reserved":0,"allocated":183140352,"data_stored":182939078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":601091795},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,1,4,3,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986269,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":585984,"kb_used_data":179440,"kb_used_omap":4,"kb_used_meta":406523,"kb_avail":93785856,"statfs":{"total":96636764160,"available":96036716544,"internally_reserved":0,"allocated":183746560,"data_stored":183545286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":416280274},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,5,6,4,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673075,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":773900,"kb_used_data":179068,"kb_used_omap":1,"kb_used_meta":594814,"kb_avail":93597940,"statfs":{"total":96636764160,"available":95844290560,"internally_reserved":0,"allocated":183365632,"data_stored":183164358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":609089989},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182506,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":774104,"kb_used_data":178804,"kb_used_omap":4,"kb_used_meta":595259,"kb_avail":93597736,"statfs":{"total":96636764160,"available":95844081664,"internally_reserved":0,"allocated":183095296,"data_stored":182894022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":609545933},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,3,0,13,1,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":182640640,"data_stored":182640640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":182919168,"data_stored":182919168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":182423552,"data_stored":182423552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":182349824,"data_stored":182349824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:42:44.140 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-05T16:42:44.140 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'10070", 'reported_seq': 10174, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:38.708140+0000', 'last_change': '2024-09-05T16:34:56.833497+0000', 'last_active': '2024-09-05T16:42:38.708140+0000', 'last_peered': '2024-09-05T16:42:38.708140+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.828289+0000', 'last_became_peered': '2024-09-05T16:34:56.828289+0000', 'last_unstale': '2024-09-05T16:42:38.708140+0000', 'last_undegraded': '2024-09-05T16:34:56.612160+0000', 'last_fullsized': '2024-09-05T16:34:56.557632+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10070, 'log_dups_size': 0, 'ondisk_log_size': 10070, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23216128, 'num_objects': 2834, 'num_object_clones': 0, 'num_object_copies': 11336, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2236, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2834, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10070, 'num_write_kb': 51616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 598}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:42:44.141 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-05T16:42:44.141 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "37'9828", 'reported_seq': 12874, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:38.442104+0000', 'last_change': '2024-09-05T16:40:28.919189+0000', 'last_active': '2024-09-05T16:42:38.442104+0000', 'last_peered': '2024-09-05T16:42:38.442104+0000', 'last_clean': '2024-09-05T16:33:53.200762+0000', 'last_became_active': '2024-09-05T16:34:56.820292+0000', 'last_became_peered': '2024-09-05T16:34:56.820292+0000', 'last_unstale': '2024-09-05T16:42:38.442104+0000', 'last_undegraded': '2024-09-05T16:34:56.605776+0000', 'last_fullsized': '2024-09-05T16:34:56.559383+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9828, 'log_dups_size': 0, 'ondisk_log_size': 9828, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22328480, 'num_objects': 2726, 'num_object_clones': 0, 'num_object_copies': 10904, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 782, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2726, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9826, 'num_write_kb': 50212, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1489, 'num_bytes_recovered': 2613248, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 1944}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:42:44.141 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:42:44.141 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'10011", 'reported_seq': 10133, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:38.574967+0000', 'last_change': '2024-09-05T16:34:56.867900+0000', 'last_active': '2024-09-05T16:42:38.574967+0000', 'last_peered': '2024-09-05T16:42:38.574967+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:42:38.574967+0000', 'last_undegraded': '2024-09-05T16:34:56.590972+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10011, 'log_dups_size': 0, 'ondisk_log_size': 10011, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22452840, 'num_objects': 2742, 'num_object_clones': 0, 'num_object_copies': 10968, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2233, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2742, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 10010, 'num_write_kb': 50999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 510}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:42:44.141 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:42:44.141 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'9813", 'reported_seq': 9912, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:38.162657+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:42:38.162657+0000', 'last_peered': '2024-09-05T16:42:38.162657+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:42:38.162657+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9813, 'log_dups_size': 0, 'ondisk_log_size': 9813, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22787232, 'num_objects': 2782, 'num_object_clones': 0, 'num_object_copies': 11128, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2147, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2782, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9811, 'num_write_kb': 50373, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 637}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:42:44.142 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:42:44.142 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'10021", 'reported_seq': 10135, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:38.637166+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:42:38.637166+0000', 'last_peered': '2024-09-05T16:42:38.637166+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:42:38.637166+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10021, 'log_dups_size': 0, 'ondisk_log_size': 10021, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22536192, 'num_objects': 2751, 'num_object_clones': 0, 'num_object_copies': 11004, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2751, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10021, 'num_write_kb': 51088, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 532}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:42:44.142 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:42:44.142 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'9864", 'reported_seq': 9972, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:38.079914+0000', 'last_change': '2024-09-05T16:34:56.903436+0000', 'last_active': '2024-09-05T16:42:38.079914+0000', 'last_peered': '2024-09-05T16:42:38.079914+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:42:38.079914+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9864, 'log_dups_size': 0, 'ondisk_log_size': 9864, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22918304, 'num_objects': 2798, 'num_object_clones': 0, 'num_object_copies': 11192, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2133, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2798, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9862, 'num_write_kb': 50636, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 667}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:42:44.142 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-05T16:42:44.143 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "37'9981", 'reported_seq': 10070, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:38.444945+0000', 'last_change': '2024-09-05T16:34:56.883319+0000', 'last_active': '2024-09-05T16:42:38.444945+0000', 'last_peered': '2024-09-05T16:42:38.444945+0000', 'last_clean': '2024-09-05T16:33:53.505305+0000', 'last_became_active': '2024-09-05T16:34:56.873389+0000', 'last_became_peered': '2024-09-05T16:34:56.873389+0000', 'last_unstale': '2024-09-05T16:42:38.444945+0000', 'last_undegraded': '2024-09-05T16:34:56.602567+0000', 'last_fullsized': '2024-09-05T16:34:56.559958+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9981, 'log_dups_size': 0, 'ondisk_log_size': 9981, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22918304, 'num_objects': 2798, 'num_object_clones': 0, 'num_object_copies': 11192, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2234, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2798, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9979, 'num_write_kb': 51108, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 566}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:42:44.143 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:42:44.143 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'10014", 'reported_seq': 10115, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:38.779138+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:42:38.779138+0000', 'last_peered': '2024-09-05T16:42:38.779138+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:42:38.779138+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10014, 'log_dups_size': 0, 'ondisk_log_size': 10014, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22604096, 'num_objects': 2760, 'num_object_clones': 0, 'num_object_copies': 11040, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2760, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10011, 'num_write_kb': 51081, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 516}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:42:44.143 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:42:44.144 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'10083", 'reported_seq': 10205, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:38.372169+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:42:38.372169+0000', 'last_peered': '2024-09-05T16:42:38.372169+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:42:38.372169+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10083, 'log_dups_size': 0, 'ondisk_log_size': 10083, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22336672, 'num_objects': 2727, 'num_object_clones': 0, 'num_object_copies': 10908, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2727, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10081, 'num_write_kb': 51230, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 482}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:42:44.144 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:42:44.144 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'10259", 'reported_seq': 10385, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:38.794695+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:42:38.794695+0000', 'last_peered': '2024-09-05T16:42:38.794695+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:42:38.794695+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "29'200", 'ondisk_log_start': "29'200", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10059, 'log_dups_size': 0, 'ondisk_log_size': 10059, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23457520, 'num_objects': 2864, 'num_object_clones': 0, 'num_object_copies': 11456, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2284, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2864, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10256, 'num_write_kb': 52476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 583}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:42:44.144 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:42:44.144 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'9953", 'reported_seq': 10051, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:38.539430+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:42:38.539430+0000', 'last_peered': '2024-09-05T16:42:38.539430+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:42:38.539430+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9953, 'log_dups_size': 0, 'ondisk_log_size': 9953, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22985296, 'num_objects': 2806, 'num_object_clones': 0, 'num_object_copies': 11224, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2806, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9952, 'num_write_kb': 51031, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 568}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:42:44.145 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:42:44.145 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'10290", 'reported_seq': 10398, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:38.680200+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:42:38.680200+0000', 'last_peered': '2024-09-05T16:42:38.680200+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:42:38.680200+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "29'200", 'ondisk_log_start': "29'200", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10090, 'log_dups_size': 0, 'ondisk_log_size': 10090, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23229600, 'num_objects': 2836, 'num_object_clones': 0, 'num_object_copies': 11344, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2836, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10288, 'num_write_kb': 52500, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 502}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:42:44.145 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:42:44.145 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'10091", 'reported_seq': 10203, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:38.680092+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:42:38.680092+0000', 'last_peered': '2024-09-05T16:42:38.680092+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:42:38.680092+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10091, 'log_dups_size': 0, 'ondisk_log_size': 10091, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22459552, 'num_objects': 2742, 'num_object_clones': 0, 'num_object_copies': 10968, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2310, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2742, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10089, 'num_write_kb': 51325, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 434}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:42:44.146 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:42:44.146 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'9993", 'reported_seq': 10101, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:38.446163+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:42:38.446163+0000', 'last_peered': '2024-09-05T16:42:38.446163+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:42:38.446163+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9993, 'log_dups_size': 0, 'ondisk_log_size': 9993, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22787232, 'num_objects': 2782, 'num_object_clones': 0, 'num_object_copies': 11128, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2782, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9991, 'num_write_kb': 51092, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 562}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:42:44.146 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:42:44.898 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:42:44.898 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 29869/178308 objects degraded (16.751%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":568,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":44577,"data_bytes":365130216,"bytes_used":2969460736,"bytes_avail":383577595904,"bytes_total":386547056640,"degraded_objects":29869,"degraded_total":178308,"degraded_ratio":0.16751351593871278,"recovering_objects_per_sec":9,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":115,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":169},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:42:44.918 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:42:44.919 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:42:45.485 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:42:45.486 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:42:45.510 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":312,"stamp":"2024-09-05T16:42:44.999057+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":356626920,"num_objects":43539,"num_object_clones":0,"num_object_copies":174156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29813,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43539,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":147298,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1824,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160422,"ondisk_log_size":160422,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2918196,"kb_used_data":699804,"kb_used_omap":14,"kb_used_meta":2218289,"kb_avail":374569164,"statfs":{"total":386547056640,"available":383558823936,"internally_reserved":0,"allocated":716599296,"data_stored":715794200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2271528503},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,1,1,8,6,10,10,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25174016,"num_objects":-3073,"num_object_clones":0,"num_object_copies":-12292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-171,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3073,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3073,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004882"},"pg_stats":[{"pgid":"1.f","version":"37'10138","reported_seq":10242,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:43.741845+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:42:43.741845+0000","last_peered":"2024-09-05T16:42:43.741845+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:42:43.741845+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"29'100","ondisk_log_start":"29'100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22659072,"num_objects":2766,"num_object_clones":0,"num_object_copies":11064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2766,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10138,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":530}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'9888","reported_seq":13049,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:43.507965+0000","last_change":"2024-09-05T16:40:28.919189+0000","last_active":"2024-09-05T16:42:43.507965+0000","last_peered":"2024-09-05T16:42:43.507965+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:42:43.507965+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9888,"log_dups_size":0,"ondisk_log_size":9888,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21836960,"num_objects":2666,"num_object_clones":0,"num_object_copies":10664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":726,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2666,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9886,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1545,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1940}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'10067","reported_seq":10189,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:43.692688+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:42:43.692688+0000","last_peered":"2024-09-05T16:42:43.692688+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:42:43.692688+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21994088,"num_objects":2686,"num_object_clones":0,"num_object_copies":10744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2686,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10066,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":454}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'10167","reported_seq":14896,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:42:43.082741+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:42:43.082741+0000","last_peered":"2024-09-05T16:42:43.082741+0000","last_clean":"2024-09-05T16:42:43.082741+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:42:43.082741+0000","last_undegraded":"2024-09-05T16:42:43.082741+0000","last_fullsized":"2024-09-05T16:42:43.082741+0000","mapping_epoch":34,"log_start":"29'100","ondisk_log_start":"29'100","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22667264,"num_objects":2767,"num_object_clones":0,"num_object_copies":11068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2767,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4467,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'9883","reported_seq":9982,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:43.764249+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:42:43.764249+0000","last_peered":"2024-09-05T16:42:43.764249+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:42:43.764249+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9883,"log_dups_size":0,"ondisk_log_size":9883,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22213792,"num_objects":2712,"num_object_clones":0,"num_object_copies":10848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2712,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9881,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":567}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'10099","reported_seq":10212,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:43.815106+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:42:43.815106+0000","last_peered":"2024-09-05T16:42:43.815106+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:42:43.815106+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21897216,"num_objects":2673,"num_object_clones":0,"num_object_copies":10692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2673,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10099,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":454}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'9933","reported_seq":10040,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:43.435875+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:42:43.435875+0000","last_peered":"2024-09-05T16:42:43.435875+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:42:43.435875+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9933,"log_dups_size":0,"ondisk_log_size":9933,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22353056,"num_objects":2729,"num_object_clones":0,"num_object_copies":10916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2729,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9931,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":598}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'10046","reported_seq":10135,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:43.314560+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:42:43.314560+0000","last_peered":"2024-09-05T16:42:43.314560+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:42:43.314560+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22385824,"num_objects":2733,"num_object_clones":0,"num_object_copies":10932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10044,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":501}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'10077","reported_seq":10178,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:43.613069+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:42:43.613069+0000","last_peered":"2024-09-05T16:42:43.613069+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:42:43.613069+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22088000,"num_objects":2697,"num_object_clones":0,"num_object_copies":10788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2697,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10074,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":453}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'10157","reported_seq":10279,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:43.692768+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:42:43.692768+0000","last_peered":"2024-09-05T16:42:43.692768+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:42:43.692768+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"29'100","ondisk_log_start":"29'100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21730464,"num_objects":2653,"num_object_clones":0,"num_object_copies":10612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2653,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10155,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":408}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'10319","reported_seq":10445,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:43.820057+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:42:43.820057+0000","last_peered":"2024-09-05T16:42:43.820057+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:42:43.820057+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"29'300","ondisk_log_start":"29'300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22966000,"num_objects":2804,"num_object_clones":0,"num_object_copies":11216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2804,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10316,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":523}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'10001","reported_seq":10099,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:43.823975+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:42:43.823975+0000","last_peered":"2024-09-05T16:42:43.823975+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:42:43.823975+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22592080,"num_objects":2758,"num_object_clones":0,"num_object_copies":11032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2758,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10000,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":520}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'10091","reported_seq":14826,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:42:43.359772+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:42:43.359772+0000","last_peered":"2024-09-05T16:42:43.359772+0000","last_clean":"2024-09-05T16:42:43.359772+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:42:43.359772+0000","last_undegraded":"2024-09-05T16:42:43.359772+0000","last_fullsized":"2024-09-05T16:42:43.359772+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22257664,"num_objects":2717,"num_object_clones":0,"num_object_copies":10868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2717,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1691,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'10350","reported_seq":10458,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:43.765375+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:42:43.765375+0000","last_peered":"2024-09-05T16:42:43.765375+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:42:43.765375+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"29'300","ondisk_log_start":"29'300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22738080,"num_objects":2776,"num_object_clones":0,"num_object_copies":11104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2776,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10348,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":442}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'10149","reported_seq":10261,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:43.712726+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:42:43.712726+0000","last_peered":"2024-09-05T16:42:43.712726+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:42:43.712726+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"29'100","ondisk_log_start":"29'100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21984416,"num_objects":2684,"num_object_clones":0,"num_object_copies":10736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2684,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10147,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":376}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'10057","reported_seq":10165,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:43.437295+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:42:43.437295+0000","last_peered":"2024-09-05T16:42:43.437295+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:42:43.437295+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22262944,"num_objects":2718,"num_object_clones":0,"num_object_copies":10872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2718,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10055,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":498}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":356626920,"num_objects":43539,"num_object_clones":0,"num_object_copies":174156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29813,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43539,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":147298,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1824,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":713445376,"data_stored":713445376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160422,"ondisk_log_size":160422,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084393,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":765900,"kb_used_data":174708,"kb_used_omap":4,"kb_used_meta":591163,"kb_avail":93605940,"statfs":{"total":96636764160,"available":95852482560,"internally_reserved":0,"allocated":178900992,"data_stored":178699718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":605351635},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,0,0,3,3,3,5,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986270,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":604200,"kb_used_data":175432,"kb_used_omap":4,"kb_used_meta":428731,"kb_avail":93767640,"statfs":{"total":96636764160,"available":96018063360,"internally_reserved":0,"allocated":179642368,"data_stored":179441094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":439021266},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4,1,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673076,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":773944,"kb_used_data":174952,"kb_used_omap":1,"kb_used_meta":598974,"kb_avail":93597896,"statfs":{"total":96636764160,"available":95844245504,"internally_reserved":0,"allocated":179150848,"data_stored":178949574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":613349829},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182507,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":774152,"kb_used_data":174712,"kb_used_omap":4,"kb_used_meta":599419,"kb_avail":93597688,"statfs":{"total":96636764160,"available":95844032512,"internally_reserved":0,"allocated":178905088,"data_stored":178703814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":613805773},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,1,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":178388992,"data_stored":178388992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":178712576,"data_stored":178712576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":178204672,"data_stored":178204672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":178139136,"data_stored":178139136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:42:48.512 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:42:49.144 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:42:49.144 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:42:49.168 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":314,"stamp":"2024-09-05T16:42:49.000701+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":348336616,"num_objects":42527,"num_object_clones":0,"num_object_copies":170108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29753,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42527,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":148310,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1884,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160334,"ondisk_log_size":160334,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2918960,"kb_used_data":683944,"kb_used_omap":14,"kb_used_meta":2234929,"kb_avail":374568400,"statfs":{"total":386547056640,"available":383558041600,"internally_reserved":0,"allocated":700358656,"data_stored":699553560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2288567863},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,9,7,1,14,17,8,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25198592,"num_objects":-3076,"num_object_clones":0,"num_object_copies":-12304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3076,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3076,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":178,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004963"},"pg_stats":[{"pgid":"1.f","version":"37'10214","reported_seq":10318,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:48.783422+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:42:48.783422+0000","last_peered":"2024-09-05T16:42:48.783422+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:42:48.783422+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"29'200","ondisk_log_start":"29'200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22036480,"num_objects":2690,"num_object_clones":0,"num_object_copies":10760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2690,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10214,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":454}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'9971","reported_seq":13251,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:48.758730+0000","last_change":"2024-09-05T16:40:28.919189+0000","last_active":"2024-09-05T16:42:48.758730+0000","last_peered":"2024-09-05T16:42:48.758730+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:42:48.758730+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9971,"log_dups_size":0,"ondisk_log_size":9971,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21157024,"num_objects":2583,"num_object_clones":0,"num_object_copies":10332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":666,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2583,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9969,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1605,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1917}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'10117","reported_seq":10238,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:48.782429+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:42:48.782429+0000","last_peered":"2024-09-05T16:42:48.782429+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:42:48.782429+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"29'100","ondisk_log_start":"29'100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21584488,"num_objects":2636,"num_object_clones":0,"num_object_copies":10544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2636,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10116,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":404}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'10213","reported_seq":14942,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:42:48.443888+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:42:48.443888+0000","last_peered":"2024-09-05T16:42:48.443888+0000","last_clean":"2024-09-05T16:42:48.443888+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:42:48.443888+0000","last_undegraded":"2024-09-05T16:42:48.443888+0000","last_fullsized":"2024-09-05T16:42:48.443888+0000","mapping_epoch":34,"log_start":"29'200","ondisk_log_start":"29'200","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22290432,"num_objects":2721,"num_object_clones":0,"num_object_copies":10884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2721,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4513,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'9942","reported_seq":10041,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:48.676223+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:42:48.676223+0000","last_peered":"2024-09-05T16:42:48.676223+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:42:48.676223+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9942,"log_dups_size":0,"ondisk_log_size":9942,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21730464,"num_objects":2653,"num_object_clones":0,"num_object_copies":10612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2653,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9940,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":508}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'10164","reported_seq":10278,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:48.740305+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:42:48.740305+0000","last_peered":"2024-09-05T16:42:48.740305+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:42:48.740305+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"29'100","ondisk_log_start":"29'100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21364736,"num_objects":2608,"num_object_clones":0,"num_object_copies":10432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2608,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10164,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":389}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'10001","reported_seq":10109,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:48.577528+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:42:48.577528+0000","last_peered":"2024-09-05T16:42:48.577528+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:42:48.577528+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21796000,"num_objects":2661,"num_object_clones":0,"num_object_copies":10644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2661,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9999,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":530}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'10124","reported_seq":10213,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:48.597217+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:42:48.597217+0000","last_peered":"2024-09-05T16:42:48.597217+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:42:48.597217+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"29'100","ondisk_log_start":"29'100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21746848,"num_objects":2655,"num_object_clones":0,"num_object_copies":10620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2655,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10122,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":423}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'10140","reported_seq":10241,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:48.841393+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:42:48.841393+0000","last_peered":"2024-09-05T16:42:48.841393+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:42:48.841393+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"29'100","ondisk_log_start":"29'100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21571904,"num_objects":2634,"num_object_clones":0,"num_object_copies":10536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2634,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10137,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":390}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'10225","reported_seq":10346,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:48.751302+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:42:48.751302+0000","last_peered":"2024-09-05T16:42:48.751302+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:42:48.751302+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"29'200","ondisk_log_start":"29'200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21173408,"num_objects":2585,"num_object_clones":0,"num_object_copies":10340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2585,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10223,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":340}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'10377","reported_seq":10504,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:48.585634+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:42:48.585634+0000","last_peered":"2024-09-05T16:42:48.585634+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:42:48.585634+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"29'300","ondisk_log_start":"29'300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22490864,"num_objects":2746,"num_object_clones":0,"num_object_copies":10984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2746,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10374,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":465}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'10060","reported_seq":10158,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:48.683932+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:42:48.683932+0000","last_peered":"2024-09-05T16:42:48.683932+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:42:48.683932+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22108752,"num_objects":2699,"num_object_clones":0,"num_object_copies":10796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2699,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10059,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":461}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'10157","reported_seq":14893,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:42:48.389715+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:42:48.389715+0000","last_peered":"2024-09-05T16:42:48.389715+0000","last_clean":"2024-09-05T16:42:48.389715+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:42:48.389715+0000","last_undegraded":"2024-09-05T16:42:48.389715+0000","last_fullsized":"2024-09-05T16:42:48.389715+0000","mapping_epoch":36,"log_start":"29'100","ondisk_log_start":"29'100","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21716992,"num_objects":2651,"num_object_clones":0,"num_object_copies":10604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2651,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1757,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'10409","reported_seq":10517,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:48.831196+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:42:48.831196+0000","last_peered":"2024-09-05T16:42:48.831196+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:42:48.831196+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"29'400","ondisk_log_start":"29'400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22254752,"num_objects":2717,"num_object_clones":0,"num_object_copies":10868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2717,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10407,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":383}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'10201","reported_seq":10314,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:48.588899+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:42:48.588899+0000","last_peered":"2024-09-05T16:42:48.588899+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:42:48.588899+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"29'200","ondisk_log_start":"29'200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21558432,"num_objects":2632,"num_object_clones":0,"num_object_copies":10528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2632,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10199,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":324}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'10119","reported_seq":10227,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:48.590976+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:42:48.590976+0000","last_peered":"2024-09-05T16:42:48.590976+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:42:48.590976+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"29'100","ondisk_log_start":"29'100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21755040,"num_objects":2656,"num_object_clones":0,"num_object_copies":10624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2656,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10117,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":436}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":348336616,"num_objects":42527,"num_object_clones":0,"num_object_copies":170108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29753,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42527,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":148310,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1884,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":696745984,"data_stored":696745984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160334,"ondisk_log_size":160334,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084394,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":766036,"kb_used_data":170692,"kb_used_omap":4,"kb_used_meta":595323,"kb_avail":93605804,"statfs":{"total":96636764160,"available":95852343296,"internally_reserved":0,"allocated":174788608,"data_stored":174587334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":609611475},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,7,0,0,2,6,3,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986271,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":604444,"kb_used_data":171524,"kb_used_omap":4,"kb_used_meta":432891,"kb_avail":93767396,"statfs":{"total":96636764160,"available":96017813504,"internally_reserved":0,"allocated":175640576,"data_stored":175439302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":443281106},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,0,4,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673077,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":774224,"kb_used_data":171072,"kb_used_omap":1,"kb_used_meta":603134,"kb_avail":93597616,"statfs":{"total":96636764160,"available":95843958784,"internally_reserved":0,"allocated":175177728,"data_stored":174976454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":617609669},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,1,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182508,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":774256,"kb_used_data":170656,"kb_used_omap":4,"kb_used_meta":603579,"kb_avail":93597584,"statfs":{"total":96636764160,"available":95843926016,"internally_reserved":0,"allocated":174751744,"data_stored":174550470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":618065613},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,4,4,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":174141440,"data_stored":174141440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":174505984,"data_stored":174505984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":174096384,"data_stored":174096384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":174002176,"data_stored":174002176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:42:49.169 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:42:49.814 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:42:49.814 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:42:49.835 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":314,"stamp":"2024-09-05T16:42:49.000701+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":348336616,"num_objects":42527,"num_object_clones":0,"num_object_copies":170108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29753,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42527,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":148310,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1884,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160334,"ondisk_log_size":160334,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2918960,"kb_used_data":683944,"kb_used_omap":14,"kb_used_meta":2234929,"kb_avail":374568400,"statfs":{"total":386547056640,"available":383558041600,"internally_reserved":0,"allocated":700358656,"data_stored":699553560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2288567863},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,9,7,1,14,17,8,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25198592,"num_objects":-3076,"num_object_clones":0,"num_object_copies":-12304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3076,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3076,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":178,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004963"},"pg_stats":[{"pgid":"1.f","version":"37'10214","reported_seq":10318,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:48.783422+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:42:48.783422+0000","last_peered":"2024-09-05T16:42:48.783422+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:42:48.783422+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"29'200","ondisk_log_start":"29'200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22036480,"num_objects":2690,"num_object_clones":0,"num_object_copies":10760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2690,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10214,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":454}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'9971","reported_seq":13251,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:48.758730+0000","last_change":"2024-09-05T16:40:28.919189+0000","last_active":"2024-09-05T16:42:48.758730+0000","last_peered":"2024-09-05T16:42:48.758730+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:42:48.758730+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9971,"log_dups_size":0,"ondisk_log_size":9971,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21157024,"num_objects":2583,"num_object_clones":0,"num_object_copies":10332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":666,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2583,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9969,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1605,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1917}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'10117","reported_seq":10238,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:48.782429+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:42:48.782429+0000","last_peered":"2024-09-05T16:42:48.782429+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:42:48.782429+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"29'100","ondisk_log_start":"29'100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21584488,"num_objects":2636,"num_object_clones":0,"num_object_copies":10544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2636,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10116,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":404}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'10213","reported_seq":14942,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:42:48.443888+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:42:48.443888+0000","last_peered":"2024-09-05T16:42:48.443888+0000","last_clean":"2024-09-05T16:42:48.443888+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:42:48.443888+0000","last_undegraded":"2024-09-05T16:42:48.443888+0000","last_fullsized":"2024-09-05T16:42:48.443888+0000","mapping_epoch":34,"log_start":"29'200","ondisk_log_start":"29'200","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22290432,"num_objects":2721,"num_object_clones":0,"num_object_copies":10884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2721,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4513,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'9942","reported_seq":10041,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:48.676223+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:42:48.676223+0000","last_peered":"2024-09-05T16:42:48.676223+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:42:48.676223+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9942,"log_dups_size":0,"ondisk_log_size":9942,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21730464,"num_objects":2653,"num_object_clones":0,"num_object_copies":10612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2653,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9940,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":508}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'10164","reported_seq":10278,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:48.740305+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:42:48.740305+0000","last_peered":"2024-09-05T16:42:48.740305+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:42:48.740305+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"29'100","ondisk_log_start":"29'100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21364736,"num_objects":2608,"num_object_clones":0,"num_object_copies":10432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2608,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10164,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":389}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'10001","reported_seq":10109,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:48.577528+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:42:48.577528+0000","last_peered":"2024-09-05T16:42:48.577528+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:42:48.577528+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21796000,"num_objects":2661,"num_object_clones":0,"num_object_copies":10644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2661,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9999,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":530}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'10124","reported_seq":10213,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:48.597217+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:42:48.597217+0000","last_peered":"2024-09-05T16:42:48.597217+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:42:48.597217+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"29'100","ondisk_log_start":"29'100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21746848,"num_objects":2655,"num_object_clones":0,"num_object_copies":10620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2655,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10122,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":423}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'10140","reported_seq":10241,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:48.841393+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:42:48.841393+0000","last_peered":"2024-09-05T16:42:48.841393+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:42:48.841393+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"29'100","ondisk_log_start":"29'100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21571904,"num_objects":2634,"num_object_clones":0,"num_object_copies":10536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2634,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10137,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":390}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'10225","reported_seq":10346,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:48.751302+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:42:48.751302+0000","last_peered":"2024-09-05T16:42:48.751302+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:42:48.751302+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"29'200","ondisk_log_start":"29'200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21173408,"num_objects":2585,"num_object_clones":0,"num_object_copies":10340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2585,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10223,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":340}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'10377","reported_seq":10504,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:48.585634+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:42:48.585634+0000","last_peered":"2024-09-05T16:42:48.585634+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:42:48.585634+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"29'300","ondisk_log_start":"29'300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22490864,"num_objects":2746,"num_object_clones":0,"num_object_copies":10984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2746,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10374,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":465}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'10060","reported_seq":10158,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:48.683932+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:42:48.683932+0000","last_peered":"2024-09-05T16:42:48.683932+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:42:48.683932+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22108752,"num_objects":2699,"num_object_clones":0,"num_object_copies":10796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2699,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10059,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":461}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'10157","reported_seq":14893,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:42:48.389715+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:42:48.389715+0000","last_peered":"2024-09-05T16:42:48.389715+0000","last_clean":"2024-09-05T16:42:48.389715+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:42:48.389715+0000","last_undegraded":"2024-09-05T16:42:48.389715+0000","last_fullsized":"2024-09-05T16:42:48.389715+0000","mapping_epoch":36,"log_start":"29'100","ondisk_log_start":"29'100","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21716992,"num_objects":2651,"num_object_clones":0,"num_object_copies":10604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2651,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1757,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'10409","reported_seq":10517,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:48.831196+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:42:48.831196+0000","last_peered":"2024-09-05T16:42:48.831196+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:42:48.831196+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"29'400","ondisk_log_start":"29'400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22254752,"num_objects":2717,"num_object_clones":0,"num_object_copies":10868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2717,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10407,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":383}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'10201","reported_seq":10314,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:48.588899+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:42:48.588899+0000","last_peered":"2024-09-05T16:42:48.588899+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:42:48.588899+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"29'200","ondisk_log_start":"29'200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21558432,"num_objects":2632,"num_object_clones":0,"num_object_copies":10528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2632,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10199,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":324}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'10119","reported_seq":10227,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:48.590976+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:42:48.590976+0000","last_peered":"2024-09-05T16:42:48.590976+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:42:48.590976+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"29'100","ondisk_log_start":"29'100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21755040,"num_objects":2656,"num_object_clones":0,"num_object_copies":10624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2656,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10117,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":436}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":348336616,"num_objects":42527,"num_object_clones":0,"num_object_copies":170108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29753,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42527,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":148310,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1884,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":696745984,"data_stored":696745984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160334,"ondisk_log_size":160334,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084394,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":766036,"kb_used_data":170692,"kb_used_omap":4,"kb_used_meta":595323,"kb_avail":93605804,"statfs":{"total":96636764160,"available":95852343296,"internally_reserved":0,"allocated":174788608,"data_stored":174587334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":609611475},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,7,0,0,2,6,3,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986271,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":604444,"kb_used_data":171524,"kb_used_omap":4,"kb_used_meta":432891,"kb_avail":93767396,"statfs":{"total":96636764160,"available":96017813504,"internally_reserved":0,"allocated":175640576,"data_stored":175439302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":443281106},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,0,4,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673077,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":774224,"kb_used_data":171072,"kb_used_omap":1,"kb_used_meta":603134,"kb_avail":93597616,"statfs":{"total":96636764160,"available":95843958784,"internally_reserved":0,"allocated":175177728,"data_stored":174976454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":617609669},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,1,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182508,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":774256,"kb_used_data":170656,"kb_used_omap":4,"kb_used_meta":603579,"kb_avail":93597584,"statfs":{"total":96636764160,"available":95843926016,"internally_reserved":0,"allocated":174751744,"data_stored":174550470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":618065613},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,4,4,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":174141440,"data_stored":174141440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":174505984,"data_stored":174505984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":174096384,"data_stored":174096384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":174002176,"data_stored":174002176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:42:49.836 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-05T16:42:49.836 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'10214", 'reported_seq': 10318, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:48.783422+0000', 'last_change': '2024-09-05T16:34:56.833497+0000', 'last_active': '2024-09-05T16:42:48.783422+0000', 'last_peered': '2024-09-05T16:42:48.783422+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.828289+0000', 'last_became_peered': '2024-09-05T16:34:56.828289+0000', 'last_unstale': '2024-09-05T16:42:48.783422+0000', 'last_undegraded': '2024-09-05T16:34:56.612160+0000', 'last_fullsized': '2024-09-05T16:34:56.557632+0000', 'mapping_epoch': 32, 'log_start': "29'200", 'ondisk_log_start': "29'200", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10014, 'log_dups_size': 0, 'ondisk_log_size': 10014, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22036480, 'num_objects': 2690, 'num_object_clones': 0, 'num_object_copies': 10760, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2236, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2690, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10214, 'num_write_kb': 51616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 454}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:42:49.836 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-05T16:42:49.836 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "37'9971", 'reported_seq': 13251, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:48.758730+0000', 'last_change': '2024-09-05T16:40:28.919189+0000', 'last_active': '2024-09-05T16:42:48.758730+0000', 'last_peered': '2024-09-05T16:42:48.758730+0000', 'last_clean': '2024-09-05T16:33:53.200762+0000', 'last_became_active': '2024-09-05T16:34:56.820292+0000', 'last_became_peered': '2024-09-05T16:34:56.820292+0000', 'last_unstale': '2024-09-05T16:42:48.758730+0000', 'last_undegraded': '2024-09-05T16:34:56.605776+0000', 'last_fullsized': '2024-09-05T16:34:56.559383+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9971, 'log_dups_size': 0, 'ondisk_log_size': 9971, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21157024, 'num_objects': 2583, 'num_object_clones': 0, 'num_object_copies': 10332, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 666, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2583, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9969, 'num_write_kb': 50212, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1605, 'num_bytes_recovered': 2613248, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 1917}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:42:49.836 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:42:49.836 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'10117", 'reported_seq': 10238, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:48.782429+0000', 'last_change': '2024-09-05T16:34:56.867900+0000', 'last_active': '2024-09-05T16:42:48.782429+0000', 'last_peered': '2024-09-05T16:42:48.782429+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:42:48.782429+0000', 'last_undegraded': '2024-09-05T16:34:56.590972+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 32, 'log_start': "29'100", 'ondisk_log_start': "29'100", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10017, 'log_dups_size': 0, 'ondisk_log_size': 10017, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21584488, 'num_objects': 2636, 'num_object_clones': 0, 'num_object_copies': 10544, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2233, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2636, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 10116, 'num_write_kb': 50999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 404}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:42:49.836 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:42:49.836 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'9942", 'reported_seq': 10041, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:48.676223+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:42:48.676223+0000', 'last_peered': '2024-09-05T16:42:48.676223+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:42:48.676223+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9942, 'log_dups_size': 0, 'ondisk_log_size': 9942, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21730464, 'num_objects': 2653, 'num_object_clones': 0, 'num_object_copies': 10612, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2147, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2653, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9940, 'num_write_kb': 50373, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 508}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:42:49.837 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:42:49.837 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'10164", 'reported_seq': 10278, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:48.740305+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:42:48.740305+0000', 'last_peered': '2024-09-05T16:42:48.740305+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:42:48.740305+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "29'100", 'ondisk_log_start': "29'100", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10064, 'log_dups_size': 0, 'ondisk_log_size': 10064, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21364736, 'num_objects': 2608, 'num_object_clones': 0, 'num_object_copies': 10432, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2608, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10164, 'num_write_kb': 51088, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 389}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:42:49.837 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:42:49.837 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'10001", 'reported_seq': 10109, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:48.577528+0000', 'last_change': '2024-09-05T16:34:56.903436+0000', 'last_active': '2024-09-05T16:42:48.577528+0000', 'last_peered': '2024-09-05T16:42:48.577528+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:42:48.577528+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10001, 'log_dups_size': 0, 'ondisk_log_size': 10001, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21796000, 'num_objects': 2661, 'num_object_clones': 0, 'num_object_copies': 10644, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2133, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2661, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9999, 'num_write_kb': 50636, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 530}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:42:49.837 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-05T16:42:49.837 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "37'10124", 'reported_seq': 10213, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:48.597217+0000', 'last_change': '2024-09-05T16:34:56.883319+0000', 'last_active': '2024-09-05T16:42:48.597217+0000', 'last_peered': '2024-09-05T16:42:48.597217+0000', 'last_clean': '2024-09-05T16:33:53.505305+0000', 'last_became_active': '2024-09-05T16:34:56.873389+0000', 'last_became_peered': '2024-09-05T16:34:56.873389+0000', 'last_unstale': '2024-09-05T16:42:48.597217+0000', 'last_undegraded': '2024-09-05T16:34:56.602567+0000', 'last_fullsized': '2024-09-05T16:34:56.559958+0000', 'mapping_epoch': 32, 'log_start': "29'100", 'ondisk_log_start': "29'100", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10024, 'log_dups_size': 0, 'ondisk_log_size': 10024, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21746848, 'num_objects': 2655, 'num_object_clones': 0, 'num_object_copies': 10620, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2234, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2655, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10122, 'num_write_kb': 51108, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 423}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:42:49.837 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:42:49.837 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'10140", 'reported_seq': 10241, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:48.841393+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:42:48.841393+0000', 'last_peered': '2024-09-05T16:42:48.841393+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:42:48.841393+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "29'100", 'ondisk_log_start': "29'100", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21571904, 'num_objects': 2634, 'num_object_clones': 0, 'num_object_copies': 10536, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2634, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10137, 'num_write_kb': 51081, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 390}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:42:49.837 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:42:49.837 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'10225", 'reported_seq': 10346, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:48.751302+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:42:48.751302+0000', 'last_peered': '2024-09-05T16:42:48.751302+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:42:48.751302+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "29'200", 'ondisk_log_start': "29'200", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10025, 'log_dups_size': 0, 'ondisk_log_size': 10025, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21173408, 'num_objects': 2585, 'num_object_clones': 0, 'num_object_copies': 10340, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2585, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10223, 'num_write_kb': 51230, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 340}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:42:49.838 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:42:49.838 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'10377", 'reported_seq': 10504, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:48.585634+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:42:48.585634+0000', 'last_peered': '2024-09-05T16:42:48.585634+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:42:48.585634+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "29'300", 'ondisk_log_start': "29'300", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10077, 'log_dups_size': 0, 'ondisk_log_size': 10077, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22490864, 'num_objects': 2746, 'num_object_clones': 0, 'num_object_copies': 10984, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2284, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2746, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10374, 'num_write_kb': 52476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 465}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:42:49.838 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:42:49.838 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'10060", 'reported_seq': 10158, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:48.683932+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:42:48.683932+0000', 'last_peered': '2024-09-05T16:42:48.683932+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:42:48.683932+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10060, 'log_dups_size': 0, 'ondisk_log_size': 10060, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22108752, 'num_objects': 2699, 'num_object_clones': 0, 'num_object_copies': 10796, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2699, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10059, 'num_write_kb': 51031, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 461}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:42:49.838 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:42:49.838 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'10409", 'reported_seq': 10517, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:48.831196+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:42:48.831196+0000', 'last_peered': '2024-09-05T16:42:48.831196+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:42:48.831196+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "29'400", 'ondisk_log_start': "29'400", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10009, 'log_dups_size': 0, 'ondisk_log_size': 10009, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22254752, 'num_objects': 2717, 'num_object_clones': 0, 'num_object_copies': 10868, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2717, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10407, 'num_write_kb': 52500, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 383}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:42:49.838 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:42:49.838 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'10201", 'reported_seq': 10314, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:48.588899+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:42:48.588899+0000', 'last_peered': '2024-09-05T16:42:48.588899+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:42:48.588899+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "29'200", 'ondisk_log_start': "29'200", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10001, 'log_dups_size': 0, 'ondisk_log_size': 10001, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21558432, 'num_objects': 2632, 'num_object_clones': 0, 'num_object_copies': 10528, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2310, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2632, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10199, 'num_write_kb': 51325, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 324}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:42:49.838 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:42:49.838 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'10119", 'reported_seq': 10227, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:48.590976+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:42:48.590976+0000', 'last_peered': '2024-09-05T16:42:48.590976+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:42:48.590976+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "29'100", 'ondisk_log_start': "29'100", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10019, 'log_dups_size': 0, 'ondisk_log_size': 10019, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21755040, 'num_objects': 2656, 'num_object_clones': 0, 'num_object_copies': 10624, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2656, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10117, 'num_write_kb': 51092, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 436}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:42:49.839 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:42:50.561 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:42:50.561 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 29753/170108 objects degraded (17.491%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":573,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":42527,"data_bytes":348336616,"bytes_used":2989015040,"bytes_avail":383558041600,"bytes_total":386547056640,"degraded_objects":29753,"degraded_total":170108,"degraded_ratio":0.17490652996919603,"recovering_objects_per_sec":14,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":178,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":256},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:42:50.584 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:42:50.585 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:42:51.225 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:42:51.225 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:42:51.246 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":315,"stamp":"2024-09-05T16:42:51.001675+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":348336616,"num_objects":42527,"num_object_clones":0,"num_object_copies":170108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29753,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42527,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":148310,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1884,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160334,"ondisk_log_size":160334,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2918960,"kb_used_data":683944,"kb_used_omap":14,"kb_used_meta":2234929,"kb_avail":374568400,"statfs":{"total":386547056640,"available":383558041600,"internally_reserved":0,"allocated":700358656,"data_stored":699553560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2288567863},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,9,7,1,14,17,8,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16793600,"num_objects":-2050,"num_object_clones":0,"num_object_copies":-8200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2050,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2050,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005146"},"pg_stats":[{"pgid":"1.f","version":"37'10214","reported_seq":10318,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:48.783422+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:42:48.783422+0000","last_peered":"2024-09-05T16:42:48.783422+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:42:48.783422+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"29'200","ondisk_log_start":"29'200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22036480,"num_objects":2690,"num_object_clones":0,"num_object_copies":10760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2690,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10214,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":454}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'9971","reported_seq":13251,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:48.758730+0000","last_change":"2024-09-05T16:40:28.919189+0000","last_active":"2024-09-05T16:42:48.758730+0000","last_peered":"2024-09-05T16:42:48.758730+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:42:48.758730+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9971,"log_dups_size":0,"ondisk_log_size":9971,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21157024,"num_objects":2583,"num_object_clones":0,"num_object_copies":10332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":666,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2583,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9969,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1605,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1917}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'10117","reported_seq":10238,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:48.782429+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:42:48.782429+0000","last_peered":"2024-09-05T16:42:48.782429+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:42:48.782429+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"29'100","ondisk_log_start":"29'100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21584488,"num_objects":2636,"num_object_clones":0,"num_object_copies":10544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2636,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10116,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":404}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'10213","reported_seq":14942,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:42:48.443888+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:42:48.443888+0000","last_peered":"2024-09-05T16:42:48.443888+0000","last_clean":"2024-09-05T16:42:48.443888+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:42:48.443888+0000","last_undegraded":"2024-09-05T16:42:48.443888+0000","last_fullsized":"2024-09-05T16:42:48.443888+0000","mapping_epoch":34,"log_start":"29'200","ondisk_log_start":"29'200","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22290432,"num_objects":2721,"num_object_clones":0,"num_object_copies":10884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2721,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4513,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'9942","reported_seq":10041,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:48.676223+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:42:48.676223+0000","last_peered":"2024-09-05T16:42:48.676223+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:42:48.676223+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9942,"log_dups_size":0,"ondisk_log_size":9942,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21730464,"num_objects":2653,"num_object_clones":0,"num_object_copies":10612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2653,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9940,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":508}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'10164","reported_seq":10278,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:48.740305+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:42:48.740305+0000","last_peered":"2024-09-05T16:42:48.740305+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:42:48.740305+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"29'100","ondisk_log_start":"29'100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21364736,"num_objects":2608,"num_object_clones":0,"num_object_copies":10432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2608,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10164,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":389}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'10001","reported_seq":10109,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:48.577528+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:42:48.577528+0000","last_peered":"2024-09-05T16:42:48.577528+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:42:48.577528+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21796000,"num_objects":2661,"num_object_clones":0,"num_object_copies":10644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2661,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9999,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":530}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'10124","reported_seq":10213,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:48.597217+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:42:48.597217+0000","last_peered":"2024-09-05T16:42:48.597217+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:42:48.597217+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"29'100","ondisk_log_start":"29'100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21746848,"num_objects":2655,"num_object_clones":0,"num_object_copies":10620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2655,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10122,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":423}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'10140","reported_seq":10241,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:48.841393+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:42:48.841393+0000","last_peered":"2024-09-05T16:42:48.841393+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:42:48.841393+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"29'100","ondisk_log_start":"29'100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21571904,"num_objects":2634,"num_object_clones":0,"num_object_copies":10536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2634,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10137,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":390}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'10225","reported_seq":10346,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:48.751302+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:42:48.751302+0000","last_peered":"2024-09-05T16:42:48.751302+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:42:48.751302+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"29'200","ondisk_log_start":"29'200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21173408,"num_objects":2585,"num_object_clones":0,"num_object_copies":10340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2585,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10223,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":340}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'10377","reported_seq":10504,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:48.585634+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:42:48.585634+0000","last_peered":"2024-09-05T16:42:48.585634+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:42:48.585634+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"29'300","ondisk_log_start":"29'300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22490864,"num_objects":2746,"num_object_clones":0,"num_object_copies":10984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2746,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10374,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":465}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'10060","reported_seq":10158,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:48.683932+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:42:48.683932+0000","last_peered":"2024-09-05T16:42:48.683932+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:42:48.683932+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22108752,"num_objects":2699,"num_object_clones":0,"num_object_copies":10796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2699,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10059,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":461}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'10157","reported_seq":14893,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:42:48.389715+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:42:48.389715+0000","last_peered":"2024-09-05T16:42:48.389715+0000","last_clean":"2024-09-05T16:42:48.389715+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:42:48.389715+0000","last_undegraded":"2024-09-05T16:42:48.389715+0000","last_fullsized":"2024-09-05T16:42:48.389715+0000","mapping_epoch":36,"log_start":"29'100","ondisk_log_start":"29'100","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21716992,"num_objects":2651,"num_object_clones":0,"num_object_copies":10604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2651,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1757,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'10409","reported_seq":10517,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:48.831196+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:42:48.831196+0000","last_peered":"2024-09-05T16:42:48.831196+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:42:48.831196+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"29'400","ondisk_log_start":"29'400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22254752,"num_objects":2717,"num_object_clones":0,"num_object_copies":10868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2717,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10407,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":383}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'10201","reported_seq":10314,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:48.588899+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:42:48.588899+0000","last_peered":"2024-09-05T16:42:48.588899+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:42:48.588899+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"29'200","ondisk_log_start":"29'200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21558432,"num_objects":2632,"num_object_clones":0,"num_object_copies":10528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2632,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10199,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":324}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'10119","reported_seq":10227,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:48.590976+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:42:48.590976+0000","last_peered":"2024-09-05T16:42:48.590976+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:42:48.590976+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"29'100","ondisk_log_start":"29'100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21755040,"num_objects":2656,"num_object_clones":0,"num_object_copies":10624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2656,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10117,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":436}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":348336616,"num_objects":42527,"num_object_clones":0,"num_object_copies":170108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29753,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42527,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":148310,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1884,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":696745984,"data_stored":696745984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160334,"ondisk_log_size":160334,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084394,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":766036,"kb_used_data":170692,"kb_used_omap":4,"kb_used_meta":595323,"kb_avail":93605804,"statfs":{"total":96636764160,"available":95852343296,"internally_reserved":0,"allocated":174788608,"data_stored":174587334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":609611475},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,7,0,0,2,6,3,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986271,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":604444,"kb_used_data":171524,"kb_used_omap":4,"kb_used_meta":432891,"kb_avail":93767396,"statfs":{"total":96636764160,"available":96017813504,"internally_reserved":0,"allocated":175640576,"data_stored":175439302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":443281106},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,0,4,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673077,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":774224,"kb_used_data":171072,"kb_used_omap":1,"kb_used_meta":603134,"kb_avail":93597616,"statfs":{"total":96636764160,"available":95843958784,"internally_reserved":0,"allocated":175177728,"data_stored":174976454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":617609669},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,1,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182508,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":774256,"kb_used_data":170656,"kb_used_omap":4,"kb_used_meta":603579,"kb_avail":93597584,"statfs":{"total":96636764160,"available":95843926016,"internally_reserved":0,"allocated":174751744,"data_stored":174550470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":618065613},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,4,4,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":174141440,"data_stored":174141440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":174505984,"data_stored":174505984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":174096384,"data_stored":174096384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":174002176,"data_stored":174002176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:42:54.249 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:42:54.826 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:42:54.826 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:42:54.861 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":316,"stamp":"2024-09-05T16:42:53.002415+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":348336616,"num_objects":42527,"num_object_clones":0,"num_object_copies":170108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29753,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42527,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":148310,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1884,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160334,"ondisk_log_size":160334,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2918960,"kb_used_data":683944,"kb_used_omap":14,"kb_used_meta":2234929,"kb_avail":374568400,"statfs":{"total":386547056640,"available":383558041600,"internally_reserved":0,"allocated":700358656,"data_stored":699553560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2288567863},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,9,7,1,14,17,8,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16793600,"num_objects":-2050,"num_object_clones":0,"num_object_copies":-8200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2050,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2050,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005071"},"pg_stats":[{"pgid":"1.f","version":"37'10214","reported_seq":10318,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:48.783422+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:42:48.783422+0000","last_peered":"2024-09-05T16:42:48.783422+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:42:48.783422+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"29'200","ondisk_log_start":"29'200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22036480,"num_objects":2690,"num_object_clones":0,"num_object_copies":10760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2690,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10214,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":454}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'9971","reported_seq":13251,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:48.758730+0000","last_change":"2024-09-05T16:40:28.919189+0000","last_active":"2024-09-05T16:42:48.758730+0000","last_peered":"2024-09-05T16:42:48.758730+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:42:48.758730+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9971,"log_dups_size":0,"ondisk_log_size":9971,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21157024,"num_objects":2583,"num_object_clones":0,"num_object_copies":10332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":666,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2583,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9969,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1605,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1917}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'10117","reported_seq":10238,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:48.782429+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:42:48.782429+0000","last_peered":"2024-09-05T16:42:48.782429+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:42:48.782429+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"29'100","ondisk_log_start":"29'100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21584488,"num_objects":2636,"num_object_clones":0,"num_object_copies":10544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2636,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10116,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":404}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'10213","reported_seq":14942,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:42:48.443888+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:42:48.443888+0000","last_peered":"2024-09-05T16:42:48.443888+0000","last_clean":"2024-09-05T16:42:48.443888+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:42:48.443888+0000","last_undegraded":"2024-09-05T16:42:48.443888+0000","last_fullsized":"2024-09-05T16:42:48.443888+0000","mapping_epoch":34,"log_start":"29'200","ondisk_log_start":"29'200","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22290432,"num_objects":2721,"num_object_clones":0,"num_object_copies":10884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2721,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4513,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'9942","reported_seq":10041,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:48.676223+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:42:48.676223+0000","last_peered":"2024-09-05T16:42:48.676223+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:42:48.676223+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9942,"log_dups_size":0,"ondisk_log_size":9942,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21730464,"num_objects":2653,"num_object_clones":0,"num_object_copies":10612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2653,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9940,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":508}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'10164","reported_seq":10278,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:48.740305+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:42:48.740305+0000","last_peered":"2024-09-05T16:42:48.740305+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:42:48.740305+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"29'100","ondisk_log_start":"29'100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21364736,"num_objects":2608,"num_object_clones":0,"num_object_copies":10432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2608,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10164,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":389}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'10001","reported_seq":10109,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:48.577528+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:42:48.577528+0000","last_peered":"2024-09-05T16:42:48.577528+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:42:48.577528+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21796000,"num_objects":2661,"num_object_clones":0,"num_object_copies":10644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2661,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9999,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":530}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'10124","reported_seq":10213,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:48.597217+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:42:48.597217+0000","last_peered":"2024-09-05T16:42:48.597217+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:42:48.597217+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"29'100","ondisk_log_start":"29'100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21746848,"num_objects":2655,"num_object_clones":0,"num_object_copies":10620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2655,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10122,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":423}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'10140","reported_seq":10241,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:48.841393+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:42:48.841393+0000","last_peered":"2024-09-05T16:42:48.841393+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:42:48.841393+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"29'100","ondisk_log_start":"29'100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21571904,"num_objects":2634,"num_object_clones":0,"num_object_copies":10536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2634,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10137,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":390}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'10225","reported_seq":10346,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:48.751302+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:42:48.751302+0000","last_peered":"2024-09-05T16:42:48.751302+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:42:48.751302+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"29'200","ondisk_log_start":"29'200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21173408,"num_objects":2585,"num_object_clones":0,"num_object_copies":10340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2585,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10223,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":340}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'10377","reported_seq":10504,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:48.585634+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:42:48.585634+0000","last_peered":"2024-09-05T16:42:48.585634+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:42:48.585634+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"29'300","ondisk_log_start":"29'300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22490864,"num_objects":2746,"num_object_clones":0,"num_object_copies":10984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2746,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10374,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":465}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'10060","reported_seq":10158,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:48.683932+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:42:48.683932+0000","last_peered":"2024-09-05T16:42:48.683932+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:42:48.683932+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22108752,"num_objects":2699,"num_object_clones":0,"num_object_copies":10796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2699,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10059,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":461}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'10157","reported_seq":14893,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:42:48.389715+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:42:48.389715+0000","last_peered":"2024-09-05T16:42:48.389715+0000","last_clean":"2024-09-05T16:42:48.389715+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:42:48.389715+0000","last_undegraded":"2024-09-05T16:42:48.389715+0000","last_fullsized":"2024-09-05T16:42:48.389715+0000","mapping_epoch":36,"log_start":"29'100","ondisk_log_start":"29'100","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21716992,"num_objects":2651,"num_object_clones":0,"num_object_copies":10604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2651,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1757,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'10409","reported_seq":10517,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:48.831196+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:42:48.831196+0000","last_peered":"2024-09-05T16:42:48.831196+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:42:48.831196+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"29'400","ondisk_log_start":"29'400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22254752,"num_objects":2717,"num_object_clones":0,"num_object_copies":10868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2717,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10407,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":383}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'10201","reported_seq":10314,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:48.588899+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:42:48.588899+0000","last_peered":"2024-09-05T16:42:48.588899+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:42:48.588899+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"29'200","ondisk_log_start":"29'200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21558432,"num_objects":2632,"num_object_clones":0,"num_object_copies":10528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2632,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10199,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":324}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'10119","reported_seq":10227,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:48.590976+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:42:48.590976+0000","last_peered":"2024-09-05T16:42:48.590976+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:42:48.590976+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"29'100","ondisk_log_start":"29'100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21755040,"num_objects":2656,"num_object_clones":0,"num_object_copies":10624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2656,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10117,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":436}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":348336616,"num_objects":42527,"num_object_clones":0,"num_object_copies":170108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29753,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42527,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":148310,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1884,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":696745984,"data_stored":696745984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160334,"ondisk_log_size":160334,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084394,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":766036,"kb_used_data":170692,"kb_used_omap":4,"kb_used_meta":595323,"kb_avail":93605804,"statfs":{"total":96636764160,"available":95852343296,"internally_reserved":0,"allocated":174788608,"data_stored":174587334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":609611475},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,7,0,0,2,6,3,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986271,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":604444,"kb_used_data":171524,"kb_used_omap":4,"kb_used_meta":432891,"kb_avail":93767396,"statfs":{"total":96636764160,"available":96017813504,"internally_reserved":0,"allocated":175640576,"data_stored":175439302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":443281106},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,0,4,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673077,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":774224,"kb_used_data":171072,"kb_used_omap":1,"kb_used_meta":603134,"kb_avail":93597616,"statfs":{"total":96636764160,"available":95843958784,"internally_reserved":0,"allocated":175177728,"data_stored":174976454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":617609669},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,1,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182508,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":774256,"kb_used_data":170656,"kb_used_omap":4,"kb_used_meta":603579,"kb_avail":93597584,"statfs":{"total":96636764160,"available":95843926016,"internally_reserved":0,"allocated":174751744,"data_stored":174550470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":618065613},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,4,4,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":174141440,"data_stored":174141440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":174505984,"data_stored":174505984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":174096384,"data_stored":174096384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":174002176,"data_stored":174002176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:42:54.863 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:42:55.490 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:42:55.491 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:42:55.512 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":317,"stamp":"2024-09-05T16:42:55.003249+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":339988968,"num_objects":41508,"num_object_clones":0,"num_object_copies":166032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29699,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41508,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":149329,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1938,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160853,"ondisk_log_size":160853,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2904448,"kb_used_data":667980,"kb_used_omap":14,"kb_used_meta":2236337,"kb_avail":374582912,"statfs":{"total":386547056640,"available":383572901888,"internally_reserved":0,"allocated":684011520,"data_stored":683206424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2290009655},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,3,0,15,13,20,11,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25141248,"num_objects":-3069,"num_object_clones":0,"num_object_copies":-12276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-170,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3069,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3069,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005052"},"pg_stats":[{"pgid":"1.f","version":"37'10276","reported_seq":10380,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:53.455351+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:42:53.455351+0000","last_peered":"2024-09-05T16:42:53.455351+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:42:53.455351+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"29'200","ondisk_log_start":"29'200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21528576,"num_objects":2628,"num_object_clones":0,"num_object_copies":10512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10276,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":392}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'10040","reported_seq":13428,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:53.775838+0000","last_change":"2024-09-05T16:40:28.919189+0000","last_active":"2024-09-05T16:42:53.775838+0000","last_peered":"2024-09-05T16:42:53.775838+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:42:53.775838+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20591776,"num_objects":2514,"num_object_clones":0,"num_object_copies":10056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":612,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2514,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10038,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1659,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1902}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'10184","reported_seq":10307,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:53.814732+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:42:53.814732+0000","last_peered":"2024-09-05T16:42:53.814732+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:42:53.814732+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"29'100","ondisk_log_start":"29'100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21035624,"num_objects":2569,"num_object_clones":0,"num_object_copies":10276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2569,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10183,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":337}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'10276","reported_seq":15005,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:42:53.455753+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:42:53.455753+0000","last_peered":"2024-09-05T16:42:53.455753+0000","last_clean":"2024-09-05T16:42:53.455753+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:42:53.455753+0000","last_undegraded":"2024-09-05T16:42:53.455753+0000","last_fullsized":"2024-09-05T16:42:53.455753+0000","mapping_epoch":34,"log_start":"29'200","ondisk_log_start":"29'200","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21774336,"num_objects":2658,"num_object_clones":0,"num_object_copies":10632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2658,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4576,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'9994","reported_seq":10091,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:53.903085+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:42:53.903085+0000","last_peered":"2024-09-05T16:42:53.903085+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:42:53.903085+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9994,"log_dups_size":0,"ondisk_log_size":9994,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21304480,"num_objects":2601,"num_object_clones":0,"num_object_copies":10404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2601,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9992,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":456}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'10220","reported_seq":10334,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:53.871106+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:42:53.871106+0000","last_peered":"2024-09-05T16:42:53.871106+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:42:53.871106+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"29'200","ondisk_log_start":"29'200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20905984,"num_objects":2552,"num_object_clones":0,"num_object_copies":10208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2552,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10220,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":333}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'10057","reported_seq":10165,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:53.710024+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:42:53.710024+0000","last_peered":"2024-09-05T16:42:53.710024+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:42:53.710024+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21337248,"num_objects":2605,"num_object_clones":0,"num_object_copies":10420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2605,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10055,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":474}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'10192","reported_seq":10281,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:53.689078+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:42:53.689078+0000","last_peered":"2024-09-05T16:42:53.689078+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:42:53.689078+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"29'100","ondisk_log_start":"29'100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21189792,"num_objects":2587,"num_object_clones":0,"num_object_copies":10348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2587,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10190,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":355}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'10201","reported_seq":10302,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:53.844246+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:42:53.844246+0000","last_peered":"2024-09-05T16:42:53.844246+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:42:53.844246+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"29'200","ondisk_log_start":"29'200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21072192,"num_objects":2573,"num_object_clones":0,"num_object_copies":10292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2573,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10198,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":329}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'10271","reported_seq":10393,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:53.691354+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:42:53.691354+0000","last_peered":"2024-09-05T16:42:53.691354+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:42:53.691354+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"29'200","ondisk_log_start":"29'200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20796576,"num_objects":2539,"num_object_clones":0,"num_object_copies":10156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2539,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10269,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":294}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'10448","reported_seq":10575,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:53.800918+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:42:53.800918+0000","last_peered":"2024-09-05T16:42:53.800918+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:42:53.800918+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"29'400","ondisk_log_start":"29'400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21909232,"num_objects":2675,"num_object_clones":0,"num_object_copies":10700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2675,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10445,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":394}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'10118","reported_seq":10215,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:53.935443+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:42:53.935443+0000","last_peered":"2024-09-05T16:42:53.935443+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:42:53.935443+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"29'100","ondisk_log_start":"29'100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21633616,"num_objects":2641,"num_object_clones":0,"num_object_copies":10564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2641,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10117,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":403}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'10236","reported_seq":14972,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:42:53.454141+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:42:53.454141+0000","last_peered":"2024-09-05T16:42:53.454141+0000","last_clean":"2024-09-05T16:42:53.454141+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:42:53.454141+0000","last_undegraded":"2024-09-05T16:42:53.454141+0000","last_fullsized":"2024-09-05T16:42:53.454141+0000","mapping_epoch":36,"log_start":"29'200","ondisk_log_start":"29'200","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21069824,"num_objects":2572,"num_object_clones":0,"num_object_copies":10288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2572,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1836,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'10479","reported_seq":10587,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:53.815055+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:42:53.815055+0000","last_peered":"2024-09-05T16:42:53.815055+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:42:53.815055+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"29'400","ondisk_log_start":"29'400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21681312,"num_objects":2647,"num_object_clones":0,"num_object_copies":10588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2647,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10477,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":313}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'10276","reported_seq":10388,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:53.674985+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:42:53.674985+0000","last_peered":"2024-09-05T16:42:53.674985+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:42:53.674985+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"29'200","ondisk_log_start":"29'200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20944032,"num_objects":2557,"num_object_clones":0,"num_object_copies":10228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2557,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10274,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":249}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'10185","reported_seq":10292,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:53.549955+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:42:53.549955+0000","last_peered":"2024-09-05T16:42:53.549955+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:42:53.549955+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"29'100","ondisk_log_start":"29'100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21214368,"num_objects":2590,"num_object_clones":0,"num_object_copies":10360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2590,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10183,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":370}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":339988968,"num_objects":41508,"num_object_clones":0,"num_object_copies":166032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29699,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41508,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":149329,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1938,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":680243200,"data_stored":680243200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160853,"ondisk_log_size":160853,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084395,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":766308,"kb_used_data":166792,"kb_used_omap":4,"kb_used_meta":599483,"kb_avail":93605532,"statfs":{"total":96636764160,"available":95852064768,"internally_reserved":0,"allocated":170795008,"data_stored":170593734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":613871315},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,5,7,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986272,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":622640,"kb_used_data":167488,"kb_used_omap":4,"kb_used_meta":455099,"kb_avail":93749200,"statfs":{"total":96636764160,"available":95999180800,"internally_reserved":0,"allocated":171507712,"data_stored":171306438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":466022098},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,7,4,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673078,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":757740,"kb_used_data":167064,"kb_used_omap":1,"kb_used_meta":590654,"kb_avail":93614100,"statfs":{"total":96636764160,"available":95860838400,"internally_reserved":0,"allocated":171073536,"data_stored":170872262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":604830149},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,6,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182509,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":757760,"kb_used_data":166636,"kb_used_omap":4,"kb_used_meta":591099,"kb_avail":93614080,"statfs":{"total":96636764160,"available":95860817920,"internally_reserved":0,"allocated":170635264,"data_stored":170433990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":605286093},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,1,1,7,4,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":170029056,"data_stored":170029056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":170389504,"data_stored":170389504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":169975808,"data_stored":169975808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":169848832,"data_stored":169848832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:42:55.513 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-05T16:42:55.513 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'10276", 'reported_seq': 10380, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:53.455351+0000', 'last_change': '2024-09-05T16:34:56.833497+0000', 'last_active': '2024-09-05T16:42:53.455351+0000', 'last_peered': '2024-09-05T16:42:53.455351+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.828289+0000', 'last_became_peered': '2024-09-05T16:34:56.828289+0000', 'last_unstale': '2024-09-05T16:42:53.455351+0000', 'last_undegraded': '2024-09-05T16:34:56.612160+0000', 'last_fullsized': '2024-09-05T16:34:56.557632+0000', 'mapping_epoch': 32, 'log_start': "29'200", 'ondisk_log_start': "29'200", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10076, 'log_dups_size': 0, 'ondisk_log_size': 10076, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21528576, 'num_objects': 2628, 'num_object_clones': 0, 'num_object_copies': 10512, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2236, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2628, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10276, 'num_write_kb': 51616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 392}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:42:55.513 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-05T16:42:55.513 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "37'10040", 'reported_seq': 13428, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:53.775838+0000', 'last_change': '2024-09-05T16:40:28.919189+0000', 'last_active': '2024-09-05T16:42:53.775838+0000', 'last_peered': '2024-09-05T16:42:53.775838+0000', 'last_clean': '2024-09-05T16:33:53.200762+0000', 'last_became_active': '2024-09-05T16:34:56.820292+0000', 'last_became_peered': '2024-09-05T16:34:56.820292+0000', 'last_unstale': '2024-09-05T16:42:53.775838+0000', 'last_undegraded': '2024-09-05T16:34:56.605776+0000', 'last_fullsized': '2024-09-05T16:34:56.559383+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20591776, 'num_objects': 2514, 'num_object_clones': 0, 'num_object_copies': 10056, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 612, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2514, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10038, 'num_write_kb': 50212, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1659, 'num_bytes_recovered': 2613248, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 1902}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:42:55.514 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:42:55.514 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'10184", 'reported_seq': 10307, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:53.814732+0000', 'last_change': '2024-09-05T16:34:56.867900+0000', 'last_active': '2024-09-05T16:42:53.814732+0000', 'last_peered': '2024-09-05T16:42:53.814732+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:42:53.814732+0000', 'last_undegraded': '2024-09-05T16:34:56.590972+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 32, 'log_start': "29'100", 'ondisk_log_start': "29'100", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10084, 'log_dups_size': 0, 'ondisk_log_size': 10084, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21035624, 'num_objects': 2569, 'num_object_clones': 0, 'num_object_copies': 10276, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2233, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2569, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 10183, 'num_write_kb': 50999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 337}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:42:55.514 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:42:55.514 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'9994", 'reported_seq': 10091, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:53.903085+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:42:53.903085+0000', 'last_peered': '2024-09-05T16:42:53.903085+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:42:53.903085+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 9994, 'log_dups_size': 0, 'ondisk_log_size': 9994, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21304480, 'num_objects': 2601, 'num_object_clones': 0, 'num_object_copies': 10404, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2147, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2601, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9992, 'num_write_kb': 50373, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 456}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:42:55.514 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:42:55.514 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'10220", 'reported_seq': 10334, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:53.871106+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:42:53.871106+0000', 'last_peered': '2024-09-05T16:42:53.871106+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:42:53.871106+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "29'200", 'ondisk_log_start': "29'200", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20905984, 'num_objects': 2552, 'num_object_clones': 0, 'num_object_copies': 10208, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2552, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10220, 'num_write_kb': 51088, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 333}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:42:55.514 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:42:55.514 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'10057", 'reported_seq': 10165, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:53.710024+0000', 'last_change': '2024-09-05T16:34:56.903436+0000', 'last_active': '2024-09-05T16:42:53.710024+0000', 'last_peered': '2024-09-05T16:42:53.710024+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:42:53.710024+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10057, 'log_dups_size': 0, 'ondisk_log_size': 10057, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21337248, 'num_objects': 2605, 'num_object_clones': 0, 'num_object_copies': 10420, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2133, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2605, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10055, 'num_write_kb': 50636, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 474}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:42:55.514 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-05T16:42:55.514 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "37'10192", 'reported_seq': 10281, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:53.689078+0000', 'last_change': '2024-09-05T16:34:56.883319+0000', 'last_active': '2024-09-05T16:42:53.689078+0000', 'last_peered': '2024-09-05T16:42:53.689078+0000', 'last_clean': '2024-09-05T16:33:53.505305+0000', 'last_became_active': '2024-09-05T16:34:56.873389+0000', 'last_became_peered': '2024-09-05T16:34:56.873389+0000', 'last_unstale': '2024-09-05T16:42:53.689078+0000', 'last_undegraded': '2024-09-05T16:34:56.602567+0000', 'last_fullsized': '2024-09-05T16:34:56.559958+0000', 'mapping_epoch': 32, 'log_start': "29'100", 'ondisk_log_start': "29'100", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10092, 'log_dups_size': 0, 'ondisk_log_size': 10092, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21189792, 'num_objects': 2587, 'num_object_clones': 0, 'num_object_copies': 10348, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2234, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2587, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10190, 'num_write_kb': 51108, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 355}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:42:55.515 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:42:55.515 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'10201", 'reported_seq': 10302, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:53.844246+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:42:53.844246+0000', 'last_peered': '2024-09-05T16:42:53.844246+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:42:53.844246+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "29'200", 'ondisk_log_start': "29'200", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10001, 'log_dups_size': 0, 'ondisk_log_size': 10001, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21072192, 'num_objects': 2573, 'num_object_clones': 0, 'num_object_copies': 10292, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2573, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10198, 'num_write_kb': 51081, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 329}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:42:55.515 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:42:55.515 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'10271", 'reported_seq': 10393, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:53.691354+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:42:53.691354+0000', 'last_peered': '2024-09-05T16:42:53.691354+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:42:53.691354+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "29'200", 'ondisk_log_start': "29'200", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10071, 'log_dups_size': 0, 'ondisk_log_size': 10071, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20796576, 'num_objects': 2539, 'num_object_clones': 0, 'num_object_copies': 10156, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2539, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10269, 'num_write_kb': 51230, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 294}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:42:55.515 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:42:55.515 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'10448", 'reported_seq': 10575, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:53.800918+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:42:53.800918+0000', 'last_peered': '2024-09-05T16:42:53.800918+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:42:53.800918+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "29'400", 'ondisk_log_start': "29'400", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10048, 'log_dups_size': 0, 'ondisk_log_size': 10048, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21909232, 'num_objects': 2675, 'num_object_clones': 0, 'num_object_copies': 10700, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2284, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2675, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10445, 'num_write_kb': 52476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 394}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:42:55.515 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:42:55.515 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'10118", 'reported_seq': 10215, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:53.935443+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:42:53.935443+0000', 'last_peered': '2024-09-05T16:42:53.935443+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:42:53.935443+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "29'100", 'ondisk_log_start': "29'100", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10018, 'log_dups_size': 0, 'ondisk_log_size': 10018, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21633616, 'num_objects': 2641, 'num_object_clones': 0, 'num_object_copies': 10564, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2641, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10117, 'num_write_kb': 51031, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 403}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:42:55.516 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:42:55.516 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'10479", 'reported_seq': 10587, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:53.815055+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:42:53.815055+0000', 'last_peered': '2024-09-05T16:42:53.815055+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:42:53.815055+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "29'400", 'ondisk_log_start': "29'400", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10079, 'log_dups_size': 0, 'ondisk_log_size': 10079, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21681312, 'num_objects': 2647, 'num_object_clones': 0, 'num_object_copies': 10588, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2647, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10477, 'num_write_kb': 52500, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 313}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:42:55.516 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:42:55.516 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'10276", 'reported_seq': 10388, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:53.674985+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:42:53.674985+0000', 'last_peered': '2024-09-05T16:42:53.674985+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:42:53.674985+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "29'200", 'ondisk_log_start': "29'200", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10076, 'log_dups_size': 0, 'ondisk_log_size': 10076, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20944032, 'num_objects': 2557, 'num_object_clones': 0, 'num_object_copies': 10228, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2310, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2557, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10274, 'num_write_kb': 51325, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 249}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:42:55.516 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:42:55.516 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'10185", 'reported_seq': 10292, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:53.549955+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:42:53.549955+0000', 'last_peered': '2024-09-05T16:42:53.549955+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:42:53.549955+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "29'100", 'ondisk_log_start': "29'100", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10085, 'log_dups_size': 0, 'ondisk_log_size': 10085, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21214368, 'num_objects': 2590, 'num_object_clones': 0, 'num_object_copies': 10360, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2590, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10183, 'num_write_kb': 51092, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 370}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:42:55.516 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:42:56.299 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:42:56.299 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 29699/166032 objects degraded (17.888%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":579,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":41508,"data_bytes":339988968,"bytes_used":2974154752,"bytes_avail":383572901888,"bytes_total":386547056640,"degraded_objects":29699,"degraded_total":166032,"degraded_ratio":0.17887515659631878,"recovering_objects_per_sec":14,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":255},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:42:56.324 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:42:56.324 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:42:56.954 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:42:56.954 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:42:56.975 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":317,"stamp":"2024-09-05T16:42:55.003249+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":339988968,"num_objects":41508,"num_object_clones":0,"num_object_copies":166032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29699,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41508,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":149329,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1938,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160853,"ondisk_log_size":160853,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2904448,"kb_used_data":667980,"kb_used_omap":14,"kb_used_meta":2236337,"kb_avail":374582912,"statfs":{"total":386547056640,"available":383572901888,"internally_reserved":0,"allocated":684011520,"data_stored":683206424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2290009655},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,3,0,15,13,20,11,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25141248,"num_objects":-3069,"num_object_clones":0,"num_object_copies":-12276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-170,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3069,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3069,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005052"},"pg_stats":[{"pgid":"1.f","version":"37'10276","reported_seq":10380,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:53.455351+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:42:53.455351+0000","last_peered":"2024-09-05T16:42:53.455351+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:42:53.455351+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"29'200","ondisk_log_start":"29'200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21528576,"num_objects":2628,"num_object_clones":0,"num_object_copies":10512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10276,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":392}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'10040","reported_seq":13428,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:53.775838+0000","last_change":"2024-09-05T16:40:28.919189+0000","last_active":"2024-09-05T16:42:53.775838+0000","last_peered":"2024-09-05T16:42:53.775838+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:42:53.775838+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20591776,"num_objects":2514,"num_object_clones":0,"num_object_copies":10056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":612,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2514,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10038,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1659,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1902}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'10184","reported_seq":10307,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:53.814732+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:42:53.814732+0000","last_peered":"2024-09-05T16:42:53.814732+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:42:53.814732+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"29'100","ondisk_log_start":"29'100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21035624,"num_objects":2569,"num_object_clones":0,"num_object_copies":10276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2569,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10183,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":337}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'10276","reported_seq":15005,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:42:53.455753+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:42:53.455753+0000","last_peered":"2024-09-05T16:42:53.455753+0000","last_clean":"2024-09-05T16:42:53.455753+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:42:53.455753+0000","last_undegraded":"2024-09-05T16:42:53.455753+0000","last_fullsized":"2024-09-05T16:42:53.455753+0000","mapping_epoch":34,"log_start":"29'200","ondisk_log_start":"29'200","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21774336,"num_objects":2658,"num_object_clones":0,"num_object_copies":10632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2658,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4576,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'9994","reported_seq":10091,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:53.903085+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:42:53.903085+0000","last_peered":"2024-09-05T16:42:53.903085+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:42:53.903085+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9994,"log_dups_size":0,"ondisk_log_size":9994,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21304480,"num_objects":2601,"num_object_clones":0,"num_object_copies":10404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2601,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9992,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":456}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'10220","reported_seq":10334,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:53.871106+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:42:53.871106+0000","last_peered":"2024-09-05T16:42:53.871106+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:42:53.871106+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"29'200","ondisk_log_start":"29'200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20905984,"num_objects":2552,"num_object_clones":0,"num_object_copies":10208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2552,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10220,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":333}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'10057","reported_seq":10165,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:53.710024+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:42:53.710024+0000","last_peered":"2024-09-05T16:42:53.710024+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:42:53.710024+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21337248,"num_objects":2605,"num_object_clones":0,"num_object_copies":10420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2605,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10055,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":474}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'10192","reported_seq":10281,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:53.689078+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:42:53.689078+0000","last_peered":"2024-09-05T16:42:53.689078+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:42:53.689078+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"29'100","ondisk_log_start":"29'100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21189792,"num_objects":2587,"num_object_clones":0,"num_object_copies":10348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2587,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10190,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":355}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'10201","reported_seq":10302,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:53.844246+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:42:53.844246+0000","last_peered":"2024-09-05T16:42:53.844246+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:42:53.844246+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"29'200","ondisk_log_start":"29'200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21072192,"num_objects":2573,"num_object_clones":0,"num_object_copies":10292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2573,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10198,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":329}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'10271","reported_seq":10393,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:53.691354+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:42:53.691354+0000","last_peered":"2024-09-05T16:42:53.691354+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:42:53.691354+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"29'200","ondisk_log_start":"29'200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20796576,"num_objects":2539,"num_object_clones":0,"num_object_copies":10156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2539,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10269,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":294}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'10448","reported_seq":10575,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:53.800918+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:42:53.800918+0000","last_peered":"2024-09-05T16:42:53.800918+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:42:53.800918+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"29'400","ondisk_log_start":"29'400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21909232,"num_objects":2675,"num_object_clones":0,"num_object_copies":10700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2675,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10445,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":394}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'10118","reported_seq":10215,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:53.935443+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:42:53.935443+0000","last_peered":"2024-09-05T16:42:53.935443+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:42:53.935443+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"29'100","ondisk_log_start":"29'100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21633616,"num_objects":2641,"num_object_clones":0,"num_object_copies":10564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2641,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10117,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":403}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'10236","reported_seq":14972,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:42:53.454141+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:42:53.454141+0000","last_peered":"2024-09-05T16:42:53.454141+0000","last_clean":"2024-09-05T16:42:53.454141+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:42:53.454141+0000","last_undegraded":"2024-09-05T16:42:53.454141+0000","last_fullsized":"2024-09-05T16:42:53.454141+0000","mapping_epoch":36,"log_start":"29'200","ondisk_log_start":"29'200","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21069824,"num_objects":2572,"num_object_clones":0,"num_object_copies":10288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2572,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1836,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'10479","reported_seq":10587,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:53.815055+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:42:53.815055+0000","last_peered":"2024-09-05T16:42:53.815055+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:42:53.815055+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"29'400","ondisk_log_start":"29'400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21681312,"num_objects":2647,"num_object_clones":0,"num_object_copies":10588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2647,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10477,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":313}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'10276","reported_seq":10388,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:53.674985+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:42:53.674985+0000","last_peered":"2024-09-05T16:42:53.674985+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:42:53.674985+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"29'200","ondisk_log_start":"29'200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20944032,"num_objects":2557,"num_object_clones":0,"num_object_copies":10228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2557,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10274,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":249}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'10185","reported_seq":10292,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:53.549955+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:42:53.549955+0000","last_peered":"2024-09-05T16:42:53.549955+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:42:53.549955+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"29'100","ondisk_log_start":"29'100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21214368,"num_objects":2590,"num_object_clones":0,"num_object_copies":10360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2590,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10183,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":370}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":339988968,"num_objects":41508,"num_object_clones":0,"num_object_copies":166032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29699,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41508,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":149329,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1938,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":680243200,"data_stored":680243200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160853,"ondisk_log_size":160853,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084395,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":766308,"kb_used_data":166792,"kb_used_omap":4,"kb_used_meta":599483,"kb_avail":93605532,"statfs":{"total":96636764160,"available":95852064768,"internally_reserved":0,"allocated":170795008,"data_stored":170593734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":613871315},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,5,7,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986272,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":622640,"kb_used_data":167488,"kb_used_omap":4,"kb_used_meta":455099,"kb_avail":93749200,"statfs":{"total":96636764160,"available":95999180800,"internally_reserved":0,"allocated":171507712,"data_stored":171306438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":466022098},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,7,4,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673078,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":757740,"kb_used_data":167064,"kb_used_omap":1,"kb_used_meta":590654,"kb_avail":93614100,"statfs":{"total":96636764160,"available":95860838400,"internally_reserved":0,"allocated":171073536,"data_stored":170872262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":604830149},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,6,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182509,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":757760,"kb_used_data":166636,"kb_used_omap":4,"kb_used_meta":591099,"kb_avail":93614080,"statfs":{"total":96636764160,"available":95860817920,"internally_reserved":0,"allocated":170635264,"data_stored":170433990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":605286093},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,1,1,7,4,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":170029056,"data_stored":170029056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":170389504,"data_stored":170389504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":169975808,"data_stored":169975808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":169848832,"data_stored":169848832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:42:59.978 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:43:00.583 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:43:00.583 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:43:00.601 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":319,"stamp":"2024-09-05T16:42:59.004616+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":335442408,"num_objects":40953,"num_object_clones":0,"num_object_copies":163812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29643,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40953,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":149884,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160508,"ondisk_log_size":160508,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2879128,"kb_used_data":655172,"kb_used_omap":14,"kb_used_meta":2223857,"kb_avail":374608232,"statfs":{"total":386547056640,"available":383598829568,"internally_reserved":0,"allocated":670896128,"data_stored":670091032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2277230135},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,6,0,0,8,17,22,12,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21184512,"num_objects":-2586,"num_object_clones":0,"num_object_copies":-10344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-170,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2586,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2586,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004657"},"pg_stats":[{"pgid":"1.f","version":"37'10276","reported_seq":10380,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:53.455351+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:42:53.455351+0000","last_peered":"2024-09-05T16:42:53.455351+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:42:53.455351+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"29'200","ondisk_log_start":"29'200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21528576,"num_objects":2628,"num_object_clones":0,"num_object_copies":10512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10276,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":392}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'10104","reported_seq":13605,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:58.896219+0000","last_change":"2024-09-05T16:40:28.919189+0000","last_active":"2024-09-05T16:42:58.896219+0000","last_peered":"2024-09-05T16:42:58.896219+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:42:58.896219+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"29'100","ondisk_log_start":"29'100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20067488,"num_objects":2450,"num_object_clones":0,"num_object_copies":9800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":556,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2450,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10102,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1715,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1894}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'10244","reported_seq":10366,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:58.814132+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:42:58.814132+0000","last_peered":"2024-09-05T16:42:58.814132+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:42:58.814132+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"29'200","ondisk_log_start":"29'200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20544104,"num_objects":2509,"num_object_clones":0,"num_object_copies":10036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2509,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10243,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":277}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'10341","reported_seq":15070,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:42:58.607371+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:42:58.607371+0000","last_peered":"2024-09-05T16:42:58.607371+0000","last_clean":"2024-09-05T16:42:58.607371+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:42:58.607371+0000","last_undegraded":"2024-09-05T16:42:58.607371+0000","last_fullsized":"2024-09-05T16:42:58.607371+0000","mapping_epoch":34,"log_start":"29'300","ondisk_log_start":"29'300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21241856,"num_objects":2593,"num_object_clones":0,"num_object_copies":10372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4641,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'9994","reported_seq":10091,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:53.903085+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:42:53.903085+0000","last_peered":"2024-09-05T16:42:53.903085+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:42:53.903085+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":9994,"log_dups_size":0,"ondisk_log_size":9994,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21304480,"num_objects":2601,"num_object_clones":0,"num_object_copies":10404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2601,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9992,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":456}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'10220","reported_seq":10334,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:53.871106+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:42:53.871106+0000","last_peered":"2024-09-05T16:42:53.871106+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:42:53.871106+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"29'200","ondisk_log_start":"29'200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20905984,"num_objects":2552,"num_object_clones":0,"num_object_copies":10208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2552,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10220,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":333}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'10119","reported_seq":10227,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:58.641253+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:42:58.641253+0000","last_peered":"2024-09-05T16:42:58.641253+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:42:58.641253+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"29'100","ondisk_log_start":"29'100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20829344,"num_objects":2543,"num_object_clones":0,"num_object_copies":10172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2543,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10117,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":412}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'10260","reported_seq":10349,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:58.775943+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:42:58.775943+0000","last_peered":"2024-09-05T16:42:58.775943+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:42:58.775943+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"29'200","ondisk_log_start":"29'200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20632736,"num_objects":2519,"num_object_clones":0,"num_object_copies":10076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2519,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10258,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":287}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'10201","reported_seq":10302,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:53.844246+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:42:53.844246+0000","last_peered":"2024-09-05T16:42:53.844246+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:42:53.844246+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"29'200","ondisk_log_start":"29'200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21072192,"num_objects":2573,"num_object_clones":0,"num_object_copies":10292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2573,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10198,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":329}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'10330","reported_seq":10452,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:58.758879+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:42:58.758879+0000","last_peered":"2024-09-05T16:42:58.758879+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:42:58.758879+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"29'300","ondisk_log_start":"29'300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20313248,"num_objects":2480,"num_object_clones":0,"num_object_copies":9920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2480,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10328,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":235}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'10448","reported_seq":10575,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:53.800918+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:42:53.800918+0000","last_peered":"2024-09-05T16:42:53.800918+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:42:53.800918+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"29'400","ondisk_log_start":"29'400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21909232,"num_objects":2675,"num_object_clones":0,"num_object_copies":10700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2675,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10445,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":394}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'10118","reported_seq":10215,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:53.935443+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:42:53.935443+0000","last_peered":"2024-09-05T16:42:53.935443+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:42:53.935443+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"29'100","ondisk_log_start":"29'100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21633616,"num_objects":2641,"num_object_clones":0,"num_object_copies":10564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2641,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10117,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":403}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'10301","reported_seq":15035,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:42:58.528946+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:42:58.528946+0000","last_peered":"2024-09-05T16:42:58.528946+0000","last_clean":"2024-09-05T16:42:58.528946+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:42:58.528946+0000","last_undegraded":"2024-09-05T16:42:58.528946+0000","last_fullsized":"2024-09-05T16:42:58.528946+0000","mapping_epoch":36,"log_start":"29'300","ondisk_log_start":"29'300","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20537344,"num_objects":2507,"num_object_clones":0,"num_object_copies":10028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2507,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1901,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'10479","reported_seq":10587,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:53.815055+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:42:53.815055+0000","last_peered":"2024-09-05T16:42:53.815055+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:42:53.815055+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"29'400","ondisk_log_start":"29'400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21681312,"num_objects":2647,"num_object_clones":0,"num_object_copies":10588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2647,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10477,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":313}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'10332","reported_seq":10445,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:58.781703+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:42:58.781703+0000","last_peered":"2024-09-05T16:42:58.781703+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:42:58.781703+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"29'300","ondisk_log_start":"29'300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20485280,"num_objects":2501,"num_object_clones":0,"num_object_copies":10004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2501,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10330,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":193}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'10241","reported_seq":10349,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:58.808822+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:42:58.808822+0000","last_peered":"2024-09-05T16:42:58.808822+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:42:58.808822+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"29'200","ondisk_log_start":"29'200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20755616,"num_objects":2534,"num_object_clones":0,"num_object_copies":10136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2534,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10239,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":314}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":335442408,"num_objects":40953,"num_object_clones":0,"num_object_copies":163812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29643,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40953,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":149884,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":667717632,"data_stored":667717632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160508,"ondisk_log_size":160508,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084395,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":766308,"kb_used_data":166792,"kb_used_omap":4,"kb_used_meta":599483,"kb_avail":93605532,"statfs":{"total":96636764160,"available":95852064768,"internally_reserved":0,"allocated":170795008,"data_stored":170593734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":613871315},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,5,7,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986273,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":605340,"kb_used_data":162692,"kb_used_omap":4,"kb_used_meta":442619,"kb_avail":93766500,"statfs":{"total":96636764160,"available":96016896000,"internally_reserved":0,"allocated":166596608,"data_stored":166395334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":453242578},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,5,3,7,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673079,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":753712,"kb_used_data":163040,"kb_used_omap":1,"kb_used_meta":590654,"kb_avail":93618128,"statfs":{"total":96636764160,"available":95864963072,"internally_reserved":0,"allocated":166952960,"data_stored":166751686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":604830149},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,1,3,8,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182510,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":753768,"kb_used_data":162648,"kb_used_omap":4,"kb_used_meta":591099,"kb_avail":93618072,"statfs":{"total":96636764160,"available":95864905728,"internally_reserved":0,"allocated":166551552,"data_stored":166350278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":605286093},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,6,0,0,1,4,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":165801984,"data_stored":165801984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":166178816,"data_stored":166178816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":165888000,"data_stored":165888000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":169848832,"data_stored":169848832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:43:00.602 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:43:01.237 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:43:01.239 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:43:01.265 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":320,"stamp":"2024-09-05T16:43:01.005531+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":331723240,"num_objects":40499,"num_object_clones":0,"num_object_copies":161996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29643,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40499,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":150338,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160662,"ondisk_log_size":160662,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2862648,"kb_used_data":651184,"kb_used_omap":14,"kb_used_meta":2211377,"kb_avail":374624712,"statfs":{"total":386547056640,"available":383615705088,"internally_reserved":0,"allocated":666812416,"data_stored":666007320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2264450615},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,6,1,1,6,16,16,11,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16613376,"num_objects":-2028,"num_object_clones":0,"num_object_copies":-8112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2028,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2028,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":110,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004830"},"pg_stats":[{"pgid":"1.f","version":"37'10344","reported_seq":10448,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:58.859267+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:42:58.859267+0000","last_peered":"2024-09-05T16:42:58.859267+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:42:58.859267+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"29'300","ondisk_log_start":"29'300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971520,"num_objects":2560,"num_object_clones":0,"num_object_copies":10240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2560,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10344,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":324}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'10104","reported_seq":13605,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:58.896219+0000","last_change":"2024-09-05T16:40:28.919189+0000","last_active":"2024-09-05T16:42:58.896219+0000","last_peered":"2024-09-05T16:42:58.896219+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:42:58.896219+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"29'100","ondisk_log_start":"29'100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20067488,"num_objects":2450,"num_object_clones":0,"num_object_copies":9800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":556,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2450,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10102,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1715,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1894}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'10244","reported_seq":10366,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:58.814132+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:42:58.814132+0000","last_peered":"2024-09-05T16:42:58.814132+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:42:58.814132+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"29'200","ondisk_log_start":"29'200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20544104,"num_objects":2509,"num_object_clones":0,"num_object_copies":10036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2509,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10243,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":277}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'10341","reported_seq":15070,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:42:58.607371+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:42:58.607371+0000","last_peered":"2024-09-05T16:42:58.607371+0000","last_clean":"2024-09-05T16:42:58.607371+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:42:58.607371+0000","last_undegraded":"2024-09-05T16:42:58.607371+0000","last_fullsized":"2024-09-05T16:42:58.607371+0000","mapping_epoch":34,"log_start":"29'300","ondisk_log_start":"29'300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21241856,"num_objects":2593,"num_object_clones":0,"num_object_copies":10372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4641,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'10060","reported_seq":10159,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:58.906318+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:42:58.906318+0000","last_peered":"2024-09-05T16:42:58.906318+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:42:58.906318+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20763808,"num_objects":2535,"num_object_clones":0,"num_object_copies":10140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2535,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10058,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":390}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'10294","reported_seq":10408,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:58.896332+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:42:58.896332+0000","last_peered":"2024-09-05T16:42:58.896332+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:42:58.896332+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"29'200","ondisk_log_start":"29'200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20299776,"num_objects":2478,"num_object_clones":0,"num_object_copies":9912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2478,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10294,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":259}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'10119","reported_seq":10227,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:58.641253+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:42:58.641253+0000","last_peered":"2024-09-05T16:42:58.641253+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:42:58.641253+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"29'100","ondisk_log_start":"29'100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20829344,"num_objects":2543,"num_object_clones":0,"num_object_copies":10172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2543,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10117,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":412}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'10260","reported_seq":10349,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:58.775943+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:42:58.775943+0000","last_peered":"2024-09-05T16:42:58.775943+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:42:58.775943+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"29'200","ondisk_log_start":"29'200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20632736,"num_objects":2519,"num_object_clones":0,"num_object_copies":10076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2519,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10258,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":287}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'10262","reported_seq":10363,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:59.005642+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:42:59.005642+0000","last_peered":"2024-09-05T16:42:59.005642+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:42:59.005642+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"29'200","ondisk_log_start":"29'200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20572480,"num_objects":2512,"num_object_clones":0,"num_object_copies":10048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2512,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10259,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":268}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'10330","reported_seq":10452,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:58.758879+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:42:58.758879+0000","last_peered":"2024-09-05T16:42:58.758879+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:42:58.758879+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"29'300","ondisk_log_start":"29'300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20313248,"num_objects":2480,"num_object_clones":0,"num_object_copies":9920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2480,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10328,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":235}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'10515","reported_seq":10642,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:58.621187+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:42:58.621187+0000","last_peered":"2024-09-05T16:42:58.621187+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:42:58.621187+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"29'500","ondisk_log_start":"29'500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21360368,"num_objects":2608,"num_object_clones":0,"num_object_copies":10432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2608,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10512,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":327}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'10176","reported_seq":10274,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:58.874000+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:42:58.874000+0000","last_peered":"2024-09-05T16:42:58.874000+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:42:58.874000+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"29'100","ondisk_log_start":"29'100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21158480,"num_objects":2583,"num_object_clones":0,"num_object_copies":10332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2583,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10175,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":345}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'10301","reported_seq":15035,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:42:58.528946+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:42:58.528946+0000","last_peered":"2024-09-05T16:42:58.528946+0000","last_clean":"2024-09-05T16:42:58.528946+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:42:58.528946+0000","last_undegraded":"2024-09-05T16:42:58.528946+0000","last_fullsized":"2024-09-05T16:42:58.528946+0000","mapping_epoch":36,"log_start":"29'300","ondisk_log_start":"29'300","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20537344,"num_objects":2507,"num_object_clones":0,"num_object_copies":10028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2507,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1901,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'10539","reported_seq":10647,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:58.973130+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:42:58.973130+0000","last_peered":"2024-09-05T16:42:58.973130+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:42:58.973130+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"29'500","ondisk_log_start":"29'500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21189792,"num_objects":2587,"num_object_clones":0,"num_object_copies":10348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2587,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10537,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":253}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'10332","reported_seq":10445,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:58.781703+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:42:58.781703+0000","last_peered":"2024-09-05T16:42:58.781703+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:42:58.781703+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"29'300","ondisk_log_start":"29'300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20485280,"num_objects":2501,"num_object_clones":0,"num_object_copies":10004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2501,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10330,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":193}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'10241","reported_seq":10349,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:58.808822+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:42:58.808822+0000","last_peered":"2024-09-05T16:42:58.808822+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:42:58.808822+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"29'200","ondisk_log_start":"29'200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20755616,"num_objects":2534,"num_object_clones":0,"num_object_copies":10136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2534,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10239,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":314}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":331723240,"num_objects":40499,"num_object_clones":0,"num_object_copies":161996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29643,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40499,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":150338,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663584768,"data_stored":663584768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160662,"ondisk_log_size":160662,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084396,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":749828,"kb_used_data":162804,"kb_used_omap":4,"kb_used_meta":587003,"kb_avail":93622012,"statfs":{"total":96636764160,"available":95868940288,"internally_reserved":0,"allocated":166711296,"data_stored":166510022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":601091795},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,1,1,1,4,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986273,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":605340,"kb_used_data":162692,"kb_used_omap":4,"kb_used_meta":442619,"kb_avail":93766500,"statfs":{"total":96636764160,"available":96016896000,"internally_reserved":0,"allocated":166596608,"data_stored":166395334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":453242578},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,5,3,7,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673079,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":753712,"kb_used_data":163040,"kb_used_omap":1,"kb_used_meta":590654,"kb_avail":93618128,"statfs":{"total":96636764160,"available":95864963072,"internally_reserved":0,"allocated":166952960,"data_stored":166751686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":604830149},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,1,3,8,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182510,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":753768,"kb_used_data":162648,"kb_used_omap":4,"kb_used_meta":591099,"kb_avail":93618072,"statfs":{"total":96636764160,"available":95864905728,"internally_reserved":0,"allocated":166551552,"data_stored":166350278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":605286093},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,6,0,0,1,4,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":165801984,"data_stored":165801984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":166178816,"data_stored":166178816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":165888000,"data_stored":165888000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":165715968,"data_stored":165715968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:43:01.266 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-05T16:43:01.266 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'10344", 'reported_seq': 10448, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:58.859267+0000', 'last_change': '2024-09-05T16:34:56.833497+0000', 'last_active': '2024-09-05T16:42:58.859267+0000', 'last_peered': '2024-09-05T16:42:58.859267+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.828289+0000', 'last_became_peered': '2024-09-05T16:34:56.828289+0000', 'last_unstale': '2024-09-05T16:42:58.859267+0000', 'last_undegraded': '2024-09-05T16:34:56.612160+0000', 'last_fullsized': '2024-09-05T16:34:56.557632+0000', 'mapping_epoch': 32, 'log_start': "29'300", 'ondisk_log_start': "29'300", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10044, 'log_dups_size': 0, 'ondisk_log_size': 10044, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20971520, 'num_objects': 2560, 'num_object_clones': 0, 'num_object_copies': 10240, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2236, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2560, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10344, 'num_write_kb': 51616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 324}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:43:01.266 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-05T16:43:01.266 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "37'10104", 'reported_seq': 13605, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:58.896219+0000', 'last_change': '2024-09-05T16:40:28.919189+0000', 'last_active': '2024-09-05T16:42:58.896219+0000', 'last_peered': '2024-09-05T16:42:58.896219+0000', 'last_clean': '2024-09-05T16:33:53.200762+0000', 'last_became_active': '2024-09-05T16:34:56.820292+0000', 'last_became_peered': '2024-09-05T16:34:56.820292+0000', 'last_unstale': '2024-09-05T16:42:58.896219+0000', 'last_undegraded': '2024-09-05T16:34:56.605776+0000', 'last_fullsized': '2024-09-05T16:34:56.559383+0000', 'mapping_epoch': 32, 'log_start': "29'100", 'ondisk_log_start': "29'100", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10004, 'log_dups_size': 0, 'ondisk_log_size': 10004, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20067488, 'num_objects': 2450, 'num_object_clones': 0, 'num_object_copies': 9800, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 556, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2450, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10102, 'num_write_kb': 50212, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1715, 'num_bytes_recovered': 2613248, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 1894}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:43:01.266 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:43:01.266 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'10244", 'reported_seq': 10366, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:58.814132+0000', 'last_change': '2024-09-05T16:34:56.867900+0000', 'last_active': '2024-09-05T16:42:58.814132+0000', 'last_peered': '2024-09-05T16:42:58.814132+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:42:58.814132+0000', 'last_undegraded': '2024-09-05T16:34:56.590972+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 32, 'log_start': "29'200", 'ondisk_log_start': "29'200", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10044, 'log_dups_size': 0, 'ondisk_log_size': 10044, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20544104, 'num_objects': 2509, 'num_object_clones': 0, 'num_object_copies': 10036, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2233, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2509, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 10243, 'num_write_kb': 50999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 277}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:43:01.266 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:43:01.266 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'10060", 'reported_seq': 10159, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:58.906318+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:42:58.906318+0000', 'last_peered': '2024-09-05T16:42:58.906318+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:42:58.906318+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10060, 'log_dups_size': 0, 'ondisk_log_size': 10060, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20763808, 'num_objects': 2535, 'num_object_clones': 0, 'num_object_copies': 10140, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2147, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2535, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10058, 'num_write_kb': 50373, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 390}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:43:01.266 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:43:01.267 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'10294", 'reported_seq': 10408, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:58.896332+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:42:58.896332+0000', 'last_peered': '2024-09-05T16:42:58.896332+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:42:58.896332+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "29'200", 'ondisk_log_start': "29'200", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10094, 'log_dups_size': 0, 'ondisk_log_size': 10094, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20299776, 'num_objects': 2478, 'num_object_clones': 0, 'num_object_copies': 9912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2478, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10294, 'num_write_kb': 51088, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 259}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:43:01.267 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:43:01.267 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'10119", 'reported_seq': 10227, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:58.641253+0000', 'last_change': '2024-09-05T16:34:56.903436+0000', 'last_active': '2024-09-05T16:42:58.641253+0000', 'last_peered': '2024-09-05T16:42:58.641253+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:42:58.641253+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "29'100", 'ondisk_log_start': "29'100", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10019, 'log_dups_size': 0, 'ondisk_log_size': 10019, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20829344, 'num_objects': 2543, 'num_object_clones': 0, 'num_object_copies': 10172, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2133, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2543, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10117, 'num_write_kb': 50636, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 412}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:43:01.267 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-05T16:43:01.267 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "37'10260", 'reported_seq': 10349, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:58.775943+0000', 'last_change': '2024-09-05T16:34:56.883319+0000', 'last_active': '2024-09-05T16:42:58.775943+0000', 'last_peered': '2024-09-05T16:42:58.775943+0000', 'last_clean': '2024-09-05T16:33:53.505305+0000', 'last_became_active': '2024-09-05T16:34:56.873389+0000', 'last_became_peered': '2024-09-05T16:34:56.873389+0000', 'last_unstale': '2024-09-05T16:42:58.775943+0000', 'last_undegraded': '2024-09-05T16:34:56.602567+0000', 'last_fullsized': '2024-09-05T16:34:56.559958+0000', 'mapping_epoch': 32, 'log_start': "29'200", 'ondisk_log_start': "29'200", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10060, 'log_dups_size': 0, 'ondisk_log_size': 10060, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20632736, 'num_objects': 2519, 'num_object_clones': 0, 'num_object_copies': 10076, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2234, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2519, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10258, 'num_write_kb': 51108, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 287}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:43:01.267 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:43:01.267 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'10262", 'reported_seq': 10363, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:59.005642+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:42:59.005642+0000', 'last_peered': '2024-09-05T16:42:59.005642+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:42:59.005642+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "29'200", 'ondisk_log_start': "29'200", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10062, 'log_dups_size': 0, 'ondisk_log_size': 10062, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20572480, 'num_objects': 2512, 'num_object_clones': 0, 'num_object_copies': 10048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2512, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10259, 'num_write_kb': 51081, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 268}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:43:01.267 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:43:01.267 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'10330", 'reported_seq': 10452, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:58.758879+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:42:58.758879+0000', 'last_peered': '2024-09-05T16:42:58.758879+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:42:58.758879+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "29'300", 'ondisk_log_start': "29'300", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20313248, 'num_objects': 2480, 'num_object_clones': 0, 'num_object_copies': 9920, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2480, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10328, 'num_write_kb': 51230, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 235}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:43:01.267 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:43:01.267 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'10515", 'reported_seq': 10642, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:58.621187+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:42:58.621187+0000', 'last_peered': '2024-09-05T16:42:58.621187+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:42:58.621187+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "29'500", 'ondisk_log_start': "29'500", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10015, 'log_dups_size': 0, 'ondisk_log_size': 10015, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21360368, 'num_objects': 2608, 'num_object_clones': 0, 'num_object_copies': 10432, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2284, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2608, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10512, 'num_write_kb': 52476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 327}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:43:01.268 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:43:01.268 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'10176", 'reported_seq': 10274, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:58.874000+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:42:58.874000+0000', 'last_peered': '2024-09-05T16:42:58.874000+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:42:58.874000+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "29'100", 'ondisk_log_start': "29'100", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10076, 'log_dups_size': 0, 'ondisk_log_size': 10076, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21158480, 'num_objects': 2583, 'num_object_clones': 0, 'num_object_copies': 10332, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2583, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10175, 'num_write_kb': 51031, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 345}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:43:01.268 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:43:01.268 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'10539", 'reported_seq': 10647, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:58.973130+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:42:58.973130+0000', 'last_peered': '2024-09-05T16:42:58.973130+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:42:58.973130+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "29'500", 'ondisk_log_start': "29'500", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10039, 'log_dups_size': 0, 'ondisk_log_size': 10039, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21189792, 'num_objects': 2587, 'num_object_clones': 0, 'num_object_copies': 10348, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2587, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10537, 'num_write_kb': 52500, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 253}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:43:01.268 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:43:01.268 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'10332", 'reported_seq': 10445, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:58.781703+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:42:58.781703+0000', 'last_peered': '2024-09-05T16:42:58.781703+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:42:58.781703+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "29'300", 'ondisk_log_start': "29'300", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10032, 'log_dups_size': 0, 'ondisk_log_size': 10032, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20485280, 'num_objects': 2501, 'num_object_clones': 0, 'num_object_copies': 10004, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2310, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2501, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10330, 'num_write_kb': 51325, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 193}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:43:01.268 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:43:01.268 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'10241", 'reported_seq': 10349, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:42:58.808822+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:42:58.808822+0000', 'last_peered': '2024-09-05T16:42:58.808822+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:42:58.808822+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "29'200", 'ondisk_log_start': "29'200", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10041, 'log_dups_size': 0, 'ondisk_log_size': 10041, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20755616, 'num_objects': 2534, 'num_object_clones': 0, 'num_object_copies': 10136, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2534, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10239, 'num_write_kb': 51092, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 314}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:43:01.269 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:43:02.022 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:43:02.023 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 29643/161996 objects degraded (18.299%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":585,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":40499,"data_bytes":331723240,"bytes_used":2931351552,"bytes_avail":383615705088,"bytes_total":386547056640,"degraded_objects":29643,"degraded_total":161996,"degraded_ratio":0.18298599965431245,"recovering_objects_per_sec":9,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":110,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":168},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:43:02.058 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:43:02.059 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:43:02.651 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:43:02.651 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:43:02.681 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":320,"stamp":"2024-09-05T16:43:01.005531+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":331723240,"num_objects":40499,"num_object_clones":0,"num_object_copies":161996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29643,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40499,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":150338,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160662,"ondisk_log_size":160662,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2862648,"kb_used_data":651184,"kb_used_omap":14,"kb_used_meta":2211377,"kb_avail":374624712,"statfs":{"total":386547056640,"available":383615705088,"internally_reserved":0,"allocated":666812416,"data_stored":666007320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2264450615},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,6,1,1,6,16,16,11,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16613376,"num_objects":-2028,"num_object_clones":0,"num_object_copies":-8112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2028,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2028,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":110,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004830"},"pg_stats":[{"pgid":"1.f","version":"37'10344","reported_seq":10448,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:58.859267+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:42:58.859267+0000","last_peered":"2024-09-05T16:42:58.859267+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:42:58.859267+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"29'300","ondisk_log_start":"29'300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971520,"num_objects":2560,"num_object_clones":0,"num_object_copies":10240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2560,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10344,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":324}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'10104","reported_seq":13605,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:58.896219+0000","last_change":"2024-09-05T16:40:28.919189+0000","last_active":"2024-09-05T16:42:58.896219+0000","last_peered":"2024-09-05T16:42:58.896219+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:42:58.896219+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"29'100","ondisk_log_start":"29'100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20067488,"num_objects":2450,"num_object_clones":0,"num_object_copies":9800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":556,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2450,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10102,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1715,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1894}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'10244","reported_seq":10366,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:58.814132+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:42:58.814132+0000","last_peered":"2024-09-05T16:42:58.814132+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:42:58.814132+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"29'200","ondisk_log_start":"29'200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20544104,"num_objects":2509,"num_object_clones":0,"num_object_copies":10036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2509,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10243,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":277}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'10341","reported_seq":15070,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:42:58.607371+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:42:58.607371+0000","last_peered":"2024-09-05T16:42:58.607371+0000","last_clean":"2024-09-05T16:42:58.607371+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:42:58.607371+0000","last_undegraded":"2024-09-05T16:42:58.607371+0000","last_fullsized":"2024-09-05T16:42:58.607371+0000","mapping_epoch":34,"log_start":"29'300","ondisk_log_start":"29'300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21241856,"num_objects":2593,"num_object_clones":0,"num_object_copies":10372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4641,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'10060","reported_seq":10159,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:58.906318+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:42:58.906318+0000","last_peered":"2024-09-05T16:42:58.906318+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:42:58.906318+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20763808,"num_objects":2535,"num_object_clones":0,"num_object_copies":10140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2535,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10058,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":390}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'10294","reported_seq":10408,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:58.896332+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:42:58.896332+0000","last_peered":"2024-09-05T16:42:58.896332+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:42:58.896332+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"29'200","ondisk_log_start":"29'200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20299776,"num_objects":2478,"num_object_clones":0,"num_object_copies":9912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2478,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10294,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":259}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'10119","reported_seq":10227,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:58.641253+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:42:58.641253+0000","last_peered":"2024-09-05T16:42:58.641253+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:42:58.641253+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"29'100","ondisk_log_start":"29'100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20829344,"num_objects":2543,"num_object_clones":0,"num_object_copies":10172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2543,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10117,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":412}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'10260","reported_seq":10349,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:58.775943+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:42:58.775943+0000","last_peered":"2024-09-05T16:42:58.775943+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:42:58.775943+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"29'200","ondisk_log_start":"29'200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20632736,"num_objects":2519,"num_object_clones":0,"num_object_copies":10076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2519,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10258,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":287}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'10262","reported_seq":10363,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:59.005642+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:42:59.005642+0000","last_peered":"2024-09-05T16:42:59.005642+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:42:59.005642+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"29'200","ondisk_log_start":"29'200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20572480,"num_objects":2512,"num_object_clones":0,"num_object_copies":10048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2512,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10259,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":268}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'10330","reported_seq":10452,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:58.758879+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:42:58.758879+0000","last_peered":"2024-09-05T16:42:58.758879+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:42:58.758879+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"29'300","ondisk_log_start":"29'300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20313248,"num_objects":2480,"num_object_clones":0,"num_object_copies":9920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2480,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10328,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":235}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'10515","reported_seq":10642,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:58.621187+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:42:58.621187+0000","last_peered":"2024-09-05T16:42:58.621187+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:42:58.621187+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"29'500","ondisk_log_start":"29'500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21360368,"num_objects":2608,"num_object_clones":0,"num_object_copies":10432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2608,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10512,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":327}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'10176","reported_seq":10274,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:58.874000+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:42:58.874000+0000","last_peered":"2024-09-05T16:42:58.874000+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:42:58.874000+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"29'100","ondisk_log_start":"29'100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21158480,"num_objects":2583,"num_object_clones":0,"num_object_copies":10332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2583,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10175,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":345}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'10301","reported_seq":15035,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:42:58.528946+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:42:58.528946+0000","last_peered":"2024-09-05T16:42:58.528946+0000","last_clean":"2024-09-05T16:42:58.528946+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:42:58.528946+0000","last_undegraded":"2024-09-05T16:42:58.528946+0000","last_fullsized":"2024-09-05T16:42:58.528946+0000","mapping_epoch":36,"log_start":"29'300","ondisk_log_start":"29'300","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20537344,"num_objects":2507,"num_object_clones":0,"num_object_copies":10028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2507,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1901,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'10539","reported_seq":10647,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:58.973130+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:42:58.973130+0000","last_peered":"2024-09-05T16:42:58.973130+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:42:58.973130+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"29'500","ondisk_log_start":"29'500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21189792,"num_objects":2587,"num_object_clones":0,"num_object_copies":10348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2587,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10537,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":253}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'10332","reported_seq":10445,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:58.781703+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:42:58.781703+0000","last_peered":"2024-09-05T16:42:58.781703+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:42:58.781703+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"29'300","ondisk_log_start":"29'300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20485280,"num_objects":2501,"num_object_clones":0,"num_object_copies":10004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2501,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10330,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":193}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'10241","reported_seq":10349,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:42:58.808822+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:42:58.808822+0000","last_peered":"2024-09-05T16:42:58.808822+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:42:58.808822+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"29'200","ondisk_log_start":"29'200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20755616,"num_objects":2534,"num_object_clones":0,"num_object_copies":10136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2534,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10239,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":314}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":331723240,"num_objects":40499,"num_object_clones":0,"num_object_copies":161996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29643,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40499,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":150338,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663584768,"data_stored":663584768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160662,"ondisk_log_size":160662,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084396,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":749828,"kb_used_data":162804,"kb_used_omap":4,"kb_used_meta":587003,"kb_avail":93622012,"statfs":{"total":96636764160,"available":95868940288,"internally_reserved":0,"allocated":166711296,"data_stored":166510022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":601091795},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,1,1,1,4,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986273,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":605340,"kb_used_data":162692,"kb_used_omap":4,"kb_used_meta":442619,"kb_avail":93766500,"statfs":{"total":96636764160,"available":96016896000,"internally_reserved":0,"allocated":166596608,"data_stored":166395334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":453242578},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,5,3,7,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673079,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":753712,"kb_used_data":163040,"kb_used_omap":1,"kb_used_meta":590654,"kb_avail":93618128,"statfs":{"total":96636764160,"available":95864963072,"internally_reserved":0,"allocated":166952960,"data_stored":166751686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":604830149},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,1,3,8,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182510,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":753768,"kb_used_data":162648,"kb_used_omap":4,"kb_used_meta":591099,"kb_avail":93618072,"statfs":{"total":96636764160,"available":95864905728,"internally_reserved":0,"allocated":166551552,"data_stored":166350278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":605286093},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,6,0,0,1,4,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":165801984,"data_stored":165801984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":166178816,"data_stored":166178816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":165888000,"data_stored":165888000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":165715968,"data_stored":165715968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:43:05.683 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:43:06.262 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:43:06.266 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:43:06.290 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":322,"stamp":"2024-09-05T16:43:05.006889+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":323383784,"num_objects":39481,"num_object_clones":0,"num_object_copies":157924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29578,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39481,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":151356,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2059,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160680,"ondisk_log_size":160680,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2863176,"kb_used_data":635048,"kb_used_omap":14,"kb_used_meta":2228017,"kb_avail":374624184,"statfs":{"total":386547056640,"available":383615164416,"internally_reserved":0,"allocated":650289152,"data_stored":649484056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2281489975},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,5,4,15,10,8,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24952832,"num_objects":-3046,"num_object_clones":0,"num_object_copies":-12184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-175,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3046,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3046,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":175,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004474"},"pg_stats":[{"pgid":"1.f","version":"37'10410","reported_seq":10514,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:04.036068+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:43:04.036068+0000","last_peered":"2024-09-05T16:43:04.036068+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:43:04.036068+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"29'400","ondisk_log_start":"29'400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20430848,"num_objects":2494,"num_object_clones":0,"num_object_copies":9976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2494,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10410,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":258}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'10163","reported_seq":13793,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:03.982646+0000","last_change":"2024-09-05T16:40:28.919189+0000","last_active":"2024-09-05T16:43:03.982646+0000","last_peered":"2024-09-05T16:43:03.982646+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:43:03.982646+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"29'100","ondisk_log_start":"29'100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19584160,"num_objects":2391,"num_object_clones":0,"num_object_copies":9564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":491,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10161,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1780,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1900}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'10303","reported_seq":10426,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:03.764481+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:43:03.764481+0000","last_peered":"2024-09-05T16:43:03.764481+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:43:03.764481+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"29'300","ondisk_log_start":"29'300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20060776,"num_objects":2450,"num_object_clones":0,"num_object_copies":9800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2450,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10302,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":218}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'10408","reported_seq":15136,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:43:03.640207+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:43:03.640207+0000","last_peered":"2024-09-05T16:43:03.640207+0000","last_clean":"2024-09-05T16:43:03.640207+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:43:03.640207+0000","last_undegraded":"2024-09-05T16:43:03.640207+0000","last_fullsized":"2024-09-05T16:43:03.640207+0000","mapping_epoch":34,"log_start":"29'400","ondisk_log_start":"29'400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20692992,"num_objects":2526,"num_object_clones":0,"num_object_copies":10104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2526,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4708,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'10121","reported_seq":10220,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:03.968411+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:43:03.968411+0000","last_peered":"2024-09-05T16:43:03.968411+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:43:03.968411+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"29'100","ondisk_log_start":"29'100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20264096,"num_objects":2474,"num_object_clones":0,"num_object_copies":9896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2474,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10119,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":329}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'10351","reported_seq":10463,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:04.045294+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:43:04.045294+0000","last_peered":"2024-09-05T16:43:04.045294+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:43:04.045294+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"29'300","ondisk_log_start":"29'300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19832832,"num_objects":2421,"num_object_clones":0,"num_object_copies":9684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2421,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10351,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":202}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'10185","reported_seq":10293,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:03.881037+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:43:03.881037+0000","last_peered":"2024-09-05T16:43:03.881037+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:43:03.881037+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"29'100","ondisk_log_start":"29'100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20288672,"num_objects":2477,"num_object_clones":0,"num_object_copies":9908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2477,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10183,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":346}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'10328","reported_seq":10417,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:03.928770+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:43:03.928770+0000","last_peered":"2024-09-05T16:43:03.928770+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:43:03.928770+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"29'300","ondisk_log_start":"29'300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20075680,"num_objects":2451,"num_object_clones":0,"num_object_copies":9804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2451,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10326,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":219}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'10338","reported_seq":10439,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:03.958373+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:43:03.958373+0000","last_peered":"2024-09-05T16:43:03.958373+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:43:03.958373+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"29'300","ondisk_log_start":"29'300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19949888,"num_objects":2436,"num_object_clones":0,"num_object_copies":9744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2436,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10335,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":192}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'10389","reported_seq":10511,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:03.632025+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:43:03.632025+0000","last_peered":"2024-09-05T16:43:03.632025+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:43:03.632025+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"29'300","ondisk_log_start":"29'300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19829920,"num_objects":2421,"num_object_clones":0,"num_object_copies":9684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2421,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10387,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":176}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'10590","reported_seq":10717,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:03.959792+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:43:03.959792+0000","last_peered":"2024-09-05T16:43:03.959792+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:43:03.959792+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"29'500","ondisk_log_start":"29'500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20745968,"num_objects":2533,"num_object_clones":0,"num_object_copies":10132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10587,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":252}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'10234","reported_seq":10333,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:04.023026+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:43:04.023026+0000","last_peered":"2024-09-05T16:43:04.023026+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:43:04.023026+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"29'200","ondisk_log_start":"29'200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20683344,"num_objects":2525,"num_object_clones":0,"num_object_copies":10100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2525,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10233,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":287}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'10354","reported_seq":15090,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:43:03.710401+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:43:03.710401+0000","last_peered":"2024-09-05T16:43:03.710401+0000","last_clean":"2024-09-05T16:43:03.710401+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:43:03.710401+0000","last_undegraded":"2024-09-05T16:43:03.710401+0000","last_fullsized":"2024-09-05T16:43:03.710401+0000","mapping_epoch":36,"log_start":"29'300","ondisk_log_start":"29'300","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20103168,"num_objects":2454,"num_object_clones":0,"num_object_copies":9816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2454,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1954,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'10607","reported_seq":10714,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:03.990940+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:43:03.990940+0000","last_peered":"2024-09-05T16:43:03.990940+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:43:03.990940+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"29'600","ondisk_log_start":"29'600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20632736,"num_objects":2519,"num_object_clones":0,"num_object_copies":10076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2519,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10605,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":185}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'10390","reported_seq":10502,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:03.787080+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:43:03.787080+0000","last_peered":"2024-09-05T16:43:03.787080+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:43:03.787080+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"29'300","ondisk_log_start":"29'300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20010144,"num_objects":2443,"num_object_clones":0,"num_object_copies":9772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2443,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10388,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":135}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'10309","reported_seq":10416,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:03.706876+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:43:03.706876+0000","last_peered":"2024-09-05T16:43:03.706876+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:43:03.706876+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"29'300","ondisk_log_start":"29'300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20198560,"num_objects":2466,"num_object_clones":0,"num_object_copies":9864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10307,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":246}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":323383784,"num_objects":39481,"num_object_clones":0,"num_object_copies":157924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29578,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39481,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":151356,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2059,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":646959104,"data_stored":646959104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160680,"ondisk_log_size":160680,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084397,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":750028,"kb_used_data":158836,"kb_used_omap":4,"kb_used_meta":591163,"kb_avail":93621812,"statfs":{"total":96636764160,"available":95868735488,"internally_reserved":0,"allocated":162648064,"data_stored":162446790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":605351635},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,7,4,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986274,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":605424,"kb_used_data":158624,"kb_used_omap":4,"kb_used_meta":446779,"kb_avail":93766416,"statfs":{"total":96636764160,"available":96016809984,"internally_reserved":0,"allocated":162430976,"data_stored":162229702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":457502418},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,2,4,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673080,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":753848,"kb_used_data":158996,"kb_used_omap":1,"kb_used_meta":594814,"kb_avail":93617992,"statfs":{"total":96636764160,"available":95864823808,"internally_reserved":0,"allocated":162811904,"data_stored":162610630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":609089989},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,0,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182511,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":753876,"kb_used_data":158592,"kb_used_omap":4,"kb_used_meta":595259,"kb_avail":93617964,"statfs":{"total":96636764160,"available":95864795136,"internally_reserved":0,"allocated":162398208,"data_stored":162196934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":609545933},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,1,3,1,2,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":161665024,"data_stored":161665024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":161976320,"data_stored":161976320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":161742848,"data_stored":161742848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":161574912,"data_stored":161574912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:43:06.292 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:43:06.894 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:43:06.895 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:43:06.919 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":322,"stamp":"2024-09-05T16:43:05.006889+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":323383784,"num_objects":39481,"num_object_clones":0,"num_object_copies":157924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29578,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39481,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":151356,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2059,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160680,"ondisk_log_size":160680,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2863176,"kb_used_data":635048,"kb_used_omap":14,"kb_used_meta":2228017,"kb_avail":374624184,"statfs":{"total":386547056640,"available":383615164416,"internally_reserved":0,"allocated":650289152,"data_stored":649484056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2281489975},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,5,4,15,10,8,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24952832,"num_objects":-3046,"num_object_clones":0,"num_object_copies":-12184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-175,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3046,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3046,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":175,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004474"},"pg_stats":[{"pgid":"1.f","version":"37'10410","reported_seq":10514,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:04.036068+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:43:04.036068+0000","last_peered":"2024-09-05T16:43:04.036068+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:43:04.036068+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"29'400","ondisk_log_start":"29'400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20430848,"num_objects":2494,"num_object_clones":0,"num_object_copies":9976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2494,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10410,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":258}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'10163","reported_seq":13793,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:03.982646+0000","last_change":"2024-09-05T16:40:28.919189+0000","last_active":"2024-09-05T16:43:03.982646+0000","last_peered":"2024-09-05T16:43:03.982646+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:43:03.982646+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"29'100","ondisk_log_start":"29'100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19584160,"num_objects":2391,"num_object_clones":0,"num_object_copies":9564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":491,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10161,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1780,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1900}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'10303","reported_seq":10426,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:03.764481+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:43:03.764481+0000","last_peered":"2024-09-05T16:43:03.764481+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:43:03.764481+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"29'300","ondisk_log_start":"29'300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20060776,"num_objects":2450,"num_object_clones":0,"num_object_copies":9800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2450,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10302,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":218}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'10408","reported_seq":15136,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:43:03.640207+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:43:03.640207+0000","last_peered":"2024-09-05T16:43:03.640207+0000","last_clean":"2024-09-05T16:43:03.640207+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:43:03.640207+0000","last_undegraded":"2024-09-05T16:43:03.640207+0000","last_fullsized":"2024-09-05T16:43:03.640207+0000","mapping_epoch":34,"log_start":"29'400","ondisk_log_start":"29'400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20692992,"num_objects":2526,"num_object_clones":0,"num_object_copies":10104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2526,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4708,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'10121","reported_seq":10220,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:03.968411+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:43:03.968411+0000","last_peered":"2024-09-05T16:43:03.968411+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:43:03.968411+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"29'100","ondisk_log_start":"29'100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20264096,"num_objects":2474,"num_object_clones":0,"num_object_copies":9896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2474,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10119,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":329}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'10351","reported_seq":10463,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:04.045294+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:43:04.045294+0000","last_peered":"2024-09-05T16:43:04.045294+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:43:04.045294+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"29'300","ondisk_log_start":"29'300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19832832,"num_objects":2421,"num_object_clones":0,"num_object_copies":9684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2421,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10351,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":202}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'10185","reported_seq":10293,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:03.881037+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:43:03.881037+0000","last_peered":"2024-09-05T16:43:03.881037+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:43:03.881037+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"29'100","ondisk_log_start":"29'100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20288672,"num_objects":2477,"num_object_clones":0,"num_object_copies":9908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2477,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10183,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":346}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'10328","reported_seq":10417,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:03.928770+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:43:03.928770+0000","last_peered":"2024-09-05T16:43:03.928770+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:43:03.928770+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"29'300","ondisk_log_start":"29'300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20075680,"num_objects":2451,"num_object_clones":0,"num_object_copies":9804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2451,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10326,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":219}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'10338","reported_seq":10439,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:03.958373+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:43:03.958373+0000","last_peered":"2024-09-05T16:43:03.958373+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:43:03.958373+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"29'300","ondisk_log_start":"29'300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19949888,"num_objects":2436,"num_object_clones":0,"num_object_copies":9744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2436,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10335,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":192}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'10389","reported_seq":10511,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:03.632025+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:43:03.632025+0000","last_peered":"2024-09-05T16:43:03.632025+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:43:03.632025+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"29'300","ondisk_log_start":"29'300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19829920,"num_objects":2421,"num_object_clones":0,"num_object_copies":9684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2421,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10387,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":176}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'10590","reported_seq":10717,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:03.959792+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:43:03.959792+0000","last_peered":"2024-09-05T16:43:03.959792+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:43:03.959792+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"29'500","ondisk_log_start":"29'500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20745968,"num_objects":2533,"num_object_clones":0,"num_object_copies":10132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10587,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":252}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'10234","reported_seq":10333,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:04.023026+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:43:04.023026+0000","last_peered":"2024-09-05T16:43:04.023026+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:43:04.023026+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"29'200","ondisk_log_start":"29'200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20683344,"num_objects":2525,"num_object_clones":0,"num_object_copies":10100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2525,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10233,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":287}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'10354","reported_seq":15090,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:43:03.710401+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:43:03.710401+0000","last_peered":"2024-09-05T16:43:03.710401+0000","last_clean":"2024-09-05T16:43:03.710401+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:43:03.710401+0000","last_undegraded":"2024-09-05T16:43:03.710401+0000","last_fullsized":"2024-09-05T16:43:03.710401+0000","mapping_epoch":36,"log_start":"29'300","ondisk_log_start":"29'300","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20103168,"num_objects":2454,"num_object_clones":0,"num_object_copies":9816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2454,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1954,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'10607","reported_seq":10714,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:03.990940+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:43:03.990940+0000","last_peered":"2024-09-05T16:43:03.990940+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:43:03.990940+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"29'600","ondisk_log_start":"29'600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20632736,"num_objects":2519,"num_object_clones":0,"num_object_copies":10076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2519,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10605,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":185}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'10390","reported_seq":10502,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:03.787080+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:43:03.787080+0000","last_peered":"2024-09-05T16:43:03.787080+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:43:03.787080+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"29'300","ondisk_log_start":"29'300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20010144,"num_objects":2443,"num_object_clones":0,"num_object_copies":9772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2443,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10388,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":135}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'10309","reported_seq":10416,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:03.706876+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:43:03.706876+0000","last_peered":"2024-09-05T16:43:03.706876+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:43:03.706876+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"29'300","ondisk_log_start":"29'300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20198560,"num_objects":2466,"num_object_clones":0,"num_object_copies":9864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10307,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":246}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":323383784,"num_objects":39481,"num_object_clones":0,"num_object_copies":157924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29578,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39481,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":151356,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2059,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":646959104,"data_stored":646959104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160680,"ondisk_log_size":160680,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084397,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":750028,"kb_used_data":158836,"kb_used_omap":4,"kb_used_meta":591163,"kb_avail":93621812,"statfs":{"total":96636764160,"available":95868735488,"internally_reserved":0,"allocated":162648064,"data_stored":162446790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":605351635},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,7,4,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986274,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":605424,"kb_used_data":158624,"kb_used_omap":4,"kb_used_meta":446779,"kb_avail":93766416,"statfs":{"total":96636764160,"available":96016809984,"internally_reserved":0,"allocated":162430976,"data_stored":162229702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":457502418},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,2,4,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673080,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":753848,"kb_used_data":158996,"kb_used_omap":1,"kb_used_meta":594814,"kb_avail":93617992,"statfs":{"total":96636764160,"available":95864823808,"internally_reserved":0,"allocated":162811904,"data_stored":162610630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":609089989},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,0,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182511,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":753876,"kb_used_data":158592,"kb_used_omap":4,"kb_used_meta":595259,"kb_avail":93617964,"statfs":{"total":96636764160,"available":95864795136,"internally_reserved":0,"allocated":162398208,"data_stored":162196934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":609545933},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,1,3,1,2,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":161665024,"data_stored":161665024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":161976320,"data_stored":161976320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":161742848,"data_stored":161742848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":161574912,"data_stored":161574912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:43:06.921 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-05T16:43:06.921 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'10410", 'reported_seq': 10514, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:04.036068+0000', 'last_change': '2024-09-05T16:34:56.833497+0000', 'last_active': '2024-09-05T16:43:04.036068+0000', 'last_peered': '2024-09-05T16:43:04.036068+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.828289+0000', 'last_became_peered': '2024-09-05T16:34:56.828289+0000', 'last_unstale': '2024-09-05T16:43:04.036068+0000', 'last_undegraded': '2024-09-05T16:34:56.612160+0000', 'last_fullsized': '2024-09-05T16:34:56.557632+0000', 'mapping_epoch': 32, 'log_start': "29'400", 'ondisk_log_start': "29'400", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10010, 'log_dups_size': 0, 'ondisk_log_size': 10010, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20430848, 'num_objects': 2494, 'num_object_clones': 0, 'num_object_copies': 9976, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2236, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2494, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10410, 'num_write_kb': 51616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 258}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:43:06.921 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-05T16:43:06.921 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "37'10163", 'reported_seq': 13793, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:03.982646+0000', 'last_change': '2024-09-05T16:40:28.919189+0000', 'last_active': '2024-09-05T16:43:03.982646+0000', 'last_peered': '2024-09-05T16:43:03.982646+0000', 'last_clean': '2024-09-05T16:33:53.200762+0000', 'last_became_active': '2024-09-05T16:34:56.820292+0000', 'last_became_peered': '2024-09-05T16:34:56.820292+0000', 'last_unstale': '2024-09-05T16:43:03.982646+0000', 'last_undegraded': '2024-09-05T16:34:56.605776+0000', 'last_fullsized': '2024-09-05T16:34:56.559383+0000', 'mapping_epoch': 32, 'log_start': "29'100", 'ondisk_log_start': "29'100", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10063, 'log_dups_size': 0, 'ondisk_log_size': 10063, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19584160, 'num_objects': 2391, 'num_object_clones': 0, 'num_object_copies': 9564, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 491, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2391, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10161, 'num_write_kb': 50212, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1780, 'num_bytes_recovered': 2613248, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 1900}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:43:06.921 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:43:06.921 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'10303", 'reported_seq': 10426, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:03.764481+0000', 'last_change': '2024-09-05T16:34:56.867900+0000', 'last_active': '2024-09-05T16:43:03.764481+0000', 'last_peered': '2024-09-05T16:43:03.764481+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:43:03.764481+0000', 'last_undegraded': '2024-09-05T16:34:56.590972+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 32, 'log_start': "29'300", 'ondisk_log_start': "29'300", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10003, 'log_dups_size': 0, 'ondisk_log_size': 10003, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20060776, 'num_objects': 2450, 'num_object_clones': 0, 'num_object_copies': 9800, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2233, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2450, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 10302, 'num_write_kb': 50999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 218}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:43:06.921 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:43:06.921 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'10121", 'reported_seq': 10220, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:03.968411+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:43:03.968411+0000', 'last_peered': '2024-09-05T16:43:03.968411+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:43:03.968411+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "29'100", 'ondisk_log_start': "29'100", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10021, 'log_dups_size': 0, 'ondisk_log_size': 10021, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20264096, 'num_objects': 2474, 'num_object_clones': 0, 'num_object_copies': 9896, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2147, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2474, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10119, 'num_write_kb': 50373, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 329}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:43:06.921 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:43:06.922 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'10351", 'reported_seq': 10463, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:04.045294+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:43:04.045294+0000', 'last_peered': '2024-09-05T16:43:04.045294+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:43:04.045294+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "29'300", 'ondisk_log_start': "29'300", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10051, 'log_dups_size': 0, 'ondisk_log_size': 10051, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19832832, 'num_objects': 2421, 'num_object_clones': 0, 'num_object_copies': 9684, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2421, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10351, 'num_write_kb': 51088, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 202}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:43:06.922 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:43:06.922 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'10185", 'reported_seq': 10293, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:03.881037+0000', 'last_change': '2024-09-05T16:34:56.903436+0000', 'last_active': '2024-09-05T16:43:03.881037+0000', 'last_peered': '2024-09-05T16:43:03.881037+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:43:03.881037+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "29'100", 'ondisk_log_start': "29'100", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10085, 'log_dups_size': 0, 'ondisk_log_size': 10085, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20288672, 'num_objects': 2477, 'num_object_clones': 0, 'num_object_copies': 9908, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2133, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2477, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10183, 'num_write_kb': 50636, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 346}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:43:06.922 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-05T16:43:06.922 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "37'10328", 'reported_seq': 10417, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:03.928770+0000', 'last_change': '2024-09-05T16:34:56.883319+0000', 'last_active': '2024-09-05T16:43:03.928770+0000', 'last_peered': '2024-09-05T16:43:03.928770+0000', 'last_clean': '2024-09-05T16:33:53.505305+0000', 'last_became_active': '2024-09-05T16:34:56.873389+0000', 'last_became_peered': '2024-09-05T16:34:56.873389+0000', 'last_unstale': '2024-09-05T16:43:03.928770+0000', 'last_undegraded': '2024-09-05T16:34:56.602567+0000', 'last_fullsized': '2024-09-05T16:34:56.559958+0000', 'mapping_epoch': 32, 'log_start': "29'300", 'ondisk_log_start': "29'300", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20075680, 'num_objects': 2451, 'num_object_clones': 0, 'num_object_copies': 9804, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2234, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2451, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10326, 'num_write_kb': 51108, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 219}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:43:06.922 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:43:06.922 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'10338", 'reported_seq': 10439, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:03.958373+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:43:03.958373+0000', 'last_peered': '2024-09-05T16:43:03.958373+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:43:03.958373+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "29'300", 'ondisk_log_start': "29'300", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10038, 'log_dups_size': 0, 'ondisk_log_size': 10038, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19949888, 'num_objects': 2436, 'num_object_clones': 0, 'num_object_copies': 9744, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2436, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10335, 'num_write_kb': 51081, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 192}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:43:06.922 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:43:06.922 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'10389", 'reported_seq': 10511, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:03.632025+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:43:03.632025+0000', 'last_peered': '2024-09-05T16:43:03.632025+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:43:03.632025+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "29'300", 'ondisk_log_start': "29'300", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10089, 'log_dups_size': 0, 'ondisk_log_size': 10089, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19829920, 'num_objects': 2421, 'num_object_clones': 0, 'num_object_copies': 9684, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2421, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10387, 'num_write_kb': 51230, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 176}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:43:06.922 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:43:06.922 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'10590", 'reported_seq': 10717, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:03.959792+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:43:03.959792+0000', 'last_peered': '2024-09-05T16:43:03.959792+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:43:03.959792+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "29'500", 'ondisk_log_start': "29'500", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10090, 'log_dups_size': 0, 'ondisk_log_size': 10090, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20745968, 'num_objects': 2533, 'num_object_clones': 0, 'num_object_copies': 10132, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2284, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2533, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10587, 'num_write_kb': 52476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 252}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:43:06.923 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:43:06.923 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'10234", 'reported_seq': 10333, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:04.023026+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:43:04.023026+0000', 'last_peered': '2024-09-05T16:43:04.023026+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:43:04.023026+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "29'200", 'ondisk_log_start': "29'200", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10034, 'log_dups_size': 0, 'ondisk_log_size': 10034, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20683344, 'num_objects': 2525, 'num_object_clones': 0, 'num_object_copies': 10100, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2525, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10233, 'num_write_kb': 51031, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 287}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:43:06.923 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:43:06.923 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'10607", 'reported_seq': 10714, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:03.990940+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:43:03.990940+0000', 'last_peered': '2024-09-05T16:43:03.990940+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:43:03.990940+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "29'600", 'ondisk_log_start': "29'600", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10007, 'log_dups_size': 0, 'ondisk_log_size': 10007, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20632736, 'num_objects': 2519, 'num_object_clones': 0, 'num_object_copies': 10076, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2519, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10605, 'num_write_kb': 52500, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 185}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:43:06.923 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:43:06.923 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'10390", 'reported_seq': 10502, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:03.787080+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:43:03.787080+0000', 'last_peered': '2024-09-05T16:43:03.787080+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:43:03.787080+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "29'300", 'ondisk_log_start': "29'300", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10090, 'log_dups_size': 0, 'ondisk_log_size': 10090, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20010144, 'num_objects': 2443, 'num_object_clones': 0, 'num_object_copies': 9772, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2310, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2443, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10388, 'num_write_kb': 51325, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 135}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:43:06.923 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:43:06.923 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'10309", 'reported_seq': 10416, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:03.706876+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:43:03.706876+0000', 'last_peered': '2024-09-05T16:43:03.706876+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:43:03.706876+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "29'300", 'ondisk_log_start': "29'300", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10009, 'log_dups_size': 0, 'ondisk_log_size': 10009, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20198560, 'num_objects': 2466, 'num_object_clones': 0, 'num_object_copies': 9864, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2466, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10307, 'num_write_kb': 51092, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 246}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:43:06.924 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:43:07.639 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:43:07.639 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 29578/157924 objects degraded (18.729%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":590,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":39481,"data_bytes":323383784,"bytes_used":2931892224,"bytes_avail":383615164416,"bytes_total":386547056640,"degraded_objects":29578,"degraded_total":157924,"degraded_ratio":0.18729262176743244,"recovering_objects_per_sec":10,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":121,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":168},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:43:07.657 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:43:07.657 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:43:08.301 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:43:08.302 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:43:08.336 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":323,"stamp":"2024-09-05T16:43:07.008662+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":323383784,"num_objects":39481,"num_object_clones":0,"num_object_copies":157924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29578,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39481,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":151356,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2059,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160680,"ondisk_log_size":160680,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2863176,"kb_used_data":635048,"kb_used_omap":14,"kb_used_meta":2228017,"kb_avail":374624184,"statfs":{"total":386547056640,"available":383615164416,"internally_reserved":0,"allocated":650289152,"data_stored":649484056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2281489975},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,5,4,15,10,8,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16605184,"num_objects":-2027,"num_object_clones":0,"num_object_copies":-8108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-121,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2027,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2027,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":121,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005413"},"pg_stats":[{"pgid":"1.f","version":"37'10410","reported_seq":10514,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:04.036068+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:43:04.036068+0000","last_peered":"2024-09-05T16:43:04.036068+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:43:04.036068+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"29'400","ondisk_log_start":"29'400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20430848,"num_objects":2494,"num_object_clones":0,"num_object_copies":9976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2494,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10410,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":258}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'10163","reported_seq":13793,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:03.982646+0000","last_change":"2024-09-05T16:40:28.919189+0000","last_active":"2024-09-05T16:43:03.982646+0000","last_peered":"2024-09-05T16:43:03.982646+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:43:03.982646+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"29'100","ondisk_log_start":"29'100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19584160,"num_objects":2391,"num_object_clones":0,"num_object_copies":9564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":491,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10161,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1780,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1900}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'10303","reported_seq":10426,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:03.764481+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:43:03.764481+0000","last_peered":"2024-09-05T16:43:03.764481+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:43:03.764481+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"29'300","ondisk_log_start":"29'300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20060776,"num_objects":2450,"num_object_clones":0,"num_object_copies":9800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2450,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10302,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":218}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'10408","reported_seq":15136,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:43:03.640207+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:43:03.640207+0000","last_peered":"2024-09-05T16:43:03.640207+0000","last_clean":"2024-09-05T16:43:03.640207+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:43:03.640207+0000","last_undegraded":"2024-09-05T16:43:03.640207+0000","last_fullsized":"2024-09-05T16:43:03.640207+0000","mapping_epoch":34,"log_start":"29'400","ondisk_log_start":"29'400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20692992,"num_objects":2526,"num_object_clones":0,"num_object_copies":10104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2526,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4708,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'10121","reported_seq":10220,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:03.968411+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:43:03.968411+0000","last_peered":"2024-09-05T16:43:03.968411+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:43:03.968411+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"29'100","ondisk_log_start":"29'100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20264096,"num_objects":2474,"num_object_clones":0,"num_object_copies":9896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2474,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10119,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":329}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'10351","reported_seq":10463,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:04.045294+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:43:04.045294+0000","last_peered":"2024-09-05T16:43:04.045294+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:43:04.045294+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"29'300","ondisk_log_start":"29'300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19832832,"num_objects":2421,"num_object_clones":0,"num_object_copies":9684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2421,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10351,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":202}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'10185","reported_seq":10293,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:03.881037+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:43:03.881037+0000","last_peered":"2024-09-05T16:43:03.881037+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:43:03.881037+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"29'100","ondisk_log_start":"29'100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20288672,"num_objects":2477,"num_object_clones":0,"num_object_copies":9908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2477,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10183,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":346}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'10328","reported_seq":10417,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:03.928770+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:43:03.928770+0000","last_peered":"2024-09-05T16:43:03.928770+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:43:03.928770+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"29'300","ondisk_log_start":"29'300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20075680,"num_objects":2451,"num_object_clones":0,"num_object_copies":9804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2451,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10326,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":219}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'10338","reported_seq":10439,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:03.958373+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:43:03.958373+0000","last_peered":"2024-09-05T16:43:03.958373+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:43:03.958373+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"29'300","ondisk_log_start":"29'300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19949888,"num_objects":2436,"num_object_clones":0,"num_object_copies":9744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2436,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10335,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":192}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'10389","reported_seq":10511,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:03.632025+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:43:03.632025+0000","last_peered":"2024-09-05T16:43:03.632025+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:43:03.632025+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"29'300","ondisk_log_start":"29'300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19829920,"num_objects":2421,"num_object_clones":0,"num_object_copies":9684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2421,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10387,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":176}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'10590","reported_seq":10717,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:03.959792+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:43:03.959792+0000","last_peered":"2024-09-05T16:43:03.959792+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:43:03.959792+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"29'500","ondisk_log_start":"29'500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20745968,"num_objects":2533,"num_object_clones":0,"num_object_copies":10132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10587,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":252}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'10234","reported_seq":10333,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:04.023026+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:43:04.023026+0000","last_peered":"2024-09-05T16:43:04.023026+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:43:04.023026+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"29'200","ondisk_log_start":"29'200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20683344,"num_objects":2525,"num_object_clones":0,"num_object_copies":10100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2525,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10233,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":287}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'10354","reported_seq":15090,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:43:03.710401+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:43:03.710401+0000","last_peered":"2024-09-05T16:43:03.710401+0000","last_clean":"2024-09-05T16:43:03.710401+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:43:03.710401+0000","last_undegraded":"2024-09-05T16:43:03.710401+0000","last_fullsized":"2024-09-05T16:43:03.710401+0000","mapping_epoch":36,"log_start":"29'300","ondisk_log_start":"29'300","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20103168,"num_objects":2454,"num_object_clones":0,"num_object_copies":9816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2454,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1954,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'10607","reported_seq":10714,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:03.990940+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:43:03.990940+0000","last_peered":"2024-09-05T16:43:03.990940+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:43:03.990940+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"29'600","ondisk_log_start":"29'600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20632736,"num_objects":2519,"num_object_clones":0,"num_object_copies":10076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2519,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10605,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":185}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'10390","reported_seq":10502,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:03.787080+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:43:03.787080+0000","last_peered":"2024-09-05T16:43:03.787080+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:43:03.787080+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"29'300","ondisk_log_start":"29'300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20010144,"num_objects":2443,"num_object_clones":0,"num_object_copies":9772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2443,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10388,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":135}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'10309","reported_seq":10416,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:03.706876+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:43:03.706876+0000","last_peered":"2024-09-05T16:43:03.706876+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:43:03.706876+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"29'300","ondisk_log_start":"29'300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20198560,"num_objects":2466,"num_object_clones":0,"num_object_copies":9864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10307,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":246}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":323383784,"num_objects":39481,"num_object_clones":0,"num_object_copies":157924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29578,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39481,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":151356,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2059,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":646959104,"data_stored":646959104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160680,"ondisk_log_size":160680,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084397,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":750028,"kb_used_data":158836,"kb_used_omap":4,"kb_used_meta":591163,"kb_avail":93621812,"statfs":{"total":96636764160,"available":95868735488,"internally_reserved":0,"allocated":162648064,"data_stored":162446790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":605351635},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,7,4,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986274,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":605424,"kb_used_data":158624,"kb_used_omap":4,"kb_used_meta":446779,"kb_avail":93766416,"statfs":{"total":96636764160,"available":96016809984,"internally_reserved":0,"allocated":162430976,"data_stored":162229702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":457502418},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,2,4,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673080,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":753848,"kb_used_data":158996,"kb_used_omap":1,"kb_used_meta":594814,"kb_avail":93617992,"statfs":{"total":96636764160,"available":95864823808,"internally_reserved":0,"allocated":162811904,"data_stored":162610630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":609089989},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,0,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182511,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":753876,"kb_used_data":158592,"kb_used_omap":4,"kb_used_meta":595259,"kb_avail":93617964,"statfs":{"total":96636764160,"available":95864795136,"internally_reserved":0,"allocated":162398208,"data_stored":162196934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":609545933},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,1,3,1,2,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":161665024,"data_stored":161665024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":161976320,"data_stored":161976320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":161742848,"data_stored":161742848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":161574912,"data_stored":161574912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:43:11.339 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:43:11.893 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:43:11.893 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:43:11.917 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":325,"stamp":"2024-09-05T16:43:11.009640+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":315159016,"num_objects":38477,"num_object_clones":0,"num_object_copies":153908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29523,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38477,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":152360,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2114,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160984,"ondisk_log_size":160984,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2917932,"kb_used_data":619032,"kb_used_omap":14,"kb_used_meta":2298801,"kb_avail":374569428,"statfs":{"total":386547056640,"available":383559094272,"internally_reserved":0,"allocated":633888768,"data_stored":633083672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2353972791},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,1,13,17,20,17,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20283392,"num_objects":-2476,"num_object_clones":0,"num_object_copies":-9904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-120,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2476,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2476,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":120,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005024"},"pg_stats":[{"pgid":"1.f","version":"37'10469","reported_seq":10573,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:08.995805+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:43:08.995805+0000","last_peered":"2024-09-05T16:43:08.995805+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:43:08.995805+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"29'400","ondisk_log_start":"29'400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19947520,"num_objects":2435,"num_object_clones":0,"num_object_copies":9740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2435,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10469,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":199}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'10223","reported_seq":13964,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:08.951721+0000","last_change":"2024-09-05T16:40:28.919189+0000","last_active":"2024-09-05T16:43:08.951721+0000","last_peered":"2024-09-05T16:43:08.951721+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:43:08.951721+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"29'200","ondisk_log_start":"29'200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19092640,"num_objects":2331,"num_object_clones":0,"num_object_copies":9324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2331,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10221,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1835,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1895}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'10366","reported_seq":10489,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:08.899695+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:43:08.899695+0000","last_peered":"2024-09-05T16:43:08.899695+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:43:08.899695+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"29'300","ondisk_log_start":"29'300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19544680,"num_objects":2387,"num_object_clones":0,"num_object_copies":9548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2387,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10365,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":155}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'10480","reported_seq":15209,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:43:08.834245+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:43:08.834245+0000","last_peered":"2024-09-05T16:43:08.834245+0000","last_clean":"2024-09-05T16:43:08.834245+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:43:08.834245+0000","last_undegraded":"2024-09-05T16:43:08.834245+0000","last_fullsized":"2024-09-05T16:43:08.834245+0000","mapping_epoch":34,"log_start":"29'400","ondisk_log_start":"29'400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20103168,"num_objects":2454,"num_object_clones":0,"num_object_copies":9816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2454,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4780,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'10179","reported_seq":10278,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:08.680468+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:43:08.680468+0000","last_peered":"2024-09-05T16:43:08.680468+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:43:08.680468+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"29'100","ondisk_log_start":"29'100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19788960,"num_objects":2416,"num_object_clones":0,"num_object_copies":9664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2416,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10177,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":271}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'10413","reported_seq":10527,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:09.024259+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:43:09.024259+0000","last_peered":"2024-09-05T16:43:09.024259+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:43:09.024259+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"29'400","ondisk_log_start":"29'400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19324928,"num_objects":2359,"num_object_clones":0,"num_object_copies":9436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2359,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10413,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":140}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'10244","reported_seq":10352,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:08.909334+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:43:08.909334+0000","last_peered":"2024-09-05T16:43:08.909334+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:43:08.909334+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"29'200","ondisk_log_start":"29'200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19805344,"num_objects":2418,"num_object_clones":0,"num_object_copies":9672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2418,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10242,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":287}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'10383","reported_seq":10472,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:08.833123+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:43:08.833123+0000","last_peered":"2024-09-05T16:43:08.833123+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:43:08.833123+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"29'300","ondisk_log_start":"29'300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19625120,"num_objects":2396,"num_object_clones":0,"num_object_copies":9584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2396,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10381,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":164}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'10398","reported_seq":10499,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:08.704946+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:43:08.704946+0000","last_peered":"2024-09-05T16:43:08.704946+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:43:08.704946+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"29'300","ondisk_log_start":"29'300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19458368,"num_objects":2376,"num_object_clones":0,"num_object_copies":9504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2376,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10395,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":132}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'10443","reported_seq":10565,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:08.744523+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:43:08.744523+0000","last_peered":"2024-09-05T16:43:08.744523+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:43:08.744523+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"29'400","ondisk_log_start":"29'400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19387552,"num_objects":2367,"num_object_clones":0,"num_object_copies":9468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10441,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":122}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'10661","reported_seq":10788,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:09.055259+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:43:09.055259+0000","last_peered":"2024-09-05T16:43:09.055259+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:43:09.055259+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"29'600","ondisk_log_start":"29'600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20164336,"num_objects":2462,"num_object_clones":0,"num_object_copies":9848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2462,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10658,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":181}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'10300","reported_seq":10399,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:08.996756+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:43:08.996756+0000","last_peered":"2024-09-05T16:43:08.996756+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:43:08.996756+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"29'200","ondisk_log_start":"29'200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20142672,"num_objects":2459,"num_object_clones":0,"num_object_copies":9836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2459,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10299,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":221}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'10423","reported_seq":15159,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:43:08.793883+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:43:08.793883+0000","last_peered":"2024-09-05T16:43:08.793883+0000","last_clean":"2024-09-05T16:43:08.793883+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:43:08.793883+0000","last_undegraded":"2024-09-05T16:43:08.793883+0000","last_fullsized":"2024-09-05T16:43:08.793883+0000","mapping_epoch":36,"log_start":"29'400","ondisk_log_start":"29'400","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19537920,"num_objects":2385,"num_object_clones":0,"num_object_copies":9540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2385,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2023,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'10667","reported_seq":10776,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:08.963069+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:43:08.963069+0000","last_peered":"2024-09-05T16:43:08.963069+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:43:08.963069+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"29'600","ondisk_log_start":"29'600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20141216,"num_objects":2459,"num_object_clones":0,"num_object_copies":9836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2459,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10665,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":125}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'10456","reported_seq":10569,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:08.848957+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:43:08.848957+0000","last_peered":"2024-09-05T16:43:08.848957+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:43:08.848957+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"29'400","ondisk_log_start":"29'400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19469472,"num_objects":2377,"num_object_clones":0,"num_object_copies":9508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2377,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10454,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":69}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'10379","reported_seq":10484,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:09.007786+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:43:09.007786+0000","last_peered":"2024-09-05T16:43:09.007786+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:43:09.007786+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"29'300","ondisk_log_start":"29'300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19625120,"num_objects":2396,"num_object_clones":0,"num_object_copies":9584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2396,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10377,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":176}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":315159016,"num_objects":38477,"num_object_clones":0,"num_object_copies":153908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29523,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38477,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":152360,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2114,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":630411264,"data_stored":630411264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160984,"ondisk_log_size":160984,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084398,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":768256,"kb_used_data":154856,"kb_used_omap":4,"kb_used_meta":613371,"kb_avail":93603584,"statfs":{"total":96636764160,"available":95850070016,"internally_reserved":0,"allocated":158572544,"data_stored":158371270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":628092627},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,0,5,6,5,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986275,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":605552,"kb_used_data":154592,"kb_used_omap":4,"kb_used_meta":450939,"kb_avail":93766288,"statfs":{"total":96636764160,"available":96016678912,"internally_reserved":0,"allocated":158302208,"data_stored":158100934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":461762258},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,2,9,7,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673081,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":772040,"kb_used_data":155000,"kb_used_omap":1,"kb_used_meta":617022,"kb_avail":93599800,"statfs":{"total":96636764160,"available":95846195200,"internally_reserved":0,"allocated":158720000,"data_stored":158518726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":631830981},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,1,0,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182512,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":772084,"kb_used_data":154584,"kb_used_omap":4,"kb_used_meta":617467,"kb_avail":93599756,"statfs":{"total":96636764160,"available":95846150144,"internally_reserved":0,"allocated":158294016,"data_stored":158092742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":632286925},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,8,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":157556736,"data_stored":157556736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":157818880,"data_stored":157818880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":157569024,"data_stored":157569024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":157466624,"data_stored":157466624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:43:11.918 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:43:12.550 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:43:12.551 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:43:12.575 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":325,"stamp":"2024-09-05T16:43:11.009640+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":315159016,"num_objects":38477,"num_object_clones":0,"num_object_copies":153908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29523,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38477,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":152360,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2114,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160984,"ondisk_log_size":160984,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2917932,"kb_used_data":619032,"kb_used_omap":14,"kb_used_meta":2298801,"kb_avail":374569428,"statfs":{"total":386547056640,"available":383559094272,"internally_reserved":0,"allocated":633888768,"data_stored":633083672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2353972791},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,1,13,17,20,17,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20283392,"num_objects":-2476,"num_object_clones":0,"num_object_copies":-9904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-120,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2476,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2476,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":120,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005024"},"pg_stats":[{"pgid":"1.f","version":"37'10469","reported_seq":10573,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:08.995805+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:43:08.995805+0000","last_peered":"2024-09-05T16:43:08.995805+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:43:08.995805+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"29'400","ondisk_log_start":"29'400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19947520,"num_objects":2435,"num_object_clones":0,"num_object_copies":9740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2435,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10469,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":199}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'10223","reported_seq":13964,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:08.951721+0000","last_change":"2024-09-05T16:40:28.919189+0000","last_active":"2024-09-05T16:43:08.951721+0000","last_peered":"2024-09-05T16:43:08.951721+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:43:08.951721+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"29'200","ondisk_log_start":"29'200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19092640,"num_objects":2331,"num_object_clones":0,"num_object_copies":9324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2331,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10221,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1835,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1895}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'10366","reported_seq":10489,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:08.899695+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:43:08.899695+0000","last_peered":"2024-09-05T16:43:08.899695+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:43:08.899695+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"29'300","ondisk_log_start":"29'300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19544680,"num_objects":2387,"num_object_clones":0,"num_object_copies":9548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2387,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10365,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":155}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'10480","reported_seq":15209,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:43:08.834245+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:43:08.834245+0000","last_peered":"2024-09-05T16:43:08.834245+0000","last_clean":"2024-09-05T16:43:08.834245+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:43:08.834245+0000","last_undegraded":"2024-09-05T16:43:08.834245+0000","last_fullsized":"2024-09-05T16:43:08.834245+0000","mapping_epoch":34,"log_start":"29'400","ondisk_log_start":"29'400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20103168,"num_objects":2454,"num_object_clones":0,"num_object_copies":9816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2454,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4780,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'10179","reported_seq":10278,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:08.680468+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:43:08.680468+0000","last_peered":"2024-09-05T16:43:08.680468+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:43:08.680468+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"29'100","ondisk_log_start":"29'100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19788960,"num_objects":2416,"num_object_clones":0,"num_object_copies":9664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2416,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10177,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":271}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'10413","reported_seq":10527,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:09.024259+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:43:09.024259+0000","last_peered":"2024-09-05T16:43:09.024259+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:43:09.024259+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"29'400","ondisk_log_start":"29'400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19324928,"num_objects":2359,"num_object_clones":0,"num_object_copies":9436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2359,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10413,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":140}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'10244","reported_seq":10352,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:08.909334+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:43:08.909334+0000","last_peered":"2024-09-05T16:43:08.909334+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:43:08.909334+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"29'200","ondisk_log_start":"29'200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19805344,"num_objects":2418,"num_object_clones":0,"num_object_copies":9672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2418,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10242,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":287}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'10383","reported_seq":10472,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:08.833123+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:43:08.833123+0000","last_peered":"2024-09-05T16:43:08.833123+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:43:08.833123+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"29'300","ondisk_log_start":"29'300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19625120,"num_objects":2396,"num_object_clones":0,"num_object_copies":9584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2396,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10381,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":164}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'10398","reported_seq":10499,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:08.704946+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:43:08.704946+0000","last_peered":"2024-09-05T16:43:08.704946+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:43:08.704946+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"29'300","ondisk_log_start":"29'300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19458368,"num_objects":2376,"num_object_clones":0,"num_object_copies":9504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2376,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10395,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":132}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'10443","reported_seq":10565,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:08.744523+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:43:08.744523+0000","last_peered":"2024-09-05T16:43:08.744523+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:43:08.744523+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"29'400","ondisk_log_start":"29'400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19387552,"num_objects":2367,"num_object_clones":0,"num_object_copies":9468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10441,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":122}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'10661","reported_seq":10788,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:09.055259+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:43:09.055259+0000","last_peered":"2024-09-05T16:43:09.055259+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:43:09.055259+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"29'600","ondisk_log_start":"29'600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20164336,"num_objects":2462,"num_object_clones":0,"num_object_copies":9848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2462,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10658,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":181}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'10300","reported_seq":10399,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:08.996756+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:43:08.996756+0000","last_peered":"2024-09-05T16:43:08.996756+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:43:08.996756+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"29'200","ondisk_log_start":"29'200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20142672,"num_objects":2459,"num_object_clones":0,"num_object_copies":9836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2459,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10299,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":221}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'10423","reported_seq":15159,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:43:08.793883+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:43:08.793883+0000","last_peered":"2024-09-05T16:43:08.793883+0000","last_clean":"2024-09-05T16:43:08.793883+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:43:08.793883+0000","last_undegraded":"2024-09-05T16:43:08.793883+0000","last_fullsized":"2024-09-05T16:43:08.793883+0000","mapping_epoch":36,"log_start":"29'400","ondisk_log_start":"29'400","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19537920,"num_objects":2385,"num_object_clones":0,"num_object_copies":9540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2385,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2023,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'10667","reported_seq":10776,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:08.963069+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:43:08.963069+0000","last_peered":"2024-09-05T16:43:08.963069+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:43:08.963069+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"29'600","ondisk_log_start":"29'600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20141216,"num_objects":2459,"num_object_clones":0,"num_object_copies":9836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2459,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10665,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":125}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'10456","reported_seq":10569,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:08.848957+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:43:08.848957+0000","last_peered":"2024-09-05T16:43:08.848957+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:43:08.848957+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"29'400","ondisk_log_start":"29'400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19469472,"num_objects":2377,"num_object_clones":0,"num_object_copies":9508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2377,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10454,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":69}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'10379","reported_seq":10484,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:09.007786+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:43:09.007786+0000","last_peered":"2024-09-05T16:43:09.007786+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:43:09.007786+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"29'300","ondisk_log_start":"29'300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19625120,"num_objects":2396,"num_object_clones":0,"num_object_copies":9584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2396,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10377,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":176}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":315159016,"num_objects":38477,"num_object_clones":0,"num_object_copies":153908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29523,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38477,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":152360,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2114,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":630411264,"data_stored":630411264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160984,"ondisk_log_size":160984,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084398,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":768256,"kb_used_data":154856,"kb_used_omap":4,"kb_used_meta":613371,"kb_avail":93603584,"statfs":{"total":96636764160,"available":95850070016,"internally_reserved":0,"allocated":158572544,"data_stored":158371270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":628092627},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,0,5,6,5,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986275,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":605552,"kb_used_data":154592,"kb_used_omap":4,"kb_used_meta":450939,"kb_avail":93766288,"statfs":{"total":96636764160,"available":96016678912,"internally_reserved":0,"allocated":158302208,"data_stored":158100934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":461762258},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,2,9,7,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673081,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":772040,"kb_used_data":155000,"kb_used_omap":1,"kb_used_meta":617022,"kb_avail":93599800,"statfs":{"total":96636764160,"available":95846195200,"internally_reserved":0,"allocated":158720000,"data_stored":158518726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":631830981},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,1,0,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182512,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":772084,"kb_used_data":154584,"kb_used_omap":4,"kb_used_meta":617467,"kb_avail":93599756,"statfs":{"total":96636764160,"available":95846150144,"internally_reserved":0,"allocated":158294016,"data_stored":158092742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":632286925},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,8,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":157556736,"data_stored":157556736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":157818880,"data_stored":157818880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":157569024,"data_stored":157569024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":157466624,"data_stored":157466624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:43:12.577 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-05T16:43:12.577 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'10469", 'reported_seq': 10573, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:08.995805+0000', 'last_change': '2024-09-05T16:34:56.833497+0000', 'last_active': '2024-09-05T16:43:08.995805+0000', 'last_peered': '2024-09-05T16:43:08.995805+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.828289+0000', 'last_became_peered': '2024-09-05T16:34:56.828289+0000', 'last_unstale': '2024-09-05T16:43:08.995805+0000', 'last_undegraded': '2024-09-05T16:34:56.612160+0000', 'last_fullsized': '2024-09-05T16:34:56.557632+0000', 'mapping_epoch': 32, 'log_start': "29'400", 'ondisk_log_start': "29'400", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10069, 'log_dups_size': 0, 'ondisk_log_size': 10069, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19947520, 'num_objects': 2435, 'num_object_clones': 0, 'num_object_copies': 9740, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2236, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2435, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10469, 'num_write_kb': 51616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 199}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:43:12.577 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-05T16:43:12.577 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "37'10223", 'reported_seq': 13964, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:08.951721+0000', 'last_change': '2024-09-05T16:40:28.919189+0000', 'last_active': '2024-09-05T16:43:08.951721+0000', 'last_peered': '2024-09-05T16:43:08.951721+0000', 'last_clean': '2024-09-05T16:33:53.200762+0000', 'last_became_active': '2024-09-05T16:34:56.820292+0000', 'last_became_peered': '2024-09-05T16:34:56.820292+0000', 'last_unstale': '2024-09-05T16:43:08.951721+0000', 'last_undegraded': '2024-09-05T16:34:56.605776+0000', 'last_fullsized': '2024-09-05T16:34:56.559383+0000', 'mapping_epoch': 32, 'log_start': "29'200", 'ondisk_log_start': "29'200", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10023, 'log_dups_size': 0, 'ondisk_log_size': 10023, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19092640, 'num_objects': 2331, 'num_object_clones': 0, 'num_object_copies': 9324, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 436, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2331, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10221, 'num_write_kb': 50212, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1835, 'num_bytes_recovered': 2613248, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 1895}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:43:12.577 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:43:12.577 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'10366", 'reported_seq': 10489, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:08.899695+0000', 'last_change': '2024-09-05T16:34:56.867900+0000', 'last_active': '2024-09-05T16:43:08.899695+0000', 'last_peered': '2024-09-05T16:43:08.899695+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:43:08.899695+0000', 'last_undegraded': '2024-09-05T16:34:56.590972+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 32, 'log_start': "29'300", 'ondisk_log_start': "29'300", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10066, 'log_dups_size': 0, 'ondisk_log_size': 10066, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19544680, 'num_objects': 2387, 'num_object_clones': 0, 'num_object_copies': 9548, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2233, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2387, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 10365, 'num_write_kb': 50999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 155}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:43:12.577 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:43:12.577 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'10179", 'reported_seq': 10278, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:08.680468+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:43:08.680468+0000', 'last_peered': '2024-09-05T16:43:08.680468+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:43:08.680468+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "29'100", 'ondisk_log_start': "29'100", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10079, 'log_dups_size': 0, 'ondisk_log_size': 10079, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19788960, 'num_objects': 2416, 'num_object_clones': 0, 'num_object_copies': 9664, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2147, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2416, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10177, 'num_write_kb': 50373, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 271}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:43:12.578 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:43:12.578 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'10413", 'reported_seq': 10527, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:09.024259+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:43:09.024259+0000', 'last_peered': '2024-09-05T16:43:09.024259+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:43:09.024259+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "29'400", 'ondisk_log_start': "29'400", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10013, 'log_dups_size': 0, 'ondisk_log_size': 10013, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19324928, 'num_objects': 2359, 'num_object_clones': 0, 'num_object_copies': 9436, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2359, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10413, 'num_write_kb': 51088, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 140}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:43:12.578 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:43:12.578 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'10244", 'reported_seq': 10352, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:08.909334+0000', 'last_change': '2024-09-05T16:34:56.903436+0000', 'last_active': '2024-09-05T16:43:08.909334+0000', 'last_peered': '2024-09-05T16:43:08.909334+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:43:08.909334+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "29'200", 'ondisk_log_start': "29'200", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10044, 'log_dups_size': 0, 'ondisk_log_size': 10044, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19805344, 'num_objects': 2418, 'num_object_clones': 0, 'num_object_copies': 9672, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2133, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2418, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10242, 'num_write_kb': 50636, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 287}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:43:12.578 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-05T16:43:12.578 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "37'10383", 'reported_seq': 10472, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:08.833123+0000', 'last_change': '2024-09-05T16:34:56.883319+0000', 'last_active': '2024-09-05T16:43:08.833123+0000', 'last_peered': '2024-09-05T16:43:08.833123+0000', 'last_clean': '2024-09-05T16:33:53.505305+0000', 'last_became_active': '2024-09-05T16:34:56.873389+0000', 'last_became_peered': '2024-09-05T16:34:56.873389+0000', 'last_unstale': '2024-09-05T16:43:08.833123+0000', 'last_undegraded': '2024-09-05T16:34:56.602567+0000', 'last_fullsized': '2024-09-05T16:34:56.559958+0000', 'mapping_epoch': 32, 'log_start': "29'300", 'ondisk_log_start': "29'300", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10083, 'log_dups_size': 0, 'ondisk_log_size': 10083, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19625120, 'num_objects': 2396, 'num_object_clones': 0, 'num_object_copies': 9584, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2234, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2396, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10381, 'num_write_kb': 51108, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 164}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:43:12.578 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:43:12.578 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'10398", 'reported_seq': 10499, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:08.704946+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:43:08.704946+0000', 'last_peered': '2024-09-05T16:43:08.704946+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:43:08.704946+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "29'300", 'ondisk_log_start': "29'300", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19458368, 'num_objects': 2376, 'num_object_clones': 0, 'num_object_copies': 9504, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2376, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10395, 'num_write_kb': 51081, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 132}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:43:12.578 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:43:12.579 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'10443", 'reported_seq': 10565, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:08.744523+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:43:08.744523+0000', 'last_peered': '2024-09-05T16:43:08.744523+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:43:08.744523+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "29'400", 'ondisk_log_start': "29'400", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10043, 'log_dups_size': 0, 'ondisk_log_size': 10043, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19387552, 'num_objects': 2367, 'num_object_clones': 0, 'num_object_copies': 9468, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10441, 'num_write_kb': 51230, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 122}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:43:12.579 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:43:12.579 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'10661", 'reported_seq': 10788, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:09.055259+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:43:09.055259+0000', 'last_peered': '2024-09-05T16:43:09.055259+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:43:09.055259+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "29'600", 'ondisk_log_start': "29'600", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10061, 'log_dups_size': 0, 'ondisk_log_size': 10061, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20164336, 'num_objects': 2462, 'num_object_clones': 0, 'num_object_copies': 9848, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2284, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2462, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10658, 'num_write_kb': 52476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 181}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:43:12.579 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:43:12.579 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'10300", 'reported_seq': 10399, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:08.996756+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:43:08.996756+0000', 'last_peered': '2024-09-05T16:43:08.996756+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:43:08.996756+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "29'200", 'ondisk_log_start': "29'200", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10100, 'log_dups_size': 0, 'ondisk_log_size': 10100, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20142672, 'num_objects': 2459, 'num_object_clones': 0, 'num_object_copies': 9836, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2459, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10299, 'num_write_kb': 51031, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 221}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:43:12.579 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:43:12.579 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'10667", 'reported_seq': 10776, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:08.963069+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:43:08.963069+0000', 'last_peered': '2024-09-05T16:43:08.963069+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:43:08.963069+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "29'600", 'ondisk_log_start': "29'600", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10067, 'log_dups_size': 0, 'ondisk_log_size': 10067, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20141216, 'num_objects': 2459, 'num_object_clones': 0, 'num_object_copies': 9836, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2459, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10665, 'num_write_kb': 52500, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 125}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:43:12.579 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:43:12.579 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'10456", 'reported_seq': 10569, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:08.848957+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:43:08.848957+0000', 'last_peered': '2024-09-05T16:43:08.848957+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:43:08.848957+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "29'400", 'ondisk_log_start': "29'400", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10056, 'log_dups_size': 0, 'ondisk_log_size': 10056, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19469472, 'num_objects': 2377, 'num_object_clones': 0, 'num_object_copies': 9508, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2310, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2377, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10454, 'num_write_kb': 51325, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 69}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:43:12.579 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:43:12.579 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'10379", 'reported_seq': 10484, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:09.007786+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:43:09.007786+0000', 'last_peered': '2024-09-05T16:43:09.007786+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:43:09.007786+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "29'300", 'ondisk_log_start': "29'300", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10079, 'log_dups_size': 0, 'ondisk_log_size': 10079, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19625120, 'num_objects': 2396, 'num_object_clones': 0, 'num_object_copies': 9584, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2396, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10377, 'num_write_kb': 51092, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 176}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:43:12.580 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:43:13.298 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:43:13.298 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 29523/153908 objects degraded (19.182%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":596,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":38477,"data_bytes":315159016,"bytes_used":2987962368,"bytes_avail":383559094272,"bytes_total":386547056640,"degraded_objects":29523,"degraded_total":153908,"degraded_ratio":0.19182238740026508,"recovering_objects_per_sec":9,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":120,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":168},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:43:13.325 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:43:13.325 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:43:13.978 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:43:13.979 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:43:14.011 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":326,"stamp":"2024-09-05T16:43:13.010548+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":315159016,"num_objects":38477,"num_object_clones":0,"num_object_copies":153908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29523,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38477,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":152360,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2114,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160984,"ondisk_log_size":160984,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2917932,"kb_used_data":619032,"kb_used_omap":14,"kb_used_meta":2298801,"kb_avail":374569428,"statfs":{"total":386547056640,"available":383559094272,"internally_reserved":0,"allocated":633888768,"data_stored":633083672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2353972791},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,1,13,17,20,17,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16564224,"num_objects":-2022,"num_object_clones":0,"num_object_copies":-8088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-120,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2022,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2022,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":120,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005016"},"pg_stats":[{"pgid":"1.f","version":"37'10469","reported_seq":10573,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:08.995805+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:43:08.995805+0000","last_peered":"2024-09-05T16:43:08.995805+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:43:08.995805+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"29'400","ondisk_log_start":"29'400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19947520,"num_objects":2435,"num_object_clones":0,"num_object_copies":9740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2435,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10469,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":199}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'10223","reported_seq":13964,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:08.951721+0000","last_change":"2024-09-05T16:40:28.919189+0000","last_active":"2024-09-05T16:43:08.951721+0000","last_peered":"2024-09-05T16:43:08.951721+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:43:08.951721+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"29'200","ondisk_log_start":"29'200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19092640,"num_objects":2331,"num_object_clones":0,"num_object_copies":9324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2331,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10221,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1835,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1895}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'10366","reported_seq":10489,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:08.899695+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:43:08.899695+0000","last_peered":"2024-09-05T16:43:08.899695+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:43:08.899695+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"29'300","ondisk_log_start":"29'300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19544680,"num_objects":2387,"num_object_clones":0,"num_object_copies":9548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2387,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10365,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":155}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'10480","reported_seq":15209,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:43:08.834245+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:43:08.834245+0000","last_peered":"2024-09-05T16:43:08.834245+0000","last_clean":"2024-09-05T16:43:08.834245+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:43:08.834245+0000","last_undegraded":"2024-09-05T16:43:08.834245+0000","last_fullsized":"2024-09-05T16:43:08.834245+0000","mapping_epoch":34,"log_start":"29'400","ondisk_log_start":"29'400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20103168,"num_objects":2454,"num_object_clones":0,"num_object_copies":9816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2454,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4780,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'10179","reported_seq":10278,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:08.680468+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:43:08.680468+0000","last_peered":"2024-09-05T16:43:08.680468+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:43:08.680468+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"29'100","ondisk_log_start":"29'100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19788960,"num_objects":2416,"num_object_clones":0,"num_object_copies":9664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2416,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10177,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":271}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'10413","reported_seq":10527,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:09.024259+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:43:09.024259+0000","last_peered":"2024-09-05T16:43:09.024259+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:43:09.024259+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"29'400","ondisk_log_start":"29'400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19324928,"num_objects":2359,"num_object_clones":0,"num_object_copies":9436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2359,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10413,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":140}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'10244","reported_seq":10352,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:08.909334+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:43:08.909334+0000","last_peered":"2024-09-05T16:43:08.909334+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:43:08.909334+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"29'200","ondisk_log_start":"29'200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19805344,"num_objects":2418,"num_object_clones":0,"num_object_copies":9672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2418,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10242,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":287}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'10383","reported_seq":10472,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:08.833123+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:43:08.833123+0000","last_peered":"2024-09-05T16:43:08.833123+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:43:08.833123+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"29'300","ondisk_log_start":"29'300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19625120,"num_objects":2396,"num_object_clones":0,"num_object_copies":9584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2396,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10381,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":164}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'10398","reported_seq":10499,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:08.704946+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:43:08.704946+0000","last_peered":"2024-09-05T16:43:08.704946+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:43:08.704946+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"29'300","ondisk_log_start":"29'300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19458368,"num_objects":2376,"num_object_clones":0,"num_object_copies":9504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2376,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10395,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":132}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'10443","reported_seq":10565,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:08.744523+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:43:08.744523+0000","last_peered":"2024-09-05T16:43:08.744523+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:43:08.744523+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"29'400","ondisk_log_start":"29'400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19387552,"num_objects":2367,"num_object_clones":0,"num_object_copies":9468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10441,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":122}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'10661","reported_seq":10788,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:09.055259+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:43:09.055259+0000","last_peered":"2024-09-05T16:43:09.055259+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:43:09.055259+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"29'600","ondisk_log_start":"29'600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20164336,"num_objects":2462,"num_object_clones":0,"num_object_copies":9848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2462,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10658,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":181}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'10300","reported_seq":10399,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:08.996756+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:43:08.996756+0000","last_peered":"2024-09-05T16:43:08.996756+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:43:08.996756+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"29'200","ondisk_log_start":"29'200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20142672,"num_objects":2459,"num_object_clones":0,"num_object_copies":9836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2459,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10299,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":221}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'10423","reported_seq":15159,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:43:08.793883+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:43:08.793883+0000","last_peered":"2024-09-05T16:43:08.793883+0000","last_clean":"2024-09-05T16:43:08.793883+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:43:08.793883+0000","last_undegraded":"2024-09-05T16:43:08.793883+0000","last_fullsized":"2024-09-05T16:43:08.793883+0000","mapping_epoch":36,"log_start":"29'400","ondisk_log_start":"29'400","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19537920,"num_objects":2385,"num_object_clones":0,"num_object_copies":9540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2385,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2023,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'10667","reported_seq":10776,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:08.963069+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:43:08.963069+0000","last_peered":"2024-09-05T16:43:08.963069+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:43:08.963069+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"29'600","ondisk_log_start":"29'600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20141216,"num_objects":2459,"num_object_clones":0,"num_object_copies":9836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2459,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10665,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":125}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'10456","reported_seq":10569,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:08.848957+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:43:08.848957+0000","last_peered":"2024-09-05T16:43:08.848957+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:43:08.848957+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"29'400","ondisk_log_start":"29'400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19469472,"num_objects":2377,"num_object_clones":0,"num_object_copies":9508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2377,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10454,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":69}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'10379","reported_seq":10484,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:09.007786+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:43:09.007786+0000","last_peered":"2024-09-05T16:43:09.007786+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:43:09.007786+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"29'300","ondisk_log_start":"29'300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19625120,"num_objects":2396,"num_object_clones":0,"num_object_copies":9584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2396,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10377,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":176}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":315159016,"num_objects":38477,"num_object_clones":0,"num_object_copies":153908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29523,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38477,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":152360,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2114,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":630411264,"data_stored":630411264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160984,"ondisk_log_size":160984,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084398,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":768256,"kb_used_data":154856,"kb_used_omap":4,"kb_used_meta":613371,"kb_avail":93603584,"statfs":{"total":96636764160,"available":95850070016,"internally_reserved":0,"allocated":158572544,"data_stored":158371270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":628092627},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,0,5,6,5,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986275,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":605552,"kb_used_data":154592,"kb_used_omap":4,"kb_used_meta":450939,"kb_avail":93766288,"statfs":{"total":96636764160,"available":96016678912,"internally_reserved":0,"allocated":158302208,"data_stored":158100934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":461762258},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,2,9,7,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673081,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":772040,"kb_used_data":155000,"kb_used_omap":1,"kb_used_meta":617022,"kb_avail":93599800,"statfs":{"total":96636764160,"available":95846195200,"internally_reserved":0,"allocated":158720000,"data_stored":158518726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":631830981},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,1,0,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182512,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":772084,"kb_used_data":154584,"kb_used_omap":4,"kb_used_meta":617467,"kb_avail":93599756,"statfs":{"total":96636764160,"available":95846150144,"internally_reserved":0,"allocated":158294016,"data_stored":158092742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":632286925},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,8,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":157556736,"data_stored":157556736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":157818880,"data_stored":157818880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":157569024,"data_stored":157569024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":157466624,"data_stored":157466624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:43:17.014 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:43:17.658 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:43:17.659 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:43:17.679 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":328,"stamp":"2024-09-05T16:43:17.012467+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":306754024,"num_objects":37451,"num_object_clones":0,"num_object_copies":149804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29464,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37451,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":153386,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2173,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160710,"ondisk_log_size":160710,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2885020,"kb_used_data":602752,"kb_used_omap":14,"kb_used_meta":2282161,"kb_avail":374602340,"statfs":{"total":386547056640,"available":383592796160,"internally_reserved":0,"allocated":617218048,"data_stored":616412952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2336933431},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,4,16,1,21,8,3,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16629760,"num_objects":-2030,"num_object_clones":0,"num_object_copies":-8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-114,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2030,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2030,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":114,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005578"},"pg_stats":[{"pgid":"1.f","version":"37'10527","reported_seq":10630,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:14.193306+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:43:14.193306+0000","last_peered":"2024-09-05T16:43:14.193306+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:43:14.193306+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"29'500","ondisk_log_start":"29'500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19472384,"num_objects":2377,"num_object_clones":0,"num_object_copies":9508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2377,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10527,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":141}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'10290","reported_seq":14148,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:14.041434+0000","last_change":"2024-09-05T16:40:28.919189+0000","last_active":"2024-09-05T16:43:14.041434+0000","last_peered":"2024-09-05T16:43:14.041434+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:43:14.041434+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"29'200","ondisk_log_start":"29'200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18543776,"num_objects":2264,"num_object_clones":0,"num_object_copies":9056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":377,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10288,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1894,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1887}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'10432","reported_seq":10555,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:13.603006+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:43:13.603006+0000","last_peered":"2024-09-05T16:43:13.603006+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:43:13.603006+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"29'400","ondisk_log_start":"29'400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19004008,"num_objects":2321,"num_object_clones":0,"num_object_copies":9284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2321,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10431,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":89}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'10535","reported_seq":15264,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:43:13.867459+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:43:13.867459+0000","last_peered":"2024-09-05T16:43:13.867459+0000","last_clean":"2024-09-05T16:43:13.867459+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:43:13.867459+0000","last_undegraded":"2024-09-05T16:43:13.867459+0000","last_fullsized":"2024-09-05T16:43:13.867459+0000","mapping_epoch":34,"log_start":"29'500","ondisk_log_start":"29'500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19652608,"num_objects":2399,"num_object_clones":0,"num_object_copies":9596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2399,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4835,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'10247","reported_seq":10346,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:13.927725+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:43:13.927725+0000","last_peered":"2024-09-05T16:43:13.927725+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:43:13.927725+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"29'200","ondisk_log_start":"29'200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19231904,"num_objects":2348,"num_object_clones":0,"num_object_copies":9392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2348,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10245,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":203}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'10483","reported_seq":10597,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:14.155011+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:43:14.155011+0000","last_peered":"2024-09-05T16:43:14.155011+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:43:14.155011+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"29'400","ondisk_log_start":"29'400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18751488,"num_objects":2289,"num_object_clones":0,"num_object_copies":9156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2289,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10483,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":70}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'10308","reported_seq":10416,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:13.958915+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:43:13.958915+0000","last_peered":"2024-09-05T16:43:13.958915+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:43:13.958915+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"29'300","ondisk_log_start":"29'300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19281056,"num_objects":2354,"num_object_clones":0,"num_object_copies":9416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2354,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10306,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":223}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'10439","reported_seq":10528,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:13.727301+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:43:13.727301+0000","last_peered":"2024-09-05T16:43:13.727301+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:43:13.727301+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"29'400","ondisk_log_start":"29'400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19166368,"num_objects":2340,"num_object_clones":0,"num_object_copies":9360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2340,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10437,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":108}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'10460","reported_seq":10561,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:13.959664+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:43:13.959664+0000","last_peered":"2024-09-05T16:43:13.959664+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:43:13.959664+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"29'400","ondisk_log_start":"29'400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18950464,"num_objects":2314,"num_object_clones":0,"num_object_copies":9256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2314,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10457,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":70}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'10509","reported_seq":10631,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:13.803517+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:43:13.803517+0000","last_peered":"2024-09-05T16:43:13.803517+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:43:13.803517+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"29'500","ondisk_log_start":"29'500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18846880,"num_objects":2301,"num_object_clones":0,"num_object_copies":9204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2301,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10507,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":56}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'10730","reported_seq":10857,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:13.999166+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:43:13.999166+0000","last_peered":"2024-09-05T16:43:13.999166+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:43:13.999166+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"29'700","ondisk_log_start":"29'700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19599088,"num_objects":2393,"num_object_clones":0,"num_object_copies":9572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2393,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10727,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":112}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'10360","reported_seq":10458,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:14.135978+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:43:14.135978+0000","last_peered":"2024-09-05T16:43:14.135978+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:43:14.135978+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"29'300","ondisk_log_start":"29'300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19651152,"num_objects":2399,"num_object_clones":0,"num_object_copies":9596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2399,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10359,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":161}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'10488","reported_seq":15224,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:43:13.608212+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:43:13.608212+0000","last_peered":"2024-09-05T16:43:13.608212+0000","last_clean":"2024-09-05T16:43:13.608212+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:43:13.608212+0000","last_undegraded":"2024-09-05T16:43:13.608212+0000","last_fullsized":"2024-09-05T16:43:13.608212+0000","mapping_epoch":36,"log_start":"29'400","ondisk_log_start":"29'400","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19005440,"num_objects":2320,"num_object_clones":0,"num_object_copies":9280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2320,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2088,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'10732","reported_seq":10841,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:14.194538+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:43:14.194538+0000","last_peered":"2024-09-05T16:43:14.194538+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:43:14.194538+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"29'700","ondisk_log_start":"29'700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19608736,"num_objects":2394,"num_object_clones":0,"num_object_copies":9576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2394,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10730,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":60}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'10525","reported_seq":10638,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:13.932161+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:43:13.932161+0000","last_peered":"2024-09-05T16:43:13.932161+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:43:13.932161+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"29'500","ondisk_log_start":"29'500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18904224,"num_objects":2308,"num_object_clones":0,"num_object_copies":9232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2308,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10523,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2308}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'10445","reported_seq":10554,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:14.006333+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:43:14.006333+0000","last_peered":"2024-09-05T16:43:14.006333+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:43:14.006333+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"29'400","ondisk_log_start":"29'400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19084448,"num_objects":2330,"num_object_clones":0,"num_object_copies":9320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2330,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10443,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":110}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":306754024,"num_objects":37451,"num_object_clones":0,"num_object_copies":149804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29464,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37451,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":153386,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2173,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":613654528,"data_stored":613654528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160710,"ondisk_log_size":160710,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084399,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":768272,"kb_used_data":150720,"kb_used_omap":4,"kb_used_meta":617531,"kb_avail":93603568,"statfs":{"total":96636764160,"available":95850053632,"internally_reserved":0,"allocated":154337280,"data_stored":154136006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":632352467},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,4,5,1,10,3,3,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986276,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":605708,"kb_used_data":150568,"kb_used_omap":4,"kb_used_meta":455099,"kb_avail":93766132,"statfs":{"total":96636764160,"available":96016519168,"internally_reserved":0,"allocated":154181632,"data_stored":153980358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":466022098},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,1,0,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673082,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":755444,"kb_used_data":150876,"kb_used_omap":1,"kb_used_meta":604542,"kb_avail":93616396,"statfs":{"total":96636764160,"available":95863189504,"internally_reserved":0,"allocated":154497024,"data_stored":154295750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":619051461},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,0,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182513,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":755596,"kb_used_data":150588,"kb_used_omap":4,"kb_used_meta":604987,"kb_avail":93616244,"statfs":{"total":96636764160,"available":95863033856,"internally_reserved":0,"allocated":154202112,"data_stored":154000838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":619507405},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,0,3,3,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":153341952,"data_stored":153341952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":153653248,"data_stored":153653248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":153407488,"data_stored":153407488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":153251840,"data_stored":153251840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:43:17.681 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:43:18.325 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:43:18.326 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:43:18.356 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":328,"stamp":"2024-09-05T16:43:17.012467+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":306754024,"num_objects":37451,"num_object_clones":0,"num_object_copies":149804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29464,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37451,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":153386,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2173,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160710,"ondisk_log_size":160710,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2885020,"kb_used_data":602752,"kb_used_omap":14,"kb_used_meta":2282161,"kb_avail":374602340,"statfs":{"total":386547056640,"available":383592796160,"internally_reserved":0,"allocated":617218048,"data_stored":616412952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2336933431},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,4,16,1,21,8,3,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16629760,"num_objects":-2030,"num_object_clones":0,"num_object_copies":-8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-114,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2030,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2030,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":114,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005578"},"pg_stats":[{"pgid":"1.f","version":"37'10527","reported_seq":10630,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:14.193306+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:43:14.193306+0000","last_peered":"2024-09-05T16:43:14.193306+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:43:14.193306+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"29'500","ondisk_log_start":"29'500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19472384,"num_objects":2377,"num_object_clones":0,"num_object_copies":9508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2377,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10527,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":141}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'10290","reported_seq":14148,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:14.041434+0000","last_change":"2024-09-05T16:40:28.919189+0000","last_active":"2024-09-05T16:43:14.041434+0000","last_peered":"2024-09-05T16:43:14.041434+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:43:14.041434+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"29'200","ondisk_log_start":"29'200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18543776,"num_objects":2264,"num_object_clones":0,"num_object_copies":9056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":377,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10288,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1894,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1887}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'10432","reported_seq":10555,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:13.603006+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:43:13.603006+0000","last_peered":"2024-09-05T16:43:13.603006+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:43:13.603006+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"29'400","ondisk_log_start":"29'400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19004008,"num_objects":2321,"num_object_clones":0,"num_object_copies":9284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2321,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10431,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":89}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'10535","reported_seq":15264,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:43:13.867459+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:43:13.867459+0000","last_peered":"2024-09-05T16:43:13.867459+0000","last_clean":"2024-09-05T16:43:13.867459+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:43:13.867459+0000","last_undegraded":"2024-09-05T16:43:13.867459+0000","last_fullsized":"2024-09-05T16:43:13.867459+0000","mapping_epoch":34,"log_start":"29'500","ondisk_log_start":"29'500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19652608,"num_objects":2399,"num_object_clones":0,"num_object_copies":9596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2399,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4835,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'10247","reported_seq":10346,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:13.927725+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:43:13.927725+0000","last_peered":"2024-09-05T16:43:13.927725+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:43:13.927725+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"29'200","ondisk_log_start":"29'200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19231904,"num_objects":2348,"num_object_clones":0,"num_object_copies":9392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2348,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10245,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":203}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'10483","reported_seq":10597,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:14.155011+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:43:14.155011+0000","last_peered":"2024-09-05T16:43:14.155011+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:43:14.155011+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"29'400","ondisk_log_start":"29'400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18751488,"num_objects":2289,"num_object_clones":0,"num_object_copies":9156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2289,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10483,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":70}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'10308","reported_seq":10416,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:13.958915+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:43:13.958915+0000","last_peered":"2024-09-05T16:43:13.958915+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:43:13.958915+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"29'300","ondisk_log_start":"29'300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19281056,"num_objects":2354,"num_object_clones":0,"num_object_copies":9416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2354,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10306,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":223}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'10439","reported_seq":10528,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:13.727301+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:43:13.727301+0000","last_peered":"2024-09-05T16:43:13.727301+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:43:13.727301+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"29'400","ondisk_log_start":"29'400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19166368,"num_objects":2340,"num_object_clones":0,"num_object_copies":9360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2340,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10437,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":108}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'10460","reported_seq":10561,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:13.959664+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:43:13.959664+0000","last_peered":"2024-09-05T16:43:13.959664+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:43:13.959664+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"29'400","ondisk_log_start":"29'400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18950464,"num_objects":2314,"num_object_clones":0,"num_object_copies":9256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2314,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10457,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":70}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'10509","reported_seq":10631,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:13.803517+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:43:13.803517+0000","last_peered":"2024-09-05T16:43:13.803517+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:43:13.803517+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"29'500","ondisk_log_start":"29'500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18846880,"num_objects":2301,"num_object_clones":0,"num_object_copies":9204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2301,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10507,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":56}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'10730","reported_seq":10857,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:13.999166+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:43:13.999166+0000","last_peered":"2024-09-05T16:43:13.999166+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:43:13.999166+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"29'700","ondisk_log_start":"29'700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19599088,"num_objects":2393,"num_object_clones":0,"num_object_copies":9572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2393,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10727,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":112}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'10360","reported_seq":10458,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:14.135978+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:43:14.135978+0000","last_peered":"2024-09-05T16:43:14.135978+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:43:14.135978+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"29'300","ondisk_log_start":"29'300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19651152,"num_objects":2399,"num_object_clones":0,"num_object_copies":9596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2399,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10359,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":161}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'10488","reported_seq":15224,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:43:13.608212+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:43:13.608212+0000","last_peered":"2024-09-05T16:43:13.608212+0000","last_clean":"2024-09-05T16:43:13.608212+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:43:13.608212+0000","last_undegraded":"2024-09-05T16:43:13.608212+0000","last_fullsized":"2024-09-05T16:43:13.608212+0000","mapping_epoch":36,"log_start":"29'400","ondisk_log_start":"29'400","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19005440,"num_objects":2320,"num_object_clones":0,"num_object_copies":9280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2320,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2088,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'10732","reported_seq":10841,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:14.194538+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:43:14.194538+0000","last_peered":"2024-09-05T16:43:14.194538+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:43:14.194538+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"29'700","ondisk_log_start":"29'700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19608736,"num_objects":2394,"num_object_clones":0,"num_object_copies":9576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2394,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10730,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":60}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'10525","reported_seq":10638,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:13.932161+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:43:13.932161+0000","last_peered":"2024-09-05T16:43:13.932161+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:43:13.932161+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"29'500","ondisk_log_start":"29'500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18904224,"num_objects":2308,"num_object_clones":0,"num_object_copies":9232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2308,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10523,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2308}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'10445","reported_seq":10554,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:14.006333+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:43:14.006333+0000","last_peered":"2024-09-05T16:43:14.006333+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:43:14.006333+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"29'400","ondisk_log_start":"29'400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19084448,"num_objects":2330,"num_object_clones":0,"num_object_copies":9320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2330,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10443,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":110}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":306754024,"num_objects":37451,"num_object_clones":0,"num_object_copies":149804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29464,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37451,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":153386,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2173,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":613654528,"data_stored":613654528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160710,"ondisk_log_size":160710,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084399,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":768272,"kb_used_data":150720,"kb_used_omap":4,"kb_used_meta":617531,"kb_avail":93603568,"statfs":{"total":96636764160,"available":95850053632,"internally_reserved":0,"allocated":154337280,"data_stored":154136006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":632352467},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,4,5,1,10,3,3,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986276,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":605708,"kb_used_data":150568,"kb_used_omap":4,"kb_used_meta":455099,"kb_avail":93766132,"statfs":{"total":96636764160,"available":96016519168,"internally_reserved":0,"allocated":154181632,"data_stored":153980358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":466022098},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,1,0,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673082,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":755444,"kb_used_data":150876,"kb_used_omap":1,"kb_used_meta":604542,"kb_avail":93616396,"statfs":{"total":96636764160,"available":95863189504,"internally_reserved":0,"allocated":154497024,"data_stored":154295750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":619051461},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,0,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182513,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":755596,"kb_used_data":150588,"kb_used_omap":4,"kb_used_meta":604987,"kb_avail":93616244,"statfs":{"total":96636764160,"available":95863033856,"internally_reserved":0,"allocated":154202112,"data_stored":154000838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":619507405},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,0,3,3,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":153341952,"data_stored":153341952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":153653248,"data_stored":153653248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":153407488,"data_stored":153407488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":153251840,"data_stored":153251840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:43:18.357 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-05T16:43:18.357 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'10527", 'reported_seq': 10630, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:14.193306+0000', 'last_change': '2024-09-05T16:34:56.833497+0000', 'last_active': '2024-09-05T16:43:14.193306+0000', 'last_peered': '2024-09-05T16:43:14.193306+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.828289+0000', 'last_became_peered': '2024-09-05T16:34:56.828289+0000', 'last_unstale': '2024-09-05T16:43:14.193306+0000', 'last_undegraded': '2024-09-05T16:34:56.612160+0000', 'last_fullsized': '2024-09-05T16:34:56.557632+0000', 'mapping_epoch': 32, 'log_start': "29'500", 'ondisk_log_start': "29'500", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10027, 'log_dups_size': 0, 'ondisk_log_size': 10027, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19472384, 'num_objects': 2377, 'num_object_clones': 0, 'num_object_copies': 9508, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2236, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2377, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10527, 'num_write_kb': 51616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 141}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:43:18.357 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-05T16:43:18.357 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "37'10290", 'reported_seq': 14148, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:14.041434+0000', 'last_change': '2024-09-05T16:40:28.919189+0000', 'last_active': '2024-09-05T16:43:14.041434+0000', 'last_peered': '2024-09-05T16:43:14.041434+0000', 'last_clean': '2024-09-05T16:33:53.200762+0000', 'last_became_active': '2024-09-05T16:34:56.820292+0000', 'last_became_peered': '2024-09-05T16:34:56.820292+0000', 'last_unstale': '2024-09-05T16:43:14.041434+0000', 'last_undegraded': '2024-09-05T16:34:56.605776+0000', 'last_fullsized': '2024-09-05T16:34:56.559383+0000', 'mapping_epoch': 32, 'log_start': "29'200", 'ondisk_log_start': "29'200", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10090, 'log_dups_size': 0, 'ondisk_log_size': 10090, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18543776, 'num_objects': 2264, 'num_object_clones': 0, 'num_object_copies': 9056, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 377, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2264, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10288, 'num_write_kb': 50212, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1894, 'num_bytes_recovered': 2613248, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 1887}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:43:18.357 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:43:18.357 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'10432", 'reported_seq': 10555, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:13.603006+0000', 'last_change': '2024-09-05T16:34:56.867900+0000', 'last_active': '2024-09-05T16:43:13.603006+0000', 'last_peered': '2024-09-05T16:43:13.603006+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:43:13.603006+0000', 'last_undegraded': '2024-09-05T16:34:56.590972+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 32, 'log_start': "29'400", 'ondisk_log_start': "29'400", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10032, 'log_dups_size': 0, 'ondisk_log_size': 10032, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19004008, 'num_objects': 2321, 'num_object_clones': 0, 'num_object_copies': 9284, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2233, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2321, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 10431, 'num_write_kb': 50999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 89}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:43:18.358 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:43:18.358 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'10247", 'reported_seq': 10346, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:13.927725+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:43:13.927725+0000', 'last_peered': '2024-09-05T16:43:13.927725+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:43:13.927725+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "29'200", 'ondisk_log_start': "29'200", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10047, 'log_dups_size': 0, 'ondisk_log_size': 10047, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19231904, 'num_objects': 2348, 'num_object_clones': 0, 'num_object_copies': 9392, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2147, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2348, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10245, 'num_write_kb': 50373, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 203}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:43:18.358 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:43:18.358 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'10483", 'reported_seq': 10597, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:14.155011+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:43:14.155011+0000', 'last_peered': '2024-09-05T16:43:14.155011+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:43:14.155011+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "29'400", 'ondisk_log_start': "29'400", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10083, 'log_dups_size': 0, 'ondisk_log_size': 10083, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18751488, 'num_objects': 2289, 'num_object_clones': 0, 'num_object_copies': 9156, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2289, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10483, 'num_write_kb': 51088, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 70}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:43:18.358 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:43:18.358 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'10308", 'reported_seq': 10416, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:13.958915+0000', 'last_change': '2024-09-05T16:34:56.903436+0000', 'last_active': '2024-09-05T16:43:13.958915+0000', 'last_peered': '2024-09-05T16:43:13.958915+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:43:13.958915+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "29'300", 'ondisk_log_start': "29'300", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10008, 'log_dups_size': 0, 'ondisk_log_size': 10008, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19281056, 'num_objects': 2354, 'num_object_clones': 0, 'num_object_copies': 9416, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2133, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2354, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10306, 'num_write_kb': 50636, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 223}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:43:18.358 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-05T16:43:18.358 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "37'10439", 'reported_seq': 10528, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:13.727301+0000', 'last_change': '2024-09-05T16:34:56.883319+0000', 'last_active': '2024-09-05T16:43:13.727301+0000', 'last_peered': '2024-09-05T16:43:13.727301+0000', 'last_clean': '2024-09-05T16:33:53.505305+0000', 'last_became_active': '2024-09-05T16:34:56.873389+0000', 'last_became_peered': '2024-09-05T16:34:56.873389+0000', 'last_unstale': '2024-09-05T16:43:13.727301+0000', 'last_undegraded': '2024-09-05T16:34:56.602567+0000', 'last_fullsized': '2024-09-05T16:34:56.559958+0000', 'mapping_epoch': 32, 'log_start': "29'400", 'ondisk_log_start': "29'400", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10039, 'log_dups_size': 0, 'ondisk_log_size': 10039, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19166368, 'num_objects': 2340, 'num_object_clones': 0, 'num_object_copies': 9360, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2234, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2340, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10437, 'num_write_kb': 51108, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 108}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:43:18.358 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:43:18.358 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'10460", 'reported_seq': 10561, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:13.959664+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:43:13.959664+0000', 'last_peered': '2024-09-05T16:43:13.959664+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:43:13.959664+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "29'400", 'ondisk_log_start': "29'400", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10060, 'log_dups_size': 0, 'ondisk_log_size': 10060, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18950464, 'num_objects': 2314, 'num_object_clones': 0, 'num_object_copies': 9256, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2314, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10457, 'num_write_kb': 51081, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 70}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:43:18.359 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:43:18.359 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'10509", 'reported_seq': 10631, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:13.803517+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:43:13.803517+0000', 'last_peered': '2024-09-05T16:43:13.803517+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:43:13.803517+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "29'500", 'ondisk_log_start': "29'500", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10009, 'log_dups_size': 0, 'ondisk_log_size': 10009, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18846880, 'num_objects': 2301, 'num_object_clones': 0, 'num_object_copies': 9204, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2301, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10507, 'num_write_kb': 51230, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 56}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:43:18.359 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:43:18.359 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'10730", 'reported_seq': 10857, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:13.999166+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:43:13.999166+0000', 'last_peered': '2024-09-05T16:43:13.999166+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:43:13.999166+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "29'700", 'ondisk_log_start': "29'700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19599088, 'num_objects': 2393, 'num_object_clones': 0, 'num_object_copies': 9572, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2284, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2393, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10727, 'num_write_kb': 52476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 112}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:43:18.359 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:43:18.359 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'10360", 'reported_seq': 10458, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:14.135978+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:43:14.135978+0000', 'last_peered': '2024-09-05T16:43:14.135978+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:43:14.135978+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "29'300", 'ondisk_log_start': "29'300", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10060, 'log_dups_size': 0, 'ondisk_log_size': 10060, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19651152, 'num_objects': 2399, 'num_object_clones': 0, 'num_object_copies': 9596, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2399, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10359, 'num_write_kb': 51031, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 161}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:43:18.359 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:43:18.359 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'10732", 'reported_seq': 10841, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:14.194538+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:43:14.194538+0000', 'last_peered': '2024-09-05T16:43:14.194538+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:43:14.194538+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "29'700", 'ondisk_log_start': "29'700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10032, 'log_dups_size': 0, 'ondisk_log_size': 10032, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19608736, 'num_objects': 2394, 'num_object_clones': 0, 'num_object_copies': 9576, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2394, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10730, 'num_write_kb': 52500, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 60}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:43:18.359 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:43:18.359 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'10525", 'reported_seq': 10638, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:13.932161+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:43:13.932161+0000', 'last_peered': '2024-09-05T16:43:13.932161+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:43:13.932161+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "29'500", 'ondisk_log_start': "29'500", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10025, 'log_dups_size': 0, 'ondisk_log_size': 10025, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18904224, 'num_objects': 2308, 'num_object_clones': 0, 'num_object_copies': 9232, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2310, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2308, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10523, 'num_write_kb': 51325, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2308}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:43:18.359 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:43:18.360 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'10445", 'reported_seq': 10554, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:14.006333+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:43:14.006333+0000', 'last_peered': '2024-09-05T16:43:14.006333+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:43:14.006333+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "29'400", 'ondisk_log_start': "29'400", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10045, 'log_dups_size': 0, 'ondisk_log_size': 10045, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19084448, 'num_objects': 2330, 'num_object_clones': 0, 'num_object_copies': 9320, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2330, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10443, 'num_write_kb': 51092, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 110}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:43:18.360 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:43:19.097 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:43:19.097 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 29464/149804 objects degraded (19.668%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":602,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":37451,"data_bytes":306754024,"bytes_used":2954260480,"bytes_avail":383592796160,"bytes_total":386547056640,"degraded_objects":29464,"degraded_total":149804,"degraded_ratio":0.19668366665776615,"recovering_objects_per_sec":9,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":114,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":169},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:43:19.132 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:43:19.132 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:43:19.696 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:43:19.697 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:43:19.730 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":329,"stamp":"2024-09-05T16:43:19.013304+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":305648104,"num_objects":37316,"num_object_clones":0,"num_object_copies":149264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29464,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37316,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":153521,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2173,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160645,"ondisk_log_size":160645,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2868480,"kb_used_data":598704,"kb_used_omap":14,"kb_used_meta":2269681,"kb_avail":374618880,"statfs":{"total":386547056640,"available":383609733120,"internally_reserved":0,"allocated":613072896,"data_stored":612267800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2324153911},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,4,15,2,20,8,4,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17735680,"num_objects":-2165,"num_object_clones":0,"num_object_copies":-8660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-114,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2165,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2165,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":114,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004641"},"pg_stats":[{"pgid":"1.f","version":"37'10527","reported_seq":10630,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:14.193306+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:43:14.193306+0000","last_peered":"2024-09-05T16:43:14.193306+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:43:14.193306+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"29'500","ondisk_log_start":"29'500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19472384,"num_objects":2377,"num_object_clones":0,"num_object_copies":9508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2377,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10527,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":141}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'10290","reported_seq":14148,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:14.041434+0000","last_change":"2024-09-05T16:40:28.919189+0000","last_active":"2024-09-05T16:43:14.041434+0000","last_peered":"2024-09-05T16:43:14.041434+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:43:14.041434+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"29'200","ondisk_log_start":"29'200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18543776,"num_objects":2264,"num_object_clones":0,"num_object_copies":9056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":377,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10288,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1894,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1887}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'10432","reported_seq":10555,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:13.603006+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:43:13.603006+0000","last_peered":"2024-09-05T16:43:13.603006+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:43:13.603006+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"29'400","ondisk_log_start":"29'400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19004008,"num_objects":2321,"num_object_clones":0,"num_object_copies":9284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2321,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10431,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":89}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'10601","reported_seq":15328,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:43:18.715081+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:43:18.715081+0000","last_peered":"2024-09-05T16:43:18.715081+0000","last_clean":"2024-09-05T16:43:18.715081+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:43:18.715081+0000","last_undegraded":"2024-09-05T16:43:18.715081+0000","last_fullsized":"2024-09-05T16:43:18.715081+0000","mapping_epoch":34,"log_start":"29'600","ondisk_log_start":"29'600","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19111936,"num_objects":2333,"num_object_clones":0,"num_object_copies":9332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2333,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4901,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'10247","reported_seq":10346,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:13.927725+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:43:13.927725+0000","last_peered":"2024-09-05T16:43:13.927725+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:43:13.927725+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"29'200","ondisk_log_start":"29'200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19231904,"num_objects":2348,"num_object_clones":0,"num_object_copies":9392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2348,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10245,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":203}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'10483","reported_seq":10597,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:14.155011+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:43:14.155011+0000","last_peered":"2024-09-05T16:43:14.155011+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:43:14.155011+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"29'400","ondisk_log_start":"29'400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18751488,"num_objects":2289,"num_object_clones":0,"num_object_copies":9156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2289,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10483,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":70}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'10308","reported_seq":10416,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:13.958915+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:43:13.958915+0000","last_peered":"2024-09-05T16:43:13.958915+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:43:13.958915+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"29'300","ondisk_log_start":"29'300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19281056,"num_objects":2354,"num_object_clones":0,"num_object_copies":9416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2354,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10306,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":223}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'10439","reported_seq":10528,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:13.727301+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:43:13.727301+0000","last_peered":"2024-09-05T16:43:13.727301+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:43:13.727301+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"29'400","ondisk_log_start":"29'400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19166368,"num_objects":2340,"num_object_clones":0,"num_object_copies":9360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2340,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10437,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":108}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'10460","reported_seq":10561,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:13.959664+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:43:13.959664+0000","last_peered":"2024-09-05T16:43:13.959664+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:43:13.959664+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"29'400","ondisk_log_start":"29'400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18950464,"num_objects":2314,"num_object_clones":0,"num_object_copies":9256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2314,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10457,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":70}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'10509","reported_seq":10631,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:13.803517+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:43:13.803517+0000","last_peered":"2024-09-05T16:43:13.803517+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:43:13.803517+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"29'500","ondisk_log_start":"29'500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18846880,"num_objects":2301,"num_object_clones":0,"num_object_copies":9204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2301,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10507,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":56}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'10730","reported_seq":10857,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:13.999166+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:43:13.999166+0000","last_peered":"2024-09-05T16:43:13.999166+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:43:13.999166+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"29'700","ondisk_log_start":"29'700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19599088,"num_objects":2393,"num_object_clones":0,"num_object_copies":9572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2393,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10727,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":112}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'10360","reported_seq":10458,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:14.135978+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:43:14.135978+0000","last_peered":"2024-09-05T16:43:14.135978+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:43:14.135978+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"29'300","ondisk_log_start":"29'300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19651152,"num_objects":2399,"num_object_clones":0,"num_object_copies":9596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2399,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10359,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":161}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'10557","reported_seq":15291,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:43:18.816933+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:43:18.816933+0000","last_peered":"2024-09-05T16:43:18.816933+0000","last_clean":"2024-09-05T16:43:18.816933+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:43:18.816933+0000","last_undegraded":"2024-09-05T16:43:18.816933+0000","last_fullsized":"2024-09-05T16:43:18.816933+0000","mapping_epoch":36,"log_start":"29'500","ondisk_log_start":"29'500","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18440192,"num_objects":2251,"num_object_clones":0,"num_object_copies":9004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2251,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2157,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'10732","reported_seq":10841,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:14.194538+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:43:14.194538+0000","last_peered":"2024-09-05T16:43:14.194538+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:43:14.194538+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"29'700","ondisk_log_start":"29'700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19608736,"num_objects":2394,"num_object_clones":0,"num_object_copies":9576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2394,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10730,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":60}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'10525","reported_seq":10638,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:13.932161+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:43:13.932161+0000","last_peered":"2024-09-05T16:43:13.932161+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:43:13.932161+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"29'500","ondisk_log_start":"29'500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18904224,"num_objects":2308,"num_object_clones":0,"num_object_copies":9232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2308,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10523,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2308}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'10445","reported_seq":10554,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:14.006333+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:43:14.006333+0000","last_peered":"2024-09-05T16:43:14.006333+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:43:14.006333+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"29'400","ondisk_log_start":"29'400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19084448,"num_objects":2330,"num_object_clones":0,"num_object_copies":9320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2330,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10443,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":110}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":305648104,"num_objects":37316,"num_object_clones":0,"num_object_copies":149264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29464,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37316,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":153521,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2173,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":609464320,"data_stored":609464320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160645,"ondisk_log_size":160645,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084399,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":768272,"kb_used_data":150720,"kb_used_omap":4,"kb_used_meta":617531,"kb_avail":93603568,"statfs":{"total":96636764160,"available":95850053632,"internally_reserved":0,"allocated":154337280,"data_stored":154136006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":632352467},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,4,5,1,10,3,3,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986277,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":589168,"kb_used_data":146520,"kb_used_omap":4,"kb_used_meta":442619,"kb_avail":93782672,"statfs":{"total":96636764160,"available":96033456128,"internally_reserved":0,"allocated":150036480,"data_stored":149835206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":453242578},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,3,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673082,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":755444,"kb_used_data":150876,"kb_used_omap":1,"kb_used_meta":604542,"kb_avail":93616396,"statfs":{"total":96636764160,"available":95863189504,"internally_reserved":0,"allocated":154497024,"data_stored":154295750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":619051461},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,0,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182513,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":755596,"kb_used_data":150588,"kb_used_omap":4,"kb_used_meta":604987,"kb_avail":93616244,"statfs":{"total":96636764160,"available":95863033856,"internally_reserved":0,"allocated":154202112,"data_stored":154000838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":619507405},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,0,3,3,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":153341952,"data_stored":153341952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":149463040,"data_stored":149463040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":153407488,"data_stored":153407488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":153251840,"data_stored":153251840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:43:22.732 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:43:23.339 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:43:23.339 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:43:23.374 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":331,"stamp":"2024-09-05T16:43:23.015067+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":298267112,"num_objects":36415,"num_object_clones":0,"num_object_copies":145660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29399,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36415,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":154422,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2238,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160846,"ondisk_log_size":160846,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2850616,"kb_used_data":584984,"kb_used_omap":14,"kb_used_meta":2265521,"kb_avail":374636744,"statfs":{"total":386547056640,"available":383628025856,"internally_reserved":0,"allocated":599023616,"data_stored":598218520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2319894071},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,15,14,7,18],"upper_bound":256},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16891904,"num_objects":-2062,"num_object_clones":0,"num_object_copies":-8248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-124,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2062,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2062,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":124,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005427"},"pg_stats":[{"pgid":"1.f","version":"37'10597","reported_seq":10699,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:19.311352+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:43:19.311352+0000","last_peered":"2024-09-05T16:43:19.311352+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:43:19.311352+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"29'500","ondisk_log_start":"29'500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18898944,"num_objects":2307,"num_object_clones":0,"num_object_copies":9228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2307,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10597,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":71}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'10357","reported_seq":14345,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:19.240432+0000","last_change":"2024-09-05T16:40:28.919189+0000","last_active":"2024-09-05T16:43:19.240432+0000","last_peered":"2024-09-05T16:43:19.240432+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:43:19.240432+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"29'300","ondisk_log_start":"29'300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17994912,"num_objects":2197,"num_object_clones":0,"num_object_copies":8788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2197,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10355,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1959,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1885}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'10497","reported_seq":10620,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:18.867165+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:43:18.867165+0000","last_peered":"2024-09-05T16:43:18.867165+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:43:18.867165+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"29'400","ondisk_log_start":"29'400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18471528,"num_objects":2256,"num_object_clones":0,"num_object_copies":9024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2256,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10496,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":24}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'10601","reported_seq":15328,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:43:18.715081+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:43:18.715081+0000","last_peered":"2024-09-05T16:43:18.715081+0000","last_clean":"2024-09-05T16:43:18.715081+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:43:18.715081+0000","last_undegraded":"2024-09-05T16:43:18.715081+0000","last_fullsized":"2024-09-05T16:43:18.715081+0000","mapping_epoch":34,"log_start":"29'600","ondisk_log_start":"29'600","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19111936,"num_objects":2333,"num_object_clones":0,"num_object_copies":9332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2333,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4901,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'10317","reported_seq":10416,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:19.105543+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:43:19.105543+0000","last_peered":"2024-09-05T16:43:19.105543+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:43:19.105543+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"29'300","ondisk_log_start":"29'300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18658464,"num_objects":2278,"num_object_clones":0,"num_object_copies":9112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2278,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10315,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":133}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'10539","reported_seq":10654,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:19.285487+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:43:19.285487+0000","last_peered":"2024-09-05T16:43:19.285487+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:43:19.285487+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"29'500","ondisk_log_start":"29'500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18292736,"num_objects":2233,"num_object_clones":0,"num_object_copies":8932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2233,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10539,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":14}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'10372","reported_seq":10480,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:18.950622+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:43:18.950622+0000","last_peered":"2024-09-05T16:43:18.950622+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:43:18.950622+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"29'300","ondisk_log_start":"29'300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18756768,"num_objects":2290,"num_object_clones":0,"num_object_copies":9160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2290,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10370,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":159}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'10492","reported_seq":10581,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:19.160569+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:43:19.160569+0000","last_peered":"2024-09-05T16:43:19.160569+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:43:19.160569+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"29'400","ondisk_log_start":"29'400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18732192,"num_objects":2287,"num_object_clones":0,"num_object_copies":9148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2287,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10490,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":55}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'10528","reported_seq":10627,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:19.287963+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:43:19.287963+0000","last_peered":"2024-09-05T16:43:19.287963+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:43:19.287963+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"29'500","ondisk_log_start":"29'500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18393408,"num_objects":2246,"num_object_clones":0,"num_object_copies":8984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2246,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10525,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'10569","reported_seq":10690,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:18.925919+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:43:18.925919+0000","last_peered":"2024-09-05T16:43:18.925919+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:43:18.925919+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"29'500","ondisk_log_start":"29'500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18355360,"num_objects":2241,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10567,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-4}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'10798","reported_seq":10923,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:19.273077+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:43:19.273077+0000","last_peered":"2024-09-05T16:43:19.273077+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:43:19.273077+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"29'700","ondisk_log_start":"29'700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19042032,"num_objects":2325,"num_object_clones":0,"num_object_copies":9300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2325,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10795,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":44}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'10421","reported_seq":10520,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:19.195876+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:43:19.195876+0000","last_peered":"2024-09-05T16:43:19.195876+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:43:19.195876+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"29'400","ondisk_log_start":"29'400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19151440,"num_objects":2338,"num_object_clones":0,"num_object_copies":9352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2338,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10420,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":100}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'10557","reported_seq":15291,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:43:18.816933+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:43:18.816933+0000","last_peered":"2024-09-05T16:43:18.816933+0000","last_clean":"2024-09-05T16:43:18.816933+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:43:18.816933+0000","last_undegraded":"2024-09-05T16:43:18.816933+0000","last_fullsized":"2024-09-05T16:43:18.816933+0000","mapping_epoch":36,"log_start":"29'500","ondisk_log_start":"29'500","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18440192,"num_objects":2251,"num_object_clones":0,"num_object_copies":9004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2251,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2157,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'10802","reported_seq":10911,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:19.249269+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:43:19.249269+0000","last_peered":"2024-09-05T16:43:19.249269+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:43:19.249269+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"29'800","ondisk_log_start":"29'800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19035296,"num_objects":2324,"num_object_clones":0,"num_object_copies":9296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2324,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10800,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-10}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'10593","reported_seq":10706,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:19.022575+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:43:19.022575+0000","last_peered":"2024-09-05T16:43:19.022575+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:43:19.022575+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"29'500","ondisk_log_start":"29'500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18347168,"num_objects":2240,"num_object_clones":0,"num_object_copies":8960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2240,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10591,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-68}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'10506","reported_seq":10615,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:19.048642+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:43:19.048642+0000","last_peered":"2024-09-05T16:43:19.048642+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:43:19.048642+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"29'500","ondisk_log_start":"29'500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18584736,"num_objects":2269,"num_object_clones":0,"num_object_copies":9076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2269,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10504,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":49}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":298267112,"num_objects":36415,"num_object_clones":0,"num_object_copies":145660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29399,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36415,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":154422,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2238,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":596881408,"data_stored":596881408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160846,"ondisk_log_size":160846,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084400,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":750968,"kb_used_data":145888,"kb_used_omap":4,"kb_used_meta":605051,"kb_avail":93620872,"statfs":{"total":96636764160,"available":95867772928,"internally_reserved":0,"allocated":149389312,"data_stored":149188038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":619572947},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,12,4,2,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986277,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":589168,"kb_used_data":146520,"kb_used_omap":4,"kb_used_meta":442619,"kb_avail":93782672,"statfs":{"total":96636764160,"available":96033456128,"internally_reserved":0,"allocated":150036480,"data_stored":149835206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":453242578},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,3,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673083,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":754808,"kb_used_data":146076,"kb_used_omap":1,"kb_used_meta":608702,"kb_avail":93617032,"statfs":{"total":96636764160,"available":95863840768,"internally_reserved":0,"allocated":149581824,"data_stored":149380550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":623311301},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,3,3,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182514,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":755672,"kb_used_data":146500,"kb_used_omap":4,"kb_used_meta":609147,"kb_avail":93616168,"statfs":{"total":96636764160,"available":95862956032,"internally_reserved":0,"allocated":150016000,"data_stored":149814726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":623767245},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,1,4,2,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":149118976,"data_stored":149118976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":149463040,"data_stored":149463040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":149250048,"data_stored":149250048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":149049344,"data_stored":149049344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:43:23.375 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:43:23.976 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:43:23.977 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:43:23.997 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":331,"stamp":"2024-09-05T16:43:23.015067+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":298267112,"num_objects":36415,"num_object_clones":0,"num_object_copies":145660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29399,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36415,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":154422,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2238,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160846,"ondisk_log_size":160846,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2850616,"kb_used_data":584984,"kb_used_omap":14,"kb_used_meta":2265521,"kb_avail":374636744,"statfs":{"total":386547056640,"available":383628025856,"internally_reserved":0,"allocated":599023616,"data_stored":598218520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2319894071},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,15,14,7,18],"upper_bound":256},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16891904,"num_objects":-2062,"num_object_clones":0,"num_object_copies":-8248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-124,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2062,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2062,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":124,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005427"},"pg_stats":[{"pgid":"1.f","version":"37'10597","reported_seq":10699,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:19.311352+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:43:19.311352+0000","last_peered":"2024-09-05T16:43:19.311352+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:43:19.311352+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"29'500","ondisk_log_start":"29'500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18898944,"num_objects":2307,"num_object_clones":0,"num_object_copies":9228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2307,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10597,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":71}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'10357","reported_seq":14345,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:19.240432+0000","last_change":"2024-09-05T16:40:28.919189+0000","last_active":"2024-09-05T16:43:19.240432+0000","last_peered":"2024-09-05T16:43:19.240432+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:43:19.240432+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"29'300","ondisk_log_start":"29'300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17994912,"num_objects":2197,"num_object_clones":0,"num_object_copies":8788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2197,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10355,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1959,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1885}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'10497","reported_seq":10620,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:18.867165+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:43:18.867165+0000","last_peered":"2024-09-05T16:43:18.867165+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:43:18.867165+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"29'400","ondisk_log_start":"29'400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18471528,"num_objects":2256,"num_object_clones":0,"num_object_copies":9024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2256,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10496,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":24}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'10601","reported_seq":15328,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:43:18.715081+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:43:18.715081+0000","last_peered":"2024-09-05T16:43:18.715081+0000","last_clean":"2024-09-05T16:43:18.715081+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:43:18.715081+0000","last_undegraded":"2024-09-05T16:43:18.715081+0000","last_fullsized":"2024-09-05T16:43:18.715081+0000","mapping_epoch":34,"log_start":"29'600","ondisk_log_start":"29'600","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19111936,"num_objects":2333,"num_object_clones":0,"num_object_copies":9332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2333,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4901,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'10317","reported_seq":10416,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:19.105543+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:43:19.105543+0000","last_peered":"2024-09-05T16:43:19.105543+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:43:19.105543+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"29'300","ondisk_log_start":"29'300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18658464,"num_objects":2278,"num_object_clones":0,"num_object_copies":9112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2278,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10315,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":133}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'10539","reported_seq":10654,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:19.285487+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:43:19.285487+0000","last_peered":"2024-09-05T16:43:19.285487+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:43:19.285487+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"29'500","ondisk_log_start":"29'500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18292736,"num_objects":2233,"num_object_clones":0,"num_object_copies":8932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2233,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10539,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":14}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'10372","reported_seq":10480,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:18.950622+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:43:18.950622+0000","last_peered":"2024-09-05T16:43:18.950622+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:43:18.950622+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"29'300","ondisk_log_start":"29'300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18756768,"num_objects":2290,"num_object_clones":0,"num_object_copies":9160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2290,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10370,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":159}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'10492","reported_seq":10581,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:19.160569+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:43:19.160569+0000","last_peered":"2024-09-05T16:43:19.160569+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:43:19.160569+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"29'400","ondisk_log_start":"29'400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18732192,"num_objects":2287,"num_object_clones":0,"num_object_copies":9148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2287,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10490,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":55}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'10528","reported_seq":10627,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:19.287963+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:43:19.287963+0000","last_peered":"2024-09-05T16:43:19.287963+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:43:19.287963+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"29'500","ondisk_log_start":"29'500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18393408,"num_objects":2246,"num_object_clones":0,"num_object_copies":8984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2246,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10525,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'10569","reported_seq":10690,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:18.925919+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:43:18.925919+0000","last_peered":"2024-09-05T16:43:18.925919+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:43:18.925919+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"29'500","ondisk_log_start":"29'500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18355360,"num_objects":2241,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10567,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-4}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'10798","reported_seq":10923,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:19.273077+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:43:19.273077+0000","last_peered":"2024-09-05T16:43:19.273077+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:43:19.273077+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"29'700","ondisk_log_start":"29'700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19042032,"num_objects":2325,"num_object_clones":0,"num_object_copies":9300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2325,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10795,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":44}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'10421","reported_seq":10520,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:19.195876+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:43:19.195876+0000","last_peered":"2024-09-05T16:43:19.195876+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:43:19.195876+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"29'400","ondisk_log_start":"29'400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19151440,"num_objects":2338,"num_object_clones":0,"num_object_copies":9352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2338,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10420,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":100}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'10557","reported_seq":15291,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:43:18.816933+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:43:18.816933+0000","last_peered":"2024-09-05T16:43:18.816933+0000","last_clean":"2024-09-05T16:43:18.816933+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:43:18.816933+0000","last_undegraded":"2024-09-05T16:43:18.816933+0000","last_fullsized":"2024-09-05T16:43:18.816933+0000","mapping_epoch":36,"log_start":"29'500","ondisk_log_start":"29'500","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18440192,"num_objects":2251,"num_object_clones":0,"num_object_copies":9004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2251,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2157,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'10802","reported_seq":10911,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:19.249269+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:43:19.249269+0000","last_peered":"2024-09-05T16:43:19.249269+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:43:19.249269+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"29'800","ondisk_log_start":"29'800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19035296,"num_objects":2324,"num_object_clones":0,"num_object_copies":9296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2324,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10800,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-10}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'10593","reported_seq":10706,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:19.022575+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:43:19.022575+0000","last_peered":"2024-09-05T16:43:19.022575+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:43:19.022575+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"29'500","ondisk_log_start":"29'500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18347168,"num_objects":2240,"num_object_clones":0,"num_object_copies":8960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2240,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10591,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-68}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'10506","reported_seq":10615,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:19.048642+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:43:19.048642+0000","last_peered":"2024-09-05T16:43:19.048642+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:43:19.048642+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"29'500","ondisk_log_start":"29'500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18584736,"num_objects":2269,"num_object_clones":0,"num_object_copies":9076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2269,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10504,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":49}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":298267112,"num_objects":36415,"num_object_clones":0,"num_object_copies":145660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29399,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36415,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":154422,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2238,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":596881408,"data_stored":596881408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160846,"ondisk_log_size":160846,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084400,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":750968,"kb_used_data":145888,"kb_used_omap":4,"kb_used_meta":605051,"kb_avail":93620872,"statfs":{"total":96636764160,"available":95867772928,"internally_reserved":0,"allocated":149389312,"data_stored":149188038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":619572947},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,12,4,2,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986277,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":589168,"kb_used_data":146520,"kb_used_omap":4,"kb_used_meta":442619,"kb_avail":93782672,"statfs":{"total":96636764160,"available":96033456128,"internally_reserved":0,"allocated":150036480,"data_stored":149835206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":453242578},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,3,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673083,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":754808,"kb_used_data":146076,"kb_used_omap":1,"kb_used_meta":608702,"kb_avail":93617032,"statfs":{"total":96636764160,"available":95863840768,"internally_reserved":0,"allocated":149581824,"data_stored":149380550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":623311301},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,3,3,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182514,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":755672,"kb_used_data":146500,"kb_used_omap":4,"kb_used_meta":609147,"kb_avail":93616168,"statfs":{"total":96636764160,"available":95862956032,"internally_reserved":0,"allocated":150016000,"data_stored":149814726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":623767245},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,1,4,2,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":149118976,"data_stored":149118976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":149463040,"data_stored":149463040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":149250048,"data_stored":149250048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":149049344,"data_stored":149049344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:43:23.998 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-05T16:43:23.998 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'10597", 'reported_seq': 10699, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:19.311352+0000', 'last_change': '2024-09-05T16:34:56.833497+0000', 'last_active': '2024-09-05T16:43:19.311352+0000', 'last_peered': '2024-09-05T16:43:19.311352+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.828289+0000', 'last_became_peered': '2024-09-05T16:34:56.828289+0000', 'last_unstale': '2024-09-05T16:43:19.311352+0000', 'last_undegraded': '2024-09-05T16:34:56.612160+0000', 'last_fullsized': '2024-09-05T16:34:56.557632+0000', 'mapping_epoch': 32, 'log_start': "29'500", 'ondisk_log_start': "29'500", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10097, 'log_dups_size': 0, 'ondisk_log_size': 10097, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18898944, 'num_objects': 2307, 'num_object_clones': 0, 'num_object_copies': 9228, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2236, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2307, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10597, 'num_write_kb': 51616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 71}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:43:23.998 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-05T16:43:23.998 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "37'10357", 'reported_seq': 14345, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:19.240432+0000', 'last_change': '2024-09-05T16:40:28.919189+0000', 'last_active': '2024-09-05T16:43:19.240432+0000', 'last_peered': '2024-09-05T16:43:19.240432+0000', 'last_clean': '2024-09-05T16:33:53.200762+0000', 'last_became_active': '2024-09-05T16:34:56.820292+0000', 'last_became_peered': '2024-09-05T16:34:56.820292+0000', 'last_unstale': '2024-09-05T16:43:19.240432+0000', 'last_undegraded': '2024-09-05T16:34:56.605776+0000', 'last_fullsized': '2024-09-05T16:34:56.559383+0000', 'mapping_epoch': 32, 'log_start': "29'300", 'ondisk_log_start': "29'300", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10057, 'log_dups_size': 0, 'ondisk_log_size': 10057, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17994912, 'num_objects': 2197, 'num_object_clones': 0, 'num_object_copies': 8788, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 312, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2197, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10355, 'num_write_kb': 50212, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1959, 'num_bytes_recovered': 2613248, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 1885}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:43:23.999 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:43:24.000 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'10497", 'reported_seq': 10620, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:18.867165+0000', 'last_change': '2024-09-05T16:34:56.867900+0000', 'last_active': '2024-09-05T16:43:18.867165+0000', 'last_peered': '2024-09-05T16:43:18.867165+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:43:18.867165+0000', 'last_undegraded': '2024-09-05T16:34:56.590972+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 32, 'log_start': "29'400", 'ondisk_log_start': "29'400", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10097, 'log_dups_size': 0, 'ondisk_log_size': 10097, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18471528, 'num_objects': 2256, 'num_object_clones': 0, 'num_object_copies': 9024, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2233, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2256, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 10496, 'num_write_kb': 50999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 24}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:43:24.000 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:43:24.000 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'10317", 'reported_seq': 10416, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:19.105543+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:43:19.105543+0000', 'last_peered': '2024-09-05T16:43:19.105543+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:43:19.105543+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "29'300", 'ondisk_log_start': "29'300", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10017, 'log_dups_size': 0, 'ondisk_log_size': 10017, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18658464, 'num_objects': 2278, 'num_object_clones': 0, 'num_object_copies': 9112, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2147, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2278, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10315, 'num_write_kb': 50373, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 133}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:43:24.000 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:43:24.000 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'10539", 'reported_seq': 10654, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:19.285487+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:43:19.285487+0000', 'last_peered': '2024-09-05T16:43:19.285487+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:43:19.285487+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "29'500", 'ondisk_log_start': "29'500", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10039, 'log_dups_size': 0, 'ondisk_log_size': 10039, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18292736, 'num_objects': 2233, 'num_object_clones': 0, 'num_object_copies': 8932, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2233, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10539, 'num_write_kb': 51088, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 14}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:43:24.000 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:43:24.000 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'10372", 'reported_seq': 10480, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:18.950622+0000', 'last_change': '2024-09-05T16:34:56.903436+0000', 'last_active': '2024-09-05T16:43:18.950622+0000', 'last_peered': '2024-09-05T16:43:18.950622+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:43:18.950622+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "29'300", 'ondisk_log_start': "29'300", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10072, 'log_dups_size': 0, 'ondisk_log_size': 10072, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18756768, 'num_objects': 2290, 'num_object_clones': 0, 'num_object_copies': 9160, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2133, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2290, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10370, 'num_write_kb': 50636, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 159}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:43:24.000 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-05T16:43:24.000 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "37'10492", 'reported_seq': 10581, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:19.160569+0000', 'last_change': '2024-09-05T16:34:56.883319+0000', 'last_active': '2024-09-05T16:43:19.160569+0000', 'last_peered': '2024-09-05T16:43:19.160569+0000', 'last_clean': '2024-09-05T16:33:53.505305+0000', 'last_became_active': '2024-09-05T16:34:56.873389+0000', 'last_became_peered': '2024-09-05T16:34:56.873389+0000', 'last_unstale': '2024-09-05T16:43:19.160569+0000', 'last_undegraded': '2024-09-05T16:34:56.602567+0000', 'last_fullsized': '2024-09-05T16:34:56.559958+0000', 'mapping_epoch': 32, 'log_start': "29'400", 'ondisk_log_start': "29'400", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10092, 'log_dups_size': 0, 'ondisk_log_size': 10092, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18732192, 'num_objects': 2287, 'num_object_clones': 0, 'num_object_copies': 9148, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2234, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2287, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10490, 'num_write_kb': 51108, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 55}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:43:24.000 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:43:24.001 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'10528", 'reported_seq': 10627, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:19.287963+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:43:19.287963+0000', 'last_peered': '2024-09-05T16:43:19.287963+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:43:19.287963+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "29'500", 'ondisk_log_start': "29'500", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18393408, 'num_objects': 2246, 'num_object_clones': 0, 'num_object_copies': 8984, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2246, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10525, 'num_write_kb': 51081, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:43:24.001 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:43:24.001 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'10569", 'reported_seq': 10690, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:18.925919+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:43:18.925919+0000', 'last_peered': '2024-09-05T16:43:18.925919+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:43:18.925919+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "29'500", 'ondisk_log_start': "29'500", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10069, 'log_dups_size': 0, 'ondisk_log_size': 10069, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18355360, 'num_objects': 2241, 'num_object_clones': 0, 'num_object_copies': 8964, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2241, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10567, 'num_write_kb': 51230, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -4}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:43:24.001 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:43:24.001 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'10798", 'reported_seq': 10923, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:19.273077+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:43:19.273077+0000', 'last_peered': '2024-09-05T16:43:19.273077+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:43:19.273077+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "29'700", 'ondisk_log_start': "29'700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19042032, 'num_objects': 2325, 'num_object_clones': 0, 'num_object_copies': 9300, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2284, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2325, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10795, 'num_write_kb': 52476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 44}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:43:24.001 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:43:24.001 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'10421", 'reported_seq': 10520, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:19.195876+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:43:19.195876+0000', 'last_peered': '2024-09-05T16:43:19.195876+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:43:19.195876+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "29'400", 'ondisk_log_start': "29'400", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10021, 'log_dups_size': 0, 'ondisk_log_size': 10021, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19151440, 'num_objects': 2338, 'num_object_clones': 0, 'num_object_copies': 9352, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2338, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10420, 'num_write_kb': 51031, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 100}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:43:24.001 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:43:24.001 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'10802", 'reported_seq': 10911, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:19.249269+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:43:19.249269+0000', 'last_peered': '2024-09-05T16:43:19.249269+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:43:19.249269+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "29'800", 'ondisk_log_start': "29'800", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10002, 'log_dups_size': 0, 'ondisk_log_size': 10002, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19035296, 'num_objects': 2324, 'num_object_clones': 0, 'num_object_copies': 9296, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2324, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10800, 'num_write_kb': 52500, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -10}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:43:24.001 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:43:24.001 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'10593", 'reported_seq': 10706, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:19.022575+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:43:19.022575+0000', 'last_peered': '2024-09-05T16:43:19.022575+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:43:19.022575+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "29'500", 'ondisk_log_start': "29'500", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10093, 'log_dups_size': 0, 'ondisk_log_size': 10093, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18347168, 'num_objects': 2240, 'num_object_clones': 0, 'num_object_copies': 8960, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2310, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2240, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10591, 'num_write_kb': 51325, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -68}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:43:24.002 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:43:24.002 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'10506", 'reported_seq': 10615, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:19.048642+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:43:19.048642+0000', 'last_peered': '2024-09-05T16:43:19.048642+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:43:19.048642+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "29'500", 'ondisk_log_start': "29'500", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10006, 'log_dups_size': 0, 'ondisk_log_size': 10006, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18584736, 'num_objects': 2269, 'num_object_clones': 0, 'num_object_copies': 9076, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2269, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10504, 'num_write_kb': 51092, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 49}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:43:24.002 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:43:24.736 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:43:24.736 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 29399/145660 objects degraded (20.183%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":607,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":36415,"data_bytes":298267112,"bytes_used":2919030784,"bytes_avail":383628025856,"bytes_total":386547056640,"degraded_objects":29399,"degraded_total":145660,"degraded_ratio":0.20183303583688039,"recovering_objects_per_sec":10,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":124,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":171},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:43:24.764 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:43:24.764 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:43:25.431 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:43:25.432 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:43:25.450 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":332,"stamp":"2024-09-05T16:43:25.015890+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":290042344,"num_objects":35411,"num_object_clones":0,"num_object_copies":141644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35411,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":155426,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2301,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160950,"ondisk_log_size":160950,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2851140,"kb_used_data":568844,"kb_used_omap":14,"kb_used_meta":2282161,"kb_avail":374636220,"statfs":{"total":386547056640,"available":383627489280,"internally_reserved":0,"allocated":582496256,"data_stored":581691160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2336933431},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,7,0,5,4,26,14,13,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25116672,"num_objects":-3066,"num_object_clones":0,"num_object_copies":-12264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-187,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3066,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3066,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":187,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005341"},"pg_stats":[{"pgid":"1.f","version":"37'10656","reported_seq":10760,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:24.265906+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:43:24.265906+0000","last_peered":"2024-09-05T16:43:24.265906+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:43:24.265906+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"29'600","ondisk_log_start":"29'600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18415616,"num_objects":2248,"num_object_clones":0,"num_object_copies":8992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2248,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10656,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":12}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'10421","reported_seq":14537,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:24.156534+0000","last_change":"2024-09-05T16:40:28.919189+0000","last_active":"2024-09-05T16:43:24.156534+0000","last_peered":"2024-09-05T16:43:24.156534+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:43:24.156534+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"29'400","ondisk_log_start":"29'400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17470624,"num_objects":2133,"num_object_clones":0,"num_object_copies":8532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":249,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2133,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10419,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2022,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1884}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'10567","reported_seq":10690,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:24.024173+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:43:24.024173+0000","last_peered":"2024-09-05T16:43:24.024173+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:43:24.024173+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"29'500","ondisk_log_start":"29'500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17898088,"num_objects":2186,"num_object_clones":0,"num_object_copies":8744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2186,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10566,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-46}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'10664","reported_seq":15393,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:43:24.009239+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:43:24.009239+0000","last_peered":"2024-09-05T16:43:24.009239+0000","last_clean":"2024-09-05T16:43:24.009239+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:43:24.009239+0000","last_undegraded":"2024-09-05T16:43:24.009239+0000","last_fullsized":"2024-09-05T16:43:24.009239+0000","mapping_epoch":34,"log_start":"29'600","ondisk_log_start":"29'600","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18595840,"num_objects":2270,"num_object_clones":0,"num_object_copies":9080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2270,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4964,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'10372","reported_seq":10471,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:24.354817+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:43:24.354817+0000","last_peered":"2024-09-05T16:43:24.354817+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:43:24.354817+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"29'300","ondisk_log_start":"29'300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18207904,"num_objects":2223,"num_object_clones":0,"num_object_copies":8892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2223,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10370,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":78}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'10600","reported_seq":10715,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:24.156924+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:43:24.156924+0000","last_peered":"2024-09-05T16:43:24.156924+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:43:24.156924+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"29'500","ondisk_log_start":"29'500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17793024,"num_objects":2172,"num_object_clones":0,"num_object_copies":8688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2172,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10600,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-47}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'10421","reported_seq":10529,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:24.066348+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:43:24.066348+0000","last_peered":"2024-09-05T16:43:24.066348+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:43:24.066348+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"29'400","ondisk_log_start":"29'400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18355360,"num_objects":2241,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10419,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":110}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'10559","reported_seq":10648,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:24.078321+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:43:24.078321+0000","last_peered":"2024-09-05T16:43:24.078321+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:43:24.078321+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"29'500","ondisk_log_start":"29'500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18183328,"num_objects":2220,"num_object_clones":0,"num_object_copies":8880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2220,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10557,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-12}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'10591","reported_seq":10692,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:24.308407+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:43:24.308407+0000","last_peered":"2024-09-05T16:43:24.308407+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:43:24.308407+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"29'500","ondisk_log_start":"29'500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17877312,"num_objects":2183,"num_object_clones":0,"num_object_copies":8732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2183,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10588,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-61}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'10629","reported_seq":10751,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:23.359569+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:43:23.359569+0000","last_peered":"2024-09-05T16:43:23.359569+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:43:23.359569+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"29'600","ondisk_log_start":"29'600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17863840,"num_objects":2181,"num_object_clones":0,"num_object_copies":8724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2181,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10627,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-64}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'10856","reported_seq":10983,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:24.250339+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:43:24.250339+0000","last_peered":"2024-09-05T16:43:24.250339+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:43:24.250339+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"29'800","ondisk_log_start":"29'800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18566896,"num_objects":2267,"num_object_clones":0,"num_object_copies":9068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2267,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10853,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-14}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'10494","reported_seq":10593,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:24.326071+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:43:24.326071+0000","last_peered":"2024-09-05T16:43:24.326071+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:43:24.326071+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"29'400","ondisk_log_start":"29'400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18553424,"num_objects":2265,"num_object_clones":0,"num_object_copies":9060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2265,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10493,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":27}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'10627","reported_seq":15362,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:43:24.037850+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:43:24.037850+0000","last_peered":"2024-09-05T16:43:24.037850+0000","last_clean":"2024-09-05T16:43:24.037850+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:43:24.037850+0000","last_undegraded":"2024-09-05T16:43:24.037850+0000","last_fullsized":"2024-09-05T16:43:24.037850+0000","mapping_epoch":36,"log_start":"29'600","ondisk_log_start":"29'600","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17866752,"num_objects":2181,"num_object_clones":0,"num_object_copies":8724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2181,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2227,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'10869","reported_seq":10978,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:24.409666+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:43:24.409666+0000","last_peered":"2024-09-05T16:43:24.409666+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:43:24.409666+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"29'800","ondisk_log_start":"29'800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18486432,"num_objects":2257,"num_object_clones":0,"num_object_copies":9028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2257,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10867,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-77}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'10655","reported_seq":10768,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:23.912903+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:43:23.912903+0000","last_peered":"2024-09-05T16:43:23.912903+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:43:23.912903+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"29'600","ondisk_log_start":"29'600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17839264,"num_objects":2178,"num_object_clones":0,"num_object_copies":8712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2178,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10653,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-130}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'10569","reported_seq":10677,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:24.240491+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:43:24.240491+0000","last_peered":"2024-09-05T16:43:24.240491+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:43:24.240491+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"29'500","ondisk_log_start":"29'500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18068640,"num_objects":2206,"num_object_clones":0,"num_object_copies":8824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2206,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10567,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-14}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":290042344,"num_objects":35411,"num_object_clones":0,"num_object_copies":141644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35411,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":155426,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2301,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":580255744,"data_stored":580255744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160950,"ondisk_log_size":160950,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084401,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":751000,"kb_used_data":141760,"kb_used_omap":4,"kb_used_meta":609211,"kb_avail":93620840,"statfs":{"total":96636764160,"available":95867740160,"internally_reserved":0,"allocated":145162240,"data_stored":144960966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":623832787},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,8,9,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986278,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":589364,"kb_used_data":142564,"kb_used_omap":4,"kb_used_meta":446779,"kb_avail":93782476,"statfs":{"total":96636764160,"available":96033255424,"internally_reserved":0,"allocated":145985536,"data_stored":145784262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":457502418},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,0,10,2,5,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673084,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":755012,"kb_used_data":142128,"kb_used_omap":1,"kb_used_meta":612862,"kb_avail":93616828,"statfs":{"total":96636764160,"available":95863631872,"internally_reserved":0,"allocated":145539072,"data_stored":145337798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":627571141},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,0,1,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182515,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":755764,"kb_used_data":142392,"kb_used_omap":4,"kb_used_meta":613307,"kb_avail":93616076,"statfs":{"total":96636764160,"available":95862861824,"internally_reserved":0,"allocated":145809408,"data_stored":145608134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":628027085},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,0,0,0,3,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":145002496,"data_stored":145002496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":145285120,"data_stored":145285120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":145117184,"data_stored":145117184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":144850944,"data_stored":144850944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:43:28.453 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:43:29.033 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:43:29.034 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:43:29.056 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":334,"stamp":"2024-09-05T16:43:29.017562+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":290042344,"num_objects":35411,"num_object_clones":0,"num_object_copies":141644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35411,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":155426,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2301,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160950,"ondisk_log_size":160950,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2851140,"kb_used_data":568844,"kb_used_omap":14,"kb_used_meta":2282161,"kb_avail":374636220,"statfs":{"total":386547056640,"available":383627489280,"internally_reserved":0,"allocated":582496256,"data_stored":581691160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2336933431},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,7,0,5,4,26,14,13,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16711680,"num_objects":-2040,"num_object_clones":0,"num_object_copies":-8160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-128,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2040,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2040,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":128,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005094"},"pg_stats":[{"pgid":"1.f","version":"37'10656","reported_seq":10760,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:24.265906+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:43:24.265906+0000","last_peered":"2024-09-05T16:43:24.265906+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:43:24.265906+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"29'600","ondisk_log_start":"29'600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18415616,"num_objects":2248,"num_object_clones":0,"num_object_copies":8992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2248,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10656,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":12}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'10421","reported_seq":14537,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:24.156534+0000","last_change":"2024-09-05T16:40:28.919189+0000","last_active":"2024-09-05T16:43:24.156534+0000","last_peered":"2024-09-05T16:43:24.156534+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:43:24.156534+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"29'400","ondisk_log_start":"29'400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17470624,"num_objects":2133,"num_object_clones":0,"num_object_copies":8532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":249,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2133,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10419,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2022,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1884}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'10567","reported_seq":10690,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:24.024173+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:43:24.024173+0000","last_peered":"2024-09-05T16:43:24.024173+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:43:24.024173+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"29'500","ondisk_log_start":"29'500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17898088,"num_objects":2186,"num_object_clones":0,"num_object_copies":8744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2186,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10566,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-46}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'10664","reported_seq":15393,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:43:24.009239+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:43:24.009239+0000","last_peered":"2024-09-05T16:43:24.009239+0000","last_clean":"2024-09-05T16:43:24.009239+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:43:24.009239+0000","last_undegraded":"2024-09-05T16:43:24.009239+0000","last_fullsized":"2024-09-05T16:43:24.009239+0000","mapping_epoch":34,"log_start":"29'600","ondisk_log_start":"29'600","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18595840,"num_objects":2270,"num_object_clones":0,"num_object_copies":9080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2270,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4964,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'10372","reported_seq":10471,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:24.354817+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:43:24.354817+0000","last_peered":"2024-09-05T16:43:24.354817+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:43:24.354817+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"29'300","ondisk_log_start":"29'300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18207904,"num_objects":2223,"num_object_clones":0,"num_object_copies":8892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2223,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10370,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":78}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'10600","reported_seq":10715,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:24.156924+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:43:24.156924+0000","last_peered":"2024-09-05T16:43:24.156924+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:43:24.156924+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"29'500","ondisk_log_start":"29'500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17793024,"num_objects":2172,"num_object_clones":0,"num_object_copies":8688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2172,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10600,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-47}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'10421","reported_seq":10529,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:24.066348+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:43:24.066348+0000","last_peered":"2024-09-05T16:43:24.066348+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:43:24.066348+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"29'400","ondisk_log_start":"29'400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18355360,"num_objects":2241,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10419,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":110}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'10559","reported_seq":10648,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:24.078321+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:43:24.078321+0000","last_peered":"2024-09-05T16:43:24.078321+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:43:24.078321+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"29'500","ondisk_log_start":"29'500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18183328,"num_objects":2220,"num_object_clones":0,"num_object_copies":8880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2220,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10557,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-12}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'10591","reported_seq":10692,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:24.308407+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:43:24.308407+0000","last_peered":"2024-09-05T16:43:24.308407+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:43:24.308407+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"29'500","ondisk_log_start":"29'500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17877312,"num_objects":2183,"num_object_clones":0,"num_object_copies":8732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2183,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10588,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-61}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'10629","reported_seq":10751,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:23.359569+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:43:23.359569+0000","last_peered":"2024-09-05T16:43:23.359569+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:43:23.359569+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"29'600","ondisk_log_start":"29'600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17863840,"num_objects":2181,"num_object_clones":0,"num_object_copies":8724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2181,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10627,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-64}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'10856","reported_seq":10983,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:24.250339+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:43:24.250339+0000","last_peered":"2024-09-05T16:43:24.250339+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:43:24.250339+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"29'800","ondisk_log_start":"29'800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18566896,"num_objects":2267,"num_object_clones":0,"num_object_copies":9068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2267,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10853,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-14}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'10494","reported_seq":10593,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:24.326071+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:43:24.326071+0000","last_peered":"2024-09-05T16:43:24.326071+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:43:24.326071+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"29'400","ondisk_log_start":"29'400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18553424,"num_objects":2265,"num_object_clones":0,"num_object_copies":9060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2265,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10493,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":27}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'10627","reported_seq":15362,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:43:24.037850+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:43:24.037850+0000","last_peered":"2024-09-05T16:43:24.037850+0000","last_clean":"2024-09-05T16:43:24.037850+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:43:24.037850+0000","last_undegraded":"2024-09-05T16:43:24.037850+0000","last_fullsized":"2024-09-05T16:43:24.037850+0000","mapping_epoch":36,"log_start":"29'600","ondisk_log_start":"29'600","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17866752,"num_objects":2181,"num_object_clones":0,"num_object_copies":8724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2181,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2227,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'10869","reported_seq":10978,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:24.409666+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:43:24.409666+0000","last_peered":"2024-09-05T16:43:24.409666+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:43:24.409666+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"29'800","ondisk_log_start":"29'800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18486432,"num_objects":2257,"num_object_clones":0,"num_object_copies":9028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2257,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10867,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-77}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'10655","reported_seq":10768,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:23.912903+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:43:23.912903+0000","last_peered":"2024-09-05T16:43:23.912903+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:43:23.912903+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"29'600","ondisk_log_start":"29'600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17839264,"num_objects":2178,"num_object_clones":0,"num_object_copies":8712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2178,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10653,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-130}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'10569","reported_seq":10677,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:24.240491+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:43:24.240491+0000","last_peered":"2024-09-05T16:43:24.240491+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:43:24.240491+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"29'500","ondisk_log_start":"29'500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18068640,"num_objects":2206,"num_object_clones":0,"num_object_copies":8824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2206,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10567,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-14}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":290042344,"num_objects":35411,"num_object_clones":0,"num_object_copies":141644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35411,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":155426,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2301,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":580255744,"data_stored":580255744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160950,"ondisk_log_size":160950,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084401,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":751000,"kb_used_data":141760,"kb_used_omap":4,"kb_used_meta":609211,"kb_avail":93620840,"statfs":{"total":96636764160,"available":95867740160,"internally_reserved":0,"allocated":145162240,"data_stored":144960966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":623832787},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,8,9,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986278,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":589364,"kb_used_data":142564,"kb_used_omap":4,"kb_used_meta":446779,"kb_avail":93782476,"statfs":{"total":96636764160,"available":96033255424,"internally_reserved":0,"allocated":145985536,"data_stored":145784262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":457502418},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,0,10,2,5,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673084,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":755012,"kb_used_data":142128,"kb_used_omap":1,"kb_used_meta":612862,"kb_avail":93616828,"statfs":{"total":96636764160,"available":95863631872,"internally_reserved":0,"allocated":145539072,"data_stored":145337798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":627571141},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,0,1,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182515,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":755764,"kb_used_data":142392,"kb_used_omap":4,"kb_used_meta":613307,"kb_avail":93616076,"statfs":{"total":96636764160,"available":95862861824,"internally_reserved":0,"allocated":145809408,"data_stored":145608134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":628027085},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,0,0,0,3,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":145002496,"data_stored":145002496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":145285120,"data_stored":145285120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":145117184,"data_stored":145117184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":144850944,"data_stored":144850944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:43:29.057 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:43:29.645 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:43:29.645 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:43:29.666 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":334,"stamp":"2024-09-05T16:43:29.017562+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":290042344,"num_objects":35411,"num_object_clones":0,"num_object_copies":141644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35411,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":155426,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2301,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160950,"ondisk_log_size":160950,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2851140,"kb_used_data":568844,"kb_used_omap":14,"kb_used_meta":2282161,"kb_avail":374636220,"statfs":{"total":386547056640,"available":383627489280,"internally_reserved":0,"allocated":582496256,"data_stored":581691160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2336933431},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,7,0,5,4,26,14,13,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16711680,"num_objects":-2040,"num_object_clones":0,"num_object_copies":-8160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-128,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2040,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2040,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":128,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005094"},"pg_stats":[{"pgid":"1.f","version":"37'10656","reported_seq":10760,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:24.265906+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:43:24.265906+0000","last_peered":"2024-09-05T16:43:24.265906+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:43:24.265906+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"29'600","ondisk_log_start":"29'600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18415616,"num_objects":2248,"num_object_clones":0,"num_object_copies":8992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2248,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10656,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":12}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'10421","reported_seq":14537,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:24.156534+0000","last_change":"2024-09-05T16:40:28.919189+0000","last_active":"2024-09-05T16:43:24.156534+0000","last_peered":"2024-09-05T16:43:24.156534+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:43:24.156534+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"29'400","ondisk_log_start":"29'400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17470624,"num_objects":2133,"num_object_clones":0,"num_object_copies":8532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":249,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2133,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10419,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2022,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1884}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'10567","reported_seq":10690,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:24.024173+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:43:24.024173+0000","last_peered":"2024-09-05T16:43:24.024173+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:43:24.024173+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"29'500","ondisk_log_start":"29'500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17898088,"num_objects":2186,"num_object_clones":0,"num_object_copies":8744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2186,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10566,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-46}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'10664","reported_seq":15393,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:43:24.009239+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:43:24.009239+0000","last_peered":"2024-09-05T16:43:24.009239+0000","last_clean":"2024-09-05T16:43:24.009239+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:43:24.009239+0000","last_undegraded":"2024-09-05T16:43:24.009239+0000","last_fullsized":"2024-09-05T16:43:24.009239+0000","mapping_epoch":34,"log_start":"29'600","ondisk_log_start":"29'600","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18595840,"num_objects":2270,"num_object_clones":0,"num_object_copies":9080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2270,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4964,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'10372","reported_seq":10471,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:24.354817+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:43:24.354817+0000","last_peered":"2024-09-05T16:43:24.354817+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:43:24.354817+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"29'300","ondisk_log_start":"29'300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18207904,"num_objects":2223,"num_object_clones":0,"num_object_copies":8892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2223,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10370,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":78}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'10600","reported_seq":10715,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:24.156924+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:43:24.156924+0000","last_peered":"2024-09-05T16:43:24.156924+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:43:24.156924+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"29'500","ondisk_log_start":"29'500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17793024,"num_objects":2172,"num_object_clones":0,"num_object_copies":8688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2172,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10600,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-47}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'10421","reported_seq":10529,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:24.066348+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:43:24.066348+0000","last_peered":"2024-09-05T16:43:24.066348+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:43:24.066348+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"29'400","ondisk_log_start":"29'400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18355360,"num_objects":2241,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10419,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":110}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'10559","reported_seq":10648,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:24.078321+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:43:24.078321+0000","last_peered":"2024-09-05T16:43:24.078321+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:43:24.078321+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"29'500","ondisk_log_start":"29'500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18183328,"num_objects":2220,"num_object_clones":0,"num_object_copies":8880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2220,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10557,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-12}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'10591","reported_seq":10692,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:24.308407+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:43:24.308407+0000","last_peered":"2024-09-05T16:43:24.308407+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:43:24.308407+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"29'500","ondisk_log_start":"29'500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17877312,"num_objects":2183,"num_object_clones":0,"num_object_copies":8732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2183,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10588,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-61}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'10629","reported_seq":10751,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:23.359569+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:43:23.359569+0000","last_peered":"2024-09-05T16:43:23.359569+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:43:23.359569+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"29'600","ondisk_log_start":"29'600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17863840,"num_objects":2181,"num_object_clones":0,"num_object_copies":8724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2181,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10627,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-64}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'10856","reported_seq":10983,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:24.250339+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:43:24.250339+0000","last_peered":"2024-09-05T16:43:24.250339+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:43:24.250339+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"29'800","ondisk_log_start":"29'800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18566896,"num_objects":2267,"num_object_clones":0,"num_object_copies":9068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2267,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10853,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-14}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'10494","reported_seq":10593,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:24.326071+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:43:24.326071+0000","last_peered":"2024-09-05T16:43:24.326071+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:43:24.326071+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"29'400","ondisk_log_start":"29'400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18553424,"num_objects":2265,"num_object_clones":0,"num_object_copies":9060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2265,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10493,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":27}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'10627","reported_seq":15362,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:43:24.037850+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:43:24.037850+0000","last_peered":"2024-09-05T16:43:24.037850+0000","last_clean":"2024-09-05T16:43:24.037850+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:43:24.037850+0000","last_undegraded":"2024-09-05T16:43:24.037850+0000","last_fullsized":"2024-09-05T16:43:24.037850+0000","mapping_epoch":36,"log_start":"29'600","ondisk_log_start":"29'600","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17866752,"num_objects":2181,"num_object_clones":0,"num_object_copies":8724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2181,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2227,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'10869","reported_seq":10978,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:24.409666+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:43:24.409666+0000","last_peered":"2024-09-05T16:43:24.409666+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:43:24.409666+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"29'800","ondisk_log_start":"29'800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18486432,"num_objects":2257,"num_object_clones":0,"num_object_copies":9028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2257,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10867,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-77}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'10655","reported_seq":10768,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:23.912903+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:43:23.912903+0000","last_peered":"2024-09-05T16:43:23.912903+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:43:23.912903+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"29'600","ondisk_log_start":"29'600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17839264,"num_objects":2178,"num_object_clones":0,"num_object_copies":8712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2178,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10653,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-130}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'10569","reported_seq":10677,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:24.240491+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:43:24.240491+0000","last_peered":"2024-09-05T16:43:24.240491+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:43:24.240491+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"29'500","ondisk_log_start":"29'500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18068640,"num_objects":2206,"num_object_clones":0,"num_object_copies":8824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2206,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10567,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-14}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":290042344,"num_objects":35411,"num_object_clones":0,"num_object_copies":141644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35411,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":155426,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2301,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":580255744,"data_stored":580255744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160950,"ondisk_log_size":160950,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084401,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":751000,"kb_used_data":141760,"kb_used_omap":4,"kb_used_meta":609211,"kb_avail":93620840,"statfs":{"total":96636764160,"available":95867740160,"internally_reserved":0,"allocated":145162240,"data_stored":144960966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":623832787},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,8,9,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986278,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":589364,"kb_used_data":142564,"kb_used_omap":4,"kb_used_meta":446779,"kb_avail":93782476,"statfs":{"total":96636764160,"available":96033255424,"internally_reserved":0,"allocated":145985536,"data_stored":145784262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":457502418},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,0,10,2,5,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673084,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":755012,"kb_used_data":142128,"kb_used_omap":1,"kb_used_meta":612862,"kb_avail":93616828,"statfs":{"total":96636764160,"available":95863631872,"internally_reserved":0,"allocated":145539072,"data_stored":145337798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":627571141},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,0,1,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182515,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":755764,"kb_used_data":142392,"kb_used_omap":4,"kb_used_meta":613307,"kb_avail":93616076,"statfs":{"total":96636764160,"available":95862861824,"internally_reserved":0,"allocated":145809408,"data_stored":145608134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":628027085},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,0,0,0,3,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":145002496,"data_stored":145002496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":145285120,"data_stored":145285120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":145117184,"data_stored":145117184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":144850944,"data_stored":144850944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:43:29.667 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-05T16:43:29.667 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'10656", 'reported_seq': 10760, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:24.265906+0000', 'last_change': '2024-09-05T16:34:56.833497+0000', 'last_active': '2024-09-05T16:43:24.265906+0000', 'last_peered': '2024-09-05T16:43:24.265906+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.828289+0000', 'last_became_peered': '2024-09-05T16:34:56.828289+0000', 'last_unstale': '2024-09-05T16:43:24.265906+0000', 'last_undegraded': '2024-09-05T16:34:56.612160+0000', 'last_fullsized': '2024-09-05T16:34:56.557632+0000', 'mapping_epoch': 32, 'log_start': "29'600", 'ondisk_log_start': "29'600", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10056, 'log_dups_size': 0, 'ondisk_log_size': 10056, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18415616, 'num_objects': 2248, 'num_object_clones': 0, 'num_object_copies': 8992, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2236, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2248, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10656, 'num_write_kb': 51616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 12}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:43:29.667 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-05T16:43:29.667 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "37'10421", 'reported_seq': 14537, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:24.156534+0000', 'last_change': '2024-09-05T16:40:28.919189+0000', 'last_active': '2024-09-05T16:43:24.156534+0000', 'last_peered': '2024-09-05T16:43:24.156534+0000', 'last_clean': '2024-09-05T16:33:53.200762+0000', 'last_became_active': '2024-09-05T16:34:56.820292+0000', 'last_became_peered': '2024-09-05T16:34:56.820292+0000', 'last_unstale': '2024-09-05T16:43:24.156534+0000', 'last_undegraded': '2024-09-05T16:34:56.605776+0000', 'last_fullsized': '2024-09-05T16:34:56.559383+0000', 'mapping_epoch': 32, 'log_start': "29'400", 'ondisk_log_start': "29'400", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10021, 'log_dups_size': 0, 'ondisk_log_size': 10021, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17470624, 'num_objects': 2133, 'num_object_clones': 0, 'num_object_copies': 8532, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 249, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2133, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10419, 'num_write_kb': 50212, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2022, 'num_bytes_recovered': 2613248, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 1884}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:43:29.667 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:43:29.667 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'10567", 'reported_seq': 10690, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:24.024173+0000', 'last_change': '2024-09-05T16:34:56.867900+0000', 'last_active': '2024-09-05T16:43:24.024173+0000', 'last_peered': '2024-09-05T16:43:24.024173+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:43:24.024173+0000', 'last_undegraded': '2024-09-05T16:34:56.590972+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 32, 'log_start': "29'500", 'ondisk_log_start': "29'500", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10067, 'log_dups_size': 0, 'ondisk_log_size': 10067, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17898088, 'num_objects': 2186, 'num_object_clones': 0, 'num_object_copies': 8744, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2233, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2186, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 10566, 'num_write_kb': 50999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -46}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:43:29.667 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:43:29.667 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'10372", 'reported_seq': 10471, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:24.354817+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:43:24.354817+0000', 'last_peered': '2024-09-05T16:43:24.354817+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:43:24.354817+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "29'300", 'ondisk_log_start': "29'300", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10072, 'log_dups_size': 0, 'ondisk_log_size': 10072, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18207904, 'num_objects': 2223, 'num_object_clones': 0, 'num_object_copies': 8892, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2147, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2223, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10370, 'num_write_kb': 50373, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 78}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:43:29.668 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:43:29.668 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'10600", 'reported_seq': 10715, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:24.156924+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:43:24.156924+0000', 'last_peered': '2024-09-05T16:43:24.156924+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:43:24.156924+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "29'500", 'ondisk_log_start': "29'500", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10100, 'log_dups_size': 0, 'ondisk_log_size': 10100, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17793024, 'num_objects': 2172, 'num_object_clones': 0, 'num_object_copies': 8688, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2172, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10600, 'num_write_kb': 51088, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -47}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:43:29.668 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:43:29.668 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'10421", 'reported_seq': 10529, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:24.066348+0000', 'last_change': '2024-09-05T16:34:56.903436+0000', 'last_active': '2024-09-05T16:43:24.066348+0000', 'last_peered': '2024-09-05T16:43:24.066348+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:43:24.066348+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "29'400", 'ondisk_log_start': "29'400", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10021, 'log_dups_size': 0, 'ondisk_log_size': 10021, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18355360, 'num_objects': 2241, 'num_object_clones': 0, 'num_object_copies': 8964, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2133, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2241, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10419, 'num_write_kb': 50636, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 110}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:43:29.668 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-05T16:43:29.668 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "37'10559", 'reported_seq': 10648, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:24.078321+0000', 'last_change': '2024-09-05T16:34:56.883319+0000', 'last_active': '2024-09-05T16:43:24.078321+0000', 'last_peered': '2024-09-05T16:43:24.078321+0000', 'last_clean': '2024-09-05T16:33:53.505305+0000', 'last_became_active': '2024-09-05T16:34:56.873389+0000', 'last_became_peered': '2024-09-05T16:34:56.873389+0000', 'last_unstale': '2024-09-05T16:43:24.078321+0000', 'last_undegraded': '2024-09-05T16:34:56.602567+0000', 'last_fullsized': '2024-09-05T16:34:56.559958+0000', 'mapping_epoch': 32, 'log_start': "29'500", 'ondisk_log_start': "29'500", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10059, 'log_dups_size': 0, 'ondisk_log_size': 10059, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18183328, 'num_objects': 2220, 'num_object_clones': 0, 'num_object_copies': 8880, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2234, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2220, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10557, 'num_write_kb': 51108, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -12}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:43:29.668 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:43:29.668 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'10591", 'reported_seq': 10692, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:24.308407+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:43:24.308407+0000', 'last_peered': '2024-09-05T16:43:24.308407+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:43:24.308407+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "29'500", 'ondisk_log_start': "29'500", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10091, 'log_dups_size': 0, 'ondisk_log_size': 10091, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17877312, 'num_objects': 2183, 'num_object_clones': 0, 'num_object_copies': 8732, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2183, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10588, 'num_write_kb': 51081, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -61}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:43:29.668 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:43:29.668 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'10629", 'reported_seq': 10751, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:23.359569+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:43:23.359569+0000', 'last_peered': '2024-09-05T16:43:23.359569+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:43:23.359569+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "29'600", 'ondisk_log_start': "29'600", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10029, 'log_dups_size': 0, 'ondisk_log_size': 10029, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17863840, 'num_objects': 2181, 'num_object_clones': 0, 'num_object_copies': 8724, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2181, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10627, 'num_write_kb': 51230, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -64}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:43:29.669 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:43:29.669 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'10856", 'reported_seq': 10983, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:24.250339+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:43:24.250339+0000', 'last_peered': '2024-09-05T16:43:24.250339+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:43:24.250339+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "29'800", 'ondisk_log_start': "29'800", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10056, 'log_dups_size': 0, 'ondisk_log_size': 10056, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18566896, 'num_objects': 2267, 'num_object_clones': 0, 'num_object_copies': 9068, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2284, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2267, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10853, 'num_write_kb': 52476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -14}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:43:29.669 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:43:29.669 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'10494", 'reported_seq': 10593, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:24.326071+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:43:24.326071+0000', 'last_peered': '2024-09-05T16:43:24.326071+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:43:24.326071+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "29'400", 'ondisk_log_start': "29'400", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10094, 'log_dups_size': 0, 'ondisk_log_size': 10094, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18553424, 'num_objects': 2265, 'num_object_clones': 0, 'num_object_copies': 9060, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2265, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10493, 'num_write_kb': 51031, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 27}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:43:29.669 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:43:29.669 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'10869", 'reported_seq': 10978, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:24.409666+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:43:24.409666+0000', 'last_peered': '2024-09-05T16:43:24.409666+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:43:24.409666+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "29'800", 'ondisk_log_start': "29'800", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10069, 'log_dups_size': 0, 'ondisk_log_size': 10069, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18486432, 'num_objects': 2257, 'num_object_clones': 0, 'num_object_copies': 9028, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2257, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10867, 'num_write_kb': 52500, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -77}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:43:29.669 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:43:29.669 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'10655", 'reported_seq': 10768, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:23.912903+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:43:23.912903+0000', 'last_peered': '2024-09-05T16:43:23.912903+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:43:23.912903+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "29'600", 'ondisk_log_start': "29'600", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10055, 'log_dups_size': 0, 'ondisk_log_size': 10055, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17839264, 'num_objects': 2178, 'num_object_clones': 0, 'num_object_copies': 8712, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2310, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2178, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10653, 'num_write_kb': 51325, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -130}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:43:29.669 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:43:29.670 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'10569", 'reported_seq': 10677, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:24.240491+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:43:24.240491+0000', 'last_peered': '2024-09-05T16:43:24.240491+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:43:24.240491+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "29'500", 'ondisk_log_start': "29'500", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10069, 'log_dups_size': 0, 'ondisk_log_size': 10069, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18068640, 'num_objects': 2206, 'num_object_clones': 0, 'num_object_copies': 8824, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2206, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10567, 'num_write_kb': 51092, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -14}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:43:29.670 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:43:30.453 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:43:30.453 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 29336/141644 objects degraded (20.711%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":613,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":35411,"data_bytes":290042344,"bytes_used":2919567360,"bytes_avail":383627489280,"bytes_total":386547056640,"degraded_objects":29336,"degraded_total":141644,"degraded_ratio":0.20711078478438905,"recovering_objects_per_sec":10,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":128,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":169},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:43:30.483 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:43:30.483 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:43:31.131 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:43:31.133 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:43:31.165 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":335,"stamp":"2024-09-05T16:43:31.018221+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":281522664,"num_objects":34371,"num_object_clones":0,"num_object_copies":137484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34371,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":156466,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2367,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160690,"ondisk_log_size":160690,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2851588,"kb_used_data":552672,"kb_used_omap":14,"kb_used_meta":2298801,"kb_avail":374635772,"statfs":{"total":386547056640,"available":383627030528,"internally_reserved":0,"allocated":565936128,"data_stored":565131032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2353972791},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,9,6,11,13,14,10,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24125440,"num_objects":-2945,"num_object_clones":0,"num_object_copies":-11780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2945,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2945,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":194,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004916"},"pg_stats":[{"pgid":"1.f","version":"37'10719","reported_seq":10824,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:29.561654+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:43:29.561654+0000","last_peered":"2024-09-05T16:43:29.561654+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:43:29.561654+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"29'700","ondisk_log_start":"29'700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17899520,"num_objects":2185,"num_object_clones":0,"num_object_copies":8740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2185,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10719,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-51}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'10472","reported_seq":14720,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:29.319544+0000","last_change":"2024-09-05T16:40:28.919189+0000","last_active":"2024-09-05T16:43:29.319544+0000","last_peered":"2024-09-05T16:43:29.319544+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:43:29.319544+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"29'400","ondisk_log_start":"29'400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17052832,"num_objects":2082,"num_object_clones":0,"num_object_copies":8328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":183,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2082,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10470,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2088,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1899}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'10625","reported_seq":10748,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:28.983052+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:43:28.983052+0000","last_peered":"2024-09-05T16:43:28.983052+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:43:28.983052+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"29'600","ondisk_log_start":"29'600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17422952,"num_objects":2128,"num_object_clones":0,"num_object_copies":8512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2128,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10624,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-104}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'10736","reported_seq":15463,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:43:29.110809+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:43:29.110809+0000","last_peered":"2024-09-05T16:43:29.110809+0000","last_clean":"2024-09-05T16:43:29.110809+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:43:29.110809+0000","last_undegraded":"2024-09-05T16:43:29.110809+0000","last_fullsized":"2024-09-05T16:43:29.110809+0000","mapping_epoch":34,"log_start":"29'700","ondisk_log_start":"29'700","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18006016,"num_objects":2198,"num_object_clones":0,"num_object_copies":8792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2198,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5036,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'10438","reported_seq":10537,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:29.415755+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:43:29.415755+0000","last_peered":"2024-09-05T16:43:29.415755+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:43:29.415755+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"29'400","ondisk_log_start":"29'400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17667232,"num_objects":2157,"num_object_clones":0,"num_object_copies":8628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2157,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10436,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":12}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'10679","reported_seq":10793,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:29.559636+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:43:29.559636+0000","last_peered":"2024-09-05T16:43:29.559636+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:43:29.559636+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"29'600","ondisk_log_start":"29'600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17145856,"num_objects":2093,"num_object_clones":0,"num_object_copies":8372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2093,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10679,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-126}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'10478","reported_seq":10586,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:29.285691+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:43:29.285691+0000","last_peered":"2024-09-05T16:43:29.285691+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:43:29.285691+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"29'400","ondisk_log_start":"29'400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17888416,"num_objects":2184,"num_object_clones":0,"num_object_copies":8736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2184,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10476,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":53}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'10610","reported_seq":10699,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:29.201196+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:43:29.201196+0000","last_peered":"2024-09-05T16:43:29.201196+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:43:29.201196+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"29'600","ondisk_log_start":"29'600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17765536,"num_objects":2169,"num_object_clones":0,"num_object_copies":8676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2169,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10608,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-63}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'10648","reported_seq":10749,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:29.249920+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:43:29.249920+0000","last_peered":"2024-09-05T16:43:29.249920+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:43:29.249920+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"29'600","ondisk_log_start":"29'600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17410368,"num_objects":2126,"num_object_clones":0,"num_object_copies":8504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2126,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10645,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-118}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'10690","reported_seq":10808,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:28.978737+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:43:28.978737+0000","last_peered":"2024-09-05T16:43:28.978737+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:43:28.978737+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"29'600","ondisk_log_start":"29'600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17364128,"num_objects":2120,"num_object_clones":0,"num_object_copies":8480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2120,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10688,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-125}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'10923","reported_seq":11050,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:29.471465+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:43:29.471465+0000","last_peered":"2024-09-05T16:43:29.471465+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:43:29.471465+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"29'900","ondisk_log_start":"29'900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18018032,"num_objects":2200,"num_object_clones":0,"num_object_copies":8800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2200,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10920,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-81}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'10563","reported_seq":10661,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:29.596275+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:43:29.596275+0000","last_peered":"2024-09-05T16:43:29.596275+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:43:29.596275+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"29'500","ondisk_log_start":"29'500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17988176,"num_objects":2196,"num_object_clones":0,"num_object_copies":8784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2196,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10562,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-42}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'10709","reported_seq":15445,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:43:29.168834+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:43:29.168834+0000","last_peered":"2024-09-05T16:43:29.168834+0000","last_clean":"2024-09-05T16:43:29.168834+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:43:29.168834+0000","last_undegraded":"2024-09-05T16:43:29.168834+0000","last_fullsized":"2024-09-05T16:43:29.168834+0000","mapping_epoch":36,"log_start":"29'700","ondisk_log_start":"29'700","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17195008,"num_objects":2099,"num_object_clones":0,"num_object_copies":8396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2099,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2309,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'10922","reported_seq":11031,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:29.414290+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:43:29.414290+0000","last_peered":"2024-09-05T16:43:29.414290+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:43:29.414290+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"29'900","ondisk_log_start":"29'900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18052256,"num_objects":2204,"num_object_clones":0,"num_object_copies":8816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2204,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10920,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-130}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'10719","reported_seq":10832,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:28.925396+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:43:28.925396+0000","last_peered":"2024-09-05T16:43:28.925396+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:43:28.925396+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"29'700","ondisk_log_start":"29'700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17314976,"num_objects":2114,"num_object_clones":0,"num_object_copies":8456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2114,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10717,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-194}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'10659","reported_seq":10768,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:29.151792+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:43:29.151792+0000","last_peered":"2024-09-05T16:43:29.151792+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:43:29.151792+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"29'600","ondisk_log_start":"29'600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17331360,"num_objects":2116,"num_object_clones":0,"num_object_copies":8464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2116,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10657,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-104}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":281522664,"num_objects":34371,"num_object_clones":0,"num_object_copies":137484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34371,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":156466,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2367,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":563331072,"data_stored":563331072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160690,"ondisk_log_size":160690,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084402,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":751288,"kb_used_data":137880,"kb_used_omap":4,"kb_used_meta":613371,"kb_avail":93620552,"statfs":{"total":96636764160,"available":95867445248,"internally_reserved":0,"allocated":141189120,"data_stored":140987846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":628092627},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,0,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986279,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":589356,"kb_used_data":138396,"kb_used_omap":4,"kb_used_meta":450939,"kb_avail":93782484,"statfs":{"total":96636764160,"available":96033263616,"internally_reserved":0,"allocated":141717504,"data_stored":141516230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":461762258},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,6,4,4,8,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673085,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":755100,"kb_used_data":138060,"kb_used_omap":1,"kb_used_meta":617022,"kb_avail":93616740,"statfs":{"total":96636764160,"available":95863541760,"internally_reserved":0,"allocated":141373440,"data_stored":141172166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":631830981},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,0,3,7,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182516,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":755844,"kb_used_data":138336,"kb_used_omap":4,"kb_used_meta":617467,"kb_avail":93615996,"statfs":{"total":96636764160,"available":95862779904,"internally_reserved":0,"allocated":141656064,"data_stored":141454790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":632286925},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,1,5,6],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":140816384,"data_stored":140816384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":140963840,"data_stored":140963840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":140963840,"data_stored":140963840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":140587008,"data_stored":140587008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:43:34.168 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:43:34.831 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:43:34.831 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:43:34.860 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":336,"stamp":"2024-09-05T16:43:33.018970+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":281522664,"num_objects":34371,"num_object_clones":0,"num_object_copies":137484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34371,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":156466,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2367,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160690,"ondisk_log_size":160690,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2851588,"kb_used_data":552672,"kb_used_omap":14,"kb_used_meta":2298801,"kb_avail":374635772,"statfs":{"total":386547056640,"available":383627030528,"internally_reserved":0,"allocated":565936128,"data_stored":565131032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2353972791},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,9,6,11,13,14,10,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16744448,"num_objects":-2044,"num_object_clones":0,"num_object_copies":-8176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2044,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2044,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":129,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004737"},"pg_stats":[{"pgid":"1.f","version":"37'10719","reported_seq":10824,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:29.561654+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:43:29.561654+0000","last_peered":"2024-09-05T16:43:29.561654+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:43:29.561654+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"29'700","ondisk_log_start":"29'700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17899520,"num_objects":2185,"num_object_clones":0,"num_object_copies":8740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2185,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10719,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-51}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'10472","reported_seq":14720,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:29.319544+0000","last_change":"2024-09-05T16:40:28.919189+0000","last_active":"2024-09-05T16:43:29.319544+0000","last_peered":"2024-09-05T16:43:29.319544+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:43:29.319544+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"29'400","ondisk_log_start":"29'400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17052832,"num_objects":2082,"num_object_clones":0,"num_object_copies":8328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":183,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2082,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10470,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2088,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1899}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'10625","reported_seq":10748,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:28.983052+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:43:28.983052+0000","last_peered":"2024-09-05T16:43:28.983052+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:43:28.983052+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"29'600","ondisk_log_start":"29'600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17422952,"num_objects":2128,"num_object_clones":0,"num_object_copies":8512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2128,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10624,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-104}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'10736","reported_seq":15463,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:43:29.110809+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:43:29.110809+0000","last_peered":"2024-09-05T16:43:29.110809+0000","last_clean":"2024-09-05T16:43:29.110809+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:43:29.110809+0000","last_undegraded":"2024-09-05T16:43:29.110809+0000","last_fullsized":"2024-09-05T16:43:29.110809+0000","mapping_epoch":34,"log_start":"29'700","ondisk_log_start":"29'700","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18006016,"num_objects":2198,"num_object_clones":0,"num_object_copies":8792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2198,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5036,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'10438","reported_seq":10537,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:29.415755+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:43:29.415755+0000","last_peered":"2024-09-05T16:43:29.415755+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:43:29.415755+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"29'400","ondisk_log_start":"29'400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17667232,"num_objects":2157,"num_object_clones":0,"num_object_copies":8628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2157,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10436,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":12}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'10679","reported_seq":10793,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:29.559636+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:43:29.559636+0000","last_peered":"2024-09-05T16:43:29.559636+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:43:29.559636+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"29'600","ondisk_log_start":"29'600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17145856,"num_objects":2093,"num_object_clones":0,"num_object_copies":8372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2093,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10679,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-126}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'10478","reported_seq":10586,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:29.285691+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:43:29.285691+0000","last_peered":"2024-09-05T16:43:29.285691+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:43:29.285691+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"29'400","ondisk_log_start":"29'400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17888416,"num_objects":2184,"num_object_clones":0,"num_object_copies":8736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2184,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10476,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":53}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'10610","reported_seq":10699,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:29.201196+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:43:29.201196+0000","last_peered":"2024-09-05T16:43:29.201196+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:43:29.201196+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"29'600","ondisk_log_start":"29'600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17765536,"num_objects":2169,"num_object_clones":0,"num_object_copies":8676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2169,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10608,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-63}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'10648","reported_seq":10749,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:29.249920+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:43:29.249920+0000","last_peered":"2024-09-05T16:43:29.249920+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:43:29.249920+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"29'600","ondisk_log_start":"29'600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17410368,"num_objects":2126,"num_object_clones":0,"num_object_copies":8504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2126,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10645,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-118}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'10690","reported_seq":10808,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:28.978737+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:43:28.978737+0000","last_peered":"2024-09-05T16:43:28.978737+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:43:28.978737+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"29'600","ondisk_log_start":"29'600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17364128,"num_objects":2120,"num_object_clones":0,"num_object_copies":8480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2120,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10688,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-125}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'10923","reported_seq":11050,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:29.471465+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:43:29.471465+0000","last_peered":"2024-09-05T16:43:29.471465+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:43:29.471465+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"29'900","ondisk_log_start":"29'900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18018032,"num_objects":2200,"num_object_clones":0,"num_object_copies":8800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2200,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10920,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-81}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'10563","reported_seq":10661,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:29.596275+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:43:29.596275+0000","last_peered":"2024-09-05T16:43:29.596275+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:43:29.596275+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"29'500","ondisk_log_start":"29'500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17988176,"num_objects":2196,"num_object_clones":0,"num_object_copies":8784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2196,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10562,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-42}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'10709","reported_seq":15445,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:43:29.168834+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:43:29.168834+0000","last_peered":"2024-09-05T16:43:29.168834+0000","last_clean":"2024-09-05T16:43:29.168834+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:43:29.168834+0000","last_undegraded":"2024-09-05T16:43:29.168834+0000","last_fullsized":"2024-09-05T16:43:29.168834+0000","mapping_epoch":36,"log_start":"29'700","ondisk_log_start":"29'700","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17195008,"num_objects":2099,"num_object_clones":0,"num_object_copies":8396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2099,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2309,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'10922","reported_seq":11031,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:29.414290+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:43:29.414290+0000","last_peered":"2024-09-05T16:43:29.414290+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:43:29.414290+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"29'900","ondisk_log_start":"29'900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18052256,"num_objects":2204,"num_object_clones":0,"num_object_copies":8816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2204,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10920,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-130}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'10719","reported_seq":10832,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:28.925396+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:43:28.925396+0000","last_peered":"2024-09-05T16:43:28.925396+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:43:28.925396+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"29'700","ondisk_log_start":"29'700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17314976,"num_objects":2114,"num_object_clones":0,"num_object_copies":8456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2114,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10717,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-194}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'10659","reported_seq":10768,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:29.151792+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:43:29.151792+0000","last_peered":"2024-09-05T16:43:29.151792+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:43:29.151792+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"29'600","ondisk_log_start":"29'600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17331360,"num_objects":2116,"num_object_clones":0,"num_object_copies":8464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2116,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10657,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-104}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":281522664,"num_objects":34371,"num_object_clones":0,"num_object_copies":137484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34371,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":156466,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2367,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":563331072,"data_stored":563331072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160690,"ondisk_log_size":160690,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084402,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":751288,"kb_used_data":137880,"kb_used_omap":4,"kb_used_meta":613371,"kb_avail":93620552,"statfs":{"total":96636764160,"available":95867445248,"internally_reserved":0,"allocated":141189120,"data_stored":140987846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":628092627},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,0,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986279,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":589356,"kb_used_data":138396,"kb_used_omap":4,"kb_used_meta":450939,"kb_avail":93782484,"statfs":{"total":96636764160,"available":96033263616,"internally_reserved":0,"allocated":141717504,"data_stored":141516230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":461762258},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,6,4,4,8,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673085,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":755100,"kb_used_data":138060,"kb_used_omap":1,"kb_used_meta":617022,"kb_avail":93616740,"statfs":{"total":96636764160,"available":95863541760,"internally_reserved":0,"allocated":141373440,"data_stored":141172166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":631830981},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,0,3,7,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182516,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":755844,"kb_used_data":138336,"kb_used_omap":4,"kb_used_meta":617467,"kb_avail":93615996,"statfs":{"total":96636764160,"available":95862779904,"internally_reserved":0,"allocated":141656064,"data_stored":141454790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":632286925},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,1,5,6],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":140816384,"data_stored":140816384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":140963840,"data_stored":140963840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":140963840,"data_stored":140963840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":140587008,"data_stored":140587008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:43:34.861 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:43:35.503 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:43:35.503 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:43:35.520 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":337,"stamp":"2024-09-05T16:43:35.019791+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":273248744,"num_objects":33361,"num_object_clones":0,"num_object_copies":133444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29218,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33361,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":157476,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2419,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161000,"ondisk_log_size":161000,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2819092,"kb_used_data":536828,"kb_used_omap":14,"kb_used_meta":2282161,"kb_avail":374668268,"statfs":{"total":386547056640,"available":383660306432,"internally_reserved":0,"allocated":549711872,"data_stored":548906776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2336933431},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,17,6,10,15,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25018368,"num_objects":-3054,"num_object_clones":0,"num_object_copies":-12216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-181,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3054,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3054,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":181,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004724"},"pg_stats":[{"pgid":"1.f","version":"37'10784","reported_seq":10889,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:34.333394+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:43:34.333394+0000","last_peered":"2024-09-05T16:43:34.333394+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:43:34.333394+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"29'700","ondisk_log_start":"29'700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17367040,"num_objects":2120,"num_object_clones":0,"num_object_copies":8480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2120,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10784,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-116}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'10543","reported_seq":14895,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:34.253768+0000","last_change":"2024-09-05T16:40:28.919189+0000","last_active":"2024-09-05T16:43:34.253768+0000","last_peered":"2024-09-05T16:43:34.253768+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:43:34.253768+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"29'500","ondisk_log_start":"29'500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16471200,"num_objects":2011,"num_object_clones":0,"num_object_copies":8044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":131,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2011,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10541,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2140,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1880}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'10684","reported_seq":10806,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:34.185504+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:43:34.185504+0000","last_peered":"2024-09-05T16:43:34.185504+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:43:34.185504+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"29'600","ondisk_log_start":"29'600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16939624,"num_objects":2069,"num_object_clones":0,"num_object_copies":8276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2069,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10683,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-163}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'10793","reported_seq":15522,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:43:34.106699+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:43:34.106699+0000","last_peered":"2024-09-05T16:43:34.106699+0000","last_clean":"2024-09-05T16:43:34.106699+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:43:34.106699+0000","last_undegraded":"2024-09-05T16:43:34.106699+0000","last_fullsized":"2024-09-05T16:43:34.106699+0000","mapping_epoch":34,"log_start":"29'700","ondisk_log_start":"29'700","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17539072,"num_objects":2141,"num_object_clones":0,"num_object_copies":8564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2141,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5093,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'10499","reported_seq":10598,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:34.541017+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:43:34.541017+0000","last_peered":"2024-09-05T16:43:34.541017+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:43:34.541017+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"29'400","ondisk_log_start":"29'400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17167520,"num_objects":2096,"num_object_clones":0,"num_object_copies":8384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2096,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10497,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-49}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'10745","reported_seq":10860,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:34.679680+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:43:34.679680+0000","last_peered":"2024-09-05T16:43:34.679680+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:43:34.679680+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"29'700","ondisk_log_start":"29'700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16605184,"num_objects":2027,"num_object_clones":0,"num_object_copies":8108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2027,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10745,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-192}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'10541","reported_seq":10649,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:34.265569+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:43:34.265569+0000","last_peered":"2024-09-05T16:43:34.265569+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:43:34.265569+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"29'500","ondisk_log_start":"29'500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17372320,"num_objects":2121,"num_object_clones":0,"num_object_copies":8484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2121,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10539,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-10}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'10684","reported_seq":10773,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:34.253405+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:43:34.253405+0000","last_peered":"2024-09-05T16:43:34.253405+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:43:34.253405+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"29'600","ondisk_log_start":"29'600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17159328,"num_objects":2095,"num_object_clones":0,"num_object_copies":8380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2095,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10682,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-137}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'10715","reported_seq":10816,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:34.632867+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:43:34.632867+0000","last_peered":"2024-09-05T16:43:34.632867+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:43:34.632867+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"29'700","ondisk_log_start":"29'700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16861504,"num_objects":2059,"num_object_clones":0,"num_object_copies":8236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2059,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10712,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-185}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'10746","reported_seq":10868,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:34.075269+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:43:34.075269+0000","last_peered":"2024-09-05T16:43:34.075269+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:43:34.075269+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"29'700","ondisk_log_start":"29'700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16905376,"num_objects":2064,"num_object_clones":0,"num_object_copies":8256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2064,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10744,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-181}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'10990","reported_seq":11117,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:34.705139+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:43:34.705139+0000","last_peered":"2024-09-05T16:43:34.705139+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:43:34.705139+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"29'900","ondisk_log_start":"29'900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17469168,"num_objects":2133,"num_object_clones":0,"num_object_copies":8532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2133,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10987,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-148}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'10624","reported_seq":10722,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:34.502304+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:43:34.502304+0000","last_peered":"2024-09-05T16:43:34.502304+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:43:34.502304+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"29'600","ondisk_log_start":"29'600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17488464,"num_objects":2135,"num_object_clones":0,"num_object_copies":8540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2135,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10623,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-103}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'10774","reported_seq":15510,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:43:34.109219+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:43:34.109219+0000","last_peered":"2024-09-05T16:43:34.109219+0000","last_clean":"2024-09-05T16:43:34.109219+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:43:34.109219+0000","last_undegraded":"2024-09-05T16:43:34.109219+0000","last_fullsized":"2024-09-05T16:43:34.109219+0000","mapping_epoch":36,"log_start":"29'700","ondisk_log_start":"29'700","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16662528,"num_objects":2034,"num_object_clones":0,"num_object_copies":8136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2034,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2374,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'10977","reported_seq":11086,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:34.664087+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:43:34.664087+0000","last_peered":"2024-09-05T16:43:34.664087+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:43:34.664087+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"29'900","ondisk_log_start":"29'900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17601696,"num_objects":2149,"num_object_clones":0,"num_object_copies":8596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2149,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10975,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-185}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'10777","reported_seq":10890,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:34.009099+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:43:34.009099+0000","last_peered":"2024-09-05T16:43:34.009099+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:43:34.009099+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"29'700","ondisk_log_start":"29'700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16839840,"num_objects":2056,"num_object_clones":0,"num_object_copies":8224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2056,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10775,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-252}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'10724","reported_seq":10832,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:34.105189+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:43:34.105189+0000","last_peered":"2024-09-05T16:43:34.105189+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:43:34.105189+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"29'700","ondisk_log_start":"29'700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16798880,"num_objects":2051,"num_object_clones":0,"num_object_copies":8204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2051,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10722,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-169}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":273248744,"num_objects":33361,"num_object_clones":0,"num_object_copies":133444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29218,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33361,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":157476,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2419,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":546725888,"data_stored":546725888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161000,"ondisk_log_size":161000,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084403,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":751508,"kb_used_data":133948,"kb_used_omap":4,"kb_used_meta":617531,"kb_avail":93620332,"statfs":{"total":96636764160,"available":95867219968,"internally_reserved":0,"allocated":137162752,"data_stored":136961478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":632352467},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,12,0,2,6,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986280,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":589532,"kb_used_data":134404,"kb_used_omap":4,"kb_used_meta":455099,"kb_avail":93782308,"statfs":{"total":96636764160,"available":96033083392,"internally_reserved":0,"allocated":137629696,"data_stored":137428422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":466022098},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,2,1,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673086,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":738696,"kb_used_data":134136,"kb_used_omap":1,"kb_used_meta":604542,"kb_avail":93633144,"statfs":{"total":96636764160,"available":95880339456,"internally_reserved":0,"allocated":137355264,"data_stored":137153990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":619051461},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182517,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":739356,"kb_used_data":134340,"kb_used_omap":4,"kb_used_meta":604987,"kb_avail":93632484,"statfs":{"total":96636764160,"available":95879663616,"internally_reserved":0,"allocated":137564160,"data_stored":137362886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":619507405},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,6,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":136716288,"data_stored":136716288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":136761344,"data_stored":136761344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":136863744,"data_stored":136863744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":136384512,"data_stored":136384512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:43:35.521 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-05T16:43:35.521 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'10784", 'reported_seq': 10889, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:34.333394+0000', 'last_change': '2024-09-05T16:34:56.833497+0000', 'last_active': '2024-09-05T16:43:34.333394+0000', 'last_peered': '2024-09-05T16:43:34.333394+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.828289+0000', 'last_became_peered': '2024-09-05T16:34:56.828289+0000', 'last_unstale': '2024-09-05T16:43:34.333394+0000', 'last_undegraded': '2024-09-05T16:34:56.612160+0000', 'last_fullsized': '2024-09-05T16:34:56.557632+0000', 'mapping_epoch': 32, 'log_start': "29'700", 'ondisk_log_start': "29'700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10084, 'log_dups_size': 0, 'ondisk_log_size': 10084, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17367040, 'num_objects': 2120, 'num_object_clones': 0, 'num_object_copies': 8480, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2236, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2120, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10784, 'num_write_kb': 51616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -116}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:43:35.521 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-05T16:43:35.521 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "37'10543", 'reported_seq': 14895, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:34.253768+0000', 'last_change': '2024-09-05T16:40:28.919189+0000', 'last_active': '2024-09-05T16:43:34.253768+0000', 'last_peered': '2024-09-05T16:43:34.253768+0000', 'last_clean': '2024-09-05T16:33:53.200762+0000', 'last_became_active': '2024-09-05T16:34:56.820292+0000', 'last_became_peered': '2024-09-05T16:34:56.820292+0000', 'last_unstale': '2024-09-05T16:43:34.253768+0000', 'last_undegraded': '2024-09-05T16:34:56.605776+0000', 'last_fullsized': '2024-09-05T16:34:56.559383+0000', 'mapping_epoch': 32, 'log_start': "29'500", 'ondisk_log_start': "29'500", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10043, 'log_dups_size': 0, 'ondisk_log_size': 10043, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16471200, 'num_objects': 2011, 'num_object_clones': 0, 'num_object_copies': 8044, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 131, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2011, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10541, 'num_write_kb': 50212, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2140, 'num_bytes_recovered': 2613248, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 1880}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:43:35.521 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:43:35.521 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'10684", 'reported_seq': 10806, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:34.185504+0000', 'last_change': '2024-09-05T16:34:56.867900+0000', 'last_active': '2024-09-05T16:43:34.185504+0000', 'last_peered': '2024-09-05T16:43:34.185504+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:43:34.185504+0000', 'last_undegraded': '2024-09-05T16:34:56.590972+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 32, 'log_start': "29'600", 'ondisk_log_start': "29'600", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10084, 'log_dups_size': 0, 'ondisk_log_size': 10084, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16939624, 'num_objects': 2069, 'num_object_clones': 0, 'num_object_copies': 8276, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2233, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2069, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 10683, 'num_write_kb': 50999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -163}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:43:35.522 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:43:35.522 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'10499", 'reported_seq': 10598, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:34.541017+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:43:34.541017+0000', 'last_peered': '2024-09-05T16:43:34.541017+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:43:34.541017+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "29'400", 'ondisk_log_start': "29'400", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10099, 'log_dups_size': 0, 'ondisk_log_size': 10099, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17167520, 'num_objects': 2096, 'num_object_clones': 0, 'num_object_copies': 8384, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2147, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2096, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10497, 'num_write_kb': 50373, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -49}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:43:35.522 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:43:35.522 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'10745", 'reported_seq': 10860, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:34.679680+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:43:34.679680+0000', 'last_peered': '2024-09-05T16:43:34.679680+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:43:34.679680+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "29'700", 'ondisk_log_start': "29'700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10045, 'log_dups_size': 0, 'ondisk_log_size': 10045, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16605184, 'num_objects': 2027, 'num_object_clones': 0, 'num_object_copies': 8108, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2027, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10745, 'num_write_kb': 51088, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -192}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:43:35.522 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:43:35.522 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'10541", 'reported_seq': 10649, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:34.265569+0000', 'last_change': '2024-09-05T16:34:56.903436+0000', 'last_active': '2024-09-05T16:43:34.265569+0000', 'last_peered': '2024-09-05T16:43:34.265569+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:43:34.265569+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "29'500", 'ondisk_log_start': "29'500", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10041, 'log_dups_size': 0, 'ondisk_log_size': 10041, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17372320, 'num_objects': 2121, 'num_object_clones': 0, 'num_object_copies': 8484, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2133, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2121, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10539, 'num_write_kb': 50636, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -10}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:43:35.522 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-05T16:43:35.522 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "37'10684", 'reported_seq': 10773, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:34.253405+0000', 'last_change': '2024-09-05T16:34:56.883319+0000', 'last_active': '2024-09-05T16:43:34.253405+0000', 'last_peered': '2024-09-05T16:43:34.253405+0000', 'last_clean': '2024-09-05T16:33:53.505305+0000', 'last_became_active': '2024-09-05T16:34:56.873389+0000', 'last_became_peered': '2024-09-05T16:34:56.873389+0000', 'last_unstale': '2024-09-05T16:43:34.253405+0000', 'last_undegraded': '2024-09-05T16:34:56.602567+0000', 'last_fullsized': '2024-09-05T16:34:56.559958+0000', 'mapping_epoch': 32, 'log_start': "29'600", 'ondisk_log_start': "29'600", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10084, 'log_dups_size': 0, 'ondisk_log_size': 10084, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17159328, 'num_objects': 2095, 'num_object_clones': 0, 'num_object_copies': 8380, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2234, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2095, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10682, 'num_write_kb': 51108, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -137}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:43:35.522 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:43:35.522 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'10715", 'reported_seq': 10816, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:34.632867+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:43:34.632867+0000', 'last_peered': '2024-09-05T16:43:34.632867+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:43:34.632867+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "29'700", 'ondisk_log_start': "29'700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10015, 'log_dups_size': 0, 'ondisk_log_size': 10015, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16861504, 'num_objects': 2059, 'num_object_clones': 0, 'num_object_copies': 8236, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2059, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10712, 'num_write_kb': 51081, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -185}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:43:35.522 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:43:35.522 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'10746", 'reported_seq': 10868, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:34.075269+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:43:34.075269+0000', 'last_peered': '2024-09-05T16:43:34.075269+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:43:34.075269+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "29'700", 'ondisk_log_start': "29'700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10046, 'log_dups_size': 0, 'ondisk_log_size': 10046, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16905376, 'num_objects': 2064, 'num_object_clones': 0, 'num_object_copies': 8256, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2064, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10744, 'num_write_kb': 51230, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -181}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:43:35.523 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:43:35.523 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'10990", 'reported_seq': 11117, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:34.705139+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:43:34.705139+0000', 'last_peered': '2024-09-05T16:43:34.705139+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:43:34.705139+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "29'900", 'ondisk_log_start': "29'900", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10090, 'log_dups_size': 0, 'ondisk_log_size': 10090, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17469168, 'num_objects': 2133, 'num_object_clones': 0, 'num_object_copies': 8532, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2284, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2133, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10987, 'num_write_kb': 52476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -148}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:43:35.523 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:43:35.523 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'10624", 'reported_seq': 10722, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:34.502304+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:43:34.502304+0000', 'last_peered': '2024-09-05T16:43:34.502304+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:43:34.502304+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "29'600", 'ondisk_log_start': "29'600", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10024, 'log_dups_size': 0, 'ondisk_log_size': 10024, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17488464, 'num_objects': 2135, 'num_object_clones': 0, 'num_object_copies': 8540, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2135, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10623, 'num_write_kb': 51031, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -103}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:43:35.523 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:43:35.523 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'10977", 'reported_seq': 11086, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:34.664087+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:43:34.664087+0000', 'last_peered': '2024-09-05T16:43:34.664087+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:43:34.664087+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "29'900", 'ondisk_log_start': "29'900", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10077, 'log_dups_size': 0, 'ondisk_log_size': 10077, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17601696, 'num_objects': 2149, 'num_object_clones': 0, 'num_object_copies': 8596, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2149, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10975, 'num_write_kb': 52500, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -185}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:43:35.523 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:43:35.523 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'10777", 'reported_seq': 10890, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:34.009099+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:43:34.009099+0000', 'last_peered': '2024-09-05T16:43:34.009099+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:43:34.009099+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "29'700", 'ondisk_log_start': "29'700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10077, 'log_dups_size': 0, 'ondisk_log_size': 10077, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16839840, 'num_objects': 2056, 'num_object_clones': 0, 'num_object_copies': 8224, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2310, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2056, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10775, 'num_write_kb': 51325, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -252}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:43:35.523 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:43:35.523 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'10724", 'reported_seq': 10832, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:34.105189+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:43:34.105189+0000', 'last_peered': '2024-09-05T16:43:34.105189+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:43:34.105189+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "29'700", 'ondisk_log_start': "29'700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10024, 'log_dups_size': 0, 'ondisk_log_size': 10024, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16798880, 'num_objects': 2051, 'num_object_clones': 0, 'num_object_copies': 8204, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2051, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10722, 'num_write_kb': 51092, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -169}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:43:35.524 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:43:36.234 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:43:36.234 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 29218/133444 objects degraded (21.895%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":619,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":33361,"data_bytes":273248744,"bytes_used":2886750208,"bytes_avail":383660306432,"bytes_total":386547056640,"degraded_objects":29218,"degraded_total":133444,"degraded_ratio":0.21895326878690688,"recovering_objects_per_sec":15,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":181,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":254},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:43:36.265 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:43:36.266 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:43:36.876 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:43:36.877 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:43:36.898 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":337,"stamp":"2024-09-05T16:43:35.019791+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":273248744,"num_objects":33361,"num_object_clones":0,"num_object_copies":133444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29218,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33361,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":157476,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2419,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161000,"ondisk_log_size":161000,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2819092,"kb_used_data":536828,"kb_used_omap":14,"kb_used_meta":2282161,"kb_avail":374668268,"statfs":{"total":386547056640,"available":383660306432,"internally_reserved":0,"allocated":549711872,"data_stored":548906776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2336933431},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,17,6,10,15,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25018368,"num_objects":-3054,"num_object_clones":0,"num_object_copies":-12216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-181,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3054,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3054,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":181,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004724"},"pg_stats":[{"pgid":"1.f","version":"37'10784","reported_seq":10889,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:34.333394+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:43:34.333394+0000","last_peered":"2024-09-05T16:43:34.333394+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:43:34.333394+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"29'700","ondisk_log_start":"29'700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17367040,"num_objects":2120,"num_object_clones":0,"num_object_copies":8480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2120,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10784,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-116}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'10543","reported_seq":14895,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:34.253768+0000","last_change":"2024-09-05T16:40:28.919189+0000","last_active":"2024-09-05T16:43:34.253768+0000","last_peered":"2024-09-05T16:43:34.253768+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:43:34.253768+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"29'500","ondisk_log_start":"29'500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16471200,"num_objects":2011,"num_object_clones":0,"num_object_copies":8044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":131,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2011,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10541,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2140,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1880}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'10684","reported_seq":10806,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:34.185504+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:43:34.185504+0000","last_peered":"2024-09-05T16:43:34.185504+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:43:34.185504+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"29'600","ondisk_log_start":"29'600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16939624,"num_objects":2069,"num_object_clones":0,"num_object_copies":8276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2069,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10683,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-163}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'10793","reported_seq":15522,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:43:34.106699+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:43:34.106699+0000","last_peered":"2024-09-05T16:43:34.106699+0000","last_clean":"2024-09-05T16:43:34.106699+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:43:34.106699+0000","last_undegraded":"2024-09-05T16:43:34.106699+0000","last_fullsized":"2024-09-05T16:43:34.106699+0000","mapping_epoch":34,"log_start":"29'700","ondisk_log_start":"29'700","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17539072,"num_objects":2141,"num_object_clones":0,"num_object_copies":8564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2141,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5093,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'10499","reported_seq":10598,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:34.541017+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:43:34.541017+0000","last_peered":"2024-09-05T16:43:34.541017+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:43:34.541017+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"29'400","ondisk_log_start":"29'400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17167520,"num_objects":2096,"num_object_clones":0,"num_object_copies":8384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2096,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10497,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-49}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'10745","reported_seq":10860,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:34.679680+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:43:34.679680+0000","last_peered":"2024-09-05T16:43:34.679680+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:43:34.679680+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"29'700","ondisk_log_start":"29'700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16605184,"num_objects":2027,"num_object_clones":0,"num_object_copies":8108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2027,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10745,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-192}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'10541","reported_seq":10649,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:34.265569+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:43:34.265569+0000","last_peered":"2024-09-05T16:43:34.265569+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:43:34.265569+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"29'500","ondisk_log_start":"29'500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17372320,"num_objects":2121,"num_object_clones":0,"num_object_copies":8484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2121,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10539,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-10}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'10684","reported_seq":10773,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:34.253405+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:43:34.253405+0000","last_peered":"2024-09-05T16:43:34.253405+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:43:34.253405+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"29'600","ondisk_log_start":"29'600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17159328,"num_objects":2095,"num_object_clones":0,"num_object_copies":8380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2095,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10682,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-137}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'10715","reported_seq":10816,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:34.632867+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:43:34.632867+0000","last_peered":"2024-09-05T16:43:34.632867+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:43:34.632867+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"29'700","ondisk_log_start":"29'700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16861504,"num_objects":2059,"num_object_clones":0,"num_object_copies":8236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2059,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10712,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-185}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'10746","reported_seq":10868,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:34.075269+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:43:34.075269+0000","last_peered":"2024-09-05T16:43:34.075269+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:43:34.075269+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"29'700","ondisk_log_start":"29'700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16905376,"num_objects":2064,"num_object_clones":0,"num_object_copies":8256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2064,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10744,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-181}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'10990","reported_seq":11117,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:34.705139+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:43:34.705139+0000","last_peered":"2024-09-05T16:43:34.705139+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:43:34.705139+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"29'900","ondisk_log_start":"29'900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17469168,"num_objects":2133,"num_object_clones":0,"num_object_copies":8532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2133,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10987,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-148}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'10624","reported_seq":10722,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:34.502304+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:43:34.502304+0000","last_peered":"2024-09-05T16:43:34.502304+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:43:34.502304+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"29'600","ondisk_log_start":"29'600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17488464,"num_objects":2135,"num_object_clones":0,"num_object_copies":8540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2135,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10623,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-103}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'10774","reported_seq":15510,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:43:34.109219+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:43:34.109219+0000","last_peered":"2024-09-05T16:43:34.109219+0000","last_clean":"2024-09-05T16:43:34.109219+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:43:34.109219+0000","last_undegraded":"2024-09-05T16:43:34.109219+0000","last_fullsized":"2024-09-05T16:43:34.109219+0000","mapping_epoch":36,"log_start":"29'700","ondisk_log_start":"29'700","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16662528,"num_objects":2034,"num_object_clones":0,"num_object_copies":8136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2034,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2374,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'10977","reported_seq":11086,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:34.664087+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:43:34.664087+0000","last_peered":"2024-09-05T16:43:34.664087+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:43:34.664087+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"29'900","ondisk_log_start":"29'900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17601696,"num_objects":2149,"num_object_clones":0,"num_object_copies":8596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2149,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10975,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-185}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'10777","reported_seq":10890,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:34.009099+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:43:34.009099+0000","last_peered":"2024-09-05T16:43:34.009099+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:43:34.009099+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"29'700","ondisk_log_start":"29'700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16839840,"num_objects":2056,"num_object_clones":0,"num_object_copies":8224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2056,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10775,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-252}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'10724","reported_seq":10832,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:34.105189+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:43:34.105189+0000","last_peered":"2024-09-05T16:43:34.105189+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:43:34.105189+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"29'700","ondisk_log_start":"29'700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16798880,"num_objects":2051,"num_object_clones":0,"num_object_copies":8204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2051,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10722,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-169}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":273248744,"num_objects":33361,"num_object_clones":0,"num_object_copies":133444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29218,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33361,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":157476,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2419,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":546725888,"data_stored":546725888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161000,"ondisk_log_size":161000,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084403,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":751508,"kb_used_data":133948,"kb_used_omap":4,"kb_used_meta":617531,"kb_avail":93620332,"statfs":{"total":96636764160,"available":95867219968,"internally_reserved":0,"allocated":137162752,"data_stored":136961478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":632352467},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,12,0,2,6,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986280,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":589532,"kb_used_data":134404,"kb_used_omap":4,"kb_used_meta":455099,"kb_avail":93782308,"statfs":{"total":96636764160,"available":96033083392,"internally_reserved":0,"allocated":137629696,"data_stored":137428422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":466022098},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,2,1,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673086,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":738696,"kb_used_data":134136,"kb_used_omap":1,"kb_used_meta":604542,"kb_avail":93633144,"statfs":{"total":96636764160,"available":95880339456,"internally_reserved":0,"allocated":137355264,"data_stored":137153990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":619051461},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182517,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":739356,"kb_used_data":134340,"kb_used_omap":4,"kb_used_meta":604987,"kb_avail":93632484,"statfs":{"total":96636764160,"available":95879663616,"internally_reserved":0,"allocated":137564160,"data_stored":137362886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":619507405},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,6,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":136716288,"data_stored":136716288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":136761344,"data_stored":136761344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":136863744,"data_stored":136863744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":136384512,"data_stored":136384512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:43:39.900 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:43:40.477 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:43:40.478 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:43:40.499 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":339,"stamp":"2024-09-05T16:43:39.021515+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":273248744,"num_objects":33361,"num_object_clones":0,"num_object_copies":133444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29218,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33361,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":157476,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2419,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161000,"ondisk_log_size":161000,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2819092,"kb_used_data":536828,"kb_used_omap":14,"kb_used_meta":2282161,"kb_avail":374668268,"statfs":{"total":386547056640,"available":383660306432,"internally_reserved":0,"allocated":549711872,"data_stored":548906776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2336933431},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,17,6,10,15,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16793600,"num_objects":-2050,"num_object_clones":0,"num_object_copies":-8200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2050,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2050,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":118,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004793"},"pg_stats":[{"pgid":"1.f","version":"37'10784","reported_seq":10889,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:34.333394+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:43:34.333394+0000","last_peered":"2024-09-05T16:43:34.333394+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:43:34.333394+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"29'700","ondisk_log_start":"29'700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17367040,"num_objects":2120,"num_object_clones":0,"num_object_copies":8480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2120,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10784,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-116}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'10543","reported_seq":14895,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:34.253768+0000","last_change":"2024-09-05T16:40:28.919189+0000","last_active":"2024-09-05T16:43:34.253768+0000","last_peered":"2024-09-05T16:43:34.253768+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:43:34.253768+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"29'500","ondisk_log_start":"29'500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16471200,"num_objects":2011,"num_object_clones":0,"num_object_copies":8044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":131,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2011,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10541,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2140,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1880}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'10684","reported_seq":10806,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:34.185504+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:43:34.185504+0000","last_peered":"2024-09-05T16:43:34.185504+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:43:34.185504+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"29'600","ondisk_log_start":"29'600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16939624,"num_objects":2069,"num_object_clones":0,"num_object_copies":8276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2069,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10683,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-163}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'10793","reported_seq":15522,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:43:34.106699+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:43:34.106699+0000","last_peered":"2024-09-05T16:43:34.106699+0000","last_clean":"2024-09-05T16:43:34.106699+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:43:34.106699+0000","last_undegraded":"2024-09-05T16:43:34.106699+0000","last_fullsized":"2024-09-05T16:43:34.106699+0000","mapping_epoch":34,"log_start":"29'700","ondisk_log_start":"29'700","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17539072,"num_objects":2141,"num_object_clones":0,"num_object_copies":8564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2141,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5093,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'10499","reported_seq":10598,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:34.541017+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:43:34.541017+0000","last_peered":"2024-09-05T16:43:34.541017+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:43:34.541017+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"29'400","ondisk_log_start":"29'400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17167520,"num_objects":2096,"num_object_clones":0,"num_object_copies":8384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2096,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10497,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-49}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'10745","reported_seq":10860,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:34.679680+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:43:34.679680+0000","last_peered":"2024-09-05T16:43:34.679680+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:43:34.679680+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"29'700","ondisk_log_start":"29'700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16605184,"num_objects":2027,"num_object_clones":0,"num_object_copies":8108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2027,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10745,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-192}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'10541","reported_seq":10649,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:34.265569+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:43:34.265569+0000","last_peered":"2024-09-05T16:43:34.265569+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:43:34.265569+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"29'500","ondisk_log_start":"29'500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17372320,"num_objects":2121,"num_object_clones":0,"num_object_copies":8484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2121,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10539,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-10}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'10684","reported_seq":10773,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:34.253405+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:43:34.253405+0000","last_peered":"2024-09-05T16:43:34.253405+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:43:34.253405+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"29'600","ondisk_log_start":"29'600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17159328,"num_objects":2095,"num_object_clones":0,"num_object_copies":8380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2095,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10682,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-137}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'10715","reported_seq":10816,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:34.632867+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:43:34.632867+0000","last_peered":"2024-09-05T16:43:34.632867+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:43:34.632867+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"29'700","ondisk_log_start":"29'700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16861504,"num_objects":2059,"num_object_clones":0,"num_object_copies":8236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2059,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10712,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-185}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'10746","reported_seq":10868,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:34.075269+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:43:34.075269+0000","last_peered":"2024-09-05T16:43:34.075269+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:43:34.075269+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"29'700","ondisk_log_start":"29'700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16905376,"num_objects":2064,"num_object_clones":0,"num_object_copies":8256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2064,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10744,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-181}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'10990","reported_seq":11117,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:34.705139+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:43:34.705139+0000","last_peered":"2024-09-05T16:43:34.705139+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:43:34.705139+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"29'900","ondisk_log_start":"29'900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17469168,"num_objects":2133,"num_object_clones":0,"num_object_copies":8532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2133,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10987,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-148}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'10624","reported_seq":10722,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:34.502304+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:43:34.502304+0000","last_peered":"2024-09-05T16:43:34.502304+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:43:34.502304+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"29'600","ondisk_log_start":"29'600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17488464,"num_objects":2135,"num_object_clones":0,"num_object_copies":8540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2135,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10623,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-103}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'10774","reported_seq":15510,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:43:34.109219+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:43:34.109219+0000","last_peered":"2024-09-05T16:43:34.109219+0000","last_clean":"2024-09-05T16:43:34.109219+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:43:34.109219+0000","last_undegraded":"2024-09-05T16:43:34.109219+0000","last_fullsized":"2024-09-05T16:43:34.109219+0000","mapping_epoch":36,"log_start":"29'700","ondisk_log_start":"29'700","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16662528,"num_objects":2034,"num_object_clones":0,"num_object_copies":8136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2034,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2374,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'10977","reported_seq":11086,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:34.664087+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:43:34.664087+0000","last_peered":"2024-09-05T16:43:34.664087+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:43:34.664087+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"29'900","ondisk_log_start":"29'900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17601696,"num_objects":2149,"num_object_clones":0,"num_object_copies":8596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2149,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10975,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-185}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'10777","reported_seq":10890,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:34.009099+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:43:34.009099+0000","last_peered":"2024-09-05T16:43:34.009099+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:43:34.009099+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"29'700","ondisk_log_start":"29'700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16839840,"num_objects":2056,"num_object_clones":0,"num_object_copies":8224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2056,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10775,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-252}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'10724","reported_seq":10832,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:34.105189+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:43:34.105189+0000","last_peered":"2024-09-05T16:43:34.105189+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:43:34.105189+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"29'700","ondisk_log_start":"29'700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16798880,"num_objects":2051,"num_object_clones":0,"num_object_copies":8204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2051,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10722,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-169}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":273248744,"num_objects":33361,"num_object_clones":0,"num_object_copies":133444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29218,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33361,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":157476,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2419,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":546725888,"data_stored":546725888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161000,"ondisk_log_size":161000,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084403,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":751508,"kb_used_data":133948,"kb_used_omap":4,"kb_used_meta":617531,"kb_avail":93620332,"statfs":{"total":96636764160,"available":95867219968,"internally_reserved":0,"allocated":137162752,"data_stored":136961478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":632352467},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,12,0,2,6,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986280,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":589532,"kb_used_data":134404,"kb_used_omap":4,"kb_used_meta":455099,"kb_avail":93782308,"statfs":{"total":96636764160,"available":96033083392,"internally_reserved":0,"allocated":137629696,"data_stored":137428422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":466022098},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,2,1,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673086,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":738696,"kb_used_data":134136,"kb_used_omap":1,"kb_used_meta":604542,"kb_avail":93633144,"statfs":{"total":96636764160,"available":95880339456,"internally_reserved":0,"allocated":137355264,"data_stored":137153990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":619051461},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182517,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":739356,"kb_used_data":134340,"kb_used_omap":4,"kb_used_meta":604987,"kb_avail":93632484,"statfs":{"total":96636764160,"available":95879663616,"internally_reserved":0,"allocated":137564160,"data_stored":137362886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":619507405},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,6,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":136716288,"data_stored":136716288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":136761344,"data_stored":136761344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":136863744,"data_stored":136863744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":136384512,"data_stored":136384512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:43:40.500 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:43:41.128 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:43:41.129 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:43:41.152 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":340,"stamp":"2024-09-05T16:43:41.022288+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":264942056,"num_objects":32347,"num_object_clones":0,"num_object_copies":129388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29141,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32347,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":158490,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2496,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160914,"ondisk_log_size":160914,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2804500,"kb_used_data":520792,"kb_used_omap":14,"kb_used_meta":2283569,"kb_avail":374682860,"statfs":{"total":386547056640,"available":383675248640,"internally_reserved":0,"allocated":533291008,"data_stored":532485912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2338375223},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,4,3,21,12,8,10],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25100288,"num_objects":-3064,"num_object_clones":0,"num_object_copies":-12256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-195,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3064,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3064,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":195,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004725"},"pg_stats":[{"pgid":"1.f","version":"37'10857","reported_seq":10959,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:39.713868+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:43:39.713868+0000","last_peered":"2024-09-05T16:43:39.713868+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:43:39.713868+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"29'800","ondisk_log_start":"29'800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16769024,"num_objects":2047,"num_object_clones":0,"num_object_copies":8188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2047,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10857,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-189}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'10596","reported_seq":15102,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:39.400954+0000","last_change":"2024-09-05T16:40:28.919189+0000","last_active":"2024-09-05T16:43:39.400954+0000","last_peered":"2024-09-05T16:43:39.400954+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:43:39.400954+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"29'500","ondisk_log_start":"29'500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16037024,"num_objects":1958,"num_object_clones":0,"num_object_copies":7832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":54,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1958,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10594,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2217,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1904}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'10748","reported_seq":10871,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:39.135340+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:43:39.135340+0000","last_peered":"2024-09-05T16:43:39.135340+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:43:39.135340+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"29'700","ondisk_log_start":"29'700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16415336,"num_objects":2005,"num_object_clones":0,"num_object_copies":8020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2005,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10747,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-227}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'10875","reported_seq":15605,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:43:39.572563+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:43:39.572563+0000","last_peered":"2024-09-05T16:43:39.572563+0000","last_clean":"2024-09-05T16:43:39.572563+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:43:39.572563+0000","last_undegraded":"2024-09-05T16:43:39.572563+0000","last_fullsized":"2024-09-05T16:43:39.572563+0000","mapping_epoch":34,"log_start":"29'800","ondisk_log_start":"29'800","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16867328,"num_objects":2059,"num_object_clones":0,"num_object_copies":8236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2059,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5175,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'10569","reported_seq":10665,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:39.739395+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:43:39.739395+0000","last_peered":"2024-09-05T16:43:39.739395+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:43:39.739395+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"29'500","ondisk_log_start":"29'500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16594080,"num_objects":2026,"num_object_clones":0,"num_object_copies":8104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2026,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10567,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-119}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'10808","reported_seq":10923,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:39.715323+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:43:39.715323+0000","last_peered":"2024-09-05T16:43:39.715323+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:43:39.715323+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"29'800","ondisk_log_start":"29'800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16089088,"num_objects":1964,"num_object_clones":0,"num_object_copies":7856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1964,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10808,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-255}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'10610","reported_seq":10716,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:39.373324+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:43:39.373324+0000","last_peered":"2024-09-05T16:43:39.373324+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:43:39.373324+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"29'600","ondisk_log_start":"29'600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16807072,"num_objects":2052,"num_object_clones":0,"num_object_copies":8208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2052,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10608,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-79}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'10748","reported_seq":10837,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:39.248309+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:43:39.248309+0000","last_peered":"2024-09-05T16:43:39.248309+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:43:39.248309+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"29'700","ondisk_log_start":"29'700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16635040,"num_objects":2031,"num_object_clones":0,"num_object_copies":8124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2031,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10746,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-201}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'10777","reported_seq":10878,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:39.744541+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:43:39.744541+0000","last_peered":"2024-09-05T16:43:39.744541+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:43:39.744541+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"29'700","ondisk_log_start":"29'700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16353600,"num_objects":1997,"num_object_clones":0,"num_object_copies":7988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10774,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-247}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'10799","reported_seq":10921,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:38.934258+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:43:38.934258+0000","last_peered":"2024-09-05T16:43:38.934258+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:43:38.934258+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"29'700","ondisk_log_start":"29'700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16471200,"num_objects":2011,"num_object_clones":0,"num_object_copies":8044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2011,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10797,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-234}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11048","reported_seq":11175,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:39.628602+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:43:39.628602+0000","last_peered":"2024-09-05T16:43:39.628602+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:43:39.628602+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"29'1000","ondisk_log_start":"29'1000","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16994032,"num_objects":2075,"num_object_clones":0,"num_object_copies":8300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2075,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11045,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-206}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'10699","reported_seq":10798,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:39.743930+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:43:39.743930+0000","last_peered":"2024-09-05T16:43:39.743930+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:43:39.743930+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"29'600","ondisk_log_start":"29'600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16874064,"num_objects":2060,"num_object_clones":0,"num_object_copies":8240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10698,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-178}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'10827","reported_seq":15563,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:43:39.422205+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:43:39.422205+0000","last_peered":"2024-09-05T16:43:39.422205+0000","last_clean":"2024-09-05T16:43:39.422205+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:43:39.422205+0000","last_undegraded":"2024-09-05T16:43:39.422205+0000","last_fullsized":"2024-09-05T16:43:39.422205+0000","mapping_epoch":36,"log_start":"29'800","ondisk_log_start":"29'800","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16228352,"num_objects":1981,"num_object_clones":0,"num_object_copies":7924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1981,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2427,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11045","reported_seq":11154,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:39.812871+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:43:39.812871+0000","last_peered":"2024-09-05T16:43:39.812871+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:43:39.812871+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"29'1000","ondisk_log_start":"29'1000","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17044640,"num_objects":2081,"num_object_clones":0,"num_object_copies":8324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2081,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11043,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-253}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'10832","reported_seq":10945,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:39.132231+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:43:39.132231+0000","last_peered":"2024-09-05T16:43:39.132231+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:43:39.132231+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"29'800","ondisk_log_start":"29'800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16389280,"num_objects":2001,"num_object_clones":0,"num_object_copies":8004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2001,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10830,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-307}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'10776","reported_seq":10885,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:39.099642+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:43:39.099642+0000","last_peered":"2024-09-05T16:43:39.099642+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:43:39.099642+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"29'700","ondisk_log_start":"29'700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16372896,"num_objects":1999,"num_object_clones":0,"num_object_copies":7996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1999,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10774,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-221}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":264942056,"num_objects":32347,"num_object_clones":0,"num_object_copies":129388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29141,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32347,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":158490,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2496,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":530137088,"data_stored":530137088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160914,"ondisk_log_size":160914,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084404,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":735024,"kb_used_data":129948,"kb_used_omap":4,"kb_used_meta":605051,"kb_avail":93636816,"statfs":{"total":96636764160,"available":95884099584,"internally_reserved":0,"allocated":133066752,"data_stored":132865478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":619572947},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,9,1,1,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986281,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":591112,"kb_used_data":130424,"kb_used_omap":4,"kb_used_meta":460667,"kb_avail":93780728,"statfs":{"total":96636764160,"available":96031465472,"internally_reserved":0,"allocated":133554176,"data_stored":133352902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":471723730},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,7,8,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673087,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":738832,"kb_used_data":130092,"kb_used_omap":1,"kb_used_meta":608702,"kb_avail":93633008,"statfs":{"total":96636764160,"available":95880200192,"internally_reserved":0,"allocated":133214208,"data_stored":133012934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":623311301},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,4,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182518,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":739532,"kb_used_data":130328,"kb_used_omap":4,"kb_used_meta":609147,"kb_avail":93632308,"statfs":{"total":96636764160,"available":95879483392,"internally_reserved":0,"allocated":133455872,"data_stored":133254598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":623767245},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,0,1,3,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":132591616,"data_stored":132591616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":132562944,"data_stored":132562944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":132784128,"data_stored":132784128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":132198400,"data_stored":132198400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:43:41.153 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-05T16:43:41.153 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'10857", 'reported_seq': 10959, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:39.713868+0000', 'last_change': '2024-09-05T16:34:56.833497+0000', 'last_active': '2024-09-05T16:43:39.713868+0000', 'last_peered': '2024-09-05T16:43:39.713868+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.828289+0000', 'last_became_peered': '2024-09-05T16:34:56.828289+0000', 'last_unstale': '2024-09-05T16:43:39.713868+0000', 'last_undegraded': '2024-09-05T16:34:56.612160+0000', 'last_fullsized': '2024-09-05T16:34:56.557632+0000', 'mapping_epoch': 32, 'log_start': "29'800", 'ondisk_log_start': "29'800", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10057, 'log_dups_size': 0, 'ondisk_log_size': 10057, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16769024, 'num_objects': 2047, 'num_object_clones': 0, 'num_object_copies': 8188, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2236, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2047, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10857, 'num_write_kb': 51616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -189}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:43:41.153 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-05T16:43:41.153 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "37'10596", 'reported_seq': 15102, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:39.400954+0000', 'last_change': '2024-09-05T16:40:28.919189+0000', 'last_active': '2024-09-05T16:43:39.400954+0000', 'last_peered': '2024-09-05T16:43:39.400954+0000', 'last_clean': '2024-09-05T16:33:53.200762+0000', 'last_became_active': '2024-09-05T16:34:56.820292+0000', 'last_became_peered': '2024-09-05T16:34:56.820292+0000', 'last_unstale': '2024-09-05T16:43:39.400954+0000', 'last_undegraded': '2024-09-05T16:34:56.605776+0000', 'last_fullsized': '2024-09-05T16:34:56.559383+0000', 'mapping_epoch': 32, 'log_start': "29'500", 'ondisk_log_start': "29'500", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10096, 'log_dups_size': 0, 'ondisk_log_size': 10096, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16037024, 'num_objects': 1958, 'num_object_clones': 0, 'num_object_copies': 7832, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 54, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1958, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10594, 'num_write_kb': 50212, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2217, 'num_bytes_recovered': 2613248, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 1904}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:43:41.153 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:43:41.153 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'10748", 'reported_seq': 10871, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:39.135340+0000', 'last_change': '2024-09-05T16:34:56.867900+0000', 'last_active': '2024-09-05T16:43:39.135340+0000', 'last_peered': '2024-09-05T16:43:39.135340+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:43:39.135340+0000', 'last_undegraded': '2024-09-05T16:34:56.590972+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 32, 'log_start': "29'700", 'ondisk_log_start': "29'700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10048, 'log_dups_size': 0, 'ondisk_log_size': 10048, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16415336, 'num_objects': 2005, 'num_object_clones': 0, 'num_object_copies': 8020, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2233, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2005, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 10747, 'num_write_kb': 50999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -227}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:43:41.153 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:43:41.153 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'10569", 'reported_seq': 10665, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:39.739395+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:43:39.739395+0000', 'last_peered': '2024-09-05T16:43:39.739395+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:43:39.739395+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "29'500", 'ondisk_log_start': "29'500", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10069, 'log_dups_size': 0, 'ondisk_log_size': 10069, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16594080, 'num_objects': 2026, 'num_object_clones': 0, 'num_object_copies': 8104, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2147, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2026, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10567, 'num_write_kb': 50373, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -119}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:43:41.154 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:43:41.154 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'10808", 'reported_seq': 10923, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:39.715323+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:43:39.715323+0000', 'last_peered': '2024-09-05T16:43:39.715323+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:43:39.715323+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "29'800", 'ondisk_log_start': "29'800", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10008, 'log_dups_size': 0, 'ondisk_log_size': 10008, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16089088, 'num_objects': 1964, 'num_object_clones': 0, 'num_object_copies': 7856, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1964, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10808, 'num_write_kb': 51088, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -255}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:43:41.154 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:43:41.154 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'10610", 'reported_seq': 10716, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:39.373324+0000', 'last_change': '2024-09-05T16:34:56.903436+0000', 'last_active': '2024-09-05T16:43:39.373324+0000', 'last_peered': '2024-09-05T16:43:39.373324+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:43:39.373324+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "29'600", 'ondisk_log_start': "29'600", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10010, 'log_dups_size': 0, 'ondisk_log_size': 10010, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16807072, 'num_objects': 2052, 'num_object_clones': 0, 'num_object_copies': 8208, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2133, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2052, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10608, 'num_write_kb': 50636, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -79}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:43:41.154 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-05T16:43:41.154 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "37'10748", 'reported_seq': 10837, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:39.248309+0000', 'last_change': '2024-09-05T16:34:56.883319+0000', 'last_active': '2024-09-05T16:43:39.248309+0000', 'last_peered': '2024-09-05T16:43:39.248309+0000', 'last_clean': '2024-09-05T16:33:53.505305+0000', 'last_became_active': '2024-09-05T16:34:56.873389+0000', 'last_became_peered': '2024-09-05T16:34:56.873389+0000', 'last_unstale': '2024-09-05T16:43:39.248309+0000', 'last_undegraded': '2024-09-05T16:34:56.602567+0000', 'last_fullsized': '2024-09-05T16:34:56.559958+0000', 'mapping_epoch': 32, 'log_start': "29'700", 'ondisk_log_start': "29'700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10048, 'log_dups_size': 0, 'ondisk_log_size': 10048, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16635040, 'num_objects': 2031, 'num_object_clones': 0, 'num_object_copies': 8124, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2234, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2031, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10746, 'num_write_kb': 51108, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -201}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:43:41.154 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:43:41.154 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'10777", 'reported_seq': 10878, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:39.744541+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:43:39.744541+0000', 'last_peered': '2024-09-05T16:43:39.744541+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:43:39.744541+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "29'700", 'ondisk_log_start': "29'700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10077, 'log_dups_size': 0, 'ondisk_log_size': 10077, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16353600, 'num_objects': 1997, 'num_object_clones': 0, 'num_object_copies': 7988, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1997, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10774, 'num_write_kb': 51081, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -247}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:43:41.154 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:43:41.154 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'10799", 'reported_seq': 10921, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:38.934258+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:43:38.934258+0000', 'last_peered': '2024-09-05T16:43:38.934258+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:43:38.934258+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "29'700", 'ondisk_log_start': "29'700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10099, 'log_dups_size': 0, 'ondisk_log_size': 10099, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16471200, 'num_objects': 2011, 'num_object_clones': 0, 'num_object_copies': 8044, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2011, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10797, 'num_write_kb': 51230, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -234}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:43:41.155 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:43:41.155 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'11048", 'reported_seq': 11175, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:39.628602+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:43:39.628602+0000', 'last_peered': '2024-09-05T16:43:39.628602+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:43:39.628602+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "29'1000", 'ondisk_log_start': "29'1000", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10048, 'log_dups_size': 0, 'ondisk_log_size': 10048, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16994032, 'num_objects': 2075, 'num_object_clones': 0, 'num_object_copies': 8300, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2284, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2075, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11045, 'num_write_kb': 52476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -206}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:43:41.155 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:43:41.155 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'10699", 'reported_seq': 10798, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:39.743930+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:43:39.743930+0000', 'last_peered': '2024-09-05T16:43:39.743930+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:43:39.743930+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "29'600", 'ondisk_log_start': "29'600", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10099, 'log_dups_size': 0, 'ondisk_log_size': 10099, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16874064, 'num_objects': 2060, 'num_object_clones': 0, 'num_object_copies': 8240, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2060, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10698, 'num_write_kb': 51031, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -178}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:43:41.155 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:43:41.155 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'11045", 'reported_seq': 11154, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:39.812871+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:43:39.812871+0000', 'last_peered': '2024-09-05T16:43:39.812871+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:43:39.812871+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "29'1000", 'ondisk_log_start': "29'1000", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10045, 'log_dups_size': 0, 'ondisk_log_size': 10045, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17044640, 'num_objects': 2081, 'num_object_clones': 0, 'num_object_copies': 8324, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2081, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11043, 'num_write_kb': 52500, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -253}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:43:41.155 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:43:41.155 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'10832", 'reported_seq': 10945, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:39.132231+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:43:39.132231+0000', 'last_peered': '2024-09-05T16:43:39.132231+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:43:39.132231+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "29'800", 'ondisk_log_start': "29'800", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10032, 'log_dups_size': 0, 'ondisk_log_size': 10032, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16389280, 'num_objects': 2001, 'num_object_clones': 0, 'num_object_copies': 8004, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2310, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2001, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10830, 'num_write_kb': 51325, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -307}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:43:41.155 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:43:41.155 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'10776", 'reported_seq': 10885, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:39.099642+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:43:39.099642+0000', 'last_peered': '2024-09-05T16:43:39.099642+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:43:39.099642+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "29'700", 'ondisk_log_start': "29'700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10076, 'log_dups_size': 0, 'ondisk_log_size': 10076, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16372896, 'num_objects': 1999, 'num_object_clones': 0, 'num_object_copies': 7996, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1999, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10774, 'num_write_kb': 51092, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -221}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:43:41.156 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:43:41.896 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:43:41.896 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 29141/129388 objects degraded (22.522%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":625,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":32347,"data_bytes":264942056,"bytes_used":2871808000,"bytes_avail":383675248640,"bytes_total":386547056640,"degraded_objects":29141,"degraded_total":129388,"degraded_ratio":0.22522181346029002,"recovering_objects_per_sec":16,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":195,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":255},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:43:41.929 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:43:41.929 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:43:42.510 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:43:42.510 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:43:42.534 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":340,"stamp":"2024-09-05T16:43:41.022288+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":264942056,"num_objects":32347,"num_object_clones":0,"num_object_copies":129388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29141,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32347,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":158490,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2496,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160914,"ondisk_log_size":160914,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2804500,"kb_used_data":520792,"kb_used_omap":14,"kb_used_meta":2283569,"kb_avail":374682860,"statfs":{"total":386547056640,"available":383675248640,"internally_reserved":0,"allocated":533291008,"data_stored":532485912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2338375223},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,4,3,21,12,8,10],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25100288,"num_objects":-3064,"num_object_clones":0,"num_object_copies":-12256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-195,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3064,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3064,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":195,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004725"},"pg_stats":[{"pgid":"1.f","version":"37'10857","reported_seq":10959,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:39.713868+0000","last_change":"2024-09-05T16:34:56.833497+0000","last_active":"2024-09-05T16:43:39.713868+0000","last_peered":"2024-09-05T16:43:39.713868+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:43:39.713868+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"29'800","ondisk_log_start":"29'800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16769024,"num_objects":2047,"num_object_clones":0,"num_object_copies":8188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2047,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10857,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-189}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'10596","reported_seq":15102,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:39.400954+0000","last_change":"2024-09-05T16:40:28.919189+0000","last_active":"2024-09-05T16:43:39.400954+0000","last_peered":"2024-09-05T16:43:39.400954+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:43:39.400954+0000","last_undegraded":"2024-09-05T16:34:56.605776+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":32,"log_start":"29'500","ondisk_log_start":"29'500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16037024,"num_objects":1958,"num_object_clones":0,"num_object_copies":7832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":54,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1958,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10594,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2217,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1904}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'10748","reported_seq":10871,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:39.135340+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:43:39.135340+0000","last_peered":"2024-09-05T16:43:39.135340+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:43:39.135340+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"29'700","ondisk_log_start":"29'700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16415336,"num_objects":2005,"num_object_clones":0,"num_object_copies":8020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2005,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10747,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-227}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'10875","reported_seq":15605,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:43:39.572563+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:43:39.572563+0000","last_peered":"2024-09-05T16:43:39.572563+0000","last_clean":"2024-09-05T16:43:39.572563+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:43:39.572563+0000","last_undegraded":"2024-09-05T16:43:39.572563+0000","last_fullsized":"2024-09-05T16:43:39.572563+0000","mapping_epoch":34,"log_start":"29'800","ondisk_log_start":"29'800","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16867328,"num_objects":2059,"num_object_clones":0,"num_object_copies":8236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2059,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5175,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'10569","reported_seq":10665,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:39.739395+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:43:39.739395+0000","last_peered":"2024-09-05T16:43:39.739395+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:43:39.739395+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"29'500","ondisk_log_start":"29'500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16594080,"num_objects":2026,"num_object_clones":0,"num_object_copies":8104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2026,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10567,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-119}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'10808","reported_seq":10923,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:39.715323+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:43:39.715323+0000","last_peered":"2024-09-05T16:43:39.715323+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:43:39.715323+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"29'800","ondisk_log_start":"29'800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16089088,"num_objects":1964,"num_object_clones":0,"num_object_copies":7856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1964,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10808,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-255}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'10610","reported_seq":10716,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:39.373324+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:43:39.373324+0000","last_peered":"2024-09-05T16:43:39.373324+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:43:39.373324+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"29'600","ondisk_log_start":"29'600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16807072,"num_objects":2052,"num_object_clones":0,"num_object_copies":8208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2052,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10608,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-79}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'10748","reported_seq":10837,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:39.248309+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:43:39.248309+0000","last_peered":"2024-09-05T16:43:39.248309+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:43:39.248309+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"29'700","ondisk_log_start":"29'700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16635040,"num_objects":2031,"num_object_clones":0,"num_object_copies":8124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2031,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10746,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-201}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'10777","reported_seq":10878,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:39.744541+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:43:39.744541+0000","last_peered":"2024-09-05T16:43:39.744541+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:43:39.744541+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"29'700","ondisk_log_start":"29'700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16353600,"num_objects":1997,"num_object_clones":0,"num_object_copies":7988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10774,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-247}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'10799","reported_seq":10921,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:38.934258+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:43:38.934258+0000","last_peered":"2024-09-05T16:43:38.934258+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:43:38.934258+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"29'700","ondisk_log_start":"29'700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16471200,"num_objects":2011,"num_object_clones":0,"num_object_copies":8044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2011,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10797,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-234}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11048","reported_seq":11175,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:39.628602+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:43:39.628602+0000","last_peered":"2024-09-05T16:43:39.628602+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:43:39.628602+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"29'1000","ondisk_log_start":"29'1000","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16994032,"num_objects":2075,"num_object_clones":0,"num_object_copies":8300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2075,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11045,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-206}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'10699","reported_seq":10798,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:39.743930+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:43:39.743930+0000","last_peered":"2024-09-05T16:43:39.743930+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:43:39.743930+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"29'600","ondisk_log_start":"29'600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16874064,"num_objects":2060,"num_object_clones":0,"num_object_copies":8240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10698,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-178}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'10827","reported_seq":15563,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-05T16:43:39.422205+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:43:39.422205+0000","last_peered":"2024-09-05T16:43:39.422205+0000","last_clean":"2024-09-05T16:43:39.422205+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:43:39.422205+0000","last_undegraded":"2024-09-05T16:43:39.422205+0000","last_fullsized":"2024-09-05T16:43:39.422205+0000","mapping_epoch":36,"log_start":"29'800","ondisk_log_start":"29'800","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16228352,"num_objects":1981,"num_object_clones":0,"num_object_copies":7924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1981,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2427,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11045","reported_seq":11154,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:39.812871+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:43:39.812871+0000","last_peered":"2024-09-05T16:43:39.812871+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:43:39.812871+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"29'1000","ondisk_log_start":"29'1000","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17044640,"num_objects":2081,"num_object_clones":0,"num_object_copies":8324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2081,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11043,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-253}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'10832","reported_seq":10945,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:39.132231+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:43:39.132231+0000","last_peered":"2024-09-05T16:43:39.132231+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:43:39.132231+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"29'800","ondisk_log_start":"29'800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16389280,"num_objects":2001,"num_object_clones":0,"num_object_copies":8004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2001,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10830,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-307}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'10776","reported_seq":10885,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:39.099642+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:43:39.099642+0000","last_peered":"2024-09-05T16:43:39.099642+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:43:39.099642+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"29'700","ondisk_log_start":"29'700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16372896,"num_objects":1999,"num_object_clones":0,"num_object_copies":7996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1999,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10774,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-221}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":264942056,"num_objects":32347,"num_object_clones":0,"num_object_copies":129388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29141,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32347,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":158490,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2496,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":530137088,"data_stored":530137088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160914,"ondisk_log_size":160914,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084404,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":735024,"kb_used_data":129948,"kb_used_omap":4,"kb_used_meta":605051,"kb_avail":93636816,"statfs":{"total":96636764160,"available":95884099584,"internally_reserved":0,"allocated":133066752,"data_stored":132865478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":619572947},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,9,1,1,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986281,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":591112,"kb_used_data":130424,"kb_used_omap":4,"kb_used_meta":460667,"kb_avail":93780728,"statfs":{"total":96636764160,"available":96031465472,"internally_reserved":0,"allocated":133554176,"data_stored":133352902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":471723730},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,7,8,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673087,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":738832,"kb_used_data":130092,"kb_used_omap":1,"kb_used_meta":608702,"kb_avail":93633008,"statfs":{"total":96636764160,"available":95880200192,"internally_reserved":0,"allocated":133214208,"data_stored":133012934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":623311301},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,4,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182518,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":739532,"kb_used_data":130328,"kb_used_omap":4,"kb_used_meta":609147,"kb_avail":93632308,"statfs":{"total":96636764160,"available":95879483392,"internally_reserved":0,"allocated":133455872,"data_stored":133254598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":623767245},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,0,1,3,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":132591616,"data_stored":132591616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":132562944,"data_stored":132562944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":132784128,"data_stored":132784128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":132198400,"data_stored":132198400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:43:45.537 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:43:45.978 INFO:tasks.ceph.osd.1.smithi087.stderr:2024-09-05T16:43:45.985+0000 7fc50bd5f640 -1 osd.1 pg_epoch: 39 pg[1.es0( v 39'10658 (29'600,39'10658] local-lis/les=38/39 n=1896 ec=15/15 lis/c=38/38 les/c/f=39/39/0 sis=38) [1,0,3,2]p1(0) r=0 lpr=38 crt=39'10658 lcod 39'10657 mlcod 39'10657 active+clean+scrubbing [ 1.es0: ] ] _scan_rollback_obs: queueing trans to clean up obsolete rollback objs 2024-09-05T16:43:46.122 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:43:46.167 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:43:46.168 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":344,"stamp":"2024-09-05T16:43:45.023628+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":256840168,"num_objects":31358,"num_object_clones":0,"num_object_copies":125432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29068,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31358,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":159479,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2569,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160703,"ondisk_log_size":160703,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2805176,"kb_used_data":504860,"kb_used_omap":14,"kb_used_meta":2300209,"kb_avail":374682184,"statfs":{"total":386547056640,"available":383674556416,"internally_reserved":0,"allocated":516976640,"data_stored":516157615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2355414583},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,4,3,3,17,13,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16408576,"num_objects":-2003,"num_object_clones":0,"num_object_copies":-8012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-150,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2003,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2003,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":150,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002991"},"pg_stats":[{"pgid":"1.f","version":"39'10934","reported_seq":11082,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:44.878295+0000","last_change":"2024-09-05T16:43:43.436935+0000","last_active":"2024-09-05T16:43:44.878295+0000","last_peered":"2024-09-05T16:43:44.878295+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:43:44.878295+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"29'900","ondisk_log_start":"29'900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16138240,"num_objects":1970,"num_object_clones":0,"num_object_copies":7880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2217,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1970,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10934,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-247}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'10640","reported_seq":15258,"reported_epoch":38,"state":"peering","last_fresh":"2024-09-05T16:43:43.602470+0000","last_change":"2024-09-05T16:43:44.622765+0000","last_active":"2024-09-05T16:43:44.622765+0000","last_peered":"2024-09-05T16:43:43.602470+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:43:43.602470+0000","last_undegraded":"2024-09-05T16:43:43.602470+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":38,"log_start":"29'600","ondisk_log_start":"29'600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15676576,"num_objects":1914,"num_object_clones":0,"num_object_copies":7656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1914,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10638,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2271,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"38'10811","reported_seq":10936,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:44.035725+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:43:44.035725+0000","last_peered":"2024-09-05T16:43:44.035725+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:43:44.035725+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"29'800","ondisk_log_start":"29'800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15899240,"num_objects":1942,"num_object_clones":0,"num_object_copies":7768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1942,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10810,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-290}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"38'10943","reported_seq":15675,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-05T16:43:44.569205+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:43:44.569205+0000","last_peered":"2024-09-05T16:43:44.569205+0000","last_clean":"2024-09-05T16:43:44.569205+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:43:44.569205+0000","last_undegraded":"2024-09-05T16:43:44.569205+0000","last_fullsized":"2024-09-05T16:43:44.569205+0000","mapping_epoch":34,"log_start":"29'900","ondisk_log_start":"29'900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16310272,"num_objects":1991,"num_object_clones":0,"num_object_copies":7964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1991,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5243,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"38'10628","reported_seq":10731,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:44.748907+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:43:44.748907+0000","last_peered":"2024-09-05T16:43:44.748907+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:43:44.748907+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"29'600","ondisk_log_start":"29'600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16110752,"num_objects":1967,"num_object_clones":0,"num_object_copies":7868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1967,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10626,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-178}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'10862","reported_seq":10981,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:44.818731+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:43:44.818731+0000","last_peered":"2024-09-05T16:43:44.818731+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:43:44.818731+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"29'800","ondisk_log_start":"29'800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15646720,"num_objects":1910,"num_object_clones":0,"num_object_copies":7640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10862,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-309}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"38'10667","reported_seq":10777,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:44.281648+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:43:44.281648+0000","last_peered":"2024-09-05T16:43:44.281648+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:43:44.281648+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"29'600","ondisk_log_start":"29'600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16340128,"num_objects":1995,"num_object_clones":0,"num_object_copies":7980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1995,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10665,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-136}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'10809","reported_seq":10900,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:44.343341+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:43:44.343341+0000","last_peered":"2024-09-05T16:43:44.343341+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:43:44.343341+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"29'800","ondisk_log_start":"29'800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16135328,"num_objects":1970,"num_object_clones":0,"num_object_copies":7880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1970,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10807,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-262}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"39'10838","reported_seq":10943,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:44.760885+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:43:44.760885+0000","last_peered":"2024-09-05T16:43:44.760885+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:43:44.760885+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"29'800","ondisk_log_start":"29'800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15853888,"num_objects":1936,"num_object_clones":0,"num_object_copies":7744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1936,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10835,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-308}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"38'10865","reported_seq":10989,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:44.082974+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:43:44.082974+0000","last_peered":"2024-09-05T16:43:44.082974+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:43:44.082974+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"29'800","ondisk_log_start":"29'800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15930528,"num_objects":1945,"num_object_clones":0,"num_object_copies":7780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1945,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10863,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-300}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"38'11109","reported_seq":11240,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:44.758410+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:43:44.758410+0000","last_peered":"2024-09-05T16:43:44.758410+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:43:44.758410+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"29'1100","ondisk_log_start":"29'1100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16494320,"num_objects":2014,"num_object_clones":0,"num_object_copies":8056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2014,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11106,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-267}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"38'10770","reported_seq":10873,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:44.767933+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:43:44.767933+0000","last_peered":"2024-09-05T16:43:44.767933+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:43:44.767933+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"29'700","ondisk_log_start":"29'700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16292432,"num_objects":1989,"num_object_clones":0,"num_object_copies":7956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1989,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10769,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-249}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"38'10892","reported_seq":15629,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-05T16:43:44.603513+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:43:44.603513+0000","last_peered":"2024-09-05T16:43:44.603513+0000","last_clean":"2024-09-05T16:43:44.603513+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:43:44.603513+0000","last_undegraded":"2024-09-05T16:43:44.603513+0000","last_fullsized":"2024-09-05T16:43:44.603513+0000","mapping_epoch":36,"log_start":"29'800","ondisk_log_start":"29'800","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15695872,"num_objects":1916,"num_object_clones":0,"num_object_copies":7664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1916,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2492,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'11118","reported_seq":11231,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:44.751790+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:43:44.751790+0000","last_peered":"2024-09-05T16:43:44.751790+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:43:44.751790+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"29'1100","ondisk_log_start":"29'1100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16446624,"num_objects":2008,"num_object_clones":0,"num_object_copies":8032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2008,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11116,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-326}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"38'10878","reported_seq":10993,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:44.183374+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:43:44.183374+0000","last_peered":"2024-09-05T16:43:44.183374+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:43:44.183374+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"29'800","ondisk_log_start":"29'800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16012448,"num_objects":1955,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10876,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-353}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"38'10839","reported_seq":10950,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:44.259252+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:43:44.259252+0000","last_peered":"2024-09-05T16:43:44.259252+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:43:44.259252+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"29'800","ondisk_log_start":"29'800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15856800,"num_objects":1936,"num_object_clones":0,"num_object_copies":7744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1936,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10837,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-284}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":256840168,"num_objects":31358,"num_object_clones":0,"num_object_copies":125432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29068,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31358,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":159479,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2569,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":513896448,"data_stored":513896448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160703,"ondisk_log_size":160703,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084405,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":735176,"kb_used_data":125936,"kb_used_omap":4,"kb_used_meta":609211,"kb_avail":93636664,"statfs":{"total":96636764160,"available":95883943936,"internally_reserved":0,"allocated":128958464,"data_stored":128752547,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":623832787},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,3,6,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986282,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":591312,"kb_used_data":126452,"kb_used_omap":4,"kb_used_meta":464827,"kb_avail":93780528,"statfs":{"total":96636764160,"available":96031260672,"internally_reserved":0,"allocated":129486848,"data_stored":129280931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":475983570},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,4,10],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673088,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":738956,"kb_used_data":126072,"kb_used_omap":1,"kb_used_meta":612862,"kb_avail":93632884,"statfs":{"total":96636764160,"available":95880073216,"internally_reserved":0,"allocated":129097728,"data_stored":128891811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":627571141},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,0,0,4,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182519,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":739732,"kb_used_data":126400,"kb_used_omap":4,"kb_used_meta":613307,"kb_avail":93632108,"statfs":{"total":96636764160,"available":95879278592,"internally_reserved":0,"allocated":129433600,"data_stored":129232326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":628027085},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,0,3,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":128536576,"data_stored":128536576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":128512000,"data_stored":128512000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":128684032,"data_stored":128684032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":128163840,"data_stored":128163840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:43:46.169 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:43:46.183 INFO:tasks.ceph.osd.1.smithi087.stderr:2024-09-05T16:43:46.190+0000 7fc50bd5f640 -1 osd.1 pg_epoch: 39 pg[1.es0( v 39'10660 (29'600,39'10660] local-lis/les=38/39 n=1894 ec=15/15 lis/c=38/38 les/c/f=39/39/0 sis=38) [1,0,3,2]p1(0) r=0 lpr=38 luod=39'10659 crt=39'10659 lcod 39'10658 mlcod 39'10658 active+clean+scrubbing [ 1.es0: ] ] _scan_rollback_obs: queueing trans to clean up obsolete rollback objs 2024-09-05T16:43:46.513 INFO:tasks.ceph.osd.1.smithi087.stderr:2024-09-05T16:43:46.520+0000 7fc50bd5f640 -1 osd.1 pg_epoch: 39 pg[1.es0( v 39'10667 (29'600,39'10667] local-lis/les=38/39 n=1887 ec=15/15 lis/c=38/38 les/c/f=39/39/0 sis=38) [1,0,3,2]p1(0) r=0 lpr=38 luod=39'10662 crt=39'10662 lcod 39'10661 mlcod 39'10661 active+clean+scrubbing [ 1.es0: ] ] _scan_rollback_obs: queueing trans to clean up obsolete rollback objs 2024-09-05T16:43:46.810 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:43:46.811 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:43:46.829 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":344,"stamp":"2024-09-05T16:43:45.023628+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":256840168,"num_objects":31358,"num_object_clones":0,"num_object_copies":125432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29068,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31358,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":159479,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2569,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160703,"ondisk_log_size":160703,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2805176,"kb_used_data":504860,"kb_used_omap":14,"kb_used_meta":2300209,"kb_avail":374682184,"statfs":{"total":386547056640,"available":383674556416,"internally_reserved":0,"allocated":516976640,"data_stored":516157615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2355414583},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,4,3,3,17,13,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16408576,"num_objects":-2003,"num_object_clones":0,"num_object_copies":-8012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-150,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2003,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2003,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":150,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002991"},"pg_stats":[{"pgid":"1.f","version":"39'10934","reported_seq":11082,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:44.878295+0000","last_change":"2024-09-05T16:43:43.436935+0000","last_active":"2024-09-05T16:43:44.878295+0000","last_peered":"2024-09-05T16:43:44.878295+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:43:44.878295+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"29'900","ondisk_log_start":"29'900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16138240,"num_objects":1970,"num_object_clones":0,"num_object_copies":7880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2217,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1970,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10934,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-247}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'10640","reported_seq":15258,"reported_epoch":38,"state":"peering","last_fresh":"2024-09-05T16:43:43.602470+0000","last_change":"2024-09-05T16:43:44.622765+0000","last_active":"2024-09-05T16:43:44.622765+0000","last_peered":"2024-09-05T16:43:43.602470+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:43:43.602470+0000","last_undegraded":"2024-09-05T16:43:43.602470+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":38,"log_start":"29'600","ondisk_log_start":"29'600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15676576,"num_objects":1914,"num_object_clones":0,"num_object_copies":7656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1914,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10638,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2271,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"38'10811","reported_seq":10936,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:44.035725+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:43:44.035725+0000","last_peered":"2024-09-05T16:43:44.035725+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:43:44.035725+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"29'800","ondisk_log_start":"29'800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15899240,"num_objects":1942,"num_object_clones":0,"num_object_copies":7768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1942,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10810,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-290}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"38'10943","reported_seq":15675,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-05T16:43:44.569205+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:43:44.569205+0000","last_peered":"2024-09-05T16:43:44.569205+0000","last_clean":"2024-09-05T16:43:44.569205+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:43:44.569205+0000","last_undegraded":"2024-09-05T16:43:44.569205+0000","last_fullsized":"2024-09-05T16:43:44.569205+0000","mapping_epoch":34,"log_start":"29'900","ondisk_log_start":"29'900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16310272,"num_objects":1991,"num_object_clones":0,"num_object_copies":7964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1991,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5243,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"38'10628","reported_seq":10731,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:44.748907+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:43:44.748907+0000","last_peered":"2024-09-05T16:43:44.748907+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:43:44.748907+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"29'600","ondisk_log_start":"29'600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16110752,"num_objects":1967,"num_object_clones":0,"num_object_copies":7868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1967,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10626,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-178}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'10862","reported_seq":10981,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:44.818731+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:43:44.818731+0000","last_peered":"2024-09-05T16:43:44.818731+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:43:44.818731+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"29'800","ondisk_log_start":"29'800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15646720,"num_objects":1910,"num_object_clones":0,"num_object_copies":7640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10862,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-309}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"38'10667","reported_seq":10777,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:44.281648+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:43:44.281648+0000","last_peered":"2024-09-05T16:43:44.281648+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:43:44.281648+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"29'600","ondisk_log_start":"29'600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16340128,"num_objects":1995,"num_object_clones":0,"num_object_copies":7980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1995,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10665,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-136}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'10809","reported_seq":10900,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:44.343341+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:43:44.343341+0000","last_peered":"2024-09-05T16:43:44.343341+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:43:44.343341+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"29'800","ondisk_log_start":"29'800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16135328,"num_objects":1970,"num_object_clones":0,"num_object_copies":7880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1970,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10807,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-262}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"39'10838","reported_seq":10943,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:44.760885+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:43:44.760885+0000","last_peered":"2024-09-05T16:43:44.760885+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:43:44.760885+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"29'800","ondisk_log_start":"29'800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15853888,"num_objects":1936,"num_object_clones":0,"num_object_copies":7744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1936,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10835,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-308}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"38'10865","reported_seq":10989,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:44.082974+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:43:44.082974+0000","last_peered":"2024-09-05T16:43:44.082974+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:43:44.082974+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"29'800","ondisk_log_start":"29'800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15930528,"num_objects":1945,"num_object_clones":0,"num_object_copies":7780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1945,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10863,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-300}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"38'11109","reported_seq":11240,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:44.758410+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:43:44.758410+0000","last_peered":"2024-09-05T16:43:44.758410+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:43:44.758410+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"29'1100","ondisk_log_start":"29'1100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16494320,"num_objects":2014,"num_object_clones":0,"num_object_copies":8056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2014,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11106,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-267}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"38'10770","reported_seq":10873,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:44.767933+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:43:44.767933+0000","last_peered":"2024-09-05T16:43:44.767933+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:43:44.767933+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"29'700","ondisk_log_start":"29'700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16292432,"num_objects":1989,"num_object_clones":0,"num_object_copies":7956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1989,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10769,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-249}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"38'10892","reported_seq":15629,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-05T16:43:44.603513+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:43:44.603513+0000","last_peered":"2024-09-05T16:43:44.603513+0000","last_clean":"2024-09-05T16:43:44.603513+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:43:44.603513+0000","last_undegraded":"2024-09-05T16:43:44.603513+0000","last_fullsized":"2024-09-05T16:43:44.603513+0000","mapping_epoch":36,"log_start":"29'800","ondisk_log_start":"29'800","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15695872,"num_objects":1916,"num_object_clones":0,"num_object_copies":7664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1916,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2492,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'11118","reported_seq":11231,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:44.751790+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:43:44.751790+0000","last_peered":"2024-09-05T16:43:44.751790+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:43:44.751790+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"29'1100","ondisk_log_start":"29'1100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16446624,"num_objects":2008,"num_object_clones":0,"num_object_copies":8032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2008,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11116,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-326}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"38'10878","reported_seq":10993,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:44.183374+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:43:44.183374+0000","last_peered":"2024-09-05T16:43:44.183374+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:43:44.183374+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"29'800","ondisk_log_start":"29'800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16012448,"num_objects":1955,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10876,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-353}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"38'10839","reported_seq":10950,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:44.259252+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:43:44.259252+0000","last_peered":"2024-09-05T16:43:44.259252+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:43:44.259252+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"29'800","ondisk_log_start":"29'800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15856800,"num_objects":1936,"num_object_clones":0,"num_object_copies":7744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1936,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10837,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-284}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":256840168,"num_objects":31358,"num_object_clones":0,"num_object_copies":125432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29068,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31358,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":159479,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2569,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":513896448,"data_stored":513896448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160703,"ondisk_log_size":160703,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084405,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":735176,"kb_used_data":125936,"kb_used_omap":4,"kb_used_meta":609211,"kb_avail":93636664,"statfs":{"total":96636764160,"available":95883943936,"internally_reserved":0,"allocated":128958464,"data_stored":128752547,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":623832787},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,3,6,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986282,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":591312,"kb_used_data":126452,"kb_used_omap":4,"kb_used_meta":464827,"kb_avail":93780528,"statfs":{"total":96636764160,"available":96031260672,"internally_reserved":0,"allocated":129486848,"data_stored":129280931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":475983570},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,4,10],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673088,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":738956,"kb_used_data":126072,"kb_used_omap":1,"kb_used_meta":612862,"kb_avail":93632884,"statfs":{"total":96636764160,"available":95880073216,"internally_reserved":0,"allocated":129097728,"data_stored":128891811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":627571141},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,0,0,4,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182519,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":739732,"kb_used_data":126400,"kb_used_omap":4,"kb_used_meta":613307,"kb_avail":93632108,"statfs":{"total":96636764160,"available":95879278592,"internally_reserved":0,"allocated":129433600,"data_stored":129232326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":628027085},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,0,3,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":128536576,"data_stored":128536576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":128512000,"data_stored":128512000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":128684032,"data_stored":128684032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":128163840,"data_stored":128163840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:43:46.830 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-05T16:43:46.830 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "39'10934", 'reported_seq': 11082, 'reported_epoch': 39, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:44.878295+0000', 'last_change': '2024-09-05T16:43:43.436935+0000', 'last_active': '2024-09-05T16:43:44.878295+0000', 'last_peered': '2024-09-05T16:43:44.878295+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.828289+0000', 'last_became_peered': '2024-09-05T16:34:56.828289+0000', 'last_unstale': '2024-09-05T16:43:44.878295+0000', 'last_undegraded': '2024-09-05T16:34:56.612160+0000', 'last_fullsized': '2024-09-05T16:34:56.557632+0000', 'mapping_epoch': 32, 'log_start': "29'900", 'ondisk_log_start': "29'900", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10034, 'log_dups_size': 0, 'ondisk_log_size': 10034, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16138240, 'num_objects': 1970, 'num_object_clones': 0, 'num_object_copies': 7880, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2217, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1970, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10934, 'num_write_kb': 51616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 45, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -247}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:43:46.831 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-05T16:43:46.831 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "37'10640", 'reported_seq': 15258, 'reported_epoch': 38, 'state': 'peering', 'last_fresh': '2024-09-05T16:43:43.602470+0000', 'last_change': '2024-09-05T16:43:44.622765+0000', 'last_active': '2024-09-05T16:43:44.622765+0000', 'last_peered': '2024-09-05T16:43:43.602470+0000', 'last_clean': '2024-09-05T16:33:53.200762+0000', 'last_became_active': '2024-09-05T16:34:56.820292+0000', 'last_became_peered': '2024-09-05T16:34:56.820292+0000', 'last_unstale': '2024-09-05T16:43:43.602470+0000', 'last_undegraded': '2024-09-05T16:43:43.602470+0000', 'last_fullsized': '2024-09-05T16:34:56.559383+0000', 'mapping_epoch': 38, 'log_start': "29'600", 'ondisk_log_start': "29'600", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15676576, 'num_objects': 1914, 'num_object_clones': 0, 'num_object_copies': 7656, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1914, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10638, 'num_write_kb': 50212, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2271, 'num_bytes_recovered': 2613248, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [1, 0, 3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-05T16:43:46.831 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:43:46.831 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "38'10811", 'reported_seq': 10936, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:44.035725+0000', 'last_change': '2024-09-05T16:34:56.867900+0000', 'last_active': '2024-09-05T16:43:44.035725+0000', 'last_peered': '2024-09-05T16:43:44.035725+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:43:44.035725+0000', 'last_undegraded': '2024-09-05T16:34:56.590972+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 32, 'log_start': "29'800", 'ondisk_log_start': "29'800", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10011, 'log_dups_size': 0, 'ondisk_log_size': 10011, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15899240, 'num_objects': 1942, 'num_object_clones': 0, 'num_object_copies': 7768, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2233, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1942, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 10810, 'num_write_kb': 50999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -290}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:43:46.831 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:43:46.831 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "38'10628", 'reported_seq': 10731, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:44.748907+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:43:44.748907+0000', 'last_peered': '2024-09-05T16:43:44.748907+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:43:44.748907+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "29'600", 'ondisk_log_start': "29'600", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16110752, 'num_objects': 1967, 'num_object_clones': 0, 'num_object_copies': 7868, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2147, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1967, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10626, 'num_write_kb': 50373, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -178}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:43:46.831 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:43:46.831 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "39'10862", 'reported_seq': 10981, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:44.818731+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:43:44.818731+0000', 'last_peered': '2024-09-05T16:43:44.818731+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:43:44.818731+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "29'800", 'ondisk_log_start': "29'800", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10062, 'log_dups_size': 0, 'ondisk_log_size': 10062, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15646720, 'num_objects': 1910, 'num_object_clones': 0, 'num_object_copies': 7640, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1910, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10862, 'num_write_kb': 51088, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -309}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:43:46.831 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:43:46.832 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "38'10667", 'reported_seq': 10777, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:44.281648+0000', 'last_change': '2024-09-05T16:34:56.903436+0000', 'last_active': '2024-09-05T16:43:44.281648+0000', 'last_peered': '2024-09-05T16:43:44.281648+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:43:44.281648+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "29'600", 'ondisk_log_start': "29'600", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10067, 'log_dups_size': 0, 'ondisk_log_size': 10067, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16340128, 'num_objects': 1995, 'num_object_clones': 0, 'num_object_copies': 7980, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2133, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1995, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10665, 'num_write_kb': 50636, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -136}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:43:46.832 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-05T16:43:46.832 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "38'10809", 'reported_seq': 10900, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:44.343341+0000', 'last_change': '2024-09-05T16:34:56.883319+0000', 'last_active': '2024-09-05T16:43:44.343341+0000', 'last_peered': '2024-09-05T16:43:44.343341+0000', 'last_clean': '2024-09-05T16:33:53.505305+0000', 'last_became_active': '2024-09-05T16:34:56.873389+0000', 'last_became_peered': '2024-09-05T16:34:56.873389+0000', 'last_unstale': '2024-09-05T16:43:44.343341+0000', 'last_undegraded': '2024-09-05T16:34:56.602567+0000', 'last_fullsized': '2024-09-05T16:34:56.559958+0000', 'mapping_epoch': 32, 'log_start': "29'800", 'ondisk_log_start': "29'800", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10009, 'log_dups_size': 0, 'ondisk_log_size': 10009, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16135328, 'num_objects': 1970, 'num_object_clones': 0, 'num_object_copies': 7880, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2234, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1970, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10807, 'num_write_kb': 51108, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -262}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:43:46.832 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:43:46.832 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "39'10838", 'reported_seq': 10943, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:44.760885+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:43:44.760885+0000', 'last_peered': '2024-09-05T16:43:44.760885+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:43:44.760885+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "29'800", 'ondisk_log_start': "29'800", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10038, 'log_dups_size': 0, 'ondisk_log_size': 10038, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15853888, 'num_objects': 1936, 'num_object_clones': 0, 'num_object_copies': 7744, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1936, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10835, 'num_write_kb': 51081, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -308}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:43:46.832 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:43:46.832 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "38'10865", 'reported_seq': 10989, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:44.082974+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:43:44.082974+0000', 'last_peered': '2024-09-05T16:43:44.082974+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:43:44.082974+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "29'800", 'ondisk_log_start': "29'800", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10065, 'log_dups_size': 0, 'ondisk_log_size': 10065, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15930528, 'num_objects': 1945, 'num_object_clones': 0, 'num_object_copies': 7780, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1945, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10863, 'num_write_kb': 51230, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -300}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:43:46.832 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:43:46.832 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "38'11109", 'reported_seq': 11240, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:44.758410+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:43:44.758410+0000', 'last_peered': '2024-09-05T16:43:44.758410+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:43:44.758410+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "29'1100", 'ondisk_log_start': "29'1100", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10009, 'log_dups_size': 0, 'ondisk_log_size': 10009, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16494320, 'num_objects': 2014, 'num_object_clones': 0, 'num_object_copies': 8056, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2284, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2014, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11106, 'num_write_kb': 52476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -267}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:43:46.832 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:43:46.832 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "38'10770", 'reported_seq': 10873, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:44.767933+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:43:44.767933+0000', 'last_peered': '2024-09-05T16:43:44.767933+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:43:44.767933+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "29'700", 'ondisk_log_start': "29'700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10070, 'log_dups_size': 0, 'ondisk_log_size': 10070, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16292432, 'num_objects': 1989, 'num_object_clones': 0, 'num_object_copies': 7956, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1989, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10769, 'num_write_kb': 51031, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -249}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:43:46.833 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:43:46.833 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "39'11118", 'reported_seq': 11231, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:44.751790+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:43:44.751790+0000', 'last_peered': '2024-09-05T16:43:44.751790+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:43:44.751790+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "29'1100", 'ondisk_log_start': "29'1100", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10018, 'log_dups_size': 0, 'ondisk_log_size': 10018, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16446624, 'num_objects': 2008, 'num_object_clones': 0, 'num_object_copies': 8032, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2008, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11116, 'num_write_kb': 52500, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -326}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:43:46.833 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:43:46.833 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "38'10878", 'reported_seq': 10993, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:44.183374+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:43:44.183374+0000', 'last_peered': '2024-09-05T16:43:44.183374+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:43:44.183374+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "29'800", 'ondisk_log_start': "29'800", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10078, 'log_dups_size': 0, 'ondisk_log_size': 10078, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16012448, 'num_objects': 1955, 'num_object_clones': 0, 'num_object_copies': 7820, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2310, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1955, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10876, 'num_write_kb': 51325, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -353}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:43:46.833 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:43:46.833 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "38'10839", 'reported_seq': 10950, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:44.259252+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:43:44.259252+0000', 'last_peered': '2024-09-05T16:43:44.259252+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:43:44.259252+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "29'800", 'ondisk_log_start': "29'800", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10039, 'log_dups_size': 0, 'ondisk_log_size': 10039, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15856800, 'num_objects': 1936, 'num_object_clones': 0, 'num_object_copies': 7744, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1936, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10837, 'num_write_kb': 51092, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -284}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:43:46.834 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:43:47.162 INFO:tasks.ceph.osd.1.smithi087.stderr:2024-09-05T16:43:47.170+0000 7fc50bd5f640 -1 osd.1 pg_epoch: 39 pg[1.es0( v 39'10675 (29'600,39'10675] local-lis/les=38/39 n=1879 ec=15/15 lis/c=38/38 les/c/f=39/39/0 sis=38) [1,0,3,2]p1(0) r=0 lpr=38 crt=39'10675 lcod 39'10674 mlcod 39'10674 active+clean+scrubbing [ 1.es0: ] ] _scan_rollback_obs: queueing trans to clean up obsolete rollback objs 2024-09-05T16:43:47.471 INFO:tasks.ceph.osd.1.smithi087.stderr:2024-09-05T16:43:47.480+0000 7fc50fd67640 -1 osd.1 pg_epoch: 39 pg[1.es0( v 39'10678 (29'600,39'10678] local-lis/les=38/39 n=1876 ec=15/15 lis/c=38/38 les/c/f=39/39/0 sis=38) [1,0,3,2]p1(0) r=0 lpr=38 luod=39'10676 crt=39'10676 lcod 39'10675 mlcod 39'10675 active+clean+scrubbing [ 1.es0: ] ] _scan_rollback_obs: queueing trans to clean up obsolete rollback objs 2024-09-05T16:43:47.617 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:43:47.617 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 29068/125432 objects degraded (23.174%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":630,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":39,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":2},{"state_name":"peering","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":31358,"data_bytes":256840168,"bytes_used":2872500224,"bytes_avail":383674556416,"bytes_total":386547056640,"inactive_pgs_ratio":0.0625,"degraded_objects":29068,"degraded_total":125432,"degraded_ratio":0.23174309586070543,"recovering_objects_per_sec":18,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":150,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":250},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:43:47.632 INFO:tasks.ceph.osd.1.smithi087.stderr:2024-09-05T16:43:47.642+0000 7fc50bd5f640 -1 osd.1 pg_epoch: 39 pg[1.es0( v 39'10680 (29'600,39'10680] local-lis/les=38/39 n=1874 ec=15/15 lis/c=38/38 les/c/f=39/39/0 sis=38) [1,0,3,2]p1(0) r=0 lpr=38 luod=39'10679 crt=39'10677 lcod 39'10678 mlcod 39'10678 active+clean+scrubbing [ 1.es0: ] ] _scan_rollback_obs: queueing trans to clean up obsolete rollback objs 2024-09-05T16:43:47.640 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:43:47.640 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:43:48.237 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:43:48.237 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:43:48.273 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":345,"stamp":"2024-09-05T16:43:47.024377+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":256840168,"num_objects":31358,"num_object_clones":0,"num_object_copies":125432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29068,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31358,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":159479,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2569,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160703,"ondisk_log_size":160703,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2805176,"kb_used_data":504860,"kb_used_omap":14,"kb_used_meta":2300209,"kb_avail":374682184,"statfs":{"total":386547056640,"available":383674556416,"internally_reserved":0,"allocated":516976640,"data_stored":516157615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2355414583},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,4,3,3,17,13,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16408576,"num_objects":-2003,"num_object_clones":0,"num_object_copies":-8012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-150,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2003,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2003,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":150,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002862"},"pg_stats":[{"pgid":"1.f","version":"39'10934","reported_seq":11082,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:44.878295+0000","last_change":"2024-09-05T16:43:43.436935+0000","last_active":"2024-09-05T16:43:44.878295+0000","last_peered":"2024-09-05T16:43:44.878295+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:43:44.878295+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"29'900","ondisk_log_start":"29'900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16138240,"num_objects":1970,"num_object_clones":0,"num_object_copies":7880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2217,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1970,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10934,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-247}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'10640","reported_seq":15258,"reported_epoch":38,"state":"peering","last_fresh":"2024-09-05T16:43:43.602470+0000","last_change":"2024-09-05T16:43:44.622765+0000","last_active":"2024-09-05T16:43:44.622765+0000","last_peered":"2024-09-05T16:43:43.602470+0000","last_clean":"2024-09-05T16:33:53.200762+0000","last_became_active":"2024-09-05T16:34:56.820292+0000","last_became_peered":"2024-09-05T16:34:56.820292+0000","last_unstale":"2024-09-05T16:43:43.602470+0000","last_undegraded":"2024-09-05T16:43:43.602470+0000","last_fullsized":"2024-09-05T16:34:56.559383+0000","mapping_epoch":38,"log_start":"29'600","ondisk_log_start":"29'600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15676576,"num_objects":1914,"num_object_clones":0,"num_object_copies":7656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1914,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10638,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2271,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"38'10811","reported_seq":10936,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:44.035725+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:43:44.035725+0000","last_peered":"2024-09-05T16:43:44.035725+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:43:44.035725+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"29'800","ondisk_log_start":"29'800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15899240,"num_objects":1942,"num_object_clones":0,"num_object_copies":7768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1942,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10810,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-290}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"38'10943","reported_seq":15675,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-05T16:43:44.569205+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:43:44.569205+0000","last_peered":"2024-09-05T16:43:44.569205+0000","last_clean":"2024-09-05T16:43:44.569205+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:43:44.569205+0000","last_undegraded":"2024-09-05T16:43:44.569205+0000","last_fullsized":"2024-09-05T16:43:44.569205+0000","mapping_epoch":34,"log_start":"29'900","ondisk_log_start":"29'900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16310272,"num_objects":1991,"num_object_clones":0,"num_object_copies":7964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1991,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5243,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"38'10628","reported_seq":10731,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:44.748907+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:43:44.748907+0000","last_peered":"2024-09-05T16:43:44.748907+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:43:44.748907+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"29'600","ondisk_log_start":"29'600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16110752,"num_objects":1967,"num_object_clones":0,"num_object_copies":7868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1967,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10626,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-178}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'10862","reported_seq":10981,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:44.818731+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:43:44.818731+0000","last_peered":"2024-09-05T16:43:44.818731+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:43:44.818731+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"29'800","ondisk_log_start":"29'800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15646720,"num_objects":1910,"num_object_clones":0,"num_object_copies":7640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10862,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-309}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"38'10667","reported_seq":10777,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:44.281648+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:43:44.281648+0000","last_peered":"2024-09-05T16:43:44.281648+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:43:44.281648+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"29'600","ondisk_log_start":"29'600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16340128,"num_objects":1995,"num_object_clones":0,"num_object_copies":7980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1995,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10665,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-136}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'10809","reported_seq":10900,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:44.343341+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:43:44.343341+0000","last_peered":"2024-09-05T16:43:44.343341+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:43:44.343341+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"29'800","ondisk_log_start":"29'800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16135328,"num_objects":1970,"num_object_clones":0,"num_object_copies":7880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1970,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10807,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-262}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"39'10838","reported_seq":10943,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:44.760885+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:43:44.760885+0000","last_peered":"2024-09-05T16:43:44.760885+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:43:44.760885+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"29'800","ondisk_log_start":"29'800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15853888,"num_objects":1936,"num_object_clones":0,"num_object_copies":7744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1936,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10835,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-308}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"38'10865","reported_seq":10989,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:44.082974+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:43:44.082974+0000","last_peered":"2024-09-05T16:43:44.082974+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:43:44.082974+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"29'800","ondisk_log_start":"29'800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15930528,"num_objects":1945,"num_object_clones":0,"num_object_copies":7780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1945,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10863,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-300}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"38'11109","reported_seq":11240,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:44.758410+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:43:44.758410+0000","last_peered":"2024-09-05T16:43:44.758410+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:43:44.758410+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"29'1100","ondisk_log_start":"29'1100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16494320,"num_objects":2014,"num_object_clones":0,"num_object_copies":8056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2014,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11106,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-267}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"38'10770","reported_seq":10873,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:44.767933+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:43:44.767933+0000","last_peered":"2024-09-05T16:43:44.767933+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:43:44.767933+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"29'700","ondisk_log_start":"29'700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16292432,"num_objects":1989,"num_object_clones":0,"num_object_copies":7956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1989,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10769,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-249}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"38'10892","reported_seq":15629,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-05T16:43:44.603513+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:43:44.603513+0000","last_peered":"2024-09-05T16:43:44.603513+0000","last_clean":"2024-09-05T16:43:44.603513+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:43:44.603513+0000","last_undegraded":"2024-09-05T16:43:44.603513+0000","last_fullsized":"2024-09-05T16:43:44.603513+0000","mapping_epoch":36,"log_start":"29'800","ondisk_log_start":"29'800","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15695872,"num_objects":1916,"num_object_clones":0,"num_object_copies":7664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1916,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2492,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'11118","reported_seq":11231,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:44.751790+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:43:44.751790+0000","last_peered":"2024-09-05T16:43:44.751790+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:43:44.751790+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"29'1100","ondisk_log_start":"29'1100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16446624,"num_objects":2008,"num_object_clones":0,"num_object_copies":8032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2008,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11116,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-326}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"38'10878","reported_seq":10993,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:44.183374+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:43:44.183374+0000","last_peered":"2024-09-05T16:43:44.183374+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:43:44.183374+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"29'800","ondisk_log_start":"29'800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16012448,"num_objects":1955,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10876,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-353}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"38'10839","reported_seq":10950,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:44.259252+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:43:44.259252+0000","last_peered":"2024-09-05T16:43:44.259252+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:43:44.259252+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"29'800","ondisk_log_start":"29'800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15856800,"num_objects":1936,"num_object_clones":0,"num_object_copies":7744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1936,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10837,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-284}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":256840168,"num_objects":31358,"num_object_clones":0,"num_object_copies":125432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29068,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31358,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":159479,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2569,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":513896448,"data_stored":513896448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160703,"ondisk_log_size":160703,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084405,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":735176,"kb_used_data":125936,"kb_used_omap":4,"kb_used_meta":609211,"kb_avail":93636664,"statfs":{"total":96636764160,"available":95883943936,"internally_reserved":0,"allocated":128958464,"data_stored":128752547,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":623832787},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,3,6,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986282,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":591312,"kb_used_data":126452,"kb_used_omap":4,"kb_used_meta":464827,"kb_avail":93780528,"statfs":{"total":96636764160,"available":96031260672,"internally_reserved":0,"allocated":129486848,"data_stored":129280931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":475983570},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,4,10],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673088,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":738956,"kb_used_data":126072,"kb_used_omap":1,"kb_used_meta":612862,"kb_avail":93632884,"statfs":{"total":96636764160,"available":95880073216,"internally_reserved":0,"allocated":129097728,"data_stored":128891811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":627571141},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,0,0,4,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182519,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":739732,"kb_used_data":126400,"kb_used_omap":4,"kb_used_meta":613307,"kb_avail":93632108,"statfs":{"total":96636764160,"available":95879278592,"internally_reserved":0,"allocated":129433600,"data_stored":129232326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":628027085},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,0,3,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":128536576,"data_stored":128536576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":128512000,"data_stored":128512000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":128684032,"data_stored":128684032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":128163840,"data_stored":128163840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:43:48.865 INFO:tasks.ceph.osd.1.smithi087.stderr:2024-09-05T16:43:48.874+0000 7fc50bd5f640 -1 osd.1 pg_epoch: 39 pg[1.es0( v 39'10692 (29'600,39'10692] local-lis/les=38/39 n=1862 ec=15/15 lis/c=38/38 les/c/f=39/39/0 sis=38) [1,0,3,2]p1(0) r=0 lpr=38 crt=39'10692 lcod 39'10691 mlcod 39'10691 active+clean+scrubbing [ 1.es0: ] ] _scan_rollback_obs: queueing trans to clean up obsolete rollback objs 2024-09-05T16:43:49.128 INFO:tasks.ceph.osd.1.smithi087.stderr:2024-09-05T16:43:49.135+0000 7fc50bd5f640 -1 osd.1 pg_epoch: 39 pg[1.es0( v 39'10698 (29'600,39'10698] local-lis/les=38/39 n=1856 ec=15/15 lis/c=38/38 les/c/f=39/39/0 sis=38) [1,0,3,2]p1(0) r=0 lpr=38 luod=39'10695 crt=39'10695 lcod 39'10694 mlcod 39'10694 active+clean+scrubbing [ 1.es0: ] ] _scan_rollback_obs: queueing trans to clean up obsolete rollback objs 2024-09-05T16:43:49.582 INFO:tasks.ceph.osd.1.smithi087.stderr:2024-09-05T16:43:49.591+0000 7fc50bd5f640 -1 osd.1 pg_epoch: 39 pg[1.es0( v 39'10706 (29'700,39'10706] local-lis/les=38/39 n=1848 ec=15/15 lis/c=38/38 les/c/f=39/39/0 sis=38) [1,0,3,2]p1(0) r=0 lpr=38 luod=39'10705 crt=39'10705 lcod 39'10704 mlcod 39'10704 active+clean+scrubbing [ 1.es0: ] ] _scan_rollback_obs: queueing trans to clean up obsolete rollback objs 2024-09-05T16:43:51.275 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:43:51.857 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:43:51.857 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:43:51.885 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":347,"stamp":"2024-09-05T16:43:51.025771+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":248893928,"num_objects":30388,"num_object_clones":0,"num_object_copies":121552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29022,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30388,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":160449,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2615,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160873,"ondisk_log_size":160873,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2824780,"kb_used_data":489732,"kb_used_omap":14,"kb_used_meta":2334897,"kb_avail":374662580,"statfs":{"total":386547056640,"available":383654481920,"internally_reserved":0,"allocated":501485568,"data_stored":500643244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2390935095},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,4,15,12,11,13],"upper_bound":256},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16048128,"num_objects":-1959,"num_object_clones":0,"num_object_copies":-7836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1959,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1959,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002758"},"pg_stats":[{"pgid":"1.f","version":"39'10992","reported_seq":11232,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:49.917113+0000","last_change":"2024-09-05T16:43:43.436935+0000","last_active":"2024-09-05T16:43:49.917113+0000","last_peered":"2024-09-05T16:43:49.917113+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:43:49.917113+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"29'900","ondisk_log_start":"29'900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15663104,"num_objects":1912,"num_object_clones":0,"num_object_copies":7648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2171,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1912,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10992,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-259}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'10709","reported_seq":15355,"reported_epoch":39,"state":"active+clean+scrubbing","last_fresh":"2024-09-05T16:43:49.558471+0000","last_change":"2024-09-05T16:43:45.841421+0000","last_active":"2024-09-05T16:43:49.558471+0000","last_peered":"2024-09-05T16:43:49.558471+0000","last_clean":"2024-09-05T16:43:49.558471+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:43:49.558471+0000","last_undegraded":"2024-09-05T16:43:49.558471+0000","last_fullsized":"2024-09-05T16:43:49.558471+0000","mapping_epoch":38,"log_start":"29'700","ondisk_log_start":"29'700","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":1788,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 3s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15111328,"num_objects":1845,"num_object_clones":0,"num_object_copies":7380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1845,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10707,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2271,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'10880","reported_seq":11007,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:49.199478+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:43:49.199478+0000","last_peered":"2024-09-05T16:43:49.199478+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:43:49.199478+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"29'800","ondisk_log_start":"29'800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15333992,"num_objects":1873,"num_object_clones":0,"num_object_copies":7492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1873,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10879,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-359}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'11001","reported_seq":15735,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:43:49.537138+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:43:49.537138+0000","last_peered":"2024-09-05T16:43:49.537138+0000","last_clean":"2024-09-05T16:43:49.537138+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:43:49.537138+0000","last_undegraded":"2024-09-05T16:43:49.537138+0000","last_fullsized":"2024-09-05T16:43:49.537138+0000","mapping_epoch":34,"log_start":"29'1000","ondisk_log_start":"29'1000","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15835136,"num_objects":1933,"num_object_clones":0,"num_object_copies":7732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1933,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5301,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'10687","reported_seq":10790,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:49.916264+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:43:49.916264+0000","last_peered":"2024-09-05T16:43:49.916264+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:43:49.916264+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"29'600","ondisk_log_start":"29'600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15627424,"num_objects":1908,"num_object_clones":0,"num_object_copies":7632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1908,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10685,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-237}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'10912","reported_seq":11031,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:49.954281+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:43:49.954281+0000","last_peered":"2024-09-05T16:43:49.954281+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:43:49.954281+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"29'900","ondisk_log_start":"29'900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15237120,"num_objects":1860,"num_object_clones":0,"num_object_copies":7440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1860,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10912,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-359}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'10732","reported_seq":10844,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:49.454994+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:43:49.454994+0000","last_peered":"2024-09-05T16:43:49.454994+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:43:49.454994+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"29'700","ondisk_log_start":"29'700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15807648,"num_objects":1930,"num_object_clones":0,"num_object_copies":7720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1930,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10730,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-201}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'10862","reported_seq":10954,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:49.421021+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:43:49.421021+0000","last_peered":"2024-09-05T16:43:49.421021+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:43:49.421021+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"29'800","ondisk_log_start":"29'800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15701152,"num_objects":1917,"num_object_clones":0,"num_object_copies":7668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1917,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10860,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-315}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"39'10893","reported_seq":10998,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:49.954747+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:43:49.954747+0000","last_peered":"2024-09-05T16:43:49.954747+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:43:49.954747+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"29'800","ondisk_log_start":"29'800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15403328,"num_objects":1881,"num_object_clones":0,"num_object_copies":7524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1881,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10890,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-363}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'10926","reported_seq":11052,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:49.061772+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:43:49.061772+0000","last_peered":"2024-09-05T16:43:49.061772+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:43:49.061772+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"29'900","ondisk_log_start":"29'900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15430816,"num_objects":1884,"num_object_clones":0,"num_object_copies":7536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1884,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10924,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-361}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'11168","reported_seq":11299,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:49.824336+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:43:49.824336+0000","last_peered":"2024-09-05T16:43:49.824336+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:43:49.824336+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"29'1100","ondisk_log_start":"29'1100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16010992,"num_objects":1955,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11165,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-326}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'10830","reported_seq":10931,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:50.004243+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:43:50.004243+0000","last_peered":"2024-09-05T16:43:50.004243+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:43:50.004243+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"29'800","ondisk_log_start":"29'800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15800912,"num_objects":1929,"num_object_clones":0,"num_object_copies":7716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1929,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10829,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-309}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'10961","reported_seq":15701,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:43:49.427973+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:43:49.427973+0000","last_peered":"2024-09-05T16:43:49.427973+0000","last_clean":"2024-09-05T16:43:49.427973+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:43:49.427973+0000","last_undegraded":"2024-09-05T16:43:49.427973+0000","last_fullsized":"2024-09-05T16:43:49.427973+0000","mapping_epoch":36,"log_start":"29'900","ondisk_log_start":"29'900","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15130624,"num_objects":1847,"num_object_clones":0,"num_object_copies":7388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1847,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2561,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'11181","reported_seq":11294,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:49.848507+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:43:49.848507+0000","last_peered":"2024-09-05T16:43:49.848507+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:43:49.848507+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"29'1100","ondisk_log_start":"29'1100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15930528,"num_objects":1945,"num_object_clones":0,"num_object_copies":7780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1945,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11179,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-389}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'10943","reported_seq":11060,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:49.267999+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:43:49.267999+0000","last_peered":"2024-09-05T16:43:49.267999+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:43:49.267999+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"29'900","ondisk_log_start":"29'900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15479968,"num_objects":1890,"num_object_clones":0,"num_object_copies":7560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1890,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10941,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-418}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'10896","reported_seq":11009,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:49.232123+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:43:49.232123+0000","last_peered":"2024-09-05T16:43:49.232123+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:43:49.232123+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"29'800","ondisk_log_start":"29'800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15389856,"num_objects":1879,"num_object_clones":0,"num_object_copies":7516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1879,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10894,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-341}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":248893928,"num_objects":30388,"num_object_clones":0,"num_object_copies":121552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29022,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30388,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":160449,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2615,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":497958912,"data_stored":497958912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160873,"ondisk_log_size":160873,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084406,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":735620,"kb_used_data":122184,"kb_used_omap":4,"kb_used_meta":613371,"kb_avail":93636220,"statfs":{"total":96636764160,"available":95883489280,"internally_reserved":0,"allocated":125116416,"data_stored":124905835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":628092627},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,4,4,1,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986283,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":609660,"kb_used_data":122596,"kb_used_omap":4,"kb_used_meta":487035,"kb_avail":93762180,"statfs":{"total":96636764160,"available":96012472320,"internally_reserved":0,"allocated":125538304,"data_stored":125327723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":498724562},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,5,0,7,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673089,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":739396,"kb_used_data":122344,"kb_used_omap":1,"kb_used_meta":617022,"kb_avail":93632444,"statfs":{"total":96636764160,"available":95879622656,"internally_reserved":0,"allocated":125280256,"data_stored":125069675,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":631830981},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,5,4,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182520,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":740104,"kb_used_data":122608,"kb_used_omap":4,"kb_used_meta":617467,"kb_avail":93631736,"statfs":{"total":96636764160,"available":95878897664,"internally_reserved":0,"allocated":125550592,"data_stored":125340011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":632286925},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,1,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":124571648,"data_stored":124571648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":124538880,"data_stored":124538880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":124731392,"data_stored":124731392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":124116992,"data_stored":124116992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:43:51.886 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:43:52.550 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:43:52.550 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:43:52.580 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":347,"stamp":"2024-09-05T16:43:51.025771+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":248893928,"num_objects":30388,"num_object_clones":0,"num_object_copies":121552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29022,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30388,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":160449,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2615,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160873,"ondisk_log_size":160873,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2824780,"kb_used_data":489732,"kb_used_omap":14,"kb_used_meta":2334897,"kb_avail":374662580,"statfs":{"total":386547056640,"available":383654481920,"internally_reserved":0,"allocated":501485568,"data_stored":500643244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2390935095},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,4,15,12,11,13],"upper_bound":256},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16048128,"num_objects":-1959,"num_object_clones":0,"num_object_copies":-7836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1959,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1959,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002758"},"pg_stats":[{"pgid":"1.f","version":"39'10992","reported_seq":11232,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:49.917113+0000","last_change":"2024-09-05T16:43:43.436935+0000","last_active":"2024-09-05T16:43:49.917113+0000","last_peered":"2024-09-05T16:43:49.917113+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:43:49.917113+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"29'900","ondisk_log_start":"29'900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15663104,"num_objects":1912,"num_object_clones":0,"num_object_copies":7648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2171,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1912,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10992,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-259}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'10709","reported_seq":15355,"reported_epoch":39,"state":"active+clean+scrubbing","last_fresh":"2024-09-05T16:43:49.558471+0000","last_change":"2024-09-05T16:43:45.841421+0000","last_active":"2024-09-05T16:43:49.558471+0000","last_peered":"2024-09-05T16:43:49.558471+0000","last_clean":"2024-09-05T16:43:49.558471+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:43:49.558471+0000","last_undegraded":"2024-09-05T16:43:49.558471+0000","last_fullsized":"2024-09-05T16:43:49.558471+0000","mapping_epoch":38,"log_start":"29'700","ondisk_log_start":"29'700","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":1788,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 3s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15111328,"num_objects":1845,"num_object_clones":0,"num_object_copies":7380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1845,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10707,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2271,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'10880","reported_seq":11007,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:49.199478+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:43:49.199478+0000","last_peered":"2024-09-05T16:43:49.199478+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:43:49.199478+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"29'800","ondisk_log_start":"29'800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15333992,"num_objects":1873,"num_object_clones":0,"num_object_copies":7492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1873,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10879,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-359}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'11001","reported_seq":15735,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:43:49.537138+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:43:49.537138+0000","last_peered":"2024-09-05T16:43:49.537138+0000","last_clean":"2024-09-05T16:43:49.537138+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:43:49.537138+0000","last_undegraded":"2024-09-05T16:43:49.537138+0000","last_fullsized":"2024-09-05T16:43:49.537138+0000","mapping_epoch":34,"log_start":"29'1000","ondisk_log_start":"29'1000","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15835136,"num_objects":1933,"num_object_clones":0,"num_object_copies":7732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1933,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5301,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'10687","reported_seq":10790,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:49.916264+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:43:49.916264+0000","last_peered":"2024-09-05T16:43:49.916264+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:43:49.916264+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"29'600","ondisk_log_start":"29'600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15627424,"num_objects":1908,"num_object_clones":0,"num_object_copies":7632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1908,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10685,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-237}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'10912","reported_seq":11031,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:49.954281+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:43:49.954281+0000","last_peered":"2024-09-05T16:43:49.954281+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:43:49.954281+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"29'900","ondisk_log_start":"29'900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15237120,"num_objects":1860,"num_object_clones":0,"num_object_copies":7440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1860,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10912,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-359}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'10732","reported_seq":10844,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:49.454994+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:43:49.454994+0000","last_peered":"2024-09-05T16:43:49.454994+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:43:49.454994+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"29'700","ondisk_log_start":"29'700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15807648,"num_objects":1930,"num_object_clones":0,"num_object_copies":7720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1930,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10730,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-201}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'10862","reported_seq":10954,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:49.421021+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:43:49.421021+0000","last_peered":"2024-09-05T16:43:49.421021+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:43:49.421021+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"29'800","ondisk_log_start":"29'800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15701152,"num_objects":1917,"num_object_clones":0,"num_object_copies":7668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1917,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10860,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-315}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"39'10893","reported_seq":10998,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:49.954747+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:43:49.954747+0000","last_peered":"2024-09-05T16:43:49.954747+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:43:49.954747+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"29'800","ondisk_log_start":"29'800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15403328,"num_objects":1881,"num_object_clones":0,"num_object_copies":7524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1881,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10890,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-363}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'10926","reported_seq":11052,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:49.061772+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:43:49.061772+0000","last_peered":"2024-09-05T16:43:49.061772+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:43:49.061772+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"29'900","ondisk_log_start":"29'900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15430816,"num_objects":1884,"num_object_clones":0,"num_object_copies":7536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1884,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10924,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-361}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'11168","reported_seq":11299,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:49.824336+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:43:49.824336+0000","last_peered":"2024-09-05T16:43:49.824336+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:43:49.824336+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"29'1100","ondisk_log_start":"29'1100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16010992,"num_objects":1955,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11165,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-326}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'10830","reported_seq":10931,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:50.004243+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:43:50.004243+0000","last_peered":"2024-09-05T16:43:50.004243+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:43:50.004243+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"29'800","ondisk_log_start":"29'800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15800912,"num_objects":1929,"num_object_clones":0,"num_object_copies":7716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1929,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10829,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-309}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'10961","reported_seq":15701,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:43:49.427973+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:43:49.427973+0000","last_peered":"2024-09-05T16:43:49.427973+0000","last_clean":"2024-09-05T16:43:49.427973+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:43:49.427973+0000","last_undegraded":"2024-09-05T16:43:49.427973+0000","last_fullsized":"2024-09-05T16:43:49.427973+0000","mapping_epoch":36,"log_start":"29'900","ondisk_log_start":"29'900","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15130624,"num_objects":1847,"num_object_clones":0,"num_object_copies":7388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1847,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2561,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'11181","reported_seq":11294,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:49.848507+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:43:49.848507+0000","last_peered":"2024-09-05T16:43:49.848507+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:43:49.848507+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"29'1100","ondisk_log_start":"29'1100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15930528,"num_objects":1945,"num_object_clones":0,"num_object_copies":7780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1945,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11179,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-389}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'10943","reported_seq":11060,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:49.267999+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:43:49.267999+0000","last_peered":"2024-09-05T16:43:49.267999+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:43:49.267999+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"29'900","ondisk_log_start":"29'900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15479968,"num_objects":1890,"num_object_clones":0,"num_object_copies":7560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1890,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10941,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-418}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'10896","reported_seq":11009,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:49.232123+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:43:49.232123+0000","last_peered":"2024-09-05T16:43:49.232123+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:43:49.232123+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"29'800","ondisk_log_start":"29'800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15389856,"num_objects":1879,"num_object_clones":0,"num_object_copies":7516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1879,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10894,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-341}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":248893928,"num_objects":30388,"num_object_clones":0,"num_object_copies":121552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29022,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30388,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":160449,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2615,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":497958912,"data_stored":497958912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160873,"ondisk_log_size":160873,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084406,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":735620,"kb_used_data":122184,"kb_used_omap":4,"kb_used_meta":613371,"kb_avail":93636220,"statfs":{"total":96636764160,"available":95883489280,"internally_reserved":0,"allocated":125116416,"data_stored":124905835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":628092627},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,4,4,1,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986283,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":609660,"kb_used_data":122596,"kb_used_omap":4,"kb_used_meta":487035,"kb_avail":93762180,"statfs":{"total":96636764160,"available":96012472320,"internally_reserved":0,"allocated":125538304,"data_stored":125327723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":498724562},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,5,0,7,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673089,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":739396,"kb_used_data":122344,"kb_used_omap":1,"kb_used_meta":617022,"kb_avail":93632444,"statfs":{"total":96636764160,"available":95879622656,"internally_reserved":0,"allocated":125280256,"data_stored":125069675,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":631830981},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,5,4,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182520,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":740104,"kb_used_data":122608,"kb_used_omap":4,"kb_used_meta":617467,"kb_avail":93631736,"statfs":{"total":96636764160,"available":95878897664,"internally_reserved":0,"allocated":125550592,"data_stored":125340011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":632286925},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,1,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":124571648,"data_stored":124571648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":124538880,"data_stored":124538880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":124731392,"data_stored":124731392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":124116992,"data_stored":124116992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:43:52.581 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-05T16:43:52.581 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "39'10992", 'reported_seq': 11232, 'reported_epoch': 39, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:49.917113+0000', 'last_change': '2024-09-05T16:43:43.436935+0000', 'last_active': '2024-09-05T16:43:49.917113+0000', 'last_peered': '2024-09-05T16:43:49.917113+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.828289+0000', 'last_became_peered': '2024-09-05T16:34:56.828289+0000', 'last_unstale': '2024-09-05T16:43:49.917113+0000', 'last_undegraded': '2024-09-05T16:34:56.612160+0000', 'last_fullsized': '2024-09-05T16:34:56.557632+0000', 'mapping_epoch': 32, 'log_start': "29'900", 'ondisk_log_start': "29'900", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10092, 'log_dups_size': 0, 'ondisk_log_size': 10092, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15663104, 'num_objects': 1912, 'num_object_clones': 0, 'num_object_copies': 7648, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2171, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1912, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10992, 'num_write_kb': 51616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 91, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -259}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:43:52.581 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-05T16:43:52.581 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "39'10709", 'reported_seq': 15355, 'reported_epoch': 39, 'state': 'active+clean+scrubbing', 'last_fresh': '2024-09-05T16:43:49.558471+0000', 'last_change': '2024-09-05T16:43:45.841421+0000', 'last_active': '2024-09-05T16:43:49.558471+0000', 'last_peered': '2024-09-05T16:43:49.558471+0000', 'last_clean': '2024-09-05T16:43:49.558471+0000', 'last_became_active': '2024-09-05T16:43:44.806229+0000', 'last_became_peered': '2024-09-05T16:43:44.806229+0000', 'last_unstale': '2024-09-05T16:43:49.558471+0000', 'last_undegraded': '2024-09-05T16:43:49.558471+0000', 'last_fullsized': '2024-09-05T16:43:49.558471+0000', 'mapping_epoch': 38, 'log_start': "29'700", 'ondisk_log_start': "29'700", 'created': 15, 'last_epoch_clean': 39, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 1788, 'log_size': 10009, 'log_dups_size': 0, 'ondisk_log_size': 10009, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'scrubbing for 3s', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15111328, 'num_objects': 1845, 'num_object_clones': 0, 'num_object_copies': 7380, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1845, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10707, 'num_write_kb': 50212, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2271, 'num_bytes_recovered': 2613248, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [1, 0, 3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-05T16:43:52.582 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:43:52.582 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "39'10880", 'reported_seq': 11007, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:49.199478+0000', 'last_change': '2024-09-05T16:34:56.867900+0000', 'last_active': '2024-09-05T16:43:49.199478+0000', 'last_peered': '2024-09-05T16:43:49.199478+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:43:49.199478+0000', 'last_undegraded': '2024-09-05T16:34:56.590972+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 32, 'log_start': "29'800", 'ondisk_log_start': "29'800", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15333992, 'num_objects': 1873, 'num_object_clones': 0, 'num_object_copies': 7492, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2233, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1873, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 10879, 'num_write_kb': 50999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -359}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:43:52.582 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:43:52.582 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "39'10687", 'reported_seq': 10790, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:49.916264+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:43:49.916264+0000', 'last_peered': '2024-09-05T16:43:49.916264+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:43:49.916264+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "29'600", 'ondisk_log_start': "29'600", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10087, 'log_dups_size': 0, 'ondisk_log_size': 10087, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15627424, 'num_objects': 1908, 'num_object_clones': 0, 'num_object_copies': 7632, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2147, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1908, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10685, 'num_write_kb': 50373, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -237}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:43:52.582 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:43:52.582 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "39'10912", 'reported_seq': 11031, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:49.954281+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:43:49.954281+0000', 'last_peered': '2024-09-05T16:43:49.954281+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:43:49.954281+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "29'900", 'ondisk_log_start': "29'900", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10012, 'log_dups_size': 0, 'ondisk_log_size': 10012, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15237120, 'num_objects': 1860, 'num_object_clones': 0, 'num_object_copies': 7440, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1860, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10912, 'num_write_kb': 51088, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -359}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:43:52.582 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:43:52.582 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "39'10732", 'reported_seq': 10844, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:49.454994+0000', 'last_change': '2024-09-05T16:34:56.903436+0000', 'last_active': '2024-09-05T16:43:49.454994+0000', 'last_peered': '2024-09-05T16:43:49.454994+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:43:49.454994+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "29'700", 'ondisk_log_start': "29'700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10032, 'log_dups_size': 0, 'ondisk_log_size': 10032, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15807648, 'num_objects': 1930, 'num_object_clones': 0, 'num_object_copies': 7720, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2133, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1930, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10730, 'num_write_kb': 50636, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -201}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:43:52.582 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-05T16:43:52.582 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "39'10862", 'reported_seq': 10954, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:49.421021+0000', 'last_change': '2024-09-05T16:34:56.883319+0000', 'last_active': '2024-09-05T16:43:49.421021+0000', 'last_peered': '2024-09-05T16:43:49.421021+0000', 'last_clean': '2024-09-05T16:33:53.505305+0000', 'last_became_active': '2024-09-05T16:34:56.873389+0000', 'last_became_peered': '2024-09-05T16:34:56.873389+0000', 'last_unstale': '2024-09-05T16:43:49.421021+0000', 'last_undegraded': '2024-09-05T16:34:56.602567+0000', 'last_fullsized': '2024-09-05T16:34:56.559958+0000', 'mapping_epoch': 32, 'log_start': "29'800", 'ondisk_log_start': "29'800", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10062, 'log_dups_size': 0, 'ondisk_log_size': 10062, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15701152, 'num_objects': 1917, 'num_object_clones': 0, 'num_object_copies': 7668, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2234, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1917, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10860, 'num_write_kb': 51108, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -315}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:43:52.582 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:43:52.582 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "39'10893", 'reported_seq': 10998, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:49.954747+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:43:49.954747+0000', 'last_peered': '2024-09-05T16:43:49.954747+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:43:49.954747+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "29'800", 'ondisk_log_start': "29'800", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10093, 'log_dups_size': 0, 'ondisk_log_size': 10093, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15403328, 'num_objects': 1881, 'num_object_clones': 0, 'num_object_copies': 7524, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1881, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10890, 'num_write_kb': 51081, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -363}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:43:52.583 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:43:52.583 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "39'10926", 'reported_seq': 11052, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:49.061772+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:43:49.061772+0000', 'last_peered': '2024-09-05T16:43:49.061772+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:43:49.061772+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "29'900", 'ondisk_log_start': "29'900", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10026, 'log_dups_size': 0, 'ondisk_log_size': 10026, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15430816, 'num_objects': 1884, 'num_object_clones': 0, 'num_object_copies': 7536, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1884, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10924, 'num_write_kb': 51230, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -361}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:43:52.583 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:43:52.583 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "39'11168", 'reported_seq': 11299, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:49.824336+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:43:49.824336+0000', 'last_peered': '2024-09-05T16:43:49.824336+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:43:49.824336+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "29'1100", 'ondisk_log_start': "29'1100", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10068, 'log_dups_size': 0, 'ondisk_log_size': 10068, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16010992, 'num_objects': 1955, 'num_object_clones': 0, 'num_object_copies': 7820, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2284, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1955, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11165, 'num_write_kb': 52476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -326}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:43:52.583 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:43:52.583 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "39'10830", 'reported_seq': 10931, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:50.004243+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:43:50.004243+0000', 'last_peered': '2024-09-05T16:43:50.004243+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:43:50.004243+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "29'800", 'ondisk_log_start': "29'800", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15800912, 'num_objects': 1929, 'num_object_clones': 0, 'num_object_copies': 7716, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1929, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10829, 'num_write_kb': 51031, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -309}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:43:52.583 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:43:52.583 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "39'11181", 'reported_seq': 11294, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:49.848507+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:43:49.848507+0000', 'last_peered': '2024-09-05T16:43:49.848507+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:43:49.848507+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "29'1100", 'ondisk_log_start': "29'1100", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10081, 'log_dups_size': 0, 'ondisk_log_size': 10081, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15930528, 'num_objects': 1945, 'num_object_clones': 0, 'num_object_copies': 7780, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1945, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11179, 'num_write_kb': 52500, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -389}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:43:52.583 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:43:52.583 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "39'10943", 'reported_seq': 11060, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:49.267999+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:43:49.267999+0000', 'last_peered': '2024-09-05T16:43:49.267999+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:43:49.267999+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "29'900", 'ondisk_log_start': "29'900", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10043, 'log_dups_size': 0, 'ondisk_log_size': 10043, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15479968, 'num_objects': 1890, 'num_object_clones': 0, 'num_object_copies': 7560, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2310, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1890, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10941, 'num_write_kb': 51325, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -418}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:43:52.583 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:43:52.583 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "39'10896", 'reported_seq': 11009, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:49.232123+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:43:49.232123+0000', 'last_peered': '2024-09-05T16:43:49.232123+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:43:49.232123+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "29'800", 'ondisk_log_start': "29'800", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10096, 'log_dups_size': 0, 'ondisk_log_size': 10096, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15389856, 'num_objects': 1879, 'num_object_clones': 0, 'num_object_copies': 7516, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1879, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10894, 'num_write_kb': 51092, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -341}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:43:52.584 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:43:53.321 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:43:53.322 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 29022/121552 objects degraded (23.876%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":636,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":39,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":2},{"state_name":"active+clean+scrubbing","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":30388,"data_bytes":248893928,"bytes_used":2892574720,"bytes_avail":383654481920,"bytes_total":386547056640,"degraded_objects":29022,"degraded_total":121552,"degraded_ratio":0.23876201132025798,"recovering_objects_per_sec":12,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":210},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:43:53.339 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:43:53.340 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:43:53.991 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:43:53.991 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:43:54.010 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":348,"stamp":"2024-09-05T16:43:53.026690+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":248893928,"num_objects":30388,"num_object_clones":0,"num_object_copies":121552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29022,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30388,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":160449,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2615,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160873,"ondisk_log_size":160873,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2824780,"kb_used_data":489732,"kb_used_omap":14,"kb_used_meta":2334897,"kb_avail":374662580,"statfs":{"total":386547056640,"available":383654481920,"internally_reserved":0,"allocated":501485568,"data_stored":500643244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2390935095},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,4,15,12,11,13],"upper_bound":256},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16048128,"num_objects":-1959,"num_object_clones":0,"num_object_copies":-7836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1959,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1959,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.298422"},"pg_stats":[{"pgid":"1.f","version":"39'10992","reported_seq":11232,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:49.917113+0000","last_change":"2024-09-05T16:43:43.436935+0000","last_active":"2024-09-05T16:43:49.917113+0000","last_peered":"2024-09-05T16:43:49.917113+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:43:49.917113+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"29'900","ondisk_log_start":"29'900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15663104,"num_objects":1912,"num_object_clones":0,"num_object_copies":7648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2171,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1912,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10992,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-259}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'10709","reported_seq":15355,"reported_epoch":39,"state":"active+clean+scrubbing","last_fresh":"2024-09-05T16:43:49.558471+0000","last_change":"2024-09-05T16:43:45.841421+0000","last_active":"2024-09-05T16:43:49.558471+0000","last_peered":"2024-09-05T16:43:49.558471+0000","last_clean":"2024-09-05T16:43:49.558471+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:43:49.558471+0000","last_undegraded":"2024-09-05T16:43:49.558471+0000","last_fullsized":"2024-09-05T16:43:49.558471+0000","mapping_epoch":38,"log_start":"29'700","ondisk_log_start":"29'700","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":1788,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 3s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15111328,"num_objects":1845,"num_object_clones":0,"num_object_copies":7380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1845,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10707,"num_write_kb":50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2271,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'10880","reported_seq":11007,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:49.199478+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:43:49.199478+0000","last_peered":"2024-09-05T16:43:49.199478+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:43:49.199478+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"29'800","ondisk_log_start":"29'800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15333992,"num_objects":1873,"num_object_clones":0,"num_object_copies":7492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1873,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10879,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-359}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'11001","reported_seq":15735,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:43:49.537138+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:43:49.537138+0000","last_peered":"2024-09-05T16:43:49.537138+0000","last_clean":"2024-09-05T16:43:49.537138+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:43:49.537138+0000","last_undegraded":"2024-09-05T16:43:49.537138+0000","last_fullsized":"2024-09-05T16:43:49.537138+0000","mapping_epoch":34,"log_start":"29'1000","ondisk_log_start":"29'1000","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15835136,"num_objects":1933,"num_object_clones":0,"num_object_copies":7732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1933,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5301,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'10687","reported_seq":10790,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:49.916264+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:43:49.916264+0000","last_peered":"2024-09-05T16:43:49.916264+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:43:49.916264+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"29'600","ondisk_log_start":"29'600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15627424,"num_objects":1908,"num_object_clones":0,"num_object_copies":7632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1908,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10685,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-237}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'10912","reported_seq":11031,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:49.954281+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:43:49.954281+0000","last_peered":"2024-09-05T16:43:49.954281+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:43:49.954281+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"29'900","ondisk_log_start":"29'900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15237120,"num_objects":1860,"num_object_clones":0,"num_object_copies":7440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1860,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10912,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-359}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'10732","reported_seq":10844,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:49.454994+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:43:49.454994+0000","last_peered":"2024-09-05T16:43:49.454994+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:43:49.454994+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"29'700","ondisk_log_start":"29'700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15807648,"num_objects":1930,"num_object_clones":0,"num_object_copies":7720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1930,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10730,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-201}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'10862","reported_seq":10954,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:49.421021+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:43:49.421021+0000","last_peered":"2024-09-05T16:43:49.421021+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:43:49.421021+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"29'800","ondisk_log_start":"29'800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15701152,"num_objects":1917,"num_object_clones":0,"num_object_copies":7668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1917,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10860,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-315}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"39'10893","reported_seq":10998,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:49.954747+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:43:49.954747+0000","last_peered":"2024-09-05T16:43:49.954747+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:43:49.954747+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"29'800","ondisk_log_start":"29'800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15403328,"num_objects":1881,"num_object_clones":0,"num_object_copies":7524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1881,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10890,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-363}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'10926","reported_seq":11052,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:49.061772+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:43:49.061772+0000","last_peered":"2024-09-05T16:43:49.061772+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:43:49.061772+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"29'900","ondisk_log_start":"29'900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15430816,"num_objects":1884,"num_object_clones":0,"num_object_copies":7536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1884,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10924,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-361}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'11168","reported_seq":11299,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:49.824336+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:43:49.824336+0000","last_peered":"2024-09-05T16:43:49.824336+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:43:49.824336+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"29'1100","ondisk_log_start":"29'1100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16010992,"num_objects":1955,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11165,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-326}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'10830","reported_seq":10931,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:50.004243+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:43:50.004243+0000","last_peered":"2024-09-05T16:43:50.004243+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:43:50.004243+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"29'800","ondisk_log_start":"29'800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15800912,"num_objects":1929,"num_object_clones":0,"num_object_copies":7716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1929,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10829,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-309}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'10961","reported_seq":15701,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:43:49.427973+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:43:49.427973+0000","last_peered":"2024-09-05T16:43:49.427973+0000","last_clean":"2024-09-05T16:43:49.427973+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:43:49.427973+0000","last_undegraded":"2024-09-05T16:43:49.427973+0000","last_fullsized":"2024-09-05T16:43:49.427973+0000","mapping_epoch":36,"log_start":"29'900","ondisk_log_start":"29'900","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15130624,"num_objects":1847,"num_object_clones":0,"num_object_copies":7388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1847,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2561,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'11181","reported_seq":11294,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:49.848507+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:43:49.848507+0000","last_peered":"2024-09-05T16:43:49.848507+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:43:49.848507+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"29'1100","ondisk_log_start":"29'1100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15930528,"num_objects":1945,"num_object_clones":0,"num_object_copies":7780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1945,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11179,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-389}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'10943","reported_seq":11060,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:49.267999+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:43:49.267999+0000","last_peered":"2024-09-05T16:43:49.267999+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:43:49.267999+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"29'900","ondisk_log_start":"29'900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15479968,"num_objects":1890,"num_object_clones":0,"num_object_copies":7560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1890,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10941,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-418}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'10896","reported_seq":11009,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:49.232123+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:43:49.232123+0000","last_peered":"2024-09-05T16:43:49.232123+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:43:49.232123+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"29'800","ondisk_log_start":"29'800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15389856,"num_objects":1879,"num_object_clones":0,"num_object_copies":7516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1879,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10894,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-341}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":248893928,"num_objects":30388,"num_object_clones":0,"num_object_copies":121552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29022,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30388,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":160449,"num_write_kb":722927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2615,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":497958912,"data_stored":497958912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160873,"ondisk_log_size":160873,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084406,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":735620,"kb_used_data":122184,"kb_used_omap":4,"kb_used_meta":613371,"kb_avail":93636220,"statfs":{"total":96636764160,"available":95883489280,"internally_reserved":0,"allocated":125116416,"data_stored":124905835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":628092627},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,4,4,1,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986283,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":609660,"kb_used_data":122596,"kb_used_omap":4,"kb_used_meta":487035,"kb_avail":93762180,"statfs":{"total":96636764160,"available":96012472320,"internally_reserved":0,"allocated":125538304,"data_stored":125327723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":498724562},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,5,0,7,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673089,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":739396,"kb_used_data":122344,"kb_used_omap":1,"kb_used_meta":617022,"kb_avail":93632444,"statfs":{"total":96636764160,"available":95879622656,"internally_reserved":0,"allocated":125280256,"data_stored":125069675,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":631830981},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,5,4,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182520,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":740104,"kb_used_data":122608,"kb_used_omap":4,"kb_used_meta":617467,"kb_avail":93631736,"statfs":{"total":96636764160,"available":95878897664,"internally_reserved":0,"allocated":125550592,"data_stored":125340011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":632286925},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,1,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":124571648,"data_stored":124571648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":124538880,"data_stored":124538880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":124731392,"data_stored":124731392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":124116992,"data_stored":124116992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:43:57.012 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:43:57.676 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:43:57.677 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:43:57.698 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":350,"stamp":"2024-09-05T16:43:57.028085+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":240590152,"num_objects":29374,"num_object_clones":0,"num_object_copies":117496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28984,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29374,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":150776,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":382,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160885,"ondisk_log_size":160885,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2846152,"kb_used_data":473668,"kb_used_omap":14,"kb_used_meta":2372401,"kb_avail":374641208,"statfs":{"total":386547056640,"available":383632596992,"internally_reserved":0,"allocated":485036032,"data_stored":484193708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2429339191},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,6,8,7,7,13,16,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16250016,"num_objects":-1984,"num_object_clones":0,"num_object_copies":-7936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-84,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1984,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-8703,"num_write_kb":-50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-2187,"num_bytes_recovered":-2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004456"},"pg_stats":[{"pgid":"1.f","version":"39'11050","reported_seq":11366,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:55.026014+0000","last_change":"2024-09-05T16:43:43.436935+0000","last_active":"2024-09-05T16:43:55.026014+0000","last_peered":"2024-09-05T16:43:55.026014+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:43:55.026014+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"29'1000","ondisk_log_start":"29'1000","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15187968,"num_objects":1854,"num_object_clones":0,"num_object_copies":7416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1854,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11050,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":129,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-279}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'10777","reported_seq":15433,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:43:54.625715+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:43:54.625715+0000","last_peered":"2024-09-05T16:43:54.625715+0000","last_clean":"2024-09-05T16:43:54.625715+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:43:54.625715+0000","last_undegraded":"2024-09-05T16:43:54.625715+0000","last_fullsized":"2024-09-05T16:43:54.625715+0000","mapping_epoch":38,"log_start":"29'700","ondisk_log_start":"29'700","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14540800,"num_objects":1775,"num_object_clones":0,"num_object_copies":7100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1775,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":90,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'10951","reported_seq":11076,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:54.263194+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:43:54.263194+0000","last_peered":"2024-09-05T16:43:54.263194+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:43:54.263194+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"29'900","ondisk_log_start":"29'900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14752360,"num_objects":1802,"num_object_clones":0,"num_object_copies":7208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1802,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10950,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-430}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'11072","reported_seq":15806,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:43:54.625827+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:43:54.625827+0000","last_peered":"2024-09-05T16:43:54.625827+0000","last_clean":"2024-09-05T16:43:54.625827+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:43:54.625827+0000","last_undegraded":"2024-09-05T16:43:54.625827+0000","last_fullsized":"2024-09-05T16:43:54.625827+0000","mapping_epoch":34,"log_start":"29'1000","ondisk_log_start":"29'1000","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15253504,"num_objects":1862,"num_object_clones":0,"num_object_copies":7448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1862,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5372,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'10754","reported_seq":10857,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:54.909046+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:43:54.909046+0000","last_peered":"2024-09-05T16:43:54.909046+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:43:54.909046+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"29'700","ondisk_log_start":"29'700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15078560,"num_objects":1841,"num_object_clones":0,"num_object_copies":7364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1841,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10752,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-304}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'10973","reported_seq":11092,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:55.025539+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:43:55.025539+0000","last_peered":"2024-09-05T16:43:55.025539+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:43:55.025539+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"29'900","ondisk_log_start":"29'900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14737408,"num_objects":1799,"num_object_clones":0,"num_object_copies":7196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1799,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10973,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-420}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'10786","reported_seq":10898,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:54.045991+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:43:54.045991+0000","last_peered":"2024-09-05T16:43:54.045991+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:43:54.045991+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"29'700","ondisk_log_start":"29'700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15365280,"num_objects":1876,"num_object_clones":0,"num_object_copies":7504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1876,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10784,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-255}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'10933","reported_seq":11026,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:54.369864+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:43:54.369864+0000","last_peered":"2024-09-05T16:43:54.369864+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:43:54.369864+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"29'900","ondisk_log_start":"29'900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15119520,"num_objects":1846,"num_object_clones":0,"num_object_copies":7384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10931,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-386}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"39'10953","reported_seq":11058,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:55.018612+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:43:55.018612+0000","last_peered":"2024-09-05T16:43:55.018612+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:43:55.018612+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"29'900","ondisk_log_start":"29'900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14911808,"num_objects":1821,"num_object_clones":0,"num_object_copies":7284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1821,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10950,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-423}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'10983","reported_seq":11109,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:54.336477+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:43:54.336477+0000","last_peered":"2024-09-05T16:43:54.336477+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:43:54.336477+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"29'900","ondisk_log_start":"29'900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14963872,"num_objects":1827,"num_object_clones":0,"num_object_copies":7308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1827,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10981,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-418}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'11236","reported_seq":11365,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:55.077659+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:43:55.077659+0000","last_peered":"2024-09-05T16:43:55.077659+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:43:55.077659+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"29'1200","ondisk_log_start":"29'1200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15453936,"num_objects":1887,"num_object_clones":0,"num_object_copies":7548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1887,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11233,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-394}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'10885","reported_seq":10988,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:54.894965+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:43:54.894965+0000","last_peered":"2024-09-05T16:43:54.894965+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:43:54.894965+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"29'800","ondisk_log_start":"29'800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15350352,"num_objects":1874,"num_object_clones":0,"num_object_copies":7496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1874,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10884,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-364}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'11024","reported_seq":15764,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:43:54.653401+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:43:54.653401+0000","last_peered":"2024-09-05T16:43:54.653401+0000","last_clean":"2024-09-05T16:43:54.653401+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:43:54.653401+0000","last_undegraded":"2024-09-05T16:43:54.653401+0000","last_fullsized":"2024-09-05T16:43:54.653401+0000","mapping_epoch":36,"log_start":"29'1000","ondisk_log_start":"29'1000","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14614528,"num_objects":1784,"num_object_clones":0,"num_object_copies":7136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1784,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2624,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'11245","reported_seq":11357,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:55.074333+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:43:55.074333+0000","last_peered":"2024-09-05T16:43:55.074333+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:43:55.074333+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"29'1200","ondisk_log_start":"29'1200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15406240,"num_objects":1881,"num_object_clones":0,"num_object_copies":7524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1881,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11243,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-453}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'11002","reported_seq":11119,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:54.247098+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:43:54.247098+0000","last_peered":"2024-09-05T16:43:54.247098+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:43:54.247098+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"29'1000","ondisk_log_start":"29'1000","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14996640,"num_objects":1831,"num_object_clones":0,"num_object_copies":7324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1831,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11000,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-477}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'10961","reported_seq":11072,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:54.396822+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:43:54.396822+0000","last_peered":"2024-09-05T16:43:54.396822+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:43:54.396822+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"29'900","ondisk_log_start":"29'900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14857376,"num_objects":1814,"num_object_clones":0,"num_object_copies":7256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1814,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10959,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-406}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":240590152,"num_objects":29374,"num_object_clones":0,"num_object_copies":117496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28984,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29374,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":150776,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":382,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":481452032,"data_stored":481452032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160885,"ondisk_log_size":160885,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084407,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":753816,"kb_used_data":118216,"kb_used_omap":4,"kb_used_meta":635579,"kb_avail":93618024,"statfs":{"total":96636764160,"available":95864856576,"internally_reserved":0,"allocated":121053184,"data_stored":120842603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":650833619},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,4,4,1,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986284,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":609856,"kb_used_data":118636,"kb_used_omap":4,"kb_used_meta":491195,"kb_avail":93761984,"statfs":{"total":96636764160,"available":96012271616,"internally_reserved":0,"allocated":121483264,"data_stored":121272683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":502984402},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,3,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673090,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":740828,"kb_used_data":118220,"kb_used_omap":1,"kb_used_meta":622590,"kb_avail":93631012,"statfs":{"total":96636764160,"available":95878156288,"internally_reserved":0,"allocated":121057280,"data_stored":120846699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":637532613},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,3,1,0,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182521,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":741652,"kb_used_data":118596,"kb_used_omap":4,"kb_used_meta":623035,"kb_avail":93630188,"statfs":{"total":96636764160,"available":95877312512,"internally_reserved":0,"allocated":121442304,"data_stored":121231723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":637988557},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,0,0,7,5,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":120483840,"data_stored":120483840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":120397824,"data_stored":120397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":120565760,"data_stored":120565760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":120004608,"data_stored":120004608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:43:57.699 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:43:58.284 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:43:58.285 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:43:58.306 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":350,"stamp":"2024-09-05T16:43:57.028085+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":240590152,"num_objects":29374,"num_object_clones":0,"num_object_copies":117496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28984,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29374,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":150776,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":382,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160885,"ondisk_log_size":160885,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2846152,"kb_used_data":473668,"kb_used_omap":14,"kb_used_meta":2372401,"kb_avail":374641208,"statfs":{"total":386547056640,"available":383632596992,"internally_reserved":0,"allocated":485036032,"data_stored":484193708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2429339191},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,6,8,7,7,13,16,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16250016,"num_objects":-1984,"num_object_clones":0,"num_object_copies":-7936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-84,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1984,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-8703,"num_write_kb":-50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-2187,"num_bytes_recovered":-2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004456"},"pg_stats":[{"pgid":"1.f","version":"39'11050","reported_seq":11366,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:55.026014+0000","last_change":"2024-09-05T16:43:43.436935+0000","last_active":"2024-09-05T16:43:55.026014+0000","last_peered":"2024-09-05T16:43:55.026014+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:43:55.026014+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"29'1000","ondisk_log_start":"29'1000","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15187968,"num_objects":1854,"num_object_clones":0,"num_object_copies":7416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1854,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11050,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":129,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-279}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'10777","reported_seq":15433,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:43:54.625715+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:43:54.625715+0000","last_peered":"2024-09-05T16:43:54.625715+0000","last_clean":"2024-09-05T16:43:54.625715+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:43:54.625715+0000","last_undegraded":"2024-09-05T16:43:54.625715+0000","last_fullsized":"2024-09-05T16:43:54.625715+0000","mapping_epoch":38,"log_start":"29'700","ondisk_log_start":"29'700","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14540800,"num_objects":1775,"num_object_clones":0,"num_object_copies":7100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1775,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":90,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'10951","reported_seq":11076,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:54.263194+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:43:54.263194+0000","last_peered":"2024-09-05T16:43:54.263194+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:43:54.263194+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"29'900","ondisk_log_start":"29'900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14752360,"num_objects":1802,"num_object_clones":0,"num_object_copies":7208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1802,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10950,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-430}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'11072","reported_seq":15806,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:43:54.625827+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:43:54.625827+0000","last_peered":"2024-09-05T16:43:54.625827+0000","last_clean":"2024-09-05T16:43:54.625827+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:43:54.625827+0000","last_undegraded":"2024-09-05T16:43:54.625827+0000","last_fullsized":"2024-09-05T16:43:54.625827+0000","mapping_epoch":34,"log_start":"29'1000","ondisk_log_start":"29'1000","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15253504,"num_objects":1862,"num_object_clones":0,"num_object_copies":7448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1862,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5372,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'10754","reported_seq":10857,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:54.909046+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:43:54.909046+0000","last_peered":"2024-09-05T16:43:54.909046+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:43:54.909046+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"29'700","ondisk_log_start":"29'700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15078560,"num_objects":1841,"num_object_clones":0,"num_object_copies":7364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1841,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10752,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-304}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'10973","reported_seq":11092,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:55.025539+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:43:55.025539+0000","last_peered":"2024-09-05T16:43:55.025539+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:43:55.025539+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"29'900","ondisk_log_start":"29'900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14737408,"num_objects":1799,"num_object_clones":0,"num_object_copies":7196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1799,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10973,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-420}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'10786","reported_seq":10898,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:54.045991+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:43:54.045991+0000","last_peered":"2024-09-05T16:43:54.045991+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:43:54.045991+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"29'700","ondisk_log_start":"29'700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15365280,"num_objects":1876,"num_object_clones":0,"num_object_copies":7504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1876,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10784,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-255}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'10933","reported_seq":11026,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:54.369864+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:43:54.369864+0000","last_peered":"2024-09-05T16:43:54.369864+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:43:54.369864+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"29'900","ondisk_log_start":"29'900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15119520,"num_objects":1846,"num_object_clones":0,"num_object_copies":7384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10931,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-386}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"39'10953","reported_seq":11058,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:55.018612+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:43:55.018612+0000","last_peered":"2024-09-05T16:43:55.018612+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:43:55.018612+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"29'900","ondisk_log_start":"29'900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14911808,"num_objects":1821,"num_object_clones":0,"num_object_copies":7284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1821,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10950,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-423}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'10983","reported_seq":11109,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:54.336477+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:43:54.336477+0000","last_peered":"2024-09-05T16:43:54.336477+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:43:54.336477+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"29'900","ondisk_log_start":"29'900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14963872,"num_objects":1827,"num_object_clones":0,"num_object_copies":7308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1827,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10981,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-418}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'11236","reported_seq":11365,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:55.077659+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:43:55.077659+0000","last_peered":"2024-09-05T16:43:55.077659+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:43:55.077659+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"29'1200","ondisk_log_start":"29'1200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15453936,"num_objects":1887,"num_object_clones":0,"num_object_copies":7548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1887,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11233,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-394}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'10885","reported_seq":10988,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:54.894965+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:43:54.894965+0000","last_peered":"2024-09-05T16:43:54.894965+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:43:54.894965+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"29'800","ondisk_log_start":"29'800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15350352,"num_objects":1874,"num_object_clones":0,"num_object_copies":7496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1874,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10884,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-364}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'11024","reported_seq":15764,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:43:54.653401+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:43:54.653401+0000","last_peered":"2024-09-05T16:43:54.653401+0000","last_clean":"2024-09-05T16:43:54.653401+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:43:54.653401+0000","last_undegraded":"2024-09-05T16:43:54.653401+0000","last_fullsized":"2024-09-05T16:43:54.653401+0000","mapping_epoch":36,"log_start":"29'1000","ondisk_log_start":"29'1000","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14614528,"num_objects":1784,"num_object_clones":0,"num_object_copies":7136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1784,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2624,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'11245","reported_seq":11357,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:55.074333+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:43:55.074333+0000","last_peered":"2024-09-05T16:43:55.074333+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:43:55.074333+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"29'1200","ondisk_log_start":"29'1200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15406240,"num_objects":1881,"num_object_clones":0,"num_object_copies":7524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1881,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11243,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-453}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'11002","reported_seq":11119,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:54.247098+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:43:54.247098+0000","last_peered":"2024-09-05T16:43:54.247098+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:43:54.247098+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"29'1000","ondisk_log_start":"29'1000","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14996640,"num_objects":1831,"num_object_clones":0,"num_object_copies":7324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1831,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11000,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-477}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'10961","reported_seq":11072,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:54.396822+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:43:54.396822+0000","last_peered":"2024-09-05T16:43:54.396822+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:43:54.396822+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"29'900","ondisk_log_start":"29'900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14857376,"num_objects":1814,"num_object_clones":0,"num_object_copies":7256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1814,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10959,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-406}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":240590152,"num_objects":29374,"num_object_clones":0,"num_object_copies":117496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28984,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29374,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":150776,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":382,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":481452032,"data_stored":481452032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160885,"ondisk_log_size":160885,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084407,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":753816,"kb_used_data":118216,"kb_used_omap":4,"kb_used_meta":635579,"kb_avail":93618024,"statfs":{"total":96636764160,"available":95864856576,"internally_reserved":0,"allocated":121053184,"data_stored":120842603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":650833619},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,4,4,1,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986284,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":609856,"kb_used_data":118636,"kb_used_omap":4,"kb_used_meta":491195,"kb_avail":93761984,"statfs":{"total":96636764160,"available":96012271616,"internally_reserved":0,"allocated":121483264,"data_stored":121272683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":502984402},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,3,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673090,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":740828,"kb_used_data":118220,"kb_used_omap":1,"kb_used_meta":622590,"kb_avail":93631012,"statfs":{"total":96636764160,"available":95878156288,"internally_reserved":0,"allocated":121057280,"data_stored":120846699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":637532613},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,3,1,0,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182521,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":741652,"kb_used_data":118596,"kb_used_omap":4,"kb_used_meta":623035,"kb_avail":93630188,"statfs":{"total":96636764160,"available":95877312512,"internally_reserved":0,"allocated":121442304,"data_stored":121231723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":637988557},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,0,0,7,5,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":120483840,"data_stored":120483840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":120397824,"data_stored":120397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":120565760,"data_stored":120565760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":120004608,"data_stored":120004608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:43:58.308 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-05T16:43:58.308 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "39'11050", 'reported_seq': 11366, 'reported_epoch': 39, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:55.026014+0000', 'last_change': '2024-09-05T16:43:43.436935+0000', 'last_active': '2024-09-05T16:43:55.026014+0000', 'last_peered': '2024-09-05T16:43:55.026014+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.828289+0000', 'last_became_peered': '2024-09-05T16:34:56.828289+0000', 'last_unstale': '2024-09-05T16:43:55.026014+0000', 'last_undegraded': '2024-09-05T16:34:56.612160+0000', 'last_fullsized': '2024-09-05T16:34:56.557632+0000', 'mapping_epoch': 32, 'log_start': "29'1000", 'ondisk_log_start': "29'1000", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15187968, 'num_objects': 1854, 'num_object_clones': 0, 'num_object_copies': 7416, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2133, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1854, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11050, 'num_write_kb': 51616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 129, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -279}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:43:58.308 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:43:58.308 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "39'10951", 'reported_seq': 11076, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:54.263194+0000', 'last_change': '2024-09-05T16:34:56.867900+0000', 'last_active': '2024-09-05T16:43:54.263194+0000', 'last_peered': '2024-09-05T16:43:54.263194+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:43:54.263194+0000', 'last_undegraded': '2024-09-05T16:34:56.590972+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 32, 'log_start': "29'900", 'ondisk_log_start': "29'900", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10051, 'log_dups_size': 0, 'ondisk_log_size': 10051, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14752360, 'num_objects': 1802, 'num_object_clones': 0, 'num_object_copies': 7208, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2233, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1802, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 10950, 'num_write_kb': 50999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -430}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:43:58.308 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:43:58.308 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "39'10754", 'reported_seq': 10857, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:54.909046+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:43:54.909046+0000', 'last_peered': '2024-09-05T16:43:54.909046+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:43:54.909046+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "29'700", 'ondisk_log_start': "29'700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10054, 'log_dups_size': 0, 'ondisk_log_size': 10054, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15078560, 'num_objects': 1841, 'num_object_clones': 0, 'num_object_copies': 7364, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2147, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1841, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10752, 'num_write_kb': 50373, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -304}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:43:58.308 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:43:58.308 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "39'10973", 'reported_seq': 11092, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:55.025539+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:43:55.025539+0000', 'last_peered': '2024-09-05T16:43:55.025539+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:43:55.025539+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "29'900", 'ondisk_log_start': "29'900", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10073, 'log_dups_size': 0, 'ondisk_log_size': 10073, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14737408, 'num_objects': 1799, 'num_object_clones': 0, 'num_object_copies': 7196, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1799, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10973, 'num_write_kb': 51088, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -420}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:43:58.308 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:43:58.309 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "39'10786", 'reported_seq': 10898, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:54.045991+0000', 'last_change': '2024-09-05T16:34:56.903436+0000', 'last_active': '2024-09-05T16:43:54.045991+0000', 'last_peered': '2024-09-05T16:43:54.045991+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:43:54.045991+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "29'700", 'ondisk_log_start': "29'700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10086, 'log_dups_size': 0, 'ondisk_log_size': 10086, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15365280, 'num_objects': 1876, 'num_object_clones': 0, 'num_object_copies': 7504, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2133, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1876, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10784, 'num_write_kb': 50636, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -255}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:43:58.309 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-05T16:43:58.309 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "39'10933", 'reported_seq': 11026, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:54.369864+0000', 'last_change': '2024-09-05T16:34:56.883319+0000', 'last_active': '2024-09-05T16:43:54.369864+0000', 'last_peered': '2024-09-05T16:43:54.369864+0000', 'last_clean': '2024-09-05T16:33:53.505305+0000', 'last_became_active': '2024-09-05T16:34:56.873389+0000', 'last_became_peered': '2024-09-05T16:34:56.873389+0000', 'last_unstale': '2024-09-05T16:43:54.369864+0000', 'last_undegraded': '2024-09-05T16:34:56.602567+0000', 'last_fullsized': '2024-09-05T16:34:56.559958+0000', 'mapping_epoch': 32, 'log_start': "29'900", 'ondisk_log_start': "29'900", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10033, 'log_dups_size': 0, 'ondisk_log_size': 10033, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15119520, 'num_objects': 1846, 'num_object_clones': 0, 'num_object_copies': 7384, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2234, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1846, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10931, 'num_write_kb': 51108, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -386}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:43:58.309 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:43:58.309 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "39'10953", 'reported_seq': 11058, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:55.018612+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:43:55.018612+0000', 'last_peered': '2024-09-05T16:43:55.018612+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:43:55.018612+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "29'900", 'ondisk_log_start': "29'900", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10053, 'log_dups_size': 0, 'ondisk_log_size': 10053, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14911808, 'num_objects': 1821, 'num_object_clones': 0, 'num_object_copies': 7284, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1821, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10950, 'num_write_kb': 51081, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -423}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:43:58.309 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:43:58.309 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "39'10983", 'reported_seq': 11109, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:54.336477+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:43:54.336477+0000', 'last_peered': '2024-09-05T16:43:54.336477+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:43:54.336477+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "29'900", 'ondisk_log_start': "29'900", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10083, 'log_dups_size': 0, 'ondisk_log_size': 10083, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14963872, 'num_objects': 1827, 'num_object_clones': 0, 'num_object_copies': 7308, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1827, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10981, 'num_write_kb': 51230, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -418}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:43:58.309 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:43:58.309 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "39'11236", 'reported_seq': 11365, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:55.077659+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:43:55.077659+0000', 'last_peered': '2024-09-05T16:43:55.077659+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:43:55.077659+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "29'1200", 'ondisk_log_start': "29'1200", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10036, 'log_dups_size': 0, 'ondisk_log_size': 10036, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15453936, 'num_objects': 1887, 'num_object_clones': 0, 'num_object_copies': 7548, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2284, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1887, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11233, 'num_write_kb': 52476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -394}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:43:58.309 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:43:58.309 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "39'10885", 'reported_seq': 10988, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:54.894965+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:43:54.894965+0000', 'last_peered': '2024-09-05T16:43:54.894965+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:43:54.894965+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "29'800", 'ondisk_log_start': "29'800", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10085, 'log_dups_size': 0, 'ondisk_log_size': 10085, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15350352, 'num_objects': 1874, 'num_object_clones': 0, 'num_object_copies': 7496, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1874, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10884, 'num_write_kb': 51031, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -364}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:43:58.310 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:43:58.310 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "39'11245", 'reported_seq': 11357, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:55.074333+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:43:55.074333+0000', 'last_peered': '2024-09-05T16:43:55.074333+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:43:55.074333+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "29'1200", 'ondisk_log_start': "29'1200", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10045, 'log_dups_size': 0, 'ondisk_log_size': 10045, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15406240, 'num_objects': 1881, 'num_object_clones': 0, 'num_object_copies': 7524, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1881, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11243, 'num_write_kb': 52500, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -453}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:43:58.310 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:43:58.310 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "39'11002", 'reported_seq': 11119, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:54.247098+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:43:54.247098+0000', 'last_peered': '2024-09-05T16:43:54.247098+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:43:54.247098+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "29'1000", 'ondisk_log_start': "29'1000", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10002, 'log_dups_size': 0, 'ondisk_log_size': 10002, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14996640, 'num_objects': 1831, 'num_object_clones': 0, 'num_object_copies': 7324, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2310, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1831, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11000, 'num_write_kb': 51325, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -477}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:43:58.310 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:43:58.310 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "39'10961", 'reported_seq': 11072, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:54.396822+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:43:54.396822+0000', 'last_peered': '2024-09-05T16:43:54.396822+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:43:54.396822+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "29'900", 'ondisk_log_start': "29'900", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10061, 'log_dups_size': 0, 'ondisk_log_size': 10061, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14857376, 'num_objects': 1814, 'num_object_clones': 0, 'num_object_copies': 7256, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1814, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10959, 'num_write_kb': 51092, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -406}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:43:58.310 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:43:59.023 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:43:59.023 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 28984/117496 objects degraded (24.668%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":642,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":39,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":29374,"data_bytes":240590152,"bytes_used":2914459648,"bytes_avail":383632596992,"bytes_total":386547056640,"degraded_objects":28984,"degraded_total":117496,"degraded_ratio":0.2466807380676789},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:43:59.047 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-09-05T16:43:59.047 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:43:59.687 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:43:59.687 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:43:59.707 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":351,"stamp":"2024-09-05T16:43:59.028837+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":240590152,"num_objects":29374,"num_object_clones":0,"num_object_copies":117496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28984,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29374,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":150776,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":382,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160885,"ondisk_log_size":160885,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2846152,"kb_used_data":473668,"kb_used_omap":14,"kb_used_meta":2372401,"kb_avail":374641208,"statfs":{"total":386547056640,"available":383632596992,"internally_reserved":0,"allocated":485036032,"data_stored":484193708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2429339191},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,6,8,7,7,13,16,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16250016,"num_objects":-1984,"num_object_clones":0,"num_object_copies":-7936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-84,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1984,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-8703,"num_write_kb":-50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-2187,"num_bytes_recovered":-2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004460"},"pg_stats":[{"pgid":"1.f","version":"39'11050","reported_seq":11366,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:55.026014+0000","last_change":"2024-09-05T16:43:43.436935+0000","last_active":"2024-09-05T16:43:55.026014+0000","last_peered":"2024-09-05T16:43:55.026014+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:43:55.026014+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"29'1000","ondisk_log_start":"29'1000","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15187968,"num_objects":1854,"num_object_clones":0,"num_object_copies":7416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1854,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11050,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":129,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-279}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'10777","reported_seq":15433,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:43:54.625715+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:43:54.625715+0000","last_peered":"2024-09-05T16:43:54.625715+0000","last_clean":"2024-09-05T16:43:54.625715+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:43:54.625715+0000","last_undegraded":"2024-09-05T16:43:54.625715+0000","last_fullsized":"2024-09-05T16:43:54.625715+0000","mapping_epoch":38,"log_start":"29'700","ondisk_log_start":"29'700","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14540800,"num_objects":1775,"num_object_clones":0,"num_object_copies":7100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1775,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":90,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'10951","reported_seq":11076,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:54.263194+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:43:54.263194+0000","last_peered":"2024-09-05T16:43:54.263194+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:43:54.263194+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"29'900","ondisk_log_start":"29'900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14752360,"num_objects":1802,"num_object_clones":0,"num_object_copies":7208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1802,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10950,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-430}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'11072","reported_seq":15806,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:43:54.625827+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:43:54.625827+0000","last_peered":"2024-09-05T16:43:54.625827+0000","last_clean":"2024-09-05T16:43:54.625827+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:43:54.625827+0000","last_undegraded":"2024-09-05T16:43:54.625827+0000","last_fullsized":"2024-09-05T16:43:54.625827+0000","mapping_epoch":34,"log_start":"29'1000","ondisk_log_start":"29'1000","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15253504,"num_objects":1862,"num_object_clones":0,"num_object_copies":7448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1862,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5372,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'10754","reported_seq":10857,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:54.909046+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:43:54.909046+0000","last_peered":"2024-09-05T16:43:54.909046+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:43:54.909046+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"29'700","ondisk_log_start":"29'700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15078560,"num_objects":1841,"num_object_clones":0,"num_object_copies":7364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1841,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10752,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-304}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'10973","reported_seq":11092,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:55.025539+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:43:55.025539+0000","last_peered":"2024-09-05T16:43:55.025539+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:43:55.025539+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"29'900","ondisk_log_start":"29'900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14737408,"num_objects":1799,"num_object_clones":0,"num_object_copies":7196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1799,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10973,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-420}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'10786","reported_seq":10898,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:54.045991+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:43:54.045991+0000","last_peered":"2024-09-05T16:43:54.045991+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:43:54.045991+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"29'700","ondisk_log_start":"29'700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15365280,"num_objects":1876,"num_object_clones":0,"num_object_copies":7504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1876,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10784,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-255}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'10933","reported_seq":11026,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:54.369864+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:43:54.369864+0000","last_peered":"2024-09-05T16:43:54.369864+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:43:54.369864+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"29'900","ondisk_log_start":"29'900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15119520,"num_objects":1846,"num_object_clones":0,"num_object_copies":7384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10931,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-386}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"39'10953","reported_seq":11058,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:55.018612+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:43:55.018612+0000","last_peered":"2024-09-05T16:43:55.018612+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:43:55.018612+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"29'900","ondisk_log_start":"29'900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14911808,"num_objects":1821,"num_object_clones":0,"num_object_copies":7284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1821,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10950,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-423}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'10983","reported_seq":11109,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:54.336477+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:43:54.336477+0000","last_peered":"2024-09-05T16:43:54.336477+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:43:54.336477+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"29'900","ondisk_log_start":"29'900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14963872,"num_objects":1827,"num_object_clones":0,"num_object_copies":7308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1827,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10981,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-418}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'11236","reported_seq":11365,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:55.077659+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:43:55.077659+0000","last_peered":"2024-09-05T16:43:55.077659+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:43:55.077659+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"29'1200","ondisk_log_start":"29'1200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15453936,"num_objects":1887,"num_object_clones":0,"num_object_copies":7548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1887,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11233,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-394}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'10885","reported_seq":10988,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:54.894965+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:43:54.894965+0000","last_peered":"2024-09-05T16:43:54.894965+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:43:54.894965+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"29'800","ondisk_log_start":"29'800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15350352,"num_objects":1874,"num_object_clones":0,"num_object_copies":7496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1874,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10884,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-364}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'11024","reported_seq":15764,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:43:54.653401+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:43:54.653401+0000","last_peered":"2024-09-05T16:43:54.653401+0000","last_clean":"2024-09-05T16:43:54.653401+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:43:54.653401+0000","last_undegraded":"2024-09-05T16:43:54.653401+0000","last_fullsized":"2024-09-05T16:43:54.653401+0000","mapping_epoch":36,"log_start":"29'1000","ondisk_log_start":"29'1000","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14614528,"num_objects":1784,"num_object_clones":0,"num_object_copies":7136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1784,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2624,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'11245","reported_seq":11357,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:55.074333+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:43:55.074333+0000","last_peered":"2024-09-05T16:43:55.074333+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:43:55.074333+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"29'1200","ondisk_log_start":"29'1200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15406240,"num_objects":1881,"num_object_clones":0,"num_object_copies":7524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1881,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11243,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-453}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'11002","reported_seq":11119,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:54.247098+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:43:54.247098+0000","last_peered":"2024-09-05T16:43:54.247098+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:43:54.247098+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"29'1000","ondisk_log_start":"29'1000","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14996640,"num_objects":1831,"num_object_clones":0,"num_object_copies":7324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1831,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11000,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-477}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'10961","reported_seq":11072,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:54.396822+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:43:54.396822+0000","last_peered":"2024-09-05T16:43:54.396822+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:43:54.396822+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"29'900","ondisk_log_start":"29'900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14857376,"num_objects":1814,"num_object_clones":0,"num_object_copies":7256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1814,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10959,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-406}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":240590152,"num_objects":29374,"num_object_clones":0,"num_object_copies":117496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28984,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29374,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":150776,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":382,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":481452032,"data_stored":481452032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160885,"ondisk_log_size":160885,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084407,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":753816,"kb_used_data":118216,"kb_used_omap":4,"kb_used_meta":635579,"kb_avail":93618024,"statfs":{"total":96636764160,"available":95864856576,"internally_reserved":0,"allocated":121053184,"data_stored":120842603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":650833619},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,4,4,1,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986284,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":609856,"kb_used_data":118636,"kb_used_omap":4,"kb_used_meta":491195,"kb_avail":93761984,"statfs":{"total":96636764160,"available":96012271616,"internally_reserved":0,"allocated":121483264,"data_stored":121272683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":502984402},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,3,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673090,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":740828,"kb_used_data":118220,"kb_used_omap":1,"kb_used_meta":622590,"kb_avail":93631012,"statfs":{"total":96636764160,"available":95878156288,"internally_reserved":0,"allocated":121057280,"data_stored":120846699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":637532613},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,3,1,0,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182521,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":741652,"kb_used_data":118596,"kb_used_omap":4,"kb_used_meta":623035,"kb_avail":93630188,"statfs":{"total":96636764160,"available":95877312512,"internally_reserved":0,"allocated":121442304,"data_stored":121231723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":637988557},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,0,0,7,5,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":120483840,"data_stored":120483840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":120397824,"data_stored":120397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":120565760,"data_stored":120565760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":120004608,"data_stored":120004608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:44:02.709 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:44:03.376 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:44:03.377 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:44:03.399 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":353,"stamp":"2024-09-05T16:44:03.030561+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":232275272,"num_objects":28359,"num_object_clones":0,"num_object_copies":113436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28940,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28359,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":151791,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":426,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160700,"ondisk_log_size":160700,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2825656,"kb_used_data":457312,"kb_used_omap":14,"kb_used_meta":2368241,"kb_avail":374661704,"statfs":{"total":386547056640,"available":383653584896,"internally_reserved":0,"allocated":468287488,"data_stored":467445164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2425079351},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,4,1,8,12,3,7,12,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16618656,"num_objects":-2029,"num_object_clones":0,"num_object_copies":-8116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-82,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2029,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-8658,"num_write_kb":-50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-2189,"num_bytes_recovered":-2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004789"},"pg_stats":[{"pgid":"1.f","version":"39'11114","reported_seq":11516,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:00.194397+0000","last_change":"2024-09-05T16:43:43.436935+0000","last_active":"2024-09-05T16:44:00.194397+0000","last_peered":"2024-09-05T16:44:00.194397+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:44:00.194397+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"29'1100","ondisk_log_start":"29'1100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14663680,"num_objects":1790,"num_object_clones":0,"num_object_copies":7160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2089,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1790,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11114,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":173,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-299}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'10842","reported_seq":15498,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:43:59.819748+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:43:59.819748+0000","last_peered":"2024-09-05T16:43:59.819748+0000","last_clean":"2024-09-05T16:43:59.819748+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:43:59.819748+0000","last_undegraded":"2024-09-05T16:43:59.819748+0000","last_fullsized":"2024-09-05T16:43:59.819748+0000","mapping_epoch":38,"log_start":"29'800","ondisk_log_start":"29'800","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14008320,"num_objects":1710,"num_object_clones":0,"num_object_copies":6840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1710,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":155,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'11011","reported_seq":11138,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:58.939572+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:43:58.939572+0000","last_peered":"2024-09-05T16:43:58.939572+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:43:58.939572+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"29'1000","ondisk_log_start":"29'1000","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14260840,"num_objects":1742,"num_object_clones":0,"num_object_copies":6968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1742,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11010,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-490}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'11137","reported_seq":15871,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:43:59.796843+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:43:59.796843+0000","last_peered":"2024-09-05T16:43:59.796843+0000","last_clean":"2024-09-05T16:43:59.796843+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:43:59.796843+0000","last_undegraded":"2024-09-05T16:43:59.796843+0000","last_fullsized":"2024-09-05T16:43:59.796843+0000","mapping_epoch":34,"log_start":"29'1100","ondisk_log_start":"29'1100","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14721024,"num_objects":1797,"num_object_clones":0,"num_object_copies":7188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1797,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5437,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'10819","reported_seq":10920,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:00.082927+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:44:00.082927+0000","last_peered":"2024-09-05T16:44:00.082927+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:44:00.082927+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"29'800","ondisk_log_start":"29'800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14546080,"num_objects":1776,"num_object_clones":0,"num_object_copies":7104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1776,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10817,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-369}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'11052","reported_seq":11171,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:00.067932+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:44:00.067932+0000","last_peered":"2024-09-05T16:44:00.067932+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:44:00.067932+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"29'1000","ondisk_log_start":"29'1000","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14090240,"num_objects":1720,"num_object_clones":0,"num_object_copies":6880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11052,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-499}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'10831","reported_seq":10943,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:59.486864+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:43:59.486864+0000","last_peered":"2024-09-05T16:43:59.486864+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:43:59.486864+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"29'800","ondisk_log_start":"29'800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14996640,"num_objects":1831,"num_object_clones":0,"num_object_copies":7324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1831,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10829,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-300}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'10999","reported_seq":11089,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:59.422669+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:43:59.422669+0000","last_peered":"2024-09-05T16:43:59.422669+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:43:59.422669+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"29'900","ondisk_log_start":"29'900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14578848,"num_objects":1780,"num_object_clones":0,"num_object_copies":7120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1780,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10997,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-452}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"39'11026","reported_seq":11130,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:00.093203+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:44:00.093203+0000","last_peered":"2024-09-05T16:44:00.093203+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:44:00.093203+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"29'1000","ondisk_log_start":"29'1000","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14313792,"num_objects":1748,"num_object_clones":0,"num_object_copies":6992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1748,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11023,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-496}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'11043","reported_seq":11169,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:59.189341+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:43:59.189341+0000","last_peered":"2024-09-05T16:43:59.189341+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:43:59.189341+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"29'1000","ondisk_log_start":"29'1000","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14472352,"num_objects":1767,"num_object_clones":0,"num_object_copies":7068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1767,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11041,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-478}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'11284","reported_seq":11415,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:00.078103+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:44:00.078103+0000","last_peered":"2024-09-05T16:44:00.078103+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:44:00.078103+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"29'1200","ondisk_log_start":"29'1200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15060720,"num_objects":1839,"num_object_clones":0,"num_object_copies":7356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1839,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11281,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-442}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'10950","reported_seq":11052,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:00.043399+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:44:00.043399+0000","last_peered":"2024-09-05T16:44:00.043399+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:44:00.043399+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"29'900","ondisk_log_start":"29'900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14817872,"num_objects":1809,"num_object_clones":0,"num_object_copies":7236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1809,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10949,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-429}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'11098","reported_seq":15838,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:43:59.794776+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:43:59.794776+0000","last_peered":"2024-09-05T16:43:59.794776+0000","last_clean":"2024-09-05T16:43:59.794776+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:43:59.794776+0000","last_undegraded":"2024-09-05T16:43:59.794776+0000","last_fullsized":"2024-09-05T16:43:59.794776+0000","mapping_epoch":36,"log_start":"29'1000","ondisk_log_start":"29'1000","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14008320,"num_objects":1710,"num_object_clones":0,"num_object_copies":6840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1710,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2698,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'11307","reported_seq":11420,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:00.135652+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:44:00.135652+0000","last_peered":"2024-09-05T16:44:00.135652+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:44:00.135652+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"29'1300","ondisk_log_start":"29'1300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14898336,"num_objects":1819,"num_object_clones":0,"num_object_copies":7276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1819,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11305,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-515}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'11055","reported_seq":11171,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:59.186125+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:43:59.186125+0000","last_peered":"2024-09-05T16:43:59.186125+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:43:59.186125+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"29'1000","ondisk_log_start":"29'1000","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14562464,"num_objects":1778,"num_object_clones":0,"num_object_copies":7112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1778,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11053,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-530}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'11032","reported_seq":11145,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:59.424141+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:43:59.424141+0000","last_peered":"2024-09-05T16:43:59.424141+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:43:59.424141+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"29'1000","ondisk_log_start":"29'1000","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14275744,"num_objects":1743,"num_object_clones":0,"num_object_copies":6972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1743,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11030,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-477}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":232275272,"num_objects":28359,"num_object_clones":0,"num_object_copies":113436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28940,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28359,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":151791,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":426,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":465006592,"data_stored":465006592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160700,"ondisk_log_size":160700,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084408,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":749760,"kb_used_data":114152,"kb_used_omap":4,"kb_used_meta":635579,"kb_avail":93622080,"statfs":{"total":96636764160,"available":95869009920,"internally_reserved":0,"allocated":116891648,"data_stored":116681067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":650833619},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,3,3,1,1,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986285,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":593304,"kb_used_data":114568,"kb_used_omap":4,"kb_used_meta":478715,"kb_avail":93778536,"statfs":{"total":96636764160,"available":96029220864,"internally_reserved":0,"allocated":117317632,"data_stored":117107051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":490204882},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,5,1,3,2,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673091,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":740928,"kb_used_data":114160,"kb_used_omap":1,"kb_used_meta":626750,"kb_avail":93630912,"statfs":{"total":96636764160,"available":95878053888,"internally_reserved":0,"allocated":116899840,"data_stored":116689259,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":641792453},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,0,0,3,1,3,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182522,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":741664,"kb_used_data":114432,"kb_used_omap":4,"kb_used_meta":627195,"kb_avail":93630176,"statfs":{"total":96636764160,"available":95877300224,"internally_reserved":0,"allocated":117178368,"data_stored":116967787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":642248397},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":116412416,"data_stored":116412416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":116264960,"data_stored":116264960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":116457472,"data_stored":116457472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":115871744,"data_stored":115871744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:44:03.400 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:44:04.059 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:44:04.059 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:44:04.075 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":353,"stamp":"2024-09-05T16:44:03.030561+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":232275272,"num_objects":28359,"num_object_clones":0,"num_object_copies":113436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28940,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28359,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":151791,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":426,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160700,"ondisk_log_size":160700,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2825656,"kb_used_data":457312,"kb_used_omap":14,"kb_used_meta":2368241,"kb_avail":374661704,"statfs":{"total":386547056640,"available":383653584896,"internally_reserved":0,"allocated":468287488,"data_stored":467445164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2425079351},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,4,1,8,12,3,7,12,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16618656,"num_objects":-2029,"num_object_clones":0,"num_object_copies":-8116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-82,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2029,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-8658,"num_write_kb":-50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-2189,"num_bytes_recovered":-2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004789"},"pg_stats":[{"pgid":"1.f","version":"39'11114","reported_seq":11516,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:00.194397+0000","last_change":"2024-09-05T16:43:43.436935+0000","last_active":"2024-09-05T16:44:00.194397+0000","last_peered":"2024-09-05T16:44:00.194397+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:44:00.194397+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"29'1100","ondisk_log_start":"29'1100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14663680,"num_objects":1790,"num_object_clones":0,"num_object_copies":7160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2089,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1790,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11114,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":173,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-299}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'10842","reported_seq":15498,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:43:59.819748+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:43:59.819748+0000","last_peered":"2024-09-05T16:43:59.819748+0000","last_clean":"2024-09-05T16:43:59.819748+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:43:59.819748+0000","last_undegraded":"2024-09-05T16:43:59.819748+0000","last_fullsized":"2024-09-05T16:43:59.819748+0000","mapping_epoch":38,"log_start":"29'800","ondisk_log_start":"29'800","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14008320,"num_objects":1710,"num_object_clones":0,"num_object_copies":6840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1710,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":155,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'11011","reported_seq":11138,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:58.939572+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:43:58.939572+0000","last_peered":"2024-09-05T16:43:58.939572+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:43:58.939572+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"29'1000","ondisk_log_start":"29'1000","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14260840,"num_objects":1742,"num_object_clones":0,"num_object_copies":6968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1742,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11010,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-490}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'11137","reported_seq":15871,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:43:59.796843+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:43:59.796843+0000","last_peered":"2024-09-05T16:43:59.796843+0000","last_clean":"2024-09-05T16:43:59.796843+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:43:59.796843+0000","last_undegraded":"2024-09-05T16:43:59.796843+0000","last_fullsized":"2024-09-05T16:43:59.796843+0000","mapping_epoch":34,"log_start":"29'1100","ondisk_log_start":"29'1100","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14721024,"num_objects":1797,"num_object_clones":0,"num_object_copies":7188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1797,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5437,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'10819","reported_seq":10920,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:00.082927+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:44:00.082927+0000","last_peered":"2024-09-05T16:44:00.082927+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:44:00.082927+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"29'800","ondisk_log_start":"29'800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14546080,"num_objects":1776,"num_object_clones":0,"num_object_copies":7104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1776,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10817,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-369}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'11052","reported_seq":11171,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:00.067932+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:44:00.067932+0000","last_peered":"2024-09-05T16:44:00.067932+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:44:00.067932+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"29'1000","ondisk_log_start":"29'1000","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14090240,"num_objects":1720,"num_object_clones":0,"num_object_copies":6880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11052,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-499}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'10831","reported_seq":10943,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:59.486864+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:43:59.486864+0000","last_peered":"2024-09-05T16:43:59.486864+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:43:59.486864+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"29'800","ondisk_log_start":"29'800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14996640,"num_objects":1831,"num_object_clones":0,"num_object_copies":7324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1831,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10829,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-300}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'10999","reported_seq":11089,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:59.422669+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:43:59.422669+0000","last_peered":"2024-09-05T16:43:59.422669+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:43:59.422669+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"29'900","ondisk_log_start":"29'900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14578848,"num_objects":1780,"num_object_clones":0,"num_object_copies":7120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1780,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10997,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-452}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"39'11026","reported_seq":11130,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:00.093203+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:44:00.093203+0000","last_peered":"2024-09-05T16:44:00.093203+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:44:00.093203+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"29'1000","ondisk_log_start":"29'1000","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14313792,"num_objects":1748,"num_object_clones":0,"num_object_copies":6992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1748,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11023,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-496}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'11043","reported_seq":11169,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:59.189341+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:43:59.189341+0000","last_peered":"2024-09-05T16:43:59.189341+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:43:59.189341+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"29'1000","ondisk_log_start":"29'1000","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14472352,"num_objects":1767,"num_object_clones":0,"num_object_copies":7068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1767,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11041,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-478}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'11284","reported_seq":11415,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:00.078103+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:44:00.078103+0000","last_peered":"2024-09-05T16:44:00.078103+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:44:00.078103+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"29'1200","ondisk_log_start":"29'1200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15060720,"num_objects":1839,"num_object_clones":0,"num_object_copies":7356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1839,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11281,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-442}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'10950","reported_seq":11052,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:00.043399+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:44:00.043399+0000","last_peered":"2024-09-05T16:44:00.043399+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:44:00.043399+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"29'900","ondisk_log_start":"29'900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14817872,"num_objects":1809,"num_object_clones":0,"num_object_copies":7236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1809,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10949,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-429}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'11098","reported_seq":15838,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:43:59.794776+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:43:59.794776+0000","last_peered":"2024-09-05T16:43:59.794776+0000","last_clean":"2024-09-05T16:43:59.794776+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:43:59.794776+0000","last_undegraded":"2024-09-05T16:43:59.794776+0000","last_fullsized":"2024-09-05T16:43:59.794776+0000","mapping_epoch":36,"log_start":"29'1000","ondisk_log_start":"29'1000","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14008320,"num_objects":1710,"num_object_clones":0,"num_object_copies":6840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1710,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2698,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'11307","reported_seq":11420,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:00.135652+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:44:00.135652+0000","last_peered":"2024-09-05T16:44:00.135652+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:44:00.135652+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"29'1300","ondisk_log_start":"29'1300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14898336,"num_objects":1819,"num_object_clones":0,"num_object_copies":7276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1819,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11305,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-515}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'11055","reported_seq":11171,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:59.186125+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:43:59.186125+0000","last_peered":"2024-09-05T16:43:59.186125+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:43:59.186125+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"29'1000","ondisk_log_start":"29'1000","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14562464,"num_objects":1778,"num_object_clones":0,"num_object_copies":7112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1778,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11053,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-530}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'11032","reported_seq":11145,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:43:59.424141+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:43:59.424141+0000","last_peered":"2024-09-05T16:43:59.424141+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:43:59.424141+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"29'1000","ondisk_log_start":"29'1000","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14275744,"num_objects":1743,"num_object_clones":0,"num_object_copies":6972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1743,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11030,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-477}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":232275272,"num_objects":28359,"num_object_clones":0,"num_object_copies":113436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28940,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28359,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":151791,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":426,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":465006592,"data_stored":465006592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160700,"ondisk_log_size":160700,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084408,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":749760,"kb_used_data":114152,"kb_used_omap":4,"kb_used_meta":635579,"kb_avail":93622080,"statfs":{"total":96636764160,"available":95869009920,"internally_reserved":0,"allocated":116891648,"data_stored":116681067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":650833619},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,3,3,1,1,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986285,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":593304,"kb_used_data":114568,"kb_used_omap":4,"kb_used_meta":478715,"kb_avail":93778536,"statfs":{"total":96636764160,"available":96029220864,"internally_reserved":0,"allocated":117317632,"data_stored":117107051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":490204882},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,5,1,3,2,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673091,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":740928,"kb_used_data":114160,"kb_used_omap":1,"kb_used_meta":626750,"kb_avail":93630912,"statfs":{"total":96636764160,"available":95878053888,"internally_reserved":0,"allocated":116899840,"data_stored":116689259,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":641792453},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,0,0,3,1,3,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182522,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":741664,"kb_used_data":114432,"kb_used_omap":4,"kb_used_meta":627195,"kb_avail":93630176,"statfs":{"total":96636764160,"available":95877300224,"internally_reserved":0,"allocated":117178368,"data_stored":116967787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":642248397},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":116412416,"data_stored":116412416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":116264960,"data_stored":116264960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":116457472,"data_stored":116457472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":115871744,"data_stored":115871744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:44:04.076 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-05T16:44:04.077 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "39'11114", 'reported_seq': 11516, 'reported_epoch': 39, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:00.194397+0000', 'last_change': '2024-09-05T16:43:43.436935+0000', 'last_active': '2024-09-05T16:44:00.194397+0000', 'last_peered': '2024-09-05T16:44:00.194397+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.828289+0000', 'last_became_peered': '2024-09-05T16:34:56.828289+0000', 'last_unstale': '2024-09-05T16:44:00.194397+0000', 'last_undegraded': '2024-09-05T16:34:56.612160+0000', 'last_fullsized': '2024-09-05T16:34:56.557632+0000', 'mapping_epoch': 32, 'log_start': "29'1100", 'ondisk_log_start': "29'1100", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10014, 'log_dups_size': 0, 'ondisk_log_size': 10014, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14663680, 'num_objects': 1790, 'num_object_clones': 0, 'num_object_copies': 7160, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2089, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1790, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11114, 'num_write_kb': 51616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 173, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -299}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:44:04.077 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:44:04.077 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "39'11011", 'reported_seq': 11138, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:58.939572+0000', 'last_change': '2024-09-05T16:34:56.867900+0000', 'last_active': '2024-09-05T16:43:58.939572+0000', 'last_peered': '2024-09-05T16:43:58.939572+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:43:58.939572+0000', 'last_undegraded': '2024-09-05T16:34:56.590972+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 32, 'log_start': "29'1000", 'ondisk_log_start': "29'1000", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10011, 'log_dups_size': 0, 'ondisk_log_size': 10011, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14260840, 'num_objects': 1742, 'num_object_clones': 0, 'num_object_copies': 6968, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2233, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1742, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 11010, 'num_write_kb': 50999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -490}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:44:04.077 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:44:04.077 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "39'10819", 'reported_seq': 10920, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:00.082927+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:44:00.082927+0000', 'last_peered': '2024-09-05T16:44:00.082927+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:44:00.082927+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "29'800", 'ondisk_log_start': "29'800", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10019, 'log_dups_size': 0, 'ondisk_log_size': 10019, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14546080, 'num_objects': 1776, 'num_object_clones': 0, 'num_object_copies': 7104, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2147, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1776, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10817, 'num_write_kb': 50373, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -369}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:44:04.077 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:44:04.077 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "39'11052", 'reported_seq': 11171, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:00.067932+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:44:00.067932+0000', 'last_peered': '2024-09-05T16:44:00.067932+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:44:00.067932+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "29'1000", 'ondisk_log_start': "29'1000", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10052, 'log_dups_size': 0, 'ondisk_log_size': 10052, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14090240, 'num_objects': 1720, 'num_object_clones': 0, 'num_object_copies': 6880, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1720, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11052, 'num_write_kb': 51088, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -499}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:44:04.077 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:44:04.077 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "39'10831", 'reported_seq': 10943, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:59.486864+0000', 'last_change': '2024-09-05T16:34:56.903436+0000', 'last_active': '2024-09-05T16:43:59.486864+0000', 'last_peered': '2024-09-05T16:43:59.486864+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:43:59.486864+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "29'800", 'ondisk_log_start': "29'800", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10031, 'log_dups_size': 0, 'ondisk_log_size': 10031, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14996640, 'num_objects': 1831, 'num_object_clones': 0, 'num_object_copies': 7324, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2133, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1831, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10829, 'num_write_kb': 50636, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -300}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:44:04.077 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-05T16:44:04.077 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "39'10999", 'reported_seq': 11089, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:59.422669+0000', 'last_change': '2024-09-05T16:34:56.883319+0000', 'last_active': '2024-09-05T16:43:59.422669+0000', 'last_peered': '2024-09-05T16:43:59.422669+0000', 'last_clean': '2024-09-05T16:33:53.505305+0000', 'last_became_active': '2024-09-05T16:34:56.873389+0000', 'last_became_peered': '2024-09-05T16:34:56.873389+0000', 'last_unstale': '2024-09-05T16:43:59.422669+0000', 'last_undegraded': '2024-09-05T16:34:56.602567+0000', 'last_fullsized': '2024-09-05T16:34:56.559958+0000', 'mapping_epoch': 32, 'log_start': "29'900", 'ondisk_log_start': "29'900", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10099, 'log_dups_size': 0, 'ondisk_log_size': 10099, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14578848, 'num_objects': 1780, 'num_object_clones': 0, 'num_object_copies': 7120, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2234, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1780, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10997, 'num_write_kb': 51108, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -452}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:44:04.078 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:44:04.078 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "39'11026", 'reported_seq': 11130, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:00.093203+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:44:00.093203+0000', 'last_peered': '2024-09-05T16:44:00.093203+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:44:00.093203+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "29'1000", 'ondisk_log_start': "29'1000", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10026, 'log_dups_size': 0, 'ondisk_log_size': 10026, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14313792, 'num_objects': 1748, 'num_object_clones': 0, 'num_object_copies': 6992, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1748, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11023, 'num_write_kb': 51081, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -496}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:44:04.078 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:44:04.078 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "39'11043", 'reported_seq': 11169, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:59.189341+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:43:59.189341+0000', 'last_peered': '2024-09-05T16:43:59.189341+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:43:59.189341+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "29'1000", 'ondisk_log_start': "29'1000", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10043, 'log_dups_size': 0, 'ondisk_log_size': 10043, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14472352, 'num_objects': 1767, 'num_object_clones': 0, 'num_object_copies': 7068, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1767, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11041, 'num_write_kb': 51230, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -478}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:44:04.078 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:44:04.078 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "39'11284", 'reported_seq': 11415, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:00.078103+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:44:00.078103+0000', 'last_peered': '2024-09-05T16:44:00.078103+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:44:00.078103+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "29'1200", 'ondisk_log_start': "29'1200", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10084, 'log_dups_size': 0, 'ondisk_log_size': 10084, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15060720, 'num_objects': 1839, 'num_object_clones': 0, 'num_object_copies': 7356, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2284, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1839, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11281, 'num_write_kb': 52476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -442}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:44:04.078 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:44:04.078 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "39'10950", 'reported_seq': 11052, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:00.043399+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:44:00.043399+0000', 'last_peered': '2024-09-05T16:44:00.043399+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:44:00.043399+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "29'900", 'ondisk_log_start': "29'900", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14817872, 'num_objects': 1809, 'num_object_clones': 0, 'num_object_copies': 7236, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1809, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10949, 'num_write_kb': 51031, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -429}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:44:04.078 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:44:04.078 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "39'11307", 'reported_seq': 11420, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:00.135652+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:44:00.135652+0000', 'last_peered': '2024-09-05T16:44:00.135652+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:44:00.135652+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "29'1300", 'ondisk_log_start': "29'1300", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10007, 'log_dups_size': 0, 'ondisk_log_size': 10007, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14898336, 'num_objects': 1819, 'num_object_clones': 0, 'num_object_copies': 7276, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1819, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11305, 'num_write_kb': 52500, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -515}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:44:04.078 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:44:04.078 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "39'11055", 'reported_seq': 11171, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:59.186125+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:43:59.186125+0000', 'last_peered': '2024-09-05T16:43:59.186125+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:43:59.186125+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "29'1000", 'ondisk_log_start': "29'1000", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10055, 'log_dups_size': 0, 'ondisk_log_size': 10055, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14562464, 'num_objects': 1778, 'num_object_clones': 0, 'num_object_copies': 7112, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2310, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1778, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11053, 'num_write_kb': 51325, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -530}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:44:04.079 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:44:04.079 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "39'11032", 'reported_seq': 11145, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:43:59.424141+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:43:59.424141+0000', 'last_peered': '2024-09-05T16:43:59.424141+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:43:59.424141+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "29'1000", 'ondisk_log_start': "29'1000", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10032, 'log_dups_size': 0, 'ondisk_log_size': 10032, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14275744, 'num_objects': 1743, 'num_object_clones': 0, 'num_object_copies': 6972, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1743, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11030, 'num_write_kb': 51092, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -477}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:44:04.079 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:44:04.806 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:44:04.809 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 28940/113436 objects degraded (25.512%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":648,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":39,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28359,"data_bytes":232275272,"bytes_used":2893471744,"bytes_avail":383653584896,"bytes_total":386547056640,"degraded_objects":28940,"degraded_total":113436,"degraded_ratio":0.25512183081208789},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:44:04.851 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-09-05T16:44:04.852 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:44:05.484 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:44:05.485 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:44:05.506 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":354,"stamp":"2024-09-05T16:44:05.031375+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":227908936,"num_objects":27826,"num_object_clones":0,"num_object_copies":111304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28940,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27826,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":152324,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":426,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160733,"ondisk_log_size":160733,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2861904,"kb_used_data":445000,"kb_used_omap":14,"kb_used_meta":2416817,"kb_avail":374625456,"statfs":{"total":386547056640,"available":383616466944,"internally_reserved":0,"allocated":455680000,"data_stored":454837676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2474821175},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,2,5,15,13,7,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20984992,"num_objects":-2562,"num_object_clones":0,"num_object_copies":-10248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-82,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2562,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-8125,"num_write_kb":-50212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-2189,"num_bytes_recovered":-2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004684"},"pg_stats":[{"pgid":"1.f","version":"39'11114","reported_seq":11516,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:00.194397+0000","last_change":"2024-09-05T16:43:43.436935+0000","last_active":"2024-09-05T16:44:00.194397+0000","last_peered":"2024-09-05T16:44:00.194397+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:44:00.194397+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"29'1100","ondisk_log_start":"29'1100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14663680,"num_objects":1790,"num_object_clones":0,"num_object_copies":7160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2089,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1790,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11114,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":173,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-299}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'10893","reported_seq":15549,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:44:04.621676+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:44:04.621676+0000","last_peered":"2024-09-05T16:44:04.621676+0000","last_clean":"2024-09-05T16:44:04.621676+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:44:04.621676+0000","last_undegraded":"2024-09-05T16:44:04.621676+0000","last_fullsized":"2024-09-05T16:44:04.621676+0000","mapping_epoch":38,"log_start":"29'800","ondisk_log_start":"29'800","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13590528,"num_objects":1659,"num_object_clones":0,"num_object_copies":6636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1659,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":206,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'11063","reported_seq":11190,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:04.254965+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:44:04.254965+0000","last_peered":"2024-09-05T16:44:04.254965+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:44:04.254965+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"29'1000","ondisk_log_start":"29'1000","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13834856,"num_objects":1690,"num_object_clones":0,"num_object_copies":6760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1690,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11062,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-542}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'11210","reported_seq":15944,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:44:04.797043+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:44:04.797043+0000","last_peered":"2024-09-05T16:44:04.797043+0000","last_clean":"2024-09-05T16:44:04.797043+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:44:04.797043+0000","last_undegraded":"2024-09-05T16:44:04.797043+0000","last_fullsized":"2024-09-05T16:44:04.797043+0000","mapping_epoch":34,"log_start":"29'1200","ondisk_log_start":"29'1200","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14123008,"num_objects":1724,"num_object_clones":0,"num_object_copies":6896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5510,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'10819","reported_seq":10920,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:00.082927+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:44:00.082927+0000","last_peered":"2024-09-05T16:44:00.082927+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:44:00.082927+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"29'800","ondisk_log_start":"29'800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14546080,"num_objects":1776,"num_object_clones":0,"num_object_copies":7104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1776,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10817,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-369}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'11052","reported_seq":11171,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:00.067932+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:44:00.067932+0000","last_peered":"2024-09-05T16:44:00.067932+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:44:00.067932+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"29'1000","ondisk_log_start":"29'1000","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14090240,"num_objects":1720,"num_object_clones":0,"num_object_copies":6880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11052,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-499}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'10888","reported_seq":11000,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:04.399057+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:44:04.399057+0000","last_peered":"2024-09-05T16:44:04.399057+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:44:04.399057+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"29'800","ondisk_log_start":"29'800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14529696,"num_objects":1774,"num_object_clones":0,"num_object_copies":7096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1774,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10886,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-357}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'11059","reported_seq":11152,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:04.349754+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:44:04.349754+0000","last_peered":"2024-09-05T16:44:04.349754+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:44:04.349754+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"29'1000","ondisk_log_start":"29'1000","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14087328,"num_objects":1720,"num_object_clones":0,"num_object_copies":6880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11057,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-512}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"39'11026","reported_seq":11130,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:00.093203+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:44:00.093203+0000","last_peered":"2024-09-05T16:44:00.093203+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:44:00.093203+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"29'1000","ondisk_log_start":"29'1000","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14313792,"num_objects":1748,"num_object_clones":0,"num_object_copies":6992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1748,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11023,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-496}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'11101","reported_seq":11226,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:04.188482+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:44:04.188482+0000","last_peered":"2024-09-05T16:44:04.188482+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:44:04.188482+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"29'1100","ondisk_log_start":"29'1100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13997216,"num_objects":1709,"num_object_clones":0,"num_object_copies":6836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1709,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11099,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-536}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'11284","reported_seq":11415,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:00.078103+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:44:00.078103+0000","last_peered":"2024-09-05T16:44:00.078103+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:44:00.078103+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"29'1200","ondisk_log_start":"29'1200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15060720,"num_objects":1839,"num_object_clones":0,"num_object_copies":7356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1839,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11281,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-442}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'10950","reported_seq":11052,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:00.043399+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:44:00.043399+0000","last_peered":"2024-09-05T16:44:00.043399+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:44:00.043399+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"29'900","ondisk_log_start":"29'900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14817872,"num_objects":1809,"num_object_clones":0,"num_object_copies":7236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1809,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10949,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-429}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'11151","reported_seq":15891,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:44:04.631808+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:44:04.631808+0000","last_peered":"2024-09-05T16:44:04.631808+0000","last_clean":"2024-09-05T16:44:04.631808+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:44:04.631808+0000","last_undegraded":"2024-09-05T16:44:04.631808+0000","last_fullsized":"2024-09-05T16:44:04.631808+0000","mapping_epoch":36,"log_start":"29'1100","ondisk_log_start":"29'1100","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13574144,"num_objects":1657,"num_object_clones":0,"num_object_copies":6628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1657,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2751,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'11307","reported_seq":11420,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:00.135652+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:44:00.135652+0000","last_peered":"2024-09-05T16:44:00.135652+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:44:00.135652+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"29'1300","ondisk_log_start":"29'1300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14898336,"num_objects":1819,"num_object_clones":0,"num_object_copies":7276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1819,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11305,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-515}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'11116","reported_seq":11233,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:04.171079+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:44:04.171079+0000","last_peered":"2024-09-05T16:44:04.171079+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:44:04.171079+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"29'1100","ondisk_log_start":"29'1100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14062752,"num_objects":1717,"num_object_clones":0,"num_object_copies":6868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1717,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11114,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-591}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'11100","reported_seq":11213,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:04.503528+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:44:04.503528+0000","last_peered":"2024-09-05T16:44:04.503528+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:44:04.503528+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"29'1000","ondisk_log_start":"29'1000","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13718688,"num_objects":1675,"num_object_clones":0,"num_object_copies":6700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1675,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11098,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-545}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":227908936,"num_objects":27826,"num_object_clones":0,"num_object_copies":111304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28940,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27826,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":152324,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":426,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":452407296,"data_stored":452407296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160733,"ondisk_log_size":160733,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084408,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":749760,"kb_used_data":114152,"kb_used_omap":4,"kb_used_meta":635579,"kb_avail":93622080,"statfs":{"total":96636764160,"available":95869009920,"internally_reserved":0,"allocated":116891648,"data_stored":116681067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":650833619},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,3,3,1,1,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986286,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":593372,"kb_used_data":110476,"kb_used_omap":4,"kb_used_meta":482875,"kb_avail":93778468,"statfs":{"total":96636764160,"available":96029151232,"internally_reserved":0,"allocated":113127424,"data_stored":112916843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":494464722},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,7,3,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673092,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":759004,"kb_used_data":110028,"kb_used_omap":1,"kb_used_meta":648958,"kb_avail":93612836,"statfs":{"total":96636764160,"available":95859544064,"internally_reserved":0,"allocated":112668672,"data_stored":112458091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":664533445},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,5,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182523,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":759768,"kb_used_data":110344,"kb_used_omap":4,"kb_used_meta":649403,"kb_avail":93612072,"statfs":{"total":96636764160,"available":95858761728,"internally_reserved":0,"allocated":112992256,"data_stored":112781675,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":664989389},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,3,4,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":112189440,"data_stored":112189440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":112066560,"data_stored":112066560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":112279552,"data_stored":112279552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":115871744,"data_stored":115871744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:44:08.508 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:44:09.153 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:44:09.155 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:44:09.180 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":356,"stamp":"2024-09-05T16:44:09.032801+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":223976776,"num_objects":27346,"num_object_clones":0,"num_object_copies":109384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28902,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27346,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":152804,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":464,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160913,"ondisk_log_size":160913,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2863364,"kb_used_data":440900,"kb_used_omap":14,"kb_used_meta":2422385,"kb_avail":374623996,"statfs":{"total":386547056640,"available":383614971904,"internally_reserved":0,"allocated":451481600,"data_stored":450639276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2480522807},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,15,13,12,6,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16613376,"num_objects":-2028,"num_object_clones":0,"num_object_copies":-8112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-82,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2028,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2028,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":82,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004716"},"pg_stats":[{"pgid":"1.f","version":"39'11185","reported_seq":11665,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:05.159173+0000","last_change":"2024-09-05T16:43:43.436935+0000","last_active":"2024-09-05T16:44:05.159173+0000","last_peered":"2024-09-05T16:44:05.159173+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:44:05.159173+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"29'1100","ondisk_log_start":"29'1100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14082048,"num_objects":1719,"num_object_clones":0,"num_object_copies":6876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2051,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1719,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11185,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":211,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-332}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'10893","reported_seq":15549,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:44:04.621676+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:44:04.621676+0000","last_peered":"2024-09-05T16:44:04.621676+0000","last_clean":"2024-09-05T16:44:04.621676+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:44:04.621676+0000","last_undegraded":"2024-09-05T16:44:04.621676+0000","last_fullsized":"2024-09-05T16:44:04.621676+0000","mapping_epoch":38,"log_start":"29'800","ondisk_log_start":"29'800","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13590528,"num_objects":1659,"num_object_clones":0,"num_object_copies":6636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1659,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":206,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'11063","reported_seq":11190,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:04.254965+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:44:04.254965+0000","last_peered":"2024-09-05T16:44:04.254965+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:44:04.254965+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"29'1000","ondisk_log_start":"29'1000","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13834856,"num_objects":1690,"num_object_clones":0,"num_object_copies":6760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1690,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11062,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-542}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'11210","reported_seq":15944,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:44:04.797043+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:44:04.797043+0000","last_peered":"2024-09-05T16:44:04.797043+0000","last_clean":"2024-09-05T16:44:04.797043+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:44:04.797043+0000","last_undegraded":"2024-09-05T16:44:04.797043+0000","last_fullsized":"2024-09-05T16:44:04.797043+0000","mapping_epoch":34,"log_start":"29'1200","ondisk_log_start":"29'1200","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14123008,"num_objects":1724,"num_object_clones":0,"num_object_copies":6896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5510,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'10893","reported_seq":10995,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:05.161839+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:44:05.161839+0000","last_peered":"2024-09-05T16:44:05.161839+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:44:05.161839+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"29'800","ondisk_log_start":"29'800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13939872,"num_objects":1702,"num_object_clones":0,"num_object_copies":6808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1702,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10891,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-443}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'11117","reported_seq":11236,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:05.190060+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:44:05.190060+0000","last_peered":"2024-09-05T16:44:05.190060+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:44:05.190060+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"29'1100","ondisk_log_start":"29'1100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13557760,"num_objects":1655,"num_object_clones":0,"num_object_copies":6620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1655,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11117,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-564}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'10888","reported_seq":11000,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:04.399057+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:44:04.399057+0000","last_peered":"2024-09-05T16:44:04.399057+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:44:04.399057+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"29'800","ondisk_log_start":"29'800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14529696,"num_objects":1774,"num_object_clones":0,"num_object_copies":7096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1774,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10886,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-357}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'11059","reported_seq":11152,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:04.349754+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:44:04.349754+0000","last_peered":"2024-09-05T16:44:04.349754+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:44:04.349754+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"29'1000","ondisk_log_start":"29'1000","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14087328,"num_objects":1720,"num_object_clones":0,"num_object_copies":6880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11057,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-512}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"39'11095","reported_seq":11200,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:05.196542+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:44:05.196542+0000","last_peered":"2024-09-05T16:44:05.196542+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:44:05.196542+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"29'1000","ondisk_log_start":"29'1000","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13748544,"num_objects":1679,"num_object_clones":0,"num_object_copies":6716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1679,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11092,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-565}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'11101","reported_seq":11226,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:04.188482+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:44:04.188482+0000","last_peered":"2024-09-05T16:44:04.188482+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:44:04.188482+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"29'1100","ondisk_log_start":"29'1100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13997216,"num_objects":1709,"num_object_clones":0,"num_object_copies":6836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1709,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11099,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-536}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'11345","reported_seq":11476,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:05.141620+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:44:05.141620+0000","last_peered":"2024-09-05T16:44:05.141620+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:44:05.141620+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"29'1300","ondisk_log_start":"29'1300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14561008,"num_objects":1778,"num_object_clones":0,"num_object_copies":7112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1778,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11342,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-503}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'11023","reported_seq":11125,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:05.072444+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:44:05.072444+0000","last_peered":"2024-09-05T16:44:05.072444+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:44:05.072444+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"29'1000","ondisk_log_start":"29'1000","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14219856,"num_objects":1736,"num_object_clones":0,"num_object_copies":6944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1736,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11022,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-502}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'11151","reported_seq":15891,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:44:04.631808+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:44:04.631808+0000","last_peered":"2024-09-05T16:44:04.631808+0000","last_clean":"2024-09-05T16:44:04.631808+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:44:04.631808+0000","last_undegraded":"2024-09-05T16:44:04.631808+0000","last_fullsized":"2024-09-05T16:44:04.631808+0000","mapping_epoch":36,"log_start":"29'1100","ondisk_log_start":"29'1100","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13574144,"num_objects":1657,"num_object_clones":0,"num_object_copies":6628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1657,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2751,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'11374","reported_seq":11487,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:05.062676+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:44:05.062676+0000","last_peered":"2024-09-05T16:44:05.062676+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:44:05.062676+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"29'1300","ondisk_log_start":"29'1300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14349472,"num_objects":1752,"num_object_clones":0,"num_object_copies":7008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11372,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-582}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'11116","reported_seq":11233,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:04.171079+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:44:04.171079+0000","last_peered":"2024-09-05T16:44:04.171079+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:44:04.171079+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"29'1100","ondisk_log_start":"29'1100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14062752,"num_objects":1717,"num_object_clones":0,"num_object_copies":6868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1717,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11114,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-591}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'11100","reported_seq":11213,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:04.503528+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:44:04.503528+0000","last_peered":"2024-09-05T16:44:04.503528+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:44:04.503528+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"29'1000","ondisk_log_start":"29'1000","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13718688,"num_objects":1675,"num_object_clones":0,"num_object_copies":6700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1675,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11098,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-545}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":223976776,"num_objects":27346,"num_object_clones":0,"num_object_copies":109384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28902,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27346,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":152804,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":464,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":448184320,"data_stored":448184320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160913,"ondisk_log_size":160913,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084409,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":751220,"kb_used_data":110052,"kb_used_omap":4,"kb_used_meta":641147,"kb_avail":93620620,"statfs":{"total":96636764160,"available":95867514880,"internally_reserved":0,"allocated":112693248,"data_stored":112482667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":656535251},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,1,6,3,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986286,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":593372,"kb_used_data":110476,"kb_used_omap":4,"kb_used_meta":482875,"kb_avail":93778468,"statfs":{"total":96636764160,"available":96029151232,"internally_reserved":0,"allocated":113127424,"data_stored":112916843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":494464722},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,7,3,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673092,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":759004,"kb_used_data":110028,"kb_used_omap":1,"kb_used_meta":648958,"kb_avail":93612836,"statfs":{"total":96636764160,"available":95859544064,"internally_reserved":0,"allocated":112668672,"data_stored":112458091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":664533445},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,5,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182523,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":759768,"kb_used_data":110344,"kb_used_omap":4,"kb_used_meta":649403,"kb_avail":93612072,"statfs":{"total":96636764160,"available":95858761728,"internally_reserved":0,"allocated":112992256,"data_stored":112781675,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":664989389},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,3,4,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":112189440,"data_stored":112189440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":112066560,"data_stored":112066560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":112279552,"data_stored":112279552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":111648768,"data_stored":111648768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:44:09.181 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:44:09.771 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:44:09.772 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:44:09.796 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":356,"stamp":"2024-09-05T16:44:09.032801+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":223976776,"num_objects":27346,"num_object_clones":0,"num_object_copies":109384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28902,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27346,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":152804,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":464,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160913,"ondisk_log_size":160913,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2863364,"kb_used_data":440900,"kb_used_omap":14,"kb_used_meta":2422385,"kb_avail":374623996,"statfs":{"total":386547056640,"available":383614971904,"internally_reserved":0,"allocated":451481600,"data_stored":450639276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2480522807},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,15,13,12,6,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16613376,"num_objects":-2028,"num_object_clones":0,"num_object_copies":-8112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-82,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2028,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2028,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":82,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004716"},"pg_stats":[{"pgid":"1.f","version":"39'11185","reported_seq":11665,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:05.159173+0000","last_change":"2024-09-05T16:43:43.436935+0000","last_active":"2024-09-05T16:44:05.159173+0000","last_peered":"2024-09-05T16:44:05.159173+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:44:05.159173+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"29'1100","ondisk_log_start":"29'1100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14082048,"num_objects":1719,"num_object_clones":0,"num_object_copies":6876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2051,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1719,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11185,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":211,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-332}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'10893","reported_seq":15549,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:44:04.621676+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:44:04.621676+0000","last_peered":"2024-09-05T16:44:04.621676+0000","last_clean":"2024-09-05T16:44:04.621676+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:44:04.621676+0000","last_undegraded":"2024-09-05T16:44:04.621676+0000","last_fullsized":"2024-09-05T16:44:04.621676+0000","mapping_epoch":38,"log_start":"29'800","ondisk_log_start":"29'800","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13590528,"num_objects":1659,"num_object_clones":0,"num_object_copies":6636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1659,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":206,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'11063","reported_seq":11190,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:04.254965+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:44:04.254965+0000","last_peered":"2024-09-05T16:44:04.254965+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:44:04.254965+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"29'1000","ondisk_log_start":"29'1000","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13834856,"num_objects":1690,"num_object_clones":0,"num_object_copies":6760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1690,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11062,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-542}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'11210","reported_seq":15944,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:44:04.797043+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:44:04.797043+0000","last_peered":"2024-09-05T16:44:04.797043+0000","last_clean":"2024-09-05T16:44:04.797043+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:44:04.797043+0000","last_undegraded":"2024-09-05T16:44:04.797043+0000","last_fullsized":"2024-09-05T16:44:04.797043+0000","mapping_epoch":34,"log_start":"29'1200","ondisk_log_start":"29'1200","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14123008,"num_objects":1724,"num_object_clones":0,"num_object_copies":6896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5510,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'10893","reported_seq":10995,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:05.161839+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:44:05.161839+0000","last_peered":"2024-09-05T16:44:05.161839+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:44:05.161839+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"29'800","ondisk_log_start":"29'800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13939872,"num_objects":1702,"num_object_clones":0,"num_object_copies":6808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1702,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10891,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-443}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'11117","reported_seq":11236,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:05.190060+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:44:05.190060+0000","last_peered":"2024-09-05T16:44:05.190060+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:44:05.190060+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"29'1100","ondisk_log_start":"29'1100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13557760,"num_objects":1655,"num_object_clones":0,"num_object_copies":6620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1655,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11117,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-564}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'10888","reported_seq":11000,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:04.399057+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:44:04.399057+0000","last_peered":"2024-09-05T16:44:04.399057+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:44:04.399057+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"29'800","ondisk_log_start":"29'800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14529696,"num_objects":1774,"num_object_clones":0,"num_object_copies":7096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1774,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10886,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-357}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'11059","reported_seq":11152,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:04.349754+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:44:04.349754+0000","last_peered":"2024-09-05T16:44:04.349754+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:44:04.349754+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"29'1000","ondisk_log_start":"29'1000","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14087328,"num_objects":1720,"num_object_clones":0,"num_object_copies":6880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11057,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-512}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"39'11095","reported_seq":11200,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:05.196542+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:44:05.196542+0000","last_peered":"2024-09-05T16:44:05.196542+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:44:05.196542+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"29'1000","ondisk_log_start":"29'1000","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13748544,"num_objects":1679,"num_object_clones":0,"num_object_copies":6716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1679,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11092,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-565}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'11101","reported_seq":11226,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:04.188482+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:44:04.188482+0000","last_peered":"2024-09-05T16:44:04.188482+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:44:04.188482+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"29'1100","ondisk_log_start":"29'1100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13997216,"num_objects":1709,"num_object_clones":0,"num_object_copies":6836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1709,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11099,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-536}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'11345","reported_seq":11476,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:05.141620+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:44:05.141620+0000","last_peered":"2024-09-05T16:44:05.141620+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:44:05.141620+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"29'1300","ondisk_log_start":"29'1300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14561008,"num_objects":1778,"num_object_clones":0,"num_object_copies":7112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1778,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11342,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-503}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'11023","reported_seq":11125,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:05.072444+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:44:05.072444+0000","last_peered":"2024-09-05T16:44:05.072444+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:44:05.072444+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"29'1000","ondisk_log_start":"29'1000","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14219856,"num_objects":1736,"num_object_clones":0,"num_object_copies":6944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1736,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11022,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-502}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'11151","reported_seq":15891,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:44:04.631808+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:44:04.631808+0000","last_peered":"2024-09-05T16:44:04.631808+0000","last_clean":"2024-09-05T16:44:04.631808+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:44:04.631808+0000","last_undegraded":"2024-09-05T16:44:04.631808+0000","last_fullsized":"2024-09-05T16:44:04.631808+0000","mapping_epoch":36,"log_start":"29'1100","ondisk_log_start":"29'1100","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13574144,"num_objects":1657,"num_object_clones":0,"num_object_copies":6628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1657,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2751,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'11374","reported_seq":11487,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:05.062676+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:44:05.062676+0000","last_peered":"2024-09-05T16:44:05.062676+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:44:05.062676+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"29'1300","ondisk_log_start":"29'1300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14349472,"num_objects":1752,"num_object_clones":0,"num_object_copies":7008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11372,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-582}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'11116","reported_seq":11233,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:04.171079+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:44:04.171079+0000","last_peered":"2024-09-05T16:44:04.171079+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:44:04.171079+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"29'1100","ondisk_log_start":"29'1100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14062752,"num_objects":1717,"num_object_clones":0,"num_object_copies":6868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1717,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11114,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-591}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'11100","reported_seq":11213,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:04.503528+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:44:04.503528+0000","last_peered":"2024-09-05T16:44:04.503528+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:44:04.503528+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"29'1000","ondisk_log_start":"29'1000","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13718688,"num_objects":1675,"num_object_clones":0,"num_object_copies":6700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1675,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11098,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-545}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":223976776,"num_objects":27346,"num_object_clones":0,"num_object_copies":109384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28902,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27346,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":152804,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":464,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":448184320,"data_stored":448184320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160913,"ondisk_log_size":160913,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084409,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":751220,"kb_used_data":110052,"kb_used_omap":4,"kb_used_meta":641147,"kb_avail":93620620,"statfs":{"total":96636764160,"available":95867514880,"internally_reserved":0,"allocated":112693248,"data_stored":112482667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":656535251},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,1,6,3,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986286,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":593372,"kb_used_data":110476,"kb_used_omap":4,"kb_used_meta":482875,"kb_avail":93778468,"statfs":{"total":96636764160,"available":96029151232,"internally_reserved":0,"allocated":113127424,"data_stored":112916843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":494464722},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,7,3,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673092,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":759004,"kb_used_data":110028,"kb_used_omap":1,"kb_used_meta":648958,"kb_avail":93612836,"statfs":{"total":96636764160,"available":95859544064,"internally_reserved":0,"allocated":112668672,"data_stored":112458091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":664533445},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,5,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182523,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":759768,"kb_used_data":110344,"kb_used_omap":4,"kb_used_meta":649403,"kb_avail":93612072,"statfs":{"total":96636764160,"available":95858761728,"internally_reserved":0,"allocated":112992256,"data_stored":112781675,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":664989389},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,3,4,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":112189440,"data_stored":112189440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":112066560,"data_stored":112066560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":112279552,"data_stored":112279552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":111648768,"data_stored":111648768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:44:09.797 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-05T16:44:09.798 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "39'11185", 'reported_seq': 11665, 'reported_epoch': 39, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:05.159173+0000', 'last_change': '2024-09-05T16:43:43.436935+0000', 'last_active': '2024-09-05T16:44:05.159173+0000', 'last_peered': '2024-09-05T16:44:05.159173+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.828289+0000', 'last_became_peered': '2024-09-05T16:34:56.828289+0000', 'last_unstale': '2024-09-05T16:44:05.159173+0000', 'last_undegraded': '2024-09-05T16:34:56.612160+0000', 'last_fullsized': '2024-09-05T16:34:56.557632+0000', 'mapping_epoch': 32, 'log_start': "29'1100", 'ondisk_log_start': "29'1100", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10085, 'log_dups_size': 0, 'ondisk_log_size': 10085, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14082048, 'num_objects': 1719, 'num_object_clones': 0, 'num_object_copies': 6876, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2051, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1719, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11185, 'num_write_kb': 51616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 211, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -332}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:44:09.798 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:44:09.798 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "39'11063", 'reported_seq': 11190, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:04.254965+0000', 'last_change': '2024-09-05T16:34:56.867900+0000', 'last_active': '2024-09-05T16:44:04.254965+0000', 'last_peered': '2024-09-05T16:44:04.254965+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:44:04.254965+0000', 'last_undegraded': '2024-09-05T16:34:56.590972+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 32, 'log_start': "29'1000", 'ondisk_log_start': "29'1000", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10063, 'log_dups_size': 0, 'ondisk_log_size': 10063, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13834856, 'num_objects': 1690, 'num_object_clones': 0, 'num_object_copies': 6760, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2233, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1690, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 11062, 'num_write_kb': 50999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -542}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:44:09.798 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:44:09.798 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "39'10893", 'reported_seq': 10995, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:05.161839+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:44:05.161839+0000', 'last_peered': '2024-09-05T16:44:05.161839+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:44:05.161839+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "29'800", 'ondisk_log_start': "29'800", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10093, 'log_dups_size': 0, 'ondisk_log_size': 10093, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13939872, 'num_objects': 1702, 'num_object_clones': 0, 'num_object_copies': 6808, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2147, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1702, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10891, 'num_write_kb': 50373, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -443}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:44:09.798 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:44:09.798 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "39'11117", 'reported_seq': 11236, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:05.190060+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:44:05.190060+0000', 'last_peered': '2024-09-05T16:44:05.190060+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:44:05.190060+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "29'1100", 'ondisk_log_start': "29'1100", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10017, 'log_dups_size': 0, 'ondisk_log_size': 10017, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13557760, 'num_objects': 1655, 'num_object_clones': 0, 'num_object_copies': 6620, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1655, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11117, 'num_write_kb': 51088, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -564}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:44:09.798 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:44:09.798 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "39'10888", 'reported_seq': 11000, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:04.399057+0000', 'last_change': '2024-09-05T16:34:56.903436+0000', 'last_active': '2024-09-05T16:44:04.399057+0000', 'last_peered': '2024-09-05T16:44:04.399057+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:44:04.399057+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "29'800", 'ondisk_log_start': "29'800", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10088, 'log_dups_size': 0, 'ondisk_log_size': 10088, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14529696, 'num_objects': 1774, 'num_object_clones': 0, 'num_object_copies': 7096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2133, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1774, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10886, 'num_write_kb': 50636, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -357}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:44:09.798 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-05T16:44:09.798 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "39'11059", 'reported_seq': 11152, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:04.349754+0000', 'last_change': '2024-09-05T16:34:56.883319+0000', 'last_active': '2024-09-05T16:44:04.349754+0000', 'last_peered': '2024-09-05T16:44:04.349754+0000', 'last_clean': '2024-09-05T16:33:53.505305+0000', 'last_became_active': '2024-09-05T16:34:56.873389+0000', 'last_became_peered': '2024-09-05T16:34:56.873389+0000', 'last_unstale': '2024-09-05T16:44:04.349754+0000', 'last_undegraded': '2024-09-05T16:34:56.602567+0000', 'last_fullsized': '2024-09-05T16:34:56.559958+0000', 'mapping_epoch': 32, 'log_start': "29'1000", 'ondisk_log_start': "29'1000", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10059, 'log_dups_size': 0, 'ondisk_log_size': 10059, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14087328, 'num_objects': 1720, 'num_object_clones': 0, 'num_object_copies': 6880, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2234, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1720, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11057, 'num_write_kb': 51108, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -512}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:44:09.799 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:44:09.799 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "39'11095", 'reported_seq': 11200, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:05.196542+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:44:05.196542+0000', 'last_peered': '2024-09-05T16:44:05.196542+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:44:05.196542+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "29'1000", 'ondisk_log_start': "29'1000", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10095, 'log_dups_size': 0, 'ondisk_log_size': 10095, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13748544, 'num_objects': 1679, 'num_object_clones': 0, 'num_object_copies': 6716, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1679, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11092, 'num_write_kb': 51081, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -565}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:44:09.799 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:44:09.799 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "39'11101", 'reported_seq': 11226, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:04.188482+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:44:04.188482+0000', 'last_peered': '2024-09-05T16:44:04.188482+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:44:04.188482+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "29'1100", 'ondisk_log_start': "29'1100", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10001, 'log_dups_size': 0, 'ondisk_log_size': 10001, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13997216, 'num_objects': 1709, 'num_object_clones': 0, 'num_object_copies': 6836, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1709, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11099, 'num_write_kb': 51230, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -536}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:44:09.799 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:44:09.799 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "39'11345", 'reported_seq': 11476, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:05.141620+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:44:05.141620+0000', 'last_peered': '2024-09-05T16:44:05.141620+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:44:05.141620+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "29'1300", 'ondisk_log_start': "29'1300", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10045, 'log_dups_size': 0, 'ondisk_log_size': 10045, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14561008, 'num_objects': 1778, 'num_object_clones': 0, 'num_object_copies': 7112, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2284, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1778, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11342, 'num_write_kb': 52476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -503}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:44:09.799 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:44:09.799 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "39'11023", 'reported_seq': 11125, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:05.072444+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:44:05.072444+0000', 'last_peered': '2024-09-05T16:44:05.072444+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:44:05.072444+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "29'1000", 'ondisk_log_start': "29'1000", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10023, 'log_dups_size': 0, 'ondisk_log_size': 10023, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14219856, 'num_objects': 1736, 'num_object_clones': 0, 'num_object_copies': 6944, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1736, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11022, 'num_write_kb': 51031, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -502}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:44:09.799 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:44:09.799 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "39'11374", 'reported_seq': 11487, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:05.062676+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:44:05.062676+0000', 'last_peered': '2024-09-05T16:44:05.062676+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:44:05.062676+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "29'1300", 'ondisk_log_start': "29'1300", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10074, 'log_dups_size': 0, 'ondisk_log_size': 10074, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14349472, 'num_objects': 1752, 'num_object_clones': 0, 'num_object_copies': 7008, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1752, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11372, 'num_write_kb': 52500, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -582}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:44:09.800 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:44:09.800 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "39'11116", 'reported_seq': 11233, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:04.171079+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:44:04.171079+0000', 'last_peered': '2024-09-05T16:44:04.171079+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:44:04.171079+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "29'1100", 'ondisk_log_start': "29'1100", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10016, 'log_dups_size': 0, 'ondisk_log_size': 10016, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14062752, 'num_objects': 1717, 'num_object_clones': 0, 'num_object_copies': 6868, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2310, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1717, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11114, 'num_write_kb': 51325, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -591}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:44:09.800 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:44:09.800 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "39'11100", 'reported_seq': 11213, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:04.503528+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:44:04.503528+0000', 'last_peered': '2024-09-05T16:44:04.503528+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:44:04.503528+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "29'1000", 'ondisk_log_start': "29'1000", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10100, 'log_dups_size': 0, 'ondisk_log_size': 10100, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13718688, 'num_objects': 1675, 'num_object_clones': 0, 'num_object_copies': 6700, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1675, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11098, 'num_write_kb': 51092, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -545}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:44:09.800 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:44:10.554 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:44:10.554 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 28902/109384 objects degraded (26.423%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":653,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":39,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":27346,"data_bytes":223976776,"bytes_used":2932084736,"bytes_avail":383614971904,"bytes_total":386547056640,"degraded_objects":28902,"degraded_total":109384,"degraded_ratio":0.26422511519052144,"recovering_objects_per_sec":6,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":82,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":168},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:44:10.586 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:44:10.586 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:44:11.187 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:44:11.188 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:44:11.221 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":357,"stamp":"2024-09-05T16:44:11.033444+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":215588168,"num_objects":26322,"num_object_clones":0,"num_object_copies":105288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26322,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":153828,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":509,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160837,"ondisk_log_size":160837,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2863860,"kb_used_data":424736,"kb_used_omap":14,"kb_used_meta":2439025,"kb_avail":374623500,"statfs":{"total":386547056640,"available":383614464000,"internally_reserved":0,"allocated":434929664,"data_stored":434087340,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2497562167},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,4,8,1,9,6,5,8,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25001984,"num_objects":-3052,"num_object_clones":0,"num_object_copies":-12208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-127,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3052,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3052,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004606"},"pg_stats":[{"pgid":"1.f","version":"39'11246","reported_seq":11816,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:10.341548+0000","last_change":"2024-09-05T16:43:43.436935+0000","last_active":"2024-09-05T16:44:10.341548+0000","last_peered":"2024-09-05T16:44:10.341548+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:44:10.341548+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"29'1200","ondisk_log_start":"29'1200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13582336,"num_objects":1658,"num_object_clones":0,"num_object_copies":6632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2006,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1658,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11246,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":256,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-348}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'10965","reported_seq":15621,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:44:09.744947+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:44:09.744947+0000","last_peered":"2024-09-05T16:44:09.744947+0000","last_clean":"2024-09-05T16:44:09.744947+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:44:09.744947+0000","last_undegraded":"2024-09-05T16:44:09.744947+0000","last_fullsized":"2024-09-05T16:44:09.744947+0000","mapping_epoch":38,"log_start":"29'900","ondisk_log_start":"29'900","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13000704,"num_objects":1587,"num_object_clones":0,"num_object_copies":6348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1587,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":278,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'11117","reported_seq":11242,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:09.414623+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:44:09.414623+0000","last_peered":"2024-09-05T16:44:09.414623+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:44:09.414623+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"29'1100","ondisk_log_start":"29'1100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13392488,"num_objects":1636,"num_object_clones":0,"num_object_copies":6544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1636,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11116,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-596}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'11269","reported_seq":16003,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:44:09.909794+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:44:09.909794+0000","last_peered":"2024-09-05T16:44:09.909794+0000","last_clean":"2024-09-05T16:44:09.909794+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:44:09.909794+0000","last_undegraded":"2024-09-05T16:44:09.909794+0000","last_fullsized":"2024-09-05T16:44:09.909794+0000","mapping_epoch":34,"log_start":"29'1200","ondisk_log_start":"29'1200","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13639680,"num_objects":1665,"num_object_clones":0,"num_object_copies":6660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5569,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'10947","reported_seq":11050,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:10.154101+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:44:10.154101+0000","last_peered":"2024-09-05T16:44:10.154101+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:44:10.154101+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"29'900","ondisk_log_start":"29'900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13497504,"num_objects":1648,"num_object_clones":0,"num_object_copies":6592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1648,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10945,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-497}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'11179","reported_seq":11298,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:09.985267+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:44:09.985267+0000","last_peered":"2024-09-05T16:44:09.985267+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:44:09.985267+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"29'1100","ondisk_log_start":"29'1100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13049856,"num_objects":1593,"num_object_clones":0,"num_object_copies":6372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11179,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-626}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'10947","reported_seq":11059,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:09.516187+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:44:09.516187+0000","last_peered":"2024-09-05T16:44:09.516187+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:44:09.516187+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"29'900","ondisk_log_start":"29'900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14046368,"num_objects":1715,"num_object_clones":0,"num_object_copies":6860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1715,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10945,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-416}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'11116","reported_seq":11209,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:09.441507+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:44:09.441507+0000","last_peered":"2024-09-05T16:44:09.441507+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:44:09.441507+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"29'1100","ondisk_log_start":"29'1100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13620384,"num_objects":1663,"num_object_clones":0,"num_object_copies":6652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11114,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-569}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"39'11168","reported_seq":11273,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:10.333759+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:44:10.333759+0000","last_peered":"2024-09-05T16:44:10.333759+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:44:10.333759+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"29'1100","ondisk_log_start":"29'1100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13150528,"num_objects":1606,"num_object_clones":0,"num_object_copies":6424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1606,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11165,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-638}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'11173","reported_seq":11299,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:09.252659+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:44:09.252659+0000","last_peered":"2024-09-05T16:44:09.252659+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:44:09.252659+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"29'1100","ondisk_log_start":"29'1100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13407392,"num_objects":1637,"num_object_clones":0,"num_object_copies":6548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1637,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11171,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-608}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'11417","reported_seq":11548,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:10.244348+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:44:10.244348+0000","last_peered":"2024-09-05T16:44:10.244348+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:44:10.244348+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"29'1400","ondisk_log_start":"29'1400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13971184,"num_objects":1706,"num_object_clones":0,"num_object_copies":6824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1706,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11414,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-575}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'11084","reported_seq":11187,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:10.340046+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:44:10.340046+0000","last_peered":"2024-09-05T16:44:10.340046+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:44:10.340046+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"29'1000","ondisk_log_start":"29'1000","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13720144,"num_objects":1675,"num_object_clones":0,"num_object_copies":6700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1675,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11083,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-563}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'11203","reported_seq":15943,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:44:09.820218+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:44:09.820218+0000","last_peered":"2024-09-05T16:44:09.820218+0000","last_clean":"2024-09-05T16:44:09.820218+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:44:09.820218+0000","last_undegraded":"2024-09-05T16:44:09.820218+0000","last_fullsized":"2024-09-05T16:44:09.820218+0000","mapping_epoch":36,"log_start":"29'1200","ondisk_log_start":"29'1200","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13148160,"num_objects":1605,"num_object_clones":0,"num_object_copies":6420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1605,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2803,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'11444","reported_seq":11557,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:10.089902+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:44:10.089902+0000","last_peered":"2024-09-05T16:44:10.089902+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:44:10.089902+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"29'1400","ondisk_log_start":"29'1400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13776032,"num_objects":1682,"num_object_clones":0,"num_object_copies":6728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1682,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11442,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-652}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'11185","reported_seq":11302,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:09.483167+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:44:09.483167+0000","last_peered":"2024-09-05T16:44:09.483167+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:44:09.483167+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"29'1100","ondisk_log_start":"29'1100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13497504,"num_objects":1648,"num_object_clones":0,"num_object_copies":6592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1648,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11183,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-660}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'11177","reported_seq":11289,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:09.680158+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:44:09.680158+0000","last_peered":"2024-09-05T16:44:09.680158+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:44:09.680158+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"29'1100","ondisk_log_start":"29'1100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13087904,"num_objects":1598,"num_object_clones":0,"num_object_copies":6392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1598,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11175,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-622}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":215588168,"num_objects":26322,"num_object_clones":0,"num_object_copies":105288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26322,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":153828,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":509,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":431472640,"data_stored":431472640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160837,"ondisk_log_size":160837,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084410,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":751348,"kb_used_data":106012,"kb_used_omap":4,"kb_used_meta":645307,"kb_avail":93620492,"statfs":{"total":96636764160,"available":95867383808,"internally_reserved":0,"allocated":108556288,"data_stored":108345707,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":660795091},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,1,2,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986287,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":593476,"kb_used_data":106420,"kb_used_omap":4,"kb_used_meta":487035,"kb_avail":93778364,"statfs":{"total":96636764160,"available":96029044736,"internally_reserved":0,"allocated":108974080,"data_stored":108763499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":498724562},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,0,1,1,5,3,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673093,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":759132,"kb_used_data":105992,"kb_used_omap":1,"kb_used_meta":653118,"kb_avail":93612708,"statfs":{"total":96636764160,"available":95859412992,"internally_reserved":0,"allocated":108535808,"data_stored":108325227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":668793285},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,5],"upper_bound":32},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182524,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":759904,"kb_used_data":106312,"kb_used_omap":4,"kb_used_meta":653563,"kb_avail":93611936,"statfs":{"total":96636764160,"available":95858622464,"internally_reserved":0,"allocated":108863488,"data_stored":108652907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":669249229},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,0,0,0,0,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":107958272,"data_stored":107958272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":107884544,"data_stored":107884544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":108126208,"data_stored":108126208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":107503616,"data_stored":107503616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:44:14.223 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:44:14.824 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:44:14.825 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:44:14.858 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":358,"stamp":"2024-09-05T16:44:13.034331+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":215588168,"num_objects":26322,"num_object_clones":0,"num_object_copies":105288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26322,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":153828,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":509,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160837,"ondisk_log_size":160837,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2863860,"kb_used_data":424736,"kb_used_omap":14,"kb_used_meta":2439025,"kb_avail":374623500,"statfs":{"total":386547056640,"available":383614464000,"internally_reserved":0,"allocated":434929664,"data_stored":434087340,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2497562167},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,4,8,1,9,6,5,8,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16687104,"num_objects":-2037,"num_object_clones":0,"num_object_copies":-8148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-83,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2037,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004680"},"pg_stats":[{"pgid":"1.f","version":"39'11246","reported_seq":11816,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:10.341548+0000","last_change":"2024-09-05T16:43:43.436935+0000","last_active":"2024-09-05T16:44:10.341548+0000","last_peered":"2024-09-05T16:44:10.341548+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:44:10.341548+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"29'1200","ondisk_log_start":"29'1200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13582336,"num_objects":1658,"num_object_clones":0,"num_object_copies":6632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2006,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1658,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11246,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":256,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-348}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'10965","reported_seq":15621,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:44:09.744947+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:44:09.744947+0000","last_peered":"2024-09-05T16:44:09.744947+0000","last_clean":"2024-09-05T16:44:09.744947+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:44:09.744947+0000","last_undegraded":"2024-09-05T16:44:09.744947+0000","last_fullsized":"2024-09-05T16:44:09.744947+0000","mapping_epoch":38,"log_start":"29'900","ondisk_log_start":"29'900","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13000704,"num_objects":1587,"num_object_clones":0,"num_object_copies":6348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1587,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":278,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'11117","reported_seq":11242,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:09.414623+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:44:09.414623+0000","last_peered":"2024-09-05T16:44:09.414623+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:44:09.414623+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"29'1100","ondisk_log_start":"29'1100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13392488,"num_objects":1636,"num_object_clones":0,"num_object_copies":6544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1636,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11116,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-596}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'11269","reported_seq":16003,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:44:09.909794+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:44:09.909794+0000","last_peered":"2024-09-05T16:44:09.909794+0000","last_clean":"2024-09-05T16:44:09.909794+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:44:09.909794+0000","last_undegraded":"2024-09-05T16:44:09.909794+0000","last_fullsized":"2024-09-05T16:44:09.909794+0000","mapping_epoch":34,"log_start":"29'1200","ondisk_log_start":"29'1200","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13639680,"num_objects":1665,"num_object_clones":0,"num_object_copies":6660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5569,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'10947","reported_seq":11050,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:10.154101+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:44:10.154101+0000","last_peered":"2024-09-05T16:44:10.154101+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:44:10.154101+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"29'900","ondisk_log_start":"29'900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13497504,"num_objects":1648,"num_object_clones":0,"num_object_copies":6592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1648,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10945,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-497}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'11179","reported_seq":11298,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:09.985267+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:44:09.985267+0000","last_peered":"2024-09-05T16:44:09.985267+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:44:09.985267+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"29'1100","ondisk_log_start":"29'1100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13049856,"num_objects":1593,"num_object_clones":0,"num_object_copies":6372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11179,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-626}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'10947","reported_seq":11059,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:09.516187+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:44:09.516187+0000","last_peered":"2024-09-05T16:44:09.516187+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:44:09.516187+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"29'900","ondisk_log_start":"29'900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14046368,"num_objects":1715,"num_object_clones":0,"num_object_copies":6860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1715,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10945,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-416}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'11116","reported_seq":11209,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:09.441507+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:44:09.441507+0000","last_peered":"2024-09-05T16:44:09.441507+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:44:09.441507+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"29'1100","ondisk_log_start":"29'1100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13620384,"num_objects":1663,"num_object_clones":0,"num_object_copies":6652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11114,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-569}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"39'11168","reported_seq":11273,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:10.333759+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:44:10.333759+0000","last_peered":"2024-09-05T16:44:10.333759+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:44:10.333759+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"29'1100","ondisk_log_start":"29'1100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13150528,"num_objects":1606,"num_object_clones":0,"num_object_copies":6424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1606,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11165,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-638}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'11173","reported_seq":11299,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:09.252659+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:44:09.252659+0000","last_peered":"2024-09-05T16:44:09.252659+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:44:09.252659+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"29'1100","ondisk_log_start":"29'1100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13407392,"num_objects":1637,"num_object_clones":0,"num_object_copies":6548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1637,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11171,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-608}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'11417","reported_seq":11548,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:10.244348+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:44:10.244348+0000","last_peered":"2024-09-05T16:44:10.244348+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:44:10.244348+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"29'1400","ondisk_log_start":"29'1400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13971184,"num_objects":1706,"num_object_clones":0,"num_object_copies":6824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1706,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11414,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-575}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'11084","reported_seq":11187,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:10.340046+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:44:10.340046+0000","last_peered":"2024-09-05T16:44:10.340046+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:44:10.340046+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"29'1000","ondisk_log_start":"29'1000","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13720144,"num_objects":1675,"num_object_clones":0,"num_object_copies":6700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1675,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11083,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-563}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'11203","reported_seq":15943,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:44:09.820218+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:44:09.820218+0000","last_peered":"2024-09-05T16:44:09.820218+0000","last_clean":"2024-09-05T16:44:09.820218+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:44:09.820218+0000","last_undegraded":"2024-09-05T16:44:09.820218+0000","last_fullsized":"2024-09-05T16:44:09.820218+0000","mapping_epoch":36,"log_start":"29'1200","ondisk_log_start":"29'1200","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13148160,"num_objects":1605,"num_object_clones":0,"num_object_copies":6420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1605,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2803,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'11444","reported_seq":11557,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:10.089902+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:44:10.089902+0000","last_peered":"2024-09-05T16:44:10.089902+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:44:10.089902+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"29'1400","ondisk_log_start":"29'1400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13776032,"num_objects":1682,"num_object_clones":0,"num_object_copies":6728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1682,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11442,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-652}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'11185","reported_seq":11302,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:09.483167+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:44:09.483167+0000","last_peered":"2024-09-05T16:44:09.483167+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:44:09.483167+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"29'1100","ondisk_log_start":"29'1100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13497504,"num_objects":1648,"num_object_clones":0,"num_object_copies":6592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1648,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11183,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-660}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'11177","reported_seq":11289,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:09.680158+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:44:09.680158+0000","last_peered":"2024-09-05T16:44:09.680158+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:44:09.680158+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"29'1100","ondisk_log_start":"29'1100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13087904,"num_objects":1598,"num_object_clones":0,"num_object_copies":6392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1598,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11175,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-622}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":215588168,"num_objects":26322,"num_object_clones":0,"num_object_copies":105288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26322,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":153828,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":509,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":431472640,"data_stored":431472640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160837,"ondisk_log_size":160837,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084410,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":751348,"kb_used_data":106012,"kb_used_omap":4,"kb_used_meta":645307,"kb_avail":93620492,"statfs":{"total":96636764160,"available":95867383808,"internally_reserved":0,"allocated":108556288,"data_stored":108345707,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":660795091},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,1,2,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986287,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":593476,"kb_used_data":106420,"kb_used_omap":4,"kb_used_meta":487035,"kb_avail":93778364,"statfs":{"total":96636764160,"available":96029044736,"internally_reserved":0,"allocated":108974080,"data_stored":108763499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":498724562},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,0,1,1,5,3,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673093,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":759132,"kb_used_data":105992,"kb_used_omap":1,"kb_used_meta":653118,"kb_avail":93612708,"statfs":{"total":96636764160,"available":95859412992,"internally_reserved":0,"allocated":108535808,"data_stored":108325227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":668793285},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,5],"upper_bound":32},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182524,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":759904,"kb_used_data":106312,"kb_used_omap":4,"kb_used_meta":653563,"kb_avail":93611936,"statfs":{"total":96636764160,"available":95858622464,"internally_reserved":0,"allocated":108863488,"data_stored":108652907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":669249229},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,0,0,0,0,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":107958272,"data_stored":107958272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":107884544,"data_stored":107884544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":108126208,"data_stored":108126208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":107503616,"data_stored":107503616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:44:14.859 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:44:15.471 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:44:15.471 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:44:15.520 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":359,"stamp":"2024-09-05T16:44:15.035138+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":212753736,"num_objects":25976,"num_object_clones":0,"num_object_copies":103904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25976,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":154174,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":509,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160783,"ondisk_log_size":160783,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2847508,"kb_used_data":416660,"kb_used_omap":14,"kb_used_meta":2430705,"kb_avail":374639852,"statfs":{"total":386547056640,"available":383631208448,"internally_reserved":0,"allocated":426659840,"data_stored":425817516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2489042487},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,4,4,9,5,8,6,7,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19521536,"num_objects":-2383,"num_object_clones":0,"num_object_copies":-9532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-83,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2383,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2383,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004576"},"pg_stats":[{"pgid":"1.f","version":"39'11246","reported_seq":11816,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:10.341548+0000","last_change":"2024-09-05T16:43:43.436935+0000","last_active":"2024-09-05T16:44:10.341548+0000","last_peered":"2024-09-05T16:44:10.341548+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:44:10.341548+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"29'1200","ondisk_log_start":"29'1200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13582336,"num_objects":1658,"num_object_clones":0,"num_object_copies":6632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2006,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1658,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11246,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":256,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-348}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'10965","reported_seq":15621,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:44:09.744947+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:44:09.744947+0000","last_peered":"2024-09-05T16:44:09.744947+0000","last_clean":"2024-09-05T16:44:09.744947+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:44:09.744947+0000","last_undegraded":"2024-09-05T16:44:09.744947+0000","last_fullsized":"2024-09-05T16:44:09.744947+0000","mapping_epoch":38,"log_start":"29'900","ondisk_log_start":"29'900","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13000704,"num_objects":1587,"num_object_clones":0,"num_object_copies":6348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1587,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":278,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'11174","reported_seq":11300,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:14.442019+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:44:14.442019+0000","last_peered":"2024-09-05T16:44:14.442019+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:44:14.442019+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"29'1100","ondisk_log_start":"29'1100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12925544,"num_objects":1579,"num_object_clones":0,"num_object_copies":6316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1579,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11173,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-653}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'11269","reported_seq":16003,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:44:09.909794+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:44:09.909794+0000","last_peered":"2024-09-05T16:44:09.909794+0000","last_clean":"2024-09-05T16:44:09.909794+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:44:09.909794+0000","last_undegraded":"2024-09-05T16:44:09.909794+0000","last_fullsized":"2024-09-05T16:44:09.909794+0000","mapping_epoch":34,"log_start":"29'1200","ondisk_log_start":"29'1200","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13639680,"num_objects":1665,"num_object_clones":0,"num_object_copies":6660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5569,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'10947","reported_seq":11050,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:10.154101+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:44:10.154101+0000","last_peered":"2024-09-05T16:44:10.154101+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:44:10.154101+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"29'900","ondisk_log_start":"29'900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13497504,"num_objects":1648,"num_object_clones":0,"num_object_copies":6592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1648,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10945,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-497}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'11179","reported_seq":11298,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:09.985267+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:44:09.985267+0000","last_peered":"2024-09-05T16:44:09.985267+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:44:09.985267+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"29'1100","ondisk_log_start":"29'1100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13049856,"num_objects":1593,"num_object_clones":0,"num_object_copies":6372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11179,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-626}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'11012","reported_seq":11124,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:14.682094+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:44:14.682094+0000","last_peered":"2024-09-05T16:44:14.682094+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:44:14.682094+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"29'1000","ondisk_log_start":"29'1000","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13513888,"num_objects":1650,"num_object_clones":0,"num_object_copies":6600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1650,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11010,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-481}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'11180","reported_seq":11273,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:14.726174+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:44:14.726174+0000","last_peered":"2024-09-05T16:44:14.726174+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:44:14.726174+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"29'1100","ondisk_log_start":"29'1100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13096096,"num_objects":1599,"num_object_clones":0,"num_object_copies":6396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11178,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-633}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"39'11168","reported_seq":11273,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:10.333759+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:44:10.333759+0000","last_peered":"2024-09-05T16:44:10.333759+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:44:10.333759+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"29'1100","ondisk_log_start":"29'1100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13150528,"num_objects":1606,"num_object_clones":0,"num_object_copies":6424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1606,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11165,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-638}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'11223","reported_seq":11348,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:14.458009+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:44:14.458009+0000","last_peered":"2024-09-05T16:44:14.458009+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:44:14.458009+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"29'1200","ondisk_log_start":"29'1200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12997792,"num_objects":1587,"num_object_clones":0,"num_object_copies":6348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1587,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11221,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-658}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'11417","reported_seq":11548,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:10.244348+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:44:10.244348+0000","last_peered":"2024-09-05T16:44:10.244348+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:44:10.244348+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"29'1400","ondisk_log_start":"29'1400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13971184,"num_objects":1706,"num_object_clones":0,"num_object_copies":6824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1706,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11414,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-575}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'11084","reported_seq":11187,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:10.340046+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:44:10.340046+0000","last_peered":"2024-09-05T16:44:10.340046+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:44:10.340046+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"29'1000","ondisk_log_start":"29'1000","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13720144,"num_objects":1675,"num_object_clones":0,"num_object_copies":6700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1675,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11083,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-563}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'11203","reported_seq":15943,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:44:09.820218+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:44:09.820218+0000","last_peered":"2024-09-05T16:44:09.820218+0000","last_clean":"2024-09-05T16:44:09.820218+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:44:09.820218+0000","last_undegraded":"2024-09-05T16:44:09.820218+0000","last_fullsized":"2024-09-05T16:44:09.820218+0000","mapping_epoch":36,"log_start":"29'1200","ondisk_log_start":"29'1200","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13148160,"num_objects":1605,"num_object_clones":0,"num_object_copies":6420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1605,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2803,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'11444","reported_seq":11557,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:10.089902+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:44:10.089902+0000","last_peered":"2024-09-05T16:44:10.089902+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:44:10.089902+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"29'1400","ondisk_log_start":"29'1400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13776032,"num_objects":1682,"num_object_clones":0,"num_object_copies":6728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1682,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11442,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-652}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'11237","reported_seq":11354,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:14.296286+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:44:14.296286+0000","last_peered":"2024-09-05T16:44:14.296286+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:44:14.296286+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"29'1200","ondisk_log_start":"29'1200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13071520,"num_objects":1596,"num_object_clones":0,"num_object_copies":6384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1596,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11235,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-712}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'11235","reported_seq":11348,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:14.727285+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:44:14.727285+0000","last_peered":"2024-09-05T16:44:14.727285+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:44:14.727285+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"29'1200","ondisk_log_start":"29'1200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12612768,"num_objects":1540,"num_object_clones":0,"num_object_copies":6160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1540,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11233,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-680}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":212753736,"num_objects":25976,"num_object_clones":0,"num_object_copies":103904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25976,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":154174,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":509,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":423170048,"data_stored":423170048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160783,"ondisk_log_size":160783,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084410,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":751348,"kb_used_data":106012,"kb_used_omap":4,"kb_used_meta":645307,"kb_avail":93620492,"statfs":{"total":96636764160,"available":95867383808,"internally_reserved":0,"allocated":108556288,"data_stored":108345707,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":660795091},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,1,2,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986287,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":593476,"kb_used_data":106420,"kb_used_omap":4,"kb_used_meta":487035,"kb_avail":93778364,"statfs":{"total":96636764160,"available":96029044736,"internally_reserved":0,"allocated":108974080,"data_stored":108763499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":498724562},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,0,1,1,5,3,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673094,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":742632,"kb_used_data":101948,"kb_used_omap":1,"kb_used_meta":640638,"kb_avail":93629208,"statfs":{"total":96636764160,"available":95876308992,"internally_reserved":0,"allocated":104394752,"data_stored":104184171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":656013765},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,3,0,1,0,2,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182525,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":760052,"kb_used_data":102280,"kb_used_omap":4,"kb_used_meta":657723,"kb_avail":93611788,"statfs":{"total":96636764160,"available":95858470912,"internally_reserved":0,"allocated":104734720,"data_stored":104524139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":673509069},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,5,1,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":103817216,"data_stored":103817216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":107884544,"data_stored":107884544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":103964672,"data_stored":103964672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":107503616,"data_stored":107503616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:44:15.521 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-05T16:44:15.521 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "39'11246", 'reported_seq': 11816, 'reported_epoch': 39, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:10.341548+0000', 'last_change': '2024-09-05T16:43:43.436935+0000', 'last_active': '2024-09-05T16:44:10.341548+0000', 'last_peered': '2024-09-05T16:44:10.341548+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.828289+0000', 'last_became_peered': '2024-09-05T16:34:56.828289+0000', 'last_unstale': '2024-09-05T16:44:10.341548+0000', 'last_undegraded': '2024-09-05T16:34:56.612160+0000', 'last_fullsized': '2024-09-05T16:34:56.557632+0000', 'mapping_epoch': 32, 'log_start': "29'1200", 'ondisk_log_start': "29'1200", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10046, 'log_dups_size': 0, 'ondisk_log_size': 10046, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13582336, 'num_objects': 1658, 'num_object_clones': 0, 'num_object_copies': 6632, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2006, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1658, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11246, 'num_write_kb': 51616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 256, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -348}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:44:15.521 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:44:15.521 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "39'11174", 'reported_seq': 11300, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:14.442019+0000', 'last_change': '2024-09-05T16:34:56.867900+0000', 'last_active': '2024-09-05T16:44:14.442019+0000', 'last_peered': '2024-09-05T16:44:14.442019+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:44:14.442019+0000', 'last_undegraded': '2024-09-05T16:34:56.590972+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 32, 'log_start': "29'1100", 'ondisk_log_start': "29'1100", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10074, 'log_dups_size': 0, 'ondisk_log_size': 10074, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12925544, 'num_objects': 1579, 'num_object_clones': 0, 'num_object_copies': 6316, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2233, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1579, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 11173, 'num_write_kb': 50999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -653}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:44:15.521 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:44:15.521 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "39'10947", 'reported_seq': 11050, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:10.154101+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:44:10.154101+0000', 'last_peered': '2024-09-05T16:44:10.154101+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:44:10.154101+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "29'900", 'ondisk_log_start': "29'900", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10047, 'log_dups_size': 0, 'ondisk_log_size': 10047, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13497504, 'num_objects': 1648, 'num_object_clones': 0, 'num_object_copies': 6592, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2147, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1648, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10945, 'num_write_kb': 50373, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -497}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:44:15.521 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:44:15.521 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "39'11179", 'reported_seq': 11298, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:09.985267+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:44:09.985267+0000', 'last_peered': '2024-09-05T16:44:09.985267+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:44:09.985267+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "29'1100", 'ondisk_log_start': "29'1100", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10079, 'log_dups_size': 0, 'ondisk_log_size': 10079, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13049856, 'num_objects': 1593, 'num_object_clones': 0, 'num_object_copies': 6372, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1593, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11179, 'num_write_kb': 51088, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -626}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:44:15.521 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:44:15.522 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "39'11012", 'reported_seq': 11124, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:14.682094+0000', 'last_change': '2024-09-05T16:34:56.903436+0000', 'last_active': '2024-09-05T16:44:14.682094+0000', 'last_peered': '2024-09-05T16:44:14.682094+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:44:14.682094+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "29'1000", 'ondisk_log_start': "29'1000", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10012, 'log_dups_size': 0, 'ondisk_log_size': 10012, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13513888, 'num_objects': 1650, 'num_object_clones': 0, 'num_object_copies': 6600, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2133, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1650, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11010, 'num_write_kb': 50636, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -481}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:44:15.522 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-05T16:44:15.522 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "39'11180", 'reported_seq': 11273, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:14.726174+0000', 'last_change': '2024-09-05T16:34:56.883319+0000', 'last_active': '2024-09-05T16:44:14.726174+0000', 'last_peered': '2024-09-05T16:44:14.726174+0000', 'last_clean': '2024-09-05T16:33:53.505305+0000', 'last_became_active': '2024-09-05T16:34:56.873389+0000', 'last_became_peered': '2024-09-05T16:34:56.873389+0000', 'last_unstale': '2024-09-05T16:44:14.726174+0000', 'last_undegraded': '2024-09-05T16:34:56.602567+0000', 'last_fullsized': '2024-09-05T16:34:56.559958+0000', 'mapping_epoch': 32, 'log_start': "29'1100", 'ondisk_log_start': "29'1100", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13096096, 'num_objects': 1599, 'num_object_clones': 0, 'num_object_copies': 6396, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2234, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1599, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11178, 'num_write_kb': 51108, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -633}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:44:15.522 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:44:15.522 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "39'11168", 'reported_seq': 11273, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:10.333759+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:44:10.333759+0000', 'last_peered': '2024-09-05T16:44:10.333759+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:44:10.333759+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "29'1100", 'ondisk_log_start': "29'1100", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10068, 'log_dups_size': 0, 'ondisk_log_size': 10068, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13150528, 'num_objects': 1606, 'num_object_clones': 0, 'num_object_copies': 6424, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1606, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11165, 'num_write_kb': 51081, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -638}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:44:15.522 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:44:15.522 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "39'11223", 'reported_seq': 11348, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:14.458009+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:44:14.458009+0000', 'last_peered': '2024-09-05T16:44:14.458009+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:44:14.458009+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "29'1200", 'ondisk_log_start': "29'1200", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10023, 'log_dups_size': 0, 'ondisk_log_size': 10023, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12997792, 'num_objects': 1587, 'num_object_clones': 0, 'num_object_copies': 6348, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1587, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11221, 'num_write_kb': 51230, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -658}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:44:15.522 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:44:15.522 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "39'11417", 'reported_seq': 11548, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:10.244348+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:44:10.244348+0000', 'last_peered': '2024-09-05T16:44:10.244348+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:44:10.244348+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "29'1400", 'ondisk_log_start': "29'1400", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10017, 'log_dups_size': 0, 'ondisk_log_size': 10017, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13971184, 'num_objects': 1706, 'num_object_clones': 0, 'num_object_copies': 6824, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2284, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1706, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11414, 'num_write_kb': 52476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -575}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:44:15.522 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:44:15.522 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "39'11084", 'reported_seq': 11187, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:10.340046+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:44:10.340046+0000', 'last_peered': '2024-09-05T16:44:10.340046+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:44:10.340046+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "29'1000", 'ondisk_log_start': "29'1000", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10084, 'log_dups_size': 0, 'ondisk_log_size': 10084, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13720144, 'num_objects': 1675, 'num_object_clones': 0, 'num_object_copies': 6700, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1675, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11083, 'num_write_kb': 51031, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -563}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:44:15.523 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:44:15.523 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "39'11444", 'reported_seq': 11557, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:10.089902+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:44:10.089902+0000', 'last_peered': '2024-09-05T16:44:10.089902+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:44:10.089902+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "29'1400", 'ondisk_log_start': "29'1400", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10044, 'log_dups_size': 0, 'ondisk_log_size': 10044, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13776032, 'num_objects': 1682, 'num_object_clones': 0, 'num_object_copies': 6728, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1682, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11442, 'num_write_kb': 52500, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -652}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:44:15.523 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:44:15.523 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "39'11237", 'reported_seq': 11354, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:14.296286+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:44:14.296286+0000', 'last_peered': '2024-09-05T16:44:14.296286+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:44:14.296286+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "29'1200", 'ondisk_log_start': "29'1200", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10037, 'log_dups_size': 0, 'ondisk_log_size': 10037, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13071520, 'num_objects': 1596, 'num_object_clones': 0, 'num_object_copies': 6384, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2310, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1596, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11235, 'num_write_kb': 51325, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -712}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:44:15.523 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:44:15.523 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "39'11235", 'reported_seq': 11348, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:14.727285+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:44:14.727285+0000', 'last_peered': '2024-09-05T16:44:14.727285+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:44:14.727285+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "29'1200", 'ondisk_log_start': "29'1200", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10035, 'log_dups_size': 0, 'ondisk_log_size': 10035, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12612768, 'num_objects': 1540, 'num_object_clones': 0, 'num_object_copies': 6160, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1540, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11233, 'num_write_kb': 51092, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -680}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:44:15.523 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:44:16.277 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:44:16.277 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 28857/103904 objects degraded (27.773%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":659,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":39,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":25976,"data_bytes":212753736,"bytes_used":2915848192,"bytes_avail":383631208448,"bytes_total":386547056640,"degraded_objects":28857,"degraded_total":103904,"degraded_ratio":0.27772751770865411,"recovering_objects_per_sec":6,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":198},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:44:16.302 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:44:16.303 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:44:16.869 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:44:16.870 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:44:16.903 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":359,"stamp":"2024-09-05T16:44:15.035138+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":212753736,"num_objects":25976,"num_object_clones":0,"num_object_copies":103904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25976,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":154174,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":509,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160783,"ondisk_log_size":160783,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2847508,"kb_used_data":416660,"kb_used_omap":14,"kb_used_meta":2430705,"kb_avail":374639852,"statfs":{"total":386547056640,"available":383631208448,"internally_reserved":0,"allocated":426659840,"data_stored":425817516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2489042487},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,4,4,9,5,8,6,7,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19521536,"num_objects":-2383,"num_object_clones":0,"num_object_copies":-9532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-83,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2383,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2383,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004576"},"pg_stats":[{"pgid":"1.f","version":"39'11246","reported_seq":11816,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:10.341548+0000","last_change":"2024-09-05T16:43:43.436935+0000","last_active":"2024-09-05T16:44:10.341548+0000","last_peered":"2024-09-05T16:44:10.341548+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:44:10.341548+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"29'1200","ondisk_log_start":"29'1200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13582336,"num_objects":1658,"num_object_clones":0,"num_object_copies":6632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2006,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1658,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11246,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":256,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-348}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'10965","reported_seq":15621,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:44:09.744947+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:44:09.744947+0000","last_peered":"2024-09-05T16:44:09.744947+0000","last_clean":"2024-09-05T16:44:09.744947+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:44:09.744947+0000","last_undegraded":"2024-09-05T16:44:09.744947+0000","last_fullsized":"2024-09-05T16:44:09.744947+0000","mapping_epoch":38,"log_start":"29'900","ondisk_log_start":"29'900","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13000704,"num_objects":1587,"num_object_clones":0,"num_object_copies":6348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1587,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":278,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'11174","reported_seq":11300,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:14.442019+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:44:14.442019+0000","last_peered":"2024-09-05T16:44:14.442019+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:44:14.442019+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"29'1100","ondisk_log_start":"29'1100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12925544,"num_objects":1579,"num_object_clones":0,"num_object_copies":6316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1579,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11173,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-653}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'11269","reported_seq":16003,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:44:09.909794+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:44:09.909794+0000","last_peered":"2024-09-05T16:44:09.909794+0000","last_clean":"2024-09-05T16:44:09.909794+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:44:09.909794+0000","last_undegraded":"2024-09-05T16:44:09.909794+0000","last_fullsized":"2024-09-05T16:44:09.909794+0000","mapping_epoch":34,"log_start":"29'1200","ondisk_log_start":"29'1200","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13639680,"num_objects":1665,"num_object_clones":0,"num_object_copies":6660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5569,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'10947","reported_seq":11050,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:10.154101+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:44:10.154101+0000","last_peered":"2024-09-05T16:44:10.154101+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:44:10.154101+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"29'900","ondisk_log_start":"29'900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13497504,"num_objects":1648,"num_object_clones":0,"num_object_copies":6592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1648,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10945,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-497}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'11179","reported_seq":11298,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:09.985267+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:44:09.985267+0000","last_peered":"2024-09-05T16:44:09.985267+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:44:09.985267+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"29'1100","ondisk_log_start":"29'1100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13049856,"num_objects":1593,"num_object_clones":0,"num_object_copies":6372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11179,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-626}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'11012","reported_seq":11124,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:14.682094+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:44:14.682094+0000","last_peered":"2024-09-05T16:44:14.682094+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:44:14.682094+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"29'1000","ondisk_log_start":"29'1000","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13513888,"num_objects":1650,"num_object_clones":0,"num_object_copies":6600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1650,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11010,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-481}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'11180","reported_seq":11273,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:14.726174+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:44:14.726174+0000","last_peered":"2024-09-05T16:44:14.726174+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:44:14.726174+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"29'1100","ondisk_log_start":"29'1100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13096096,"num_objects":1599,"num_object_clones":0,"num_object_copies":6396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11178,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-633}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"39'11168","reported_seq":11273,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:10.333759+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:44:10.333759+0000","last_peered":"2024-09-05T16:44:10.333759+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:44:10.333759+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"29'1100","ondisk_log_start":"29'1100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13150528,"num_objects":1606,"num_object_clones":0,"num_object_copies":6424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1606,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11165,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-638}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'11223","reported_seq":11348,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:14.458009+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:44:14.458009+0000","last_peered":"2024-09-05T16:44:14.458009+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:44:14.458009+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"29'1200","ondisk_log_start":"29'1200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12997792,"num_objects":1587,"num_object_clones":0,"num_object_copies":6348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1587,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11221,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-658}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'11417","reported_seq":11548,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:10.244348+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:44:10.244348+0000","last_peered":"2024-09-05T16:44:10.244348+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:44:10.244348+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"29'1400","ondisk_log_start":"29'1400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13971184,"num_objects":1706,"num_object_clones":0,"num_object_copies":6824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1706,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11414,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-575}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'11084","reported_seq":11187,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:10.340046+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:44:10.340046+0000","last_peered":"2024-09-05T16:44:10.340046+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:44:10.340046+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"29'1000","ondisk_log_start":"29'1000","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13720144,"num_objects":1675,"num_object_clones":0,"num_object_copies":6700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1675,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11083,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-563}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'11203","reported_seq":15943,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:44:09.820218+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:44:09.820218+0000","last_peered":"2024-09-05T16:44:09.820218+0000","last_clean":"2024-09-05T16:44:09.820218+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:44:09.820218+0000","last_undegraded":"2024-09-05T16:44:09.820218+0000","last_fullsized":"2024-09-05T16:44:09.820218+0000","mapping_epoch":36,"log_start":"29'1200","ondisk_log_start":"29'1200","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13148160,"num_objects":1605,"num_object_clones":0,"num_object_copies":6420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1605,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2803,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'11444","reported_seq":11557,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:10.089902+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:44:10.089902+0000","last_peered":"2024-09-05T16:44:10.089902+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:44:10.089902+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"29'1400","ondisk_log_start":"29'1400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13776032,"num_objects":1682,"num_object_clones":0,"num_object_copies":6728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1682,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11442,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-652}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'11237","reported_seq":11354,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:14.296286+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:44:14.296286+0000","last_peered":"2024-09-05T16:44:14.296286+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:44:14.296286+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"29'1200","ondisk_log_start":"29'1200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13071520,"num_objects":1596,"num_object_clones":0,"num_object_copies":6384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1596,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11235,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-712}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'11235","reported_seq":11348,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:14.727285+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:44:14.727285+0000","last_peered":"2024-09-05T16:44:14.727285+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:44:14.727285+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"29'1200","ondisk_log_start":"29'1200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12612768,"num_objects":1540,"num_object_clones":0,"num_object_copies":6160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1540,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11233,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-680}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":212753736,"num_objects":25976,"num_object_clones":0,"num_object_copies":103904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25976,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":154174,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":509,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":423170048,"data_stored":423170048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160783,"ondisk_log_size":160783,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084410,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":751348,"kb_used_data":106012,"kb_used_omap":4,"kb_used_meta":645307,"kb_avail":93620492,"statfs":{"total":96636764160,"available":95867383808,"internally_reserved":0,"allocated":108556288,"data_stored":108345707,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":660795091},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,1,2,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986287,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":593476,"kb_used_data":106420,"kb_used_omap":4,"kb_used_meta":487035,"kb_avail":93778364,"statfs":{"total":96636764160,"available":96029044736,"internally_reserved":0,"allocated":108974080,"data_stored":108763499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":498724562},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,0,1,1,5,3,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673094,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":742632,"kb_used_data":101948,"kb_used_omap":1,"kb_used_meta":640638,"kb_avail":93629208,"statfs":{"total":96636764160,"available":95876308992,"internally_reserved":0,"allocated":104394752,"data_stored":104184171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":656013765},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,3,0,1,0,2,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182525,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":760052,"kb_used_data":102280,"kb_used_omap":4,"kb_used_meta":657723,"kb_avail":93611788,"statfs":{"total":96636764160,"available":95858470912,"internally_reserved":0,"allocated":104734720,"data_stored":104524139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":673509069},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,5,1,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":103817216,"data_stored":103817216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":107884544,"data_stored":107884544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":103964672,"data_stored":103964672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":107503616,"data_stored":107503616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:44:19.904 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:44:20.533 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:44:20.533 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:44:20.553 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":361,"stamp":"2024-09-05T16:44:19.036809+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":207224136,"num_objects":25301,"num_object_clones":0,"num_object_copies":101204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28814,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25301,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":154849,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":552,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160658,"ondisk_log_size":160658,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2850428,"kb_used_data":407104,"kb_used_omap":14,"kb_used_meta":2443185,"kb_avail":374636932,"statfs":{"total":386547056640,"available":383628218368,"internally_reserved":0,"allocated":416874496,"data_stored":416032172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2501822007},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,4,10,7,4,8,6,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16752640,"num_objects":-2045,"num_object_clones":0,"num_object_copies":-8180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-88,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2045,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2045,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":88,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004581"},"pg_stats":[{"pgid":"1.f","version":"39'11313","reported_seq":11969,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:15.511045+0000","last_change":"2024-09-05T16:43:43.436935+0000","last_active":"2024-09-05T16:44:15.511045+0000","last_peered":"2024-09-05T16:44:15.511045+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:44:15.511045+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"29'1300","ondisk_log_start":"29'1300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13033472,"num_objects":1591,"num_object_clones":0,"num_object_copies":6364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1591,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11313,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":299,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-372}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'11034","reported_seq":15690,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:44:14.845289+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:44:14.845289+0000","last_peered":"2024-09-05T16:44:14.845289+0000","last_clean":"2024-09-05T16:44:14.845289+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:44:14.845289+0000","last_undegraded":"2024-09-05T16:44:14.845289+0000","last_fullsized":"2024-09-05T16:44:14.845289+0000","mapping_epoch":38,"log_start":"29'1000","ondisk_log_start":"29'1000","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12435456,"num_objects":1518,"num_object_clones":0,"num_object_copies":6072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1518,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":347,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'11174","reported_seq":11300,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:14.442019+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:44:14.442019+0000","last_peered":"2024-09-05T16:44:14.442019+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:44:14.442019+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"29'1100","ondisk_log_start":"29'1100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12925544,"num_objects":1579,"num_object_clones":0,"num_object_copies":6316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1579,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11173,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-653}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'11350","reported_seq":16084,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:44:15.102523+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:44:15.102523+0000","last_peered":"2024-09-05T16:44:15.102523+0000","last_clean":"2024-09-05T16:44:15.102523+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:44:15.102523+0000","last_undegraded":"2024-09-05T16:44:15.102523+0000","last_fullsized":"2024-09-05T16:44:15.102523+0000","mapping_epoch":34,"log_start":"29'1300","ondisk_log_start":"29'1300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12976128,"num_objects":1584,"num_object_clones":0,"num_object_copies":6336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1584,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5650,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'11023","reported_seq":11126,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:15.296722+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:44:15.296722+0000","last_peered":"2024-09-05T16:44:15.296722+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:44:15.296722+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"29'1000","ondisk_log_start":"29'1000","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12874912,"num_objects":1572,"num_object_clones":0,"num_object_copies":6288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1572,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11021,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-573}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'11234","reported_seq":11353,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:15.462879+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:44:15.462879+0000","last_peered":"2024-09-05T16:44:15.462879+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:44:15.462879+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"29'1200","ondisk_log_start":"29'1200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12599296,"num_objects":1538,"num_object_clones":0,"num_object_copies":6152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1538,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11234,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-681}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'11012","reported_seq":11124,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:14.682094+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:44:14.682094+0000","last_peered":"2024-09-05T16:44:14.682094+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:44:14.682094+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"29'1000","ondisk_log_start":"29'1000","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13513888,"num_objects":1650,"num_object_clones":0,"num_object_copies":6600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1650,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11010,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-481}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'11180","reported_seq":11273,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:14.726174+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:44:14.726174+0000","last_peered":"2024-09-05T16:44:14.726174+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:44:14.726174+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"29'1100","ondisk_log_start":"29'1100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13096096,"num_objects":1599,"num_object_clones":0,"num_object_copies":6396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11178,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-633}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"39'11234","reported_seq":11339,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:15.386125+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:44:15.386125+0000","last_peered":"2024-09-05T16:44:15.386125+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:44:15.386125+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"29'1200","ondisk_log_start":"29'1200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12609856,"num_objects":1540,"num_object_clones":0,"num_object_copies":6160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1540,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11231,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-704}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'11223","reported_seq":11348,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:14.458009+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:44:14.458009+0000","last_peered":"2024-09-05T16:44:14.458009+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:44:14.458009+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"29'1200","ondisk_log_start":"29'1200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12997792,"num_objects":1587,"num_object_clones":0,"num_object_copies":6348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1587,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11221,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-658}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'11487","reported_seq":11618,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:15.430920+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:44:15.430920+0000","last_peered":"2024-09-05T16:44:15.430920+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:44:15.430920+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"29'1400","ondisk_log_start":"29'1400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13397744,"num_objects":1636,"num_object_clones":0,"num_object_copies":6544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11484,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-645}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'11148","reported_seq":11251,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:15.520838+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:44:15.520838+0000","last_peered":"2024-09-05T16:44:15.520838+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:44:15.520838+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"29'1100","ondisk_log_start":"29'1100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13195856,"num_objects":1611,"num_object_clones":0,"num_object_copies":6444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1611,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11147,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-627}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'11254","reported_seq":15994,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:44:15.102411+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:44:15.102411+0000","last_peered":"2024-09-05T16:44:15.102411+0000","last_clean":"2024-09-05T16:44:15.102411+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:44:15.102411+0000","last_undegraded":"2024-09-05T16:44:15.102411+0000","last_fullsized":"2024-09-05T16:44:15.102411+0000","mapping_epoch":36,"log_start":"29'1200","ondisk_log_start":"29'1200","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12730368,"num_objects":1554,"num_object_clones":0,"num_object_copies":6216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1554,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2854,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'11520","reported_seq":11633,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:15.317932+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:44:15.317932+0000","last_peered":"2024-09-05T16:44:15.317932+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:44:15.317932+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"29'1500","ondisk_log_start":"29'1500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13153440,"num_objects":1606,"num_object_clones":0,"num_object_copies":6424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1606,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11518,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-728}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'11237","reported_seq":11354,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:14.296286+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:44:14.296286+0000","last_peered":"2024-09-05T16:44:14.296286+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:44:14.296286+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"29'1200","ondisk_log_start":"29'1200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13071520,"num_objects":1596,"num_object_clones":0,"num_object_copies":6384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1596,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11235,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-712}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'11235","reported_seq":11348,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:14.727285+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:44:14.727285+0000","last_peered":"2024-09-05T16:44:14.727285+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:44:14.727285+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"29'1200","ondisk_log_start":"29'1200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12612768,"num_objects":1540,"num_object_clones":0,"num_object_copies":6160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1540,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11233,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-680}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":207224136,"num_objects":25301,"num_object_clones":0,"num_object_copies":101204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28814,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25301,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":154849,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":552,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":414605312,"data_stored":414605312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160658,"ondisk_log_size":160658,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084411,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":754940,"kb_used_data":101288,"kb_used_omap":4,"kb_used_meta":653627,"kb_avail":93616900,"statfs":{"total":96636764160,"available":95863705600,"internally_reserved":0,"allocated":103718912,"data_stored":103508331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":669314771},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,1,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986288,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":592804,"kb_used_data":101588,"kb_used_omap":4,"kb_used_meta":491195,"kb_avail":93779036,"statfs":{"total":96636764160,"available":96029732864,"internally_reserved":0,"allocated":104026112,"data_stored":103815531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":502984402},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,1,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673094,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":742632,"kb_used_data":101948,"kb_used_omap":1,"kb_used_meta":640638,"kb_avail":93629208,"statfs":{"total":96636764160,"available":95876308992,"internally_reserved":0,"allocated":104394752,"data_stored":104184171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":656013765},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,3,0,1,0,2,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182525,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":760052,"kb_used_data":102280,"kb_used_omap":4,"kb_used_meta":657723,"kb_avail":93611788,"statfs":{"total":96636764160,"available":95858470912,"internally_reserved":0,"allocated":104734720,"data_stored":104524139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":673509069},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,5,1,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":103817216,"data_stored":103817216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":103596032,"data_stored":103596032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":103964672,"data_stored":103964672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":103227392,"data_stored":103227392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:44:20.554 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:44:21.201 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:44:21.202 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:44:21.221 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":362,"stamp":"2024-09-05T16:44:21.037439+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":198843720,"num_objects":24278,"num_object_clones":0,"num_object_copies":97112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28769,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24278,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":155872,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160981,"ondisk_log_size":160981,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2818324,"kb_used_data":391692,"kb_used_omap":14,"kb_used_meta":2426545,"kb_avail":374669036,"statfs":{"total":386547056640,"available":383661092864,"internally_reserved":0,"allocated":401092608,"data_stored":400250284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2484782647},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,0,0,5,6,16,22,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25133056,"num_objects":-3068,"num_object_clones":0,"num_object_copies":-12272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3068,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3068,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":133,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004637"},"pg_stats":[{"pgid":"1.f","version":"39'11361","reported_seq":12107,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:20.580463+0000","last_change":"2024-09-05T16:43:43.436935+0000","last_active":"2024-09-05T16:44:20.580463+0000","last_peered":"2024-09-05T16:44:20.580463+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:44:20.580463+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"29'1300","ondisk_log_start":"29'1300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12640256,"num_objects":1543,"num_object_clones":0,"num_object_copies":6172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1918,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1543,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11361,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":344,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-375}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'11107","reported_seq":15763,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:44:20.309521+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:44:20.309521+0000","last_peered":"2024-09-05T16:44:20.309521+0000","last_clean":"2024-09-05T16:44:20.309521+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:44:20.309521+0000","last_undegraded":"2024-09-05T16:44:20.309521+0000","last_fullsized":"2024-09-05T16:44:20.309521+0000","mapping_epoch":38,"log_start":"29'1100","ondisk_log_start":"29'1100","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11837440,"num_objects":1445,"num_object_clones":0,"num_object_copies":5780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1445,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":420,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'11240","reported_seq":11367,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:19.506006+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:44:19.506006+0000","last_peered":"2024-09-05T16:44:19.506006+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:44:19.506006+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"29'1200","ondisk_log_start":"29'1200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12384872,"num_objects":1513,"num_object_clones":0,"num_object_copies":6052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1513,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11239,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-719}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'11420","reported_seq":16154,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:44:20.174040+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:44:20.174040+0000","last_peered":"2024-09-05T16:44:20.174040+0000","last_clean":"2024-09-05T16:44:20.174040+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:44:20.174040+0000","last_undegraded":"2024-09-05T16:44:20.174040+0000","last_fullsized":"2024-09-05T16:44:20.174040+0000","mapping_epoch":34,"log_start":"29'1400","ondisk_log_start":"29'1400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12402688,"num_objects":1514,"num_object_clones":0,"num_object_copies":6056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1514,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5720,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'11094","reported_seq":11197,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:20.442925+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:44:20.442925+0000","last_peered":"2024-09-05T16:44:20.442925+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:44:20.442925+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"29'1000","ondisk_log_start":"29'1000","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12293280,"num_objects":1501,"num_object_clones":0,"num_object_copies":6004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1501,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11092,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-644}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'11295","reported_seq":11413,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:20.338238+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:44:20.338238+0000","last_peered":"2024-09-05T16:44:20.338238+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:44:20.338238+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"29'1200","ondisk_log_start":"29'1200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12099584,"num_objects":1477,"num_object_clones":0,"num_object_copies":5908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1477,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11295,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-742}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'11072","reported_seq":11184,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:19.514538+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:44:19.514538+0000","last_peered":"2024-09-05T16:44:19.514538+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:44:19.514538+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"29'1000","ondisk_log_start":"29'1000","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13022368,"num_objects":1590,"num_object_clones":0,"num_object_copies":6360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1590,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11070,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-541}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'11251","reported_seq":11344,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:19.727482+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:44:19.727482+0000","last_peered":"2024-09-05T16:44:19.727482+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:44:19.727482+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"29'1200","ondisk_log_start":"29'1200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12514464,"num_objects":1528,"num_object_clones":0,"num_object_copies":6112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1528,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11249,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-704}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"39'11300","reported_seq":11405,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:20.246221+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:44:20.246221+0000","last_peered":"2024-09-05T16:44:20.246221+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:44:20.246221+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"29'1200","ondisk_log_start":"29'1200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12069184,"num_objects":1474,"num_object_clones":0,"num_object_copies":5896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1474,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11297,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-770}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'11289","reported_seq":11414,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:19.553254+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:44:19.553254+0000","last_peered":"2024-09-05T16:44:19.553254+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:44:19.553254+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"29'1200","ondisk_log_start":"29'1200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12457120,"num_objects":1521,"num_object_clones":0,"num_object_copies":6084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1521,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11287,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-724}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'11550","reported_seq":11681,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:20.580702+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:44:20.580702+0000","last_peered":"2024-09-05T16:44:20.580702+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:44:20.580702+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"29'1500","ondisk_log_start":"29'1500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12881648,"num_objects":1573,"num_object_clones":0,"num_object_copies":6292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1573,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11547,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-708}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'11196","reported_seq":11299,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:20.429771+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:44:20.429771+0000","last_peered":"2024-09-05T16:44:20.429771+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:44:20.429771+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"29'1100","ondisk_log_start":"29'1100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12802640,"num_objects":1563,"num_object_clones":0,"num_object_copies":6252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1563,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11195,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-675}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'11318","reported_seq":16058,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:44:20.115574+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:44:20.115574+0000","last_peered":"2024-09-05T16:44:20.115574+0000","last_clean":"2024-09-05T16:44:20.115574+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:44:20.115574+0000","last_undegraded":"2024-09-05T16:44:20.115574+0000","last_fullsized":"2024-09-05T16:44:20.115574+0000","mapping_epoch":36,"log_start":"29'1300","ondisk_log_start":"29'1300","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12206080,"num_objects":1490,"num_object_clones":0,"num_object_copies":5960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2918,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'11586","reported_seq":11698,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:20.586288+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:44:20.586288+0000","last_peered":"2024-09-05T16:44:20.586288+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:44:20.586288+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"29'1500","ondisk_log_start":"29'1500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12612768,"num_objects":1540,"num_object_clones":0,"num_object_copies":6160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1540,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11584,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-794}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'11308","reported_seq":11425,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:19.581517+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:44:19.581517+0000","last_peered":"2024-09-05T16:44:19.581517+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:44:19.581517+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"29'1300","ondisk_log_start":"29'1300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12489888,"num_objects":1525,"num_object_clones":0,"num_object_copies":6100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1525,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11306,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-783}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'11294","reported_seq":11407,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:19.789528+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:44:19.789528+0000","last_peered":"2024-09-05T16:44:19.789528+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:44:19.789528+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"29'1200","ondisk_log_start":"29'1200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12129440,"num_objects":1481,"num_object_clones":0,"num_object_copies":5924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1481,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11292,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-739}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":198843720,"num_objects":24278,"num_object_clones":0,"num_object_copies":97112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28769,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24278,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":155872,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":397963264,"data_stored":397963264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160981,"ondisk_log_size":160981,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084412,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":754904,"kb_used_data":97096,"kb_used_omap":4,"kb_used_meta":657787,"kb_avail":93616936,"statfs":{"total":96636764160,"available":95863742464,"internally_reserved":0,"allocated":99426304,"data_stored":99215723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":673574611},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,0,0,1,2,1,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986289,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":576956,"kb_used_data":98220,"kb_used_omap":4,"kb_used_meta":478715,"kb_avail":93794884,"statfs":{"total":96636764160,"available":96045961216,"internally_reserved":0,"allocated":100577280,"data_stored":100366699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":490204882},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,8,10,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673095,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":742820,"kb_used_data":97996,"kb_used_omap":1,"kb_used_meta":644798,"kb_avail":93629020,"statfs":{"total":96636764160,"available":95876116480,"internally_reserved":0,"allocated":100347904,"data_stored":100137323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":660273605},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,7,7,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182526,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":743644,"kb_used_data":98380,"kb_used_omap":4,"kb_used_meta":645243,"kb_avail":93628196,"statfs":{"total":96636764160,"available":95875272704,"internally_reserved":0,"allocated":100741120,"data_stored":100530539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":660729549},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,1,0,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":99663872,"data_stored":99663872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":99356672,"data_stored":99356672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":99860480,"data_stored":99860480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":99082240,"data_stored":99082240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:44:21.222 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-05T16:44:21.222 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "39'11361", 'reported_seq': 12107, 'reported_epoch': 39, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:20.580463+0000', 'last_change': '2024-09-05T16:43:43.436935+0000', 'last_active': '2024-09-05T16:44:20.580463+0000', 'last_peered': '2024-09-05T16:44:20.580463+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.828289+0000', 'last_became_peered': '2024-09-05T16:34:56.828289+0000', 'last_unstale': '2024-09-05T16:44:20.580463+0000', 'last_undegraded': '2024-09-05T16:34:56.612160+0000', 'last_fullsized': '2024-09-05T16:34:56.557632+0000', 'mapping_epoch': 32, 'log_start': "29'1300", 'ondisk_log_start': "29'1300", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10061, 'log_dups_size': 0, 'ondisk_log_size': 10061, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12640256, 'num_objects': 1543, 'num_object_clones': 0, 'num_object_copies': 6172, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1918, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1543, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11361, 'num_write_kb': 51616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 344, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -375}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:44:21.222 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:44:21.222 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "39'11240", 'reported_seq': 11367, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:19.506006+0000', 'last_change': '2024-09-05T16:34:56.867900+0000', 'last_active': '2024-09-05T16:44:19.506006+0000', 'last_peered': '2024-09-05T16:44:19.506006+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:44:19.506006+0000', 'last_undegraded': '2024-09-05T16:34:56.590972+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 32, 'log_start': "29'1200", 'ondisk_log_start': "29'1200", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12384872, 'num_objects': 1513, 'num_object_clones': 0, 'num_object_copies': 6052, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2233, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1513, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 11239, 'num_write_kb': 50999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -719}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:44:21.222 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:44:21.223 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "39'11094", 'reported_seq': 11197, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:20.442925+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:44:20.442925+0000', 'last_peered': '2024-09-05T16:44:20.442925+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:44:20.442925+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "29'1000", 'ondisk_log_start': "29'1000", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10094, 'log_dups_size': 0, 'ondisk_log_size': 10094, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12293280, 'num_objects': 1501, 'num_object_clones': 0, 'num_object_copies': 6004, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2147, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1501, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11092, 'num_write_kb': 50373, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -644}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:44:21.223 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:44:21.223 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "39'11295", 'reported_seq': 11413, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:20.338238+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:44:20.338238+0000', 'last_peered': '2024-09-05T16:44:20.338238+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:44:20.338238+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "29'1200", 'ondisk_log_start': "29'1200", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10095, 'log_dups_size': 0, 'ondisk_log_size': 10095, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12099584, 'num_objects': 1477, 'num_object_clones': 0, 'num_object_copies': 5908, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1477, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11295, 'num_write_kb': 51088, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -742}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:44:21.223 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:44:21.223 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "39'11072", 'reported_seq': 11184, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:19.514538+0000', 'last_change': '2024-09-05T16:34:56.903436+0000', 'last_active': '2024-09-05T16:44:19.514538+0000', 'last_peered': '2024-09-05T16:44:19.514538+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:44:19.514538+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "29'1000", 'ondisk_log_start': "29'1000", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10072, 'log_dups_size': 0, 'ondisk_log_size': 10072, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13022368, 'num_objects': 1590, 'num_object_clones': 0, 'num_object_copies': 6360, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2133, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1590, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11070, 'num_write_kb': 50636, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -541}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:44:21.223 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-05T16:44:21.223 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "39'11251", 'reported_seq': 11344, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:19.727482+0000', 'last_change': '2024-09-05T16:34:56.883319+0000', 'last_active': '2024-09-05T16:44:19.727482+0000', 'last_peered': '2024-09-05T16:44:19.727482+0000', 'last_clean': '2024-09-05T16:33:53.505305+0000', 'last_became_active': '2024-09-05T16:34:56.873389+0000', 'last_became_peered': '2024-09-05T16:34:56.873389+0000', 'last_unstale': '2024-09-05T16:44:19.727482+0000', 'last_undegraded': '2024-09-05T16:34:56.602567+0000', 'last_fullsized': '2024-09-05T16:34:56.559958+0000', 'mapping_epoch': 32, 'log_start': "29'1200", 'ondisk_log_start': "29'1200", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10051, 'log_dups_size': 0, 'ondisk_log_size': 10051, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12514464, 'num_objects': 1528, 'num_object_clones': 0, 'num_object_copies': 6112, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2234, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1528, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11249, 'num_write_kb': 51108, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -704}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:44:21.223 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:44:21.223 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "39'11300", 'reported_seq': 11405, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:20.246221+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:44:20.246221+0000', 'last_peered': '2024-09-05T16:44:20.246221+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:44:20.246221+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "29'1200", 'ondisk_log_start': "29'1200", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10100, 'log_dups_size': 0, 'ondisk_log_size': 10100, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12069184, 'num_objects': 1474, 'num_object_clones': 0, 'num_object_copies': 5896, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1474, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11297, 'num_write_kb': 51081, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -770}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:44:21.223 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:44:21.224 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "39'11289", 'reported_seq': 11414, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:19.553254+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:44:19.553254+0000', 'last_peered': '2024-09-05T16:44:19.553254+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:44:19.553254+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "29'1200", 'ondisk_log_start': "29'1200", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10089, 'log_dups_size': 0, 'ondisk_log_size': 10089, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12457120, 'num_objects': 1521, 'num_object_clones': 0, 'num_object_copies': 6084, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1521, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11287, 'num_write_kb': 51230, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -724}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:44:21.224 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:44:21.224 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "39'11550", 'reported_seq': 11681, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:20.580702+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:44:20.580702+0000', 'last_peered': '2024-09-05T16:44:20.580702+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:44:20.580702+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "29'1500", 'ondisk_log_start': "29'1500", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12881648, 'num_objects': 1573, 'num_object_clones': 0, 'num_object_copies': 6292, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2284, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1573, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11547, 'num_write_kb': 52476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -708}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:44:21.224 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:44:21.224 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "39'11196", 'reported_seq': 11299, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:20.429771+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:44:20.429771+0000', 'last_peered': '2024-09-05T16:44:20.429771+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:44:20.429771+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "29'1100", 'ondisk_log_start': "29'1100", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10096, 'log_dups_size': 0, 'ondisk_log_size': 10096, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12802640, 'num_objects': 1563, 'num_object_clones': 0, 'num_object_copies': 6252, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1563, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11195, 'num_write_kb': 51031, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -675}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:44:21.224 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:44:21.224 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "39'11586", 'reported_seq': 11698, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:20.586288+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:44:20.586288+0000', 'last_peered': '2024-09-05T16:44:20.586288+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:44:20.586288+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "29'1500", 'ondisk_log_start': "29'1500", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10086, 'log_dups_size': 0, 'ondisk_log_size': 10086, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12612768, 'num_objects': 1540, 'num_object_clones': 0, 'num_object_copies': 6160, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1540, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11584, 'num_write_kb': 52500, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -794}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:44:21.224 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:44:21.224 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "39'11308", 'reported_seq': 11425, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:19.581517+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:44:19.581517+0000', 'last_peered': '2024-09-05T16:44:19.581517+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:44:19.581517+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "29'1300", 'ondisk_log_start': "29'1300", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10008, 'log_dups_size': 0, 'ondisk_log_size': 10008, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12489888, 'num_objects': 1525, 'num_object_clones': 0, 'num_object_copies': 6100, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2310, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1525, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11306, 'num_write_kb': 51325, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -783}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:44:21.224 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:44:21.225 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "39'11294", 'reported_seq': 11407, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:19.789528+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:44:19.789528+0000', 'last_peered': '2024-09-05T16:44:19.789528+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:44:19.789528+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "29'1200", 'ondisk_log_start': "29'1200", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10094, 'log_dups_size': 0, 'ondisk_log_size': 10094, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12129440, 'num_objects': 1481, 'num_object_clones': 0, 'num_object_copies': 5924, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1481, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11292, 'num_write_kb': 51092, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -739}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:44:21.225 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:44:21.947 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:44:21.947 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 28769/97112 objects degraded (29.625%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":665,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":39,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":24278,"data_bytes":198843720,"bytes_used":2885963776,"bytes_avail":383661092864,"bytes_total":386547056640,"degraded_objects":28769,"degraded_total":97112,"degraded_ratio":0.29624557212290964,"recovering_objects_per_sec":11,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":133,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":255},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:44:21.972 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:44:21.972 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:44:22.566 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:44:22.566 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:44:22.593 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":362,"stamp":"2024-09-05T16:44:21.037439+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":198843720,"num_objects":24278,"num_object_clones":0,"num_object_copies":97112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28769,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24278,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":155872,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160981,"ondisk_log_size":160981,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2818324,"kb_used_data":391692,"kb_used_omap":14,"kb_used_meta":2426545,"kb_avail":374669036,"statfs":{"total":386547056640,"available":383661092864,"internally_reserved":0,"allocated":401092608,"data_stored":400250284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2484782647},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,0,0,5,6,16,22,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25133056,"num_objects":-3068,"num_object_clones":0,"num_object_copies":-12272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3068,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3068,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":133,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004637"},"pg_stats":[{"pgid":"1.f","version":"39'11361","reported_seq":12107,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:20.580463+0000","last_change":"2024-09-05T16:43:43.436935+0000","last_active":"2024-09-05T16:44:20.580463+0000","last_peered":"2024-09-05T16:44:20.580463+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:44:20.580463+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"29'1300","ondisk_log_start":"29'1300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12640256,"num_objects":1543,"num_object_clones":0,"num_object_copies":6172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1918,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1543,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11361,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":344,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-375}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'11107","reported_seq":15763,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:44:20.309521+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:44:20.309521+0000","last_peered":"2024-09-05T16:44:20.309521+0000","last_clean":"2024-09-05T16:44:20.309521+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:44:20.309521+0000","last_undegraded":"2024-09-05T16:44:20.309521+0000","last_fullsized":"2024-09-05T16:44:20.309521+0000","mapping_epoch":38,"log_start":"29'1100","ondisk_log_start":"29'1100","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11837440,"num_objects":1445,"num_object_clones":0,"num_object_copies":5780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1445,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":420,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'11240","reported_seq":11367,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:19.506006+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:44:19.506006+0000","last_peered":"2024-09-05T16:44:19.506006+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:44:19.506006+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"29'1200","ondisk_log_start":"29'1200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12384872,"num_objects":1513,"num_object_clones":0,"num_object_copies":6052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1513,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11239,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-719}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'11420","reported_seq":16154,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:44:20.174040+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:44:20.174040+0000","last_peered":"2024-09-05T16:44:20.174040+0000","last_clean":"2024-09-05T16:44:20.174040+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:44:20.174040+0000","last_undegraded":"2024-09-05T16:44:20.174040+0000","last_fullsized":"2024-09-05T16:44:20.174040+0000","mapping_epoch":34,"log_start":"29'1400","ondisk_log_start":"29'1400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12402688,"num_objects":1514,"num_object_clones":0,"num_object_copies":6056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1514,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5720,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'11094","reported_seq":11197,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:20.442925+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:44:20.442925+0000","last_peered":"2024-09-05T16:44:20.442925+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:44:20.442925+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"29'1000","ondisk_log_start":"29'1000","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12293280,"num_objects":1501,"num_object_clones":0,"num_object_copies":6004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1501,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11092,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-644}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'11295","reported_seq":11413,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:20.338238+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:44:20.338238+0000","last_peered":"2024-09-05T16:44:20.338238+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:44:20.338238+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"29'1200","ondisk_log_start":"29'1200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12099584,"num_objects":1477,"num_object_clones":0,"num_object_copies":5908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1477,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11295,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-742}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'11072","reported_seq":11184,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:19.514538+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:44:19.514538+0000","last_peered":"2024-09-05T16:44:19.514538+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:44:19.514538+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"29'1000","ondisk_log_start":"29'1000","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13022368,"num_objects":1590,"num_object_clones":0,"num_object_copies":6360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1590,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11070,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-541}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'11251","reported_seq":11344,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:19.727482+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:44:19.727482+0000","last_peered":"2024-09-05T16:44:19.727482+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:44:19.727482+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"29'1200","ondisk_log_start":"29'1200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12514464,"num_objects":1528,"num_object_clones":0,"num_object_copies":6112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1528,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11249,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-704}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"39'11300","reported_seq":11405,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:20.246221+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:44:20.246221+0000","last_peered":"2024-09-05T16:44:20.246221+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:44:20.246221+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"29'1200","ondisk_log_start":"29'1200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12069184,"num_objects":1474,"num_object_clones":0,"num_object_copies":5896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1474,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11297,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-770}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'11289","reported_seq":11414,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:19.553254+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:44:19.553254+0000","last_peered":"2024-09-05T16:44:19.553254+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:44:19.553254+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"29'1200","ondisk_log_start":"29'1200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12457120,"num_objects":1521,"num_object_clones":0,"num_object_copies":6084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1521,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11287,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-724}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'11550","reported_seq":11681,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:20.580702+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:44:20.580702+0000","last_peered":"2024-09-05T16:44:20.580702+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:44:20.580702+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"29'1500","ondisk_log_start":"29'1500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12881648,"num_objects":1573,"num_object_clones":0,"num_object_copies":6292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1573,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11547,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-708}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'11196","reported_seq":11299,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:20.429771+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:44:20.429771+0000","last_peered":"2024-09-05T16:44:20.429771+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:44:20.429771+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"29'1100","ondisk_log_start":"29'1100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12802640,"num_objects":1563,"num_object_clones":0,"num_object_copies":6252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1563,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11195,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-675}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'11318","reported_seq":16058,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:44:20.115574+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:44:20.115574+0000","last_peered":"2024-09-05T16:44:20.115574+0000","last_clean":"2024-09-05T16:44:20.115574+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:44:20.115574+0000","last_undegraded":"2024-09-05T16:44:20.115574+0000","last_fullsized":"2024-09-05T16:44:20.115574+0000","mapping_epoch":36,"log_start":"29'1300","ondisk_log_start":"29'1300","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12206080,"num_objects":1490,"num_object_clones":0,"num_object_copies":5960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2918,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'11586","reported_seq":11698,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:20.586288+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:44:20.586288+0000","last_peered":"2024-09-05T16:44:20.586288+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:44:20.586288+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"29'1500","ondisk_log_start":"29'1500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12612768,"num_objects":1540,"num_object_clones":0,"num_object_copies":6160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1540,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11584,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-794}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'11308","reported_seq":11425,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:19.581517+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:44:19.581517+0000","last_peered":"2024-09-05T16:44:19.581517+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:44:19.581517+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"29'1300","ondisk_log_start":"29'1300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12489888,"num_objects":1525,"num_object_clones":0,"num_object_copies":6100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1525,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11306,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-783}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'11294","reported_seq":11407,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:19.789528+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:44:19.789528+0000","last_peered":"2024-09-05T16:44:19.789528+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:44:19.789528+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"29'1200","ondisk_log_start":"29'1200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12129440,"num_objects":1481,"num_object_clones":0,"num_object_copies":5924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1481,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11292,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-739}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":198843720,"num_objects":24278,"num_object_clones":0,"num_object_copies":97112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28769,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24278,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":155872,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":397963264,"data_stored":397963264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160981,"ondisk_log_size":160981,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084412,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":754904,"kb_used_data":97096,"kb_used_omap":4,"kb_used_meta":657787,"kb_avail":93616936,"statfs":{"total":96636764160,"available":95863742464,"internally_reserved":0,"allocated":99426304,"data_stored":99215723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":673574611},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,0,0,1,2,1,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986289,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":576956,"kb_used_data":98220,"kb_used_omap":4,"kb_used_meta":478715,"kb_avail":93794884,"statfs":{"total":96636764160,"available":96045961216,"internally_reserved":0,"allocated":100577280,"data_stored":100366699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":490204882},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,8,10,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673095,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":742820,"kb_used_data":97996,"kb_used_omap":1,"kb_used_meta":644798,"kb_avail":93629020,"statfs":{"total":96636764160,"available":95876116480,"internally_reserved":0,"allocated":100347904,"data_stored":100137323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":660273605},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,7,7,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182526,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":743644,"kb_used_data":98380,"kb_used_omap":4,"kb_used_meta":645243,"kb_avail":93628196,"statfs":{"total":96636764160,"available":95875272704,"internally_reserved":0,"allocated":100741120,"data_stored":100530539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":660729549},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,1,0,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":99663872,"data_stored":99663872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":99356672,"data_stored":99356672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":99860480,"data_stored":99860480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":99082240,"data_stored":99082240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:44:25.595 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:44:26.242 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:44:26.242 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:44:26.269 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":364,"stamp":"2024-09-05T16:44:25.038997+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":195829064,"num_objects":23910,"num_object_clones":0,"num_object_copies":95640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28769,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23910,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":156240,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160849,"ondisk_log_size":160849,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2814528,"kb_used_data":383732,"kb_used_omap":14,"kb_used_meta":2430705,"kb_avail":374672832,"statfs":{"total":386547056640,"available":383664979968,"internally_reserved":0,"allocated":392941568,"data_stored":392099244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2489042487},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,0,0,5,7,18,26,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19759104,"num_objects":-2412,"num_object_clones":0,"num_object_copies":-9648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-88,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2412,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2412,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":88,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004665"},"pg_stats":[{"pgid":"1.f","version":"39'11361","reported_seq":12107,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:20.580463+0000","last_change":"2024-09-05T16:43:43.436935+0000","last_active":"2024-09-05T16:44:20.580463+0000","last_peered":"2024-09-05T16:44:20.580463+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:44:20.580463+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"29'1300","ondisk_log_start":"29'1300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12640256,"num_objects":1543,"num_object_clones":0,"num_object_copies":6172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1918,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1543,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11361,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":344,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-375}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'11107","reported_seq":15763,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:44:20.309521+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:44:20.309521+0000","last_peered":"2024-09-05T16:44:20.309521+0000","last_clean":"2024-09-05T16:44:20.309521+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:44:20.309521+0000","last_undegraded":"2024-09-05T16:44:20.309521+0000","last_fullsized":"2024-09-05T16:44:20.309521+0000","mapping_epoch":38,"log_start":"29'1100","ondisk_log_start":"29'1100","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11837440,"num_objects":1445,"num_object_clones":0,"num_object_copies":5780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1445,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":420,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'11305","reported_seq":11432,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:24.521135+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:44:24.521135+0000","last_peered":"2024-09-05T16:44:24.521135+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:44:24.521135+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"29'1300","ondisk_log_start":"29'1300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11852392,"num_objects":1448,"num_object_clones":0,"num_object_copies":5792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1448,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11304,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-784}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'11420","reported_seq":16154,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:44:20.174040+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:44:20.174040+0000","last_peered":"2024-09-05T16:44:20.174040+0000","last_clean":"2024-09-05T16:44:20.174040+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:44:20.174040+0000","last_undegraded":"2024-09-05T16:44:20.174040+0000","last_fullsized":"2024-09-05T16:44:20.174040+0000","mapping_epoch":34,"log_start":"29'1400","ondisk_log_start":"29'1400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12402688,"num_objects":1514,"num_object_clones":0,"num_object_copies":6056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1514,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5720,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'11094","reported_seq":11197,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:20.442925+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:44:20.442925+0000","last_peered":"2024-09-05T16:44:20.442925+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:44:20.442925+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"29'1000","ondisk_log_start":"29'1000","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12293280,"num_objects":1501,"num_object_clones":0,"num_object_copies":6004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1501,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11092,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-644}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'11295","reported_seq":11413,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:20.338238+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:44:20.338238+0000","last_peered":"2024-09-05T16:44:20.338238+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:44:20.338238+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"29'1200","ondisk_log_start":"29'1200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12099584,"num_objects":1477,"num_object_clones":0,"num_object_copies":5908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1477,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11295,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-742}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'11146","reported_seq":11258,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:24.724062+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:44:24.724062+0000","last_peered":"2024-09-05T16:44:24.724062+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:44:24.724062+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"29'1100","ondisk_log_start":"29'1100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12416160,"num_objects":1516,"num_object_clones":0,"num_object_copies":6064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1516,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11144,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-615}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'11316","reported_seq":11407,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:24.865194+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:44:24.865194+0000","last_peered":"2024-09-05T16:44:24.865194+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:44:24.865194+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"29'1300","ondisk_log_start":"29'1300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11981984,"num_objects":1463,"num_object_clones":0,"num_object_copies":5852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1463,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11314,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-769}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"39'11300","reported_seq":11405,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:20.246221+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:44:20.246221+0000","last_peered":"2024-09-05T16:44:20.246221+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:44:20.246221+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"29'1200","ondisk_log_start":"29'1200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12069184,"num_objects":1474,"num_object_clones":0,"num_object_copies":5896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1474,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11297,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-770}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'11338","reported_seq":11465,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:24.635655+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:44:24.635655+0000","last_peered":"2024-09-05T16:44:24.635655+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:44:24.635655+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"29'1300","ondisk_log_start":"29'1300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12055712,"num_objects":1472,"num_object_clones":0,"num_object_copies":5888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1472,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11336,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-773}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'11550","reported_seq":11681,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:20.580702+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:44:20.580702+0000","last_peered":"2024-09-05T16:44:20.580702+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:44:20.580702+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"29'1500","ondisk_log_start":"29'1500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12881648,"num_objects":1573,"num_object_clones":0,"num_object_copies":6292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1573,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11547,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-708}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'11196","reported_seq":11299,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:20.429771+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:44:20.429771+0000","last_peered":"2024-09-05T16:44:20.429771+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:44:20.429771+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"29'1100","ondisk_log_start":"29'1100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12802640,"num_objects":1563,"num_object_clones":0,"num_object_copies":6252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1563,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11195,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-675}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'11318","reported_seq":16058,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:44:20.115574+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:44:20.115574+0000","last_peered":"2024-09-05T16:44:20.115574+0000","last_clean":"2024-09-05T16:44:20.115574+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:44:20.115574+0000","last_undegraded":"2024-09-05T16:44:20.115574+0000","last_fullsized":"2024-09-05T16:44:20.115574+0000","mapping_epoch":36,"log_start":"29'1300","ondisk_log_start":"29'1300","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12206080,"num_objects":1490,"num_object_clones":0,"num_object_copies":5960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2918,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'11586","reported_seq":11698,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:20.586288+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:44:20.586288+0000","last_peered":"2024-09-05T16:44:20.586288+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:44:20.586288+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"29'1500","ondisk_log_start":"29'1500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12612768,"num_objects":1540,"num_object_clones":0,"num_object_copies":6160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1540,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11584,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-794}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'11370","reported_seq":11487,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:24.559136+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:44:24.559136+0000","last_peered":"2024-09-05T16:44:24.559136+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:44:24.559136+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"29'1300","ondisk_log_start":"29'1300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11981984,"num_objects":1463,"num_object_clones":0,"num_object_copies":5852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1463,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11368,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-845}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'11347","reported_seq":11459,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:24.805836+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:44:24.805836+0000","last_peered":"2024-09-05T16:44:24.805836+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:44:24.805836+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"29'1300","ondisk_log_start":"29'1300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11695264,"num_objects":1428,"num_object_clones":0,"num_object_copies":5712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1428,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11345,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-792}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":195829064,"num_objects":23910,"num_object_clones":0,"num_object_copies":95640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28769,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23910,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":156240,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":389787648,"data_stored":389787648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160849,"ondisk_log_size":160849,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084412,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":754904,"kb_used_data":97096,"kb_used_omap":4,"kb_used_meta":657787,"kb_avail":93616936,"statfs":{"total":96636764160,"available":95863742464,"internally_reserved":0,"allocated":99426304,"data_stored":99215723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":673574611},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,0,0,1,2,1,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986289,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":576956,"kb_used_data":98220,"kb_used_omap":4,"kb_used_meta":478715,"kb_avail":93794884,"statfs":{"total":96636764160,"available":96045961216,"internally_reserved":0,"allocated":100577280,"data_stored":100366699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":490204882},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,8,10,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673096,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":742968,"kb_used_data":93988,"kb_used_omap":1,"kb_used_meta":648958,"kb_avail":93628872,"statfs":{"total":96636764160,"available":95875964928,"internally_reserved":0,"allocated":96243712,"data_stored":96033131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":664533445},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,0,7,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182527,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":739700,"kb_used_data":94428,"kb_used_omap":4,"kb_used_meta":645243,"kb_avail":93632140,"statfs":{"total":96636764160,"available":95879311360,"internally_reserved":0,"allocated":96694272,"data_stored":96483691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":660729549},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,2,10,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":95559680,"data_stored":95559680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":99356672,"data_stored":99356672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":95789056,"data_stored":95789056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":99082240,"data_stored":99082240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:44:26.271 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:44:26.890 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:44:26.890 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:44:26.920 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":364,"stamp":"2024-09-05T16:44:25.038997+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":195829064,"num_objects":23910,"num_object_clones":0,"num_object_copies":95640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28769,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23910,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":156240,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160849,"ondisk_log_size":160849,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2814528,"kb_used_data":383732,"kb_used_omap":14,"kb_used_meta":2430705,"kb_avail":374672832,"statfs":{"total":386547056640,"available":383664979968,"internally_reserved":0,"allocated":392941568,"data_stored":392099244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2489042487},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,0,0,5,7,18,26,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19759104,"num_objects":-2412,"num_object_clones":0,"num_object_copies":-9648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-88,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2412,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2412,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":88,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004665"},"pg_stats":[{"pgid":"1.f","version":"39'11361","reported_seq":12107,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:20.580463+0000","last_change":"2024-09-05T16:43:43.436935+0000","last_active":"2024-09-05T16:44:20.580463+0000","last_peered":"2024-09-05T16:44:20.580463+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:44:20.580463+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"29'1300","ondisk_log_start":"29'1300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12640256,"num_objects":1543,"num_object_clones":0,"num_object_copies":6172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1918,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1543,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11361,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":344,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-375}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'11107","reported_seq":15763,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:44:20.309521+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:44:20.309521+0000","last_peered":"2024-09-05T16:44:20.309521+0000","last_clean":"2024-09-05T16:44:20.309521+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:44:20.309521+0000","last_undegraded":"2024-09-05T16:44:20.309521+0000","last_fullsized":"2024-09-05T16:44:20.309521+0000","mapping_epoch":38,"log_start":"29'1100","ondisk_log_start":"29'1100","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11837440,"num_objects":1445,"num_object_clones":0,"num_object_copies":5780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1445,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":420,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'11305","reported_seq":11432,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:24.521135+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:44:24.521135+0000","last_peered":"2024-09-05T16:44:24.521135+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:44:24.521135+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"29'1300","ondisk_log_start":"29'1300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11852392,"num_objects":1448,"num_object_clones":0,"num_object_copies":5792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1448,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11304,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-784}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'11420","reported_seq":16154,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:44:20.174040+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:44:20.174040+0000","last_peered":"2024-09-05T16:44:20.174040+0000","last_clean":"2024-09-05T16:44:20.174040+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:44:20.174040+0000","last_undegraded":"2024-09-05T16:44:20.174040+0000","last_fullsized":"2024-09-05T16:44:20.174040+0000","mapping_epoch":34,"log_start":"29'1400","ondisk_log_start":"29'1400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12402688,"num_objects":1514,"num_object_clones":0,"num_object_copies":6056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1514,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5720,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'11094","reported_seq":11197,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:20.442925+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:44:20.442925+0000","last_peered":"2024-09-05T16:44:20.442925+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:44:20.442925+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"29'1000","ondisk_log_start":"29'1000","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12293280,"num_objects":1501,"num_object_clones":0,"num_object_copies":6004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1501,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11092,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-644}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'11295","reported_seq":11413,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:20.338238+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:44:20.338238+0000","last_peered":"2024-09-05T16:44:20.338238+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:44:20.338238+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"29'1200","ondisk_log_start":"29'1200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12099584,"num_objects":1477,"num_object_clones":0,"num_object_copies":5908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1477,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11295,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-742}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'11146","reported_seq":11258,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:24.724062+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:44:24.724062+0000","last_peered":"2024-09-05T16:44:24.724062+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:44:24.724062+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"29'1100","ondisk_log_start":"29'1100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12416160,"num_objects":1516,"num_object_clones":0,"num_object_copies":6064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1516,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11144,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-615}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'11316","reported_seq":11407,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:24.865194+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:44:24.865194+0000","last_peered":"2024-09-05T16:44:24.865194+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:44:24.865194+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"29'1300","ondisk_log_start":"29'1300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11981984,"num_objects":1463,"num_object_clones":0,"num_object_copies":5852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1463,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11314,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-769}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"39'11300","reported_seq":11405,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:20.246221+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:44:20.246221+0000","last_peered":"2024-09-05T16:44:20.246221+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:44:20.246221+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"29'1200","ondisk_log_start":"29'1200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12069184,"num_objects":1474,"num_object_clones":0,"num_object_copies":5896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1474,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11297,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-770}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'11338","reported_seq":11465,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:24.635655+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:44:24.635655+0000","last_peered":"2024-09-05T16:44:24.635655+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:44:24.635655+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"29'1300","ondisk_log_start":"29'1300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12055712,"num_objects":1472,"num_object_clones":0,"num_object_copies":5888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1472,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11336,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-773}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'11550","reported_seq":11681,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:20.580702+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:44:20.580702+0000","last_peered":"2024-09-05T16:44:20.580702+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:44:20.580702+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"29'1500","ondisk_log_start":"29'1500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12881648,"num_objects":1573,"num_object_clones":0,"num_object_copies":6292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1573,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11547,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-708}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'11196","reported_seq":11299,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:20.429771+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:44:20.429771+0000","last_peered":"2024-09-05T16:44:20.429771+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:44:20.429771+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"29'1100","ondisk_log_start":"29'1100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12802640,"num_objects":1563,"num_object_clones":0,"num_object_copies":6252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1563,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11195,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-675}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'11318","reported_seq":16058,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:44:20.115574+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:44:20.115574+0000","last_peered":"2024-09-05T16:44:20.115574+0000","last_clean":"2024-09-05T16:44:20.115574+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:44:20.115574+0000","last_undegraded":"2024-09-05T16:44:20.115574+0000","last_fullsized":"2024-09-05T16:44:20.115574+0000","mapping_epoch":36,"log_start":"29'1300","ondisk_log_start":"29'1300","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12206080,"num_objects":1490,"num_object_clones":0,"num_object_copies":5960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2918,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'11586","reported_seq":11698,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:20.586288+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:44:20.586288+0000","last_peered":"2024-09-05T16:44:20.586288+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:44:20.586288+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"29'1500","ondisk_log_start":"29'1500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12612768,"num_objects":1540,"num_object_clones":0,"num_object_copies":6160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1540,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11584,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-794}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'11370","reported_seq":11487,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:24.559136+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:44:24.559136+0000","last_peered":"2024-09-05T16:44:24.559136+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:44:24.559136+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"29'1300","ondisk_log_start":"29'1300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11981984,"num_objects":1463,"num_object_clones":0,"num_object_copies":5852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1463,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11368,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-845}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'11347","reported_seq":11459,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:24.805836+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:44:24.805836+0000","last_peered":"2024-09-05T16:44:24.805836+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:44:24.805836+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"29'1300","ondisk_log_start":"29'1300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11695264,"num_objects":1428,"num_object_clones":0,"num_object_copies":5712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1428,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11345,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-792}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":195829064,"num_objects":23910,"num_object_clones":0,"num_object_copies":95640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28769,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23910,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":156240,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":389787648,"data_stored":389787648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160849,"ondisk_log_size":160849,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084412,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":754904,"kb_used_data":97096,"kb_used_omap":4,"kb_used_meta":657787,"kb_avail":93616936,"statfs":{"total":96636764160,"available":95863742464,"internally_reserved":0,"allocated":99426304,"data_stored":99215723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":673574611},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,0,0,1,2,1,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986289,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":576956,"kb_used_data":98220,"kb_used_omap":4,"kb_used_meta":478715,"kb_avail":93794884,"statfs":{"total":96636764160,"available":96045961216,"internally_reserved":0,"allocated":100577280,"data_stored":100366699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":490204882},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,8,10,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673096,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":742968,"kb_used_data":93988,"kb_used_omap":1,"kb_used_meta":648958,"kb_avail":93628872,"statfs":{"total":96636764160,"available":95875964928,"internally_reserved":0,"allocated":96243712,"data_stored":96033131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":664533445},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,0,7,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182527,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":739700,"kb_used_data":94428,"kb_used_omap":4,"kb_used_meta":645243,"kb_avail":93632140,"statfs":{"total":96636764160,"available":95879311360,"internally_reserved":0,"allocated":96694272,"data_stored":96483691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":660729549},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,2,10,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":95559680,"data_stored":95559680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":99356672,"data_stored":99356672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":95789056,"data_stored":95789056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":99082240,"data_stored":99082240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:44:26.921 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-05T16:44:26.921 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "39'11361", 'reported_seq': 12107, 'reported_epoch': 39, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:20.580463+0000', 'last_change': '2024-09-05T16:43:43.436935+0000', 'last_active': '2024-09-05T16:44:20.580463+0000', 'last_peered': '2024-09-05T16:44:20.580463+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.828289+0000', 'last_became_peered': '2024-09-05T16:34:56.828289+0000', 'last_unstale': '2024-09-05T16:44:20.580463+0000', 'last_undegraded': '2024-09-05T16:34:56.612160+0000', 'last_fullsized': '2024-09-05T16:34:56.557632+0000', 'mapping_epoch': 32, 'log_start': "29'1300", 'ondisk_log_start': "29'1300", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10061, 'log_dups_size': 0, 'ondisk_log_size': 10061, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12640256, 'num_objects': 1543, 'num_object_clones': 0, 'num_object_copies': 6172, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1918, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1543, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11361, 'num_write_kb': 51616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 344, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -375}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:44:26.921 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:44:26.921 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "39'11305", 'reported_seq': 11432, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:24.521135+0000', 'last_change': '2024-09-05T16:34:56.867900+0000', 'last_active': '2024-09-05T16:44:24.521135+0000', 'last_peered': '2024-09-05T16:44:24.521135+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:44:24.521135+0000', 'last_undegraded': '2024-09-05T16:34:56.590972+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 32, 'log_start': "29'1300", 'ondisk_log_start': "29'1300", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10005, 'log_dups_size': 0, 'ondisk_log_size': 10005, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11852392, 'num_objects': 1448, 'num_object_clones': 0, 'num_object_copies': 5792, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2233, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1448, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 11304, 'num_write_kb': 50999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -784}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:44:26.921 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:44:26.921 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "39'11094", 'reported_seq': 11197, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:20.442925+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:44:20.442925+0000', 'last_peered': '2024-09-05T16:44:20.442925+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:44:20.442925+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "29'1000", 'ondisk_log_start': "29'1000", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10094, 'log_dups_size': 0, 'ondisk_log_size': 10094, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12293280, 'num_objects': 1501, 'num_object_clones': 0, 'num_object_copies': 6004, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2147, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1501, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11092, 'num_write_kb': 50373, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -644}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:44:26.921 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:44:26.921 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "39'11295", 'reported_seq': 11413, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:20.338238+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:44:20.338238+0000', 'last_peered': '2024-09-05T16:44:20.338238+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:44:20.338238+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "29'1200", 'ondisk_log_start': "29'1200", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10095, 'log_dups_size': 0, 'ondisk_log_size': 10095, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12099584, 'num_objects': 1477, 'num_object_clones': 0, 'num_object_copies': 5908, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1477, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11295, 'num_write_kb': 51088, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -742}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:44:26.922 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:44:26.922 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "39'11146", 'reported_seq': 11258, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:24.724062+0000', 'last_change': '2024-09-05T16:34:56.903436+0000', 'last_active': '2024-09-05T16:44:24.724062+0000', 'last_peered': '2024-09-05T16:44:24.724062+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:44:24.724062+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "29'1100", 'ondisk_log_start': "29'1100", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10046, 'log_dups_size': 0, 'ondisk_log_size': 10046, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12416160, 'num_objects': 1516, 'num_object_clones': 0, 'num_object_copies': 6064, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2133, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1516, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11144, 'num_write_kb': 50636, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -615}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:44:26.922 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-05T16:44:26.922 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "39'11316", 'reported_seq': 11407, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:24.865194+0000', 'last_change': '2024-09-05T16:34:56.883319+0000', 'last_active': '2024-09-05T16:44:24.865194+0000', 'last_peered': '2024-09-05T16:44:24.865194+0000', 'last_clean': '2024-09-05T16:33:53.505305+0000', 'last_became_active': '2024-09-05T16:34:56.873389+0000', 'last_became_peered': '2024-09-05T16:34:56.873389+0000', 'last_unstale': '2024-09-05T16:44:24.865194+0000', 'last_undegraded': '2024-09-05T16:34:56.602567+0000', 'last_fullsized': '2024-09-05T16:34:56.559958+0000', 'mapping_epoch': 32, 'log_start': "29'1300", 'ondisk_log_start': "29'1300", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10016, 'log_dups_size': 0, 'ondisk_log_size': 10016, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11981984, 'num_objects': 1463, 'num_object_clones': 0, 'num_object_copies': 5852, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2234, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1463, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11314, 'num_write_kb': 51108, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -769}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:44:26.922 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:44:26.922 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "39'11300", 'reported_seq': 11405, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:20.246221+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:44:20.246221+0000', 'last_peered': '2024-09-05T16:44:20.246221+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:44:20.246221+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "29'1200", 'ondisk_log_start': "29'1200", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10100, 'log_dups_size': 0, 'ondisk_log_size': 10100, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12069184, 'num_objects': 1474, 'num_object_clones': 0, 'num_object_copies': 5896, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1474, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11297, 'num_write_kb': 51081, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -770}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:44:26.922 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:44:26.922 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "39'11338", 'reported_seq': 11465, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:24.635655+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:44:24.635655+0000', 'last_peered': '2024-09-05T16:44:24.635655+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:44:24.635655+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "29'1300", 'ondisk_log_start': "29'1300", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10038, 'log_dups_size': 0, 'ondisk_log_size': 10038, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12055712, 'num_objects': 1472, 'num_object_clones': 0, 'num_object_copies': 5888, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1472, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11336, 'num_write_kb': 51230, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -773}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:44:26.922 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:44:26.922 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "39'11550", 'reported_seq': 11681, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:20.580702+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:44:20.580702+0000', 'last_peered': '2024-09-05T16:44:20.580702+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:44:20.580702+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "29'1500", 'ondisk_log_start': "29'1500", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12881648, 'num_objects': 1573, 'num_object_clones': 0, 'num_object_copies': 6292, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2284, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1573, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11547, 'num_write_kb': 52476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -708}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:44:26.922 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:44:26.922 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "39'11196", 'reported_seq': 11299, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:20.429771+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:44:20.429771+0000', 'last_peered': '2024-09-05T16:44:20.429771+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:44:20.429771+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "29'1100", 'ondisk_log_start': "29'1100", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10096, 'log_dups_size': 0, 'ondisk_log_size': 10096, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12802640, 'num_objects': 1563, 'num_object_clones': 0, 'num_object_copies': 6252, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1563, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11195, 'num_write_kb': 51031, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -675}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:44:26.923 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:44:26.923 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "39'11586", 'reported_seq': 11698, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:20.586288+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:44:20.586288+0000', 'last_peered': '2024-09-05T16:44:20.586288+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:44:20.586288+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "29'1500", 'ondisk_log_start': "29'1500", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10086, 'log_dups_size': 0, 'ondisk_log_size': 10086, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12612768, 'num_objects': 1540, 'num_object_clones': 0, 'num_object_copies': 6160, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1540, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11584, 'num_write_kb': 52500, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -794}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:44:26.923 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:44:26.923 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "39'11370", 'reported_seq': 11487, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:24.559136+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:44:24.559136+0000', 'last_peered': '2024-09-05T16:44:24.559136+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:44:24.559136+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "29'1300", 'ondisk_log_start': "29'1300", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10070, 'log_dups_size': 0, 'ondisk_log_size': 10070, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11981984, 'num_objects': 1463, 'num_object_clones': 0, 'num_object_copies': 5852, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2310, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1463, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11368, 'num_write_kb': 51325, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -845}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:44:26.923 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:44:26.923 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "39'11347", 'reported_seq': 11459, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:24.805836+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:44:24.805836+0000', 'last_peered': '2024-09-05T16:44:24.805836+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:44:24.805836+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "29'1300", 'ondisk_log_start': "29'1300", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10047, 'log_dups_size': 0, 'ondisk_log_size': 10047, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11695264, 'num_objects': 1428, 'num_object_clones': 0, 'num_object_copies': 5712, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1428, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11345, 'num_write_kb': 51092, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -792}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:44:26.923 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:44:27.647 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:44:27.647 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 28732/93020 objects degraded (30.888%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":670,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":39,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":23255,"data_bytes":190463304,"bytes_used":2864480256,"bytes_avail":383682576384,"bytes_total":386547056640,"degraded_objects":28732,"degraded_total":93020,"degraded_ratio":0.30887981079337778,"recovering_objects_per_sec":10,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":125,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":226},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:44:27.671 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:44:27.671 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:44:28.274 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:44:28.274 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:44:28.308 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":365,"stamp":"2024-09-05T16:44:27.039740+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":190463304,"num_objects":23255,"num_object_clones":0,"num_object_copies":93020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28732,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23255,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":156895,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":634,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160804,"ondisk_log_size":160804,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2797344,"kb_used_data":374868,"kb_used_omap":14,"kb_used_meta":2422385,"kb_avail":374690016,"statfs":{"total":386547056640,"available":383682576384,"internally_reserved":0,"allocated":383864832,"data_stored":383022508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2480522807},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,1,8,4,15,19,14,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22290432,"num_objects":-2721,"num_object_clones":0,"num_object_copies":-10884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2721,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2721,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":125,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004602"},"pg_stats":[{"pgid":"1.f","version":"39'11429","reported_seq":12249,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:25.731722+0000","last_change":"2024-09-05T16:43:43.436935+0000","last_active":"2024-09-05T16:44:25.731722+0000","last_peered":"2024-09-05T16:44:25.731722+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:44:25.731722+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"29'1400","ondisk_log_start":"29'1400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12083200,"num_objects":1475,"num_object_clones":0,"num_object_copies":5900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1881,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1475,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11429,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":381,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-406}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'11168","reported_seq":15824,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:44:25.357488+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:44:25.357488+0000","last_peered":"2024-09-05T16:44:25.357488+0000","last_clean":"2024-09-05T16:44:25.357488+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:44:25.357488+0000","last_undegraded":"2024-09-05T16:44:25.357488+0000","last_fullsized":"2024-09-05T16:44:25.357488+0000","mapping_epoch":38,"log_start":"29'1100","ondisk_log_start":"29'1100","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11337728,"num_objects":1384,"num_object_clones":0,"num_object_copies":5536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1384,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":481,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'11305","reported_seq":11432,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:24.521135+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:44:24.521135+0000","last_peered":"2024-09-05T16:44:24.521135+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:44:24.521135+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"29'1300","ondisk_log_start":"29'1300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11852392,"num_objects":1448,"num_object_clones":0,"num_object_copies":5792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1448,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11304,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-784}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'11487","reported_seq":16221,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:44:25.452888+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:44:25.452888+0000","last_peered":"2024-09-05T16:44:25.452888+0000","last_clean":"2024-09-05T16:44:25.452888+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:44:25.452888+0000","last_undegraded":"2024-09-05T16:44:25.452888+0000","last_fullsized":"2024-09-05T16:44:25.452888+0000","mapping_epoch":34,"log_start":"29'1400","ondisk_log_start":"29'1400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11853824,"num_objects":1447,"num_object_clones":0,"num_object_copies":5788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1447,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5787,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'11166","reported_seq":11270,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:25.675508+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:44:25.675508+0000","last_peered":"2024-09-05T16:44:25.675508+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:44:25.675508+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"29'1100","ondisk_log_start":"29'1100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11703456,"num_objects":1429,"num_object_clones":0,"num_object_copies":5716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1429,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11164,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-716}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'11356","reported_seq":11475,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:25.613378+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:44:25.613378+0000","last_peered":"2024-09-05T16:44:25.613378+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:44:25.613378+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"29'1300","ondisk_log_start":"29'1300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11599872,"num_objects":1416,"num_object_clones":0,"num_object_copies":5664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1416,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11356,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-803}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'11146","reported_seq":11258,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:24.724062+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:44:24.724062+0000","last_peered":"2024-09-05T16:44:24.724062+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:44:24.724062+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"29'1100","ondisk_log_start":"29'1100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12416160,"num_objects":1516,"num_object_clones":0,"num_object_copies":6064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1516,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11144,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-615}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'11316","reported_seq":11407,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:24.865194+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:44:24.865194+0000","last_peered":"2024-09-05T16:44:24.865194+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:44:24.865194+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"29'1300","ondisk_log_start":"29'1300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11981984,"num_objects":1463,"num_object_clones":0,"num_object_copies":5852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1463,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11314,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-769}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"39'11351","reported_seq":11456,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:25.431172+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:44:25.431172+0000","last_peered":"2024-09-05T16:44:25.431172+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:44:25.431172+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"29'1300","ondisk_log_start":"29'1300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11651392,"num_objects":1423,"num_object_clones":0,"num_object_copies":5692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1423,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11348,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-821}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'11338","reported_seq":11465,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:24.635655+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:44:24.635655+0000","last_peered":"2024-09-05T16:44:24.635655+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:44:24.635655+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"29'1300","ondisk_log_start":"29'1300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12055712,"num_objects":1472,"num_object_clones":0,"num_object_copies":5888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1472,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11336,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-773}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'11623","reported_seq":11752,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:25.691885+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:44:25.691885+0000","last_peered":"2024-09-05T16:44:25.691885+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:44:25.691885+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"29'1600","ondisk_log_start":"29'1600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12283632,"num_objects":1500,"num_object_clones":0,"num_object_copies":6000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1500,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11620,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-781}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'11261","reported_seq":11364,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:25.704900+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:44:25.704900+0000","last_peered":"2024-09-05T16:44:25.704900+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:44:25.704900+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"29'1200","ondisk_log_start":"29'1200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12270160,"num_objects":1498,"num_object_clones":0,"num_object_copies":5992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1498,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11260,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-740}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'11390","reported_seq":16130,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:44:25.260994+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:44:25.260994+0000","last_peered":"2024-09-05T16:44:25.260994+0000","last_clean":"2024-09-05T16:44:25.260994+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:44:25.260994+0000","last_undegraded":"2024-09-05T16:44:25.260994+0000","last_fullsized":"2024-09-05T16:44:25.260994+0000","mapping_epoch":36,"log_start":"29'1300","ondisk_log_start":"29'1300","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11616256,"num_objects":1418,"num_object_clones":0,"num_object_copies":5672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1418,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2990,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'11651","reported_seq":11764,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:25.669666+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:44:25.669666+0000","last_peered":"2024-09-05T16:44:25.669666+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:44:25.669666+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"29'1600","ondisk_log_start":"29'1600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12080288,"num_objects":1475,"num_object_clones":0,"num_object_copies":5900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1475,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11649,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-859}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'11370","reported_seq":11487,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:24.559136+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:44:24.559136+0000","last_peered":"2024-09-05T16:44:24.559136+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:44:24.559136+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"29'1300","ondisk_log_start":"29'1300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11981984,"num_objects":1463,"num_object_clones":0,"num_object_copies":5852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1463,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11368,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-845}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'11347","reported_seq":11459,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:24.805836+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:44:24.805836+0000","last_peered":"2024-09-05T16:44:24.805836+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:44:24.805836+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"29'1300","ondisk_log_start":"29'1300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11695264,"num_objects":1428,"num_object_clones":0,"num_object_copies":5712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1428,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11345,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-792}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":190463304,"num_objects":23255,"num_object_clones":0,"num_object_copies":93020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28732,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23255,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":156895,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":634,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":381415424,"data_stored":381415424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160804,"ondisk_log_size":160804,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084413,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":738452,"kb_used_data":93124,"kb_used_omap":4,"kb_used_meta":645307,"kb_avail":93633388,"statfs":{"total":96636764160,"available":95880589312,"internally_reserved":0,"allocated":95358976,"data_stored":95148395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":660795091},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,2,1,2,4,7,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986290,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":576224,"kb_used_data":93328,"kb_used_omap":4,"kb_used_meta":482875,"kb_avail":93795616,"statfs":{"total":96636764160,"available":96046710784,"internally_reserved":0,"allocated":95567872,"data_stored":95357291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":494464722},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,0,4,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673096,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":742968,"kb_used_data":93988,"kb_used_omap":1,"kb_used_meta":648958,"kb_avail":93628872,"statfs":{"total":96636764160,"available":95875964928,"internally_reserved":0,"allocated":96243712,"data_stored":96033131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":664533445},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,0,7,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182527,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":739700,"kb_used_data":94428,"kb_used_omap":4,"kb_used_meta":645243,"kb_avail":93632140,"statfs":{"total":96636764160,"available":95879311360,"internally_reserved":0,"allocated":96694272,"data_stored":96483691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":660729549},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,2,10,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":95559680,"data_stored":95559680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":95203328,"data_stored":95203328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":95789056,"data_stored":95789056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":94863360,"data_stored":94863360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:44:31.311 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:44:31.909 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:44:31.909 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:44:31.932 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":367,"stamp":"2024-09-05T16:44:31.044114+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":182361416,"num_objects":22266,"num_object_clones":0,"num_object_copies":89064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28693,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22266,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":157884,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":673,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160693,"ondisk_log_size":160693,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2819724,"kb_used_data":358392,"kb_used_omap":14,"kb_used_meta":2461233,"kb_avail":374667636,"statfs":{"total":386547056640,"available":383659659264,"internally_reserved":0,"allocated":366993408,"data_stored":366151084,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2520303159},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,3,2,3,19,5,14,12,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24862720,"num_objects":-3035,"num_object_clones":0,"num_object_copies":-12140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-121,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3035,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3035,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":121,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007304"},"pg_stats":[{"pgid":"1.f","version":"39'11492","reported_seq":12389,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:30.770940+0000","last_change":"2024-09-05T16:43:43.436935+0000","last_active":"2024-09-05T16:44:30.770940+0000","last_peered":"2024-09-05T16:44:30.770940+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:44:30.770940+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"29'1400","ondisk_log_start":"29'1400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11567104,"num_objects":1412,"num_object_clones":0,"num_object_copies":5648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1842,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1412,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11492,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":420,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-430}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'11234","reported_seq":15890,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:44:30.345431+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:44:30.345431+0000","last_peered":"2024-09-05T16:44:30.345431+0000","last_clean":"2024-09-05T16:44:30.345431+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:44:30.345431+0000","last_undegraded":"2024-09-05T16:44:30.345431+0000","last_fullsized":"2024-09-05T16:44:30.345431+0000","mapping_epoch":38,"log_start":"29'1200","ondisk_log_start":"29'1200","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10797056,"num_objects":1318,"num_object_clones":0,"num_object_copies":5272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1318,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":547,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'11359","reported_seq":11486,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:29.628713+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:44:29.628713+0000","last_peered":"2024-09-05T16:44:29.628713+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:44:29.628713+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"29'1300","ondisk_log_start":"29'1300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11410024,"num_objects":1394,"num_object_clones":0,"num_object_copies":5576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1394,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11358,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-838}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'11533","reported_seq":16267,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:44:30.241026+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:44:30.241026+0000","last_peered":"2024-09-05T16:44:30.241026+0000","last_clean":"2024-09-05T16:44:30.241026+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:44:30.241026+0000","last_undegraded":"2024-09-05T16:44:30.241026+0000","last_fullsized":"2024-09-05T16:44:30.241026+0000","mapping_epoch":34,"log_start":"29'1500","ondisk_log_start":"29'1500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11476992,"num_objects":1401,"num_object_clones":0,"num_object_copies":5604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5833,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'11239","reported_seq":11343,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:30.707854+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:44:30.707854+0000","last_peered":"2024-09-05T16:44:30.707854+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:44:30.707854+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"29'1200","ondisk_log_start":"29'1200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11105440,"num_objects":1356,"num_object_clones":0,"num_object_copies":5424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1356,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11237,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-789}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'11417","reported_seq":11536,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:30.638533+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:44:30.638533+0000","last_peered":"2024-09-05T16:44:30.638533+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:44:30.638533+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"29'1400","ondisk_log_start":"29'1400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11100160,"num_objects":1355,"num_object_clones":0,"num_object_copies":5420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1355,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11417,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-864}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'11205","reported_seq":11317,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:29.721977+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:44:29.721977+0000","last_peered":"2024-09-05T16:44:29.721977+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:44:29.721977+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"29'1200","ondisk_log_start":"29'1200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11932832,"num_objects":1457,"num_object_clones":0,"num_object_copies":5828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1457,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11203,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-674}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'11375","reported_seq":11468,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:29.657989+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:44:29.657989+0000","last_peered":"2024-09-05T16:44:29.657989+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:44:29.657989+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"29'1300","ondisk_log_start":"29'1300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11498656,"num_objects":1404,"num_object_clones":0,"num_object_copies":5616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1404,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11373,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-828}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"39'11430","reported_seq":11535,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:30.679336+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:44:30.679336+0000","last_peered":"2024-09-05T16:44:30.679336+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:44:30.679336+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"29'1400","ondisk_log_start":"29'1400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11004224,"num_objects":1344,"num_object_clones":0,"num_object_copies":5376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1344,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11427,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-900}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'11398","reported_seq":11525,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:29.659623+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:44:29.659623+0000","last_peered":"2024-09-05T16:44:29.659623+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:44:29.659623+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"29'1300","ondisk_log_start":"29'1300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11564192,"num_objects":1412,"num_object_clones":0,"num_object_copies":5648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1412,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11396,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-833}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'11681","reported_seq":11811,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:30.611233+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:44:30.611233+0000","last_peered":"2024-09-05T16:44:30.611233+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:44:30.611233+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"29'1600","ondisk_log_start":"29'1600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11808496,"num_objects":1442,"num_object_clones":0,"num_object_copies":5768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1442,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-839}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'11327","reported_seq":11430,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:30.773385+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:44:30.773385+0000","last_peered":"2024-09-05T16:44:30.773385+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:44:30.773385+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"29'1300","ondisk_log_start":"29'1300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11729488,"num_objects":1432,"num_object_clones":0,"num_object_copies":5728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1432,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11326,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-806}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'11456","reported_seq":16195,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:44:30.464210+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:44:30.464210+0000","last_peered":"2024-09-05T16:44:30.464210+0000","last_clean":"2024-09-05T16:44:30.464210+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:44:30.464210+0000","last_undegraded":"2024-09-05T16:44:30.464210+0000","last_fullsized":"2024-09-05T16:44:30.464210+0000","mapping_epoch":36,"log_start":"29'1400","ondisk_log_start":"29'1400","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11075584,"num_objects":1352,"num_object_clones":0,"num_object_copies":5408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1352,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3056,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'11702","reported_seq":11813,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:30.707448+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:44:30.707448+0000","last_peered":"2024-09-05T16:44:30.707448+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:44:30.707448+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"29'1700","ondisk_log_start":"29'1700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11662496,"num_objects":1424,"num_object_clones":0,"num_object_copies":5696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1424,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11700,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-910}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'11435","reported_seq":11551,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:29.715022+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:44:29.715022+0000","last_peered":"2024-09-05T16:44:29.715022+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:44:29.715022+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"29'1400","ondisk_log_start":"29'1400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11449504,"num_objects":1398,"num_object_clones":0,"num_object_copies":5592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1398,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11433,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-910}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'11410","reported_seq":11523,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:29.866955+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:44:29.866955+0000","last_peered":"2024-09-05T16:44:29.866955+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:44:29.866955+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"29'1400","ondisk_log_start":"29'1400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11179168,"num_objects":1365,"num_object_clones":0,"num_object_copies":5460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11408,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-855}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":182361416,"num_objects":22266,"num_object_clones":0,"num_object_copies":89064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28693,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22266,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":157884,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":673,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":365060096,"data_stored":365060096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160693,"ondisk_log_size":160693,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084414,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":738712,"kb_used_data":89220,"kb_used_omap":4,"kb_used_meta":649467,"kb_avail":93633128,"statfs":{"total":96636764160,"available":95880323072,"internally_reserved":0,"allocated":91361280,"data_stored":91150699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":665054931},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1,3,0,3,5,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986291,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":594492,"kb_used_data":89388,"kb_used_omap":4,"kb_used_meta":505083,"kb_avail":93777348,"statfs":{"total":96636764160,"available":96028004352,"internally_reserved":0,"allocated":91533312,"data_stored":91322731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":517205714},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,6,1,4,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673097,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":743212,"kb_used_data":90068,"kb_used_omap":1,"kb_used_meta":653118,"kb_avail":93628628,"statfs":{"total":96636764160,"available":95875715072,"internally_reserved":0,"allocated":92229632,"data_stored":92019051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":668793285},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,0,2,4,7,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182528,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":743308,"kb_used_data":89716,"kb_used_omap":4,"kb_used_meta":653563,"kb_avail":93628532,"statfs":{"total":96636764160,"available":95875616768,"internally_reserved":0,"allocated":91869184,"data_stored":91658603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":669249229},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,8,0,0,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":91475968,"data_stored":91475968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":91131904,"data_stored":91131904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":91705344,"data_stored":91705344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":90746880,"data_stored":90746880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:44:31.933 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:44:32.515 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:44:32.517 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:44:32.549 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":367,"stamp":"2024-09-05T16:44:31.044114+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":182361416,"num_objects":22266,"num_object_clones":0,"num_object_copies":89064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28693,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22266,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":157884,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":673,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160693,"ondisk_log_size":160693,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2819724,"kb_used_data":358392,"kb_used_omap":14,"kb_used_meta":2461233,"kb_avail":374667636,"statfs":{"total":386547056640,"available":383659659264,"internally_reserved":0,"allocated":366993408,"data_stored":366151084,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2520303159},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,3,2,3,19,5,14,12,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24862720,"num_objects":-3035,"num_object_clones":0,"num_object_copies":-12140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-121,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3035,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3035,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":121,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007304"},"pg_stats":[{"pgid":"1.f","version":"39'11492","reported_seq":12389,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:30.770940+0000","last_change":"2024-09-05T16:43:43.436935+0000","last_active":"2024-09-05T16:44:30.770940+0000","last_peered":"2024-09-05T16:44:30.770940+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:44:30.770940+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"29'1400","ondisk_log_start":"29'1400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11567104,"num_objects":1412,"num_object_clones":0,"num_object_copies":5648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1842,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1412,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11492,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":420,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-430}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'11234","reported_seq":15890,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:44:30.345431+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:44:30.345431+0000","last_peered":"2024-09-05T16:44:30.345431+0000","last_clean":"2024-09-05T16:44:30.345431+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:44:30.345431+0000","last_undegraded":"2024-09-05T16:44:30.345431+0000","last_fullsized":"2024-09-05T16:44:30.345431+0000","mapping_epoch":38,"log_start":"29'1200","ondisk_log_start":"29'1200","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10797056,"num_objects":1318,"num_object_clones":0,"num_object_copies":5272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1318,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":547,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'11359","reported_seq":11486,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:29.628713+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:44:29.628713+0000","last_peered":"2024-09-05T16:44:29.628713+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:44:29.628713+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"29'1300","ondisk_log_start":"29'1300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11410024,"num_objects":1394,"num_object_clones":0,"num_object_copies":5576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1394,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11358,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-838}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'11533","reported_seq":16267,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:44:30.241026+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:44:30.241026+0000","last_peered":"2024-09-05T16:44:30.241026+0000","last_clean":"2024-09-05T16:44:30.241026+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:44:30.241026+0000","last_undegraded":"2024-09-05T16:44:30.241026+0000","last_fullsized":"2024-09-05T16:44:30.241026+0000","mapping_epoch":34,"log_start":"29'1500","ondisk_log_start":"29'1500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11476992,"num_objects":1401,"num_object_clones":0,"num_object_copies":5604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5833,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'11239","reported_seq":11343,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:30.707854+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:44:30.707854+0000","last_peered":"2024-09-05T16:44:30.707854+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:44:30.707854+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"29'1200","ondisk_log_start":"29'1200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11105440,"num_objects":1356,"num_object_clones":0,"num_object_copies":5424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1356,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11237,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-789}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'11417","reported_seq":11536,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:30.638533+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:44:30.638533+0000","last_peered":"2024-09-05T16:44:30.638533+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:44:30.638533+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"29'1400","ondisk_log_start":"29'1400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11100160,"num_objects":1355,"num_object_clones":0,"num_object_copies":5420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1355,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11417,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-864}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'11205","reported_seq":11317,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:29.721977+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:44:29.721977+0000","last_peered":"2024-09-05T16:44:29.721977+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:44:29.721977+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"29'1200","ondisk_log_start":"29'1200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11932832,"num_objects":1457,"num_object_clones":0,"num_object_copies":5828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1457,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11203,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-674}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'11375","reported_seq":11468,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:29.657989+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:44:29.657989+0000","last_peered":"2024-09-05T16:44:29.657989+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:44:29.657989+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"29'1300","ondisk_log_start":"29'1300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11498656,"num_objects":1404,"num_object_clones":0,"num_object_copies":5616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1404,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11373,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-828}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"39'11430","reported_seq":11535,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:30.679336+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:44:30.679336+0000","last_peered":"2024-09-05T16:44:30.679336+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:44:30.679336+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"29'1400","ondisk_log_start":"29'1400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11004224,"num_objects":1344,"num_object_clones":0,"num_object_copies":5376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1344,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11427,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-900}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'11398","reported_seq":11525,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:29.659623+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:44:29.659623+0000","last_peered":"2024-09-05T16:44:29.659623+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:44:29.659623+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"29'1300","ondisk_log_start":"29'1300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11564192,"num_objects":1412,"num_object_clones":0,"num_object_copies":5648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1412,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11396,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-833}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'11681","reported_seq":11811,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:30.611233+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:44:30.611233+0000","last_peered":"2024-09-05T16:44:30.611233+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:44:30.611233+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"29'1600","ondisk_log_start":"29'1600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11808496,"num_objects":1442,"num_object_clones":0,"num_object_copies":5768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1442,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-839}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'11327","reported_seq":11430,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:30.773385+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:44:30.773385+0000","last_peered":"2024-09-05T16:44:30.773385+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:44:30.773385+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"29'1300","ondisk_log_start":"29'1300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11729488,"num_objects":1432,"num_object_clones":0,"num_object_copies":5728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1432,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11326,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-806}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'11456","reported_seq":16195,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:44:30.464210+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:44:30.464210+0000","last_peered":"2024-09-05T16:44:30.464210+0000","last_clean":"2024-09-05T16:44:30.464210+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:44:30.464210+0000","last_undegraded":"2024-09-05T16:44:30.464210+0000","last_fullsized":"2024-09-05T16:44:30.464210+0000","mapping_epoch":36,"log_start":"29'1400","ondisk_log_start":"29'1400","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11075584,"num_objects":1352,"num_object_clones":0,"num_object_copies":5408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1352,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3056,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'11702","reported_seq":11813,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:30.707448+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:44:30.707448+0000","last_peered":"2024-09-05T16:44:30.707448+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:44:30.707448+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"29'1700","ondisk_log_start":"29'1700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11662496,"num_objects":1424,"num_object_clones":0,"num_object_copies":5696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1424,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11700,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-910}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'11435","reported_seq":11551,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:29.715022+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:44:29.715022+0000","last_peered":"2024-09-05T16:44:29.715022+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:44:29.715022+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"29'1400","ondisk_log_start":"29'1400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11449504,"num_objects":1398,"num_object_clones":0,"num_object_copies":5592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1398,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11433,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-910}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'11410","reported_seq":11523,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:29.866955+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:44:29.866955+0000","last_peered":"2024-09-05T16:44:29.866955+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:44:29.866955+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"29'1400","ondisk_log_start":"29'1400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11179168,"num_objects":1365,"num_object_clones":0,"num_object_copies":5460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11408,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-855}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":182361416,"num_objects":22266,"num_object_clones":0,"num_object_copies":89064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28693,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22266,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":157884,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":673,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":365060096,"data_stored":365060096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160693,"ondisk_log_size":160693,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084414,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":738712,"kb_used_data":89220,"kb_used_omap":4,"kb_used_meta":649467,"kb_avail":93633128,"statfs":{"total":96636764160,"available":95880323072,"internally_reserved":0,"allocated":91361280,"data_stored":91150699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":665054931},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1,3,0,3,5,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986291,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":594492,"kb_used_data":89388,"kb_used_omap":4,"kb_used_meta":505083,"kb_avail":93777348,"statfs":{"total":96636764160,"available":96028004352,"internally_reserved":0,"allocated":91533312,"data_stored":91322731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":517205714},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,6,1,4,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673097,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":743212,"kb_used_data":90068,"kb_used_omap":1,"kb_used_meta":653118,"kb_avail":93628628,"statfs":{"total":96636764160,"available":95875715072,"internally_reserved":0,"allocated":92229632,"data_stored":92019051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":668793285},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,0,2,4,7,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182528,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":743308,"kb_used_data":89716,"kb_used_omap":4,"kb_used_meta":653563,"kb_avail":93628532,"statfs":{"total":96636764160,"available":95875616768,"internally_reserved":0,"allocated":91869184,"data_stored":91658603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":669249229},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,8,0,0,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":91475968,"data_stored":91475968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":91131904,"data_stored":91131904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":91705344,"data_stored":91705344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":90746880,"data_stored":90746880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:44:32.550 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-05T16:44:32.550 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "39'11492", 'reported_seq': 12389, 'reported_epoch': 39, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:30.770940+0000', 'last_change': '2024-09-05T16:43:43.436935+0000', 'last_active': '2024-09-05T16:44:30.770940+0000', 'last_peered': '2024-09-05T16:44:30.770940+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.828289+0000', 'last_became_peered': '2024-09-05T16:34:56.828289+0000', 'last_unstale': '2024-09-05T16:44:30.770940+0000', 'last_undegraded': '2024-09-05T16:34:56.612160+0000', 'last_fullsized': '2024-09-05T16:34:56.557632+0000', 'mapping_epoch': 32, 'log_start': "29'1400", 'ondisk_log_start': "29'1400", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10092, 'log_dups_size': 0, 'ondisk_log_size': 10092, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11567104, 'num_objects': 1412, 'num_object_clones': 0, 'num_object_copies': 5648, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1842, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1412, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11492, 'num_write_kb': 51616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 420, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -430}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:44:32.550 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:44:32.550 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "39'11359", 'reported_seq': 11486, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:29.628713+0000', 'last_change': '2024-09-05T16:34:56.867900+0000', 'last_active': '2024-09-05T16:44:29.628713+0000', 'last_peered': '2024-09-05T16:44:29.628713+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:44:29.628713+0000', 'last_undegraded': '2024-09-05T16:34:56.590972+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 32, 'log_start': "29'1300", 'ondisk_log_start': "29'1300", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10059, 'log_dups_size': 0, 'ondisk_log_size': 10059, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11410024, 'num_objects': 1394, 'num_object_clones': 0, 'num_object_copies': 5576, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2233, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1394, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 11358, 'num_write_kb': 50999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -838}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:44:32.551 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:44:32.551 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "39'11239", 'reported_seq': 11343, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:30.707854+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:44:30.707854+0000', 'last_peered': '2024-09-05T16:44:30.707854+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:44:30.707854+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "29'1200", 'ondisk_log_start': "29'1200", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10039, 'log_dups_size': 0, 'ondisk_log_size': 10039, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11105440, 'num_objects': 1356, 'num_object_clones': 0, 'num_object_copies': 5424, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2147, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1356, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11237, 'num_write_kb': 50373, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -789}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:44:32.551 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:44:32.551 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "39'11417", 'reported_seq': 11536, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:30.638533+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:44:30.638533+0000', 'last_peered': '2024-09-05T16:44:30.638533+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:44:30.638533+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "29'1400", 'ondisk_log_start': "29'1400", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10017, 'log_dups_size': 0, 'ondisk_log_size': 10017, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11100160, 'num_objects': 1355, 'num_object_clones': 0, 'num_object_copies': 5420, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1355, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11417, 'num_write_kb': 51088, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -864}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:44:32.551 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:44:32.551 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "39'11205", 'reported_seq': 11317, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:29.721977+0000', 'last_change': '2024-09-05T16:34:56.903436+0000', 'last_active': '2024-09-05T16:44:29.721977+0000', 'last_peered': '2024-09-05T16:44:29.721977+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:44:29.721977+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "29'1200", 'ondisk_log_start': "29'1200", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10005, 'log_dups_size': 0, 'ondisk_log_size': 10005, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11932832, 'num_objects': 1457, 'num_object_clones': 0, 'num_object_copies': 5828, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2133, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1457, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11203, 'num_write_kb': 50636, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -674}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:44:32.551 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-05T16:44:32.551 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "39'11375", 'reported_seq': 11468, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:29.657989+0000', 'last_change': '2024-09-05T16:34:56.883319+0000', 'last_active': '2024-09-05T16:44:29.657989+0000', 'last_peered': '2024-09-05T16:44:29.657989+0000', 'last_clean': '2024-09-05T16:33:53.505305+0000', 'last_became_active': '2024-09-05T16:34:56.873389+0000', 'last_became_peered': '2024-09-05T16:34:56.873389+0000', 'last_unstale': '2024-09-05T16:44:29.657989+0000', 'last_undegraded': '2024-09-05T16:34:56.602567+0000', 'last_fullsized': '2024-09-05T16:34:56.559958+0000', 'mapping_epoch': 32, 'log_start': "29'1300", 'ondisk_log_start': "29'1300", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10075, 'log_dups_size': 0, 'ondisk_log_size': 10075, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11498656, 'num_objects': 1404, 'num_object_clones': 0, 'num_object_copies': 5616, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2234, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1404, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11373, 'num_write_kb': 51108, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -828}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:44:32.551 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:44:32.551 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "39'11430", 'reported_seq': 11535, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:30.679336+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:44:30.679336+0000', 'last_peered': '2024-09-05T16:44:30.679336+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:44:30.679336+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "29'1400", 'ondisk_log_start': "29'1400", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11004224, 'num_objects': 1344, 'num_object_clones': 0, 'num_object_copies': 5376, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1344, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11427, 'num_write_kb': 51081, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -900}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:44:32.552 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:44:32.552 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "39'11398", 'reported_seq': 11525, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:29.659623+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:44:29.659623+0000', 'last_peered': '2024-09-05T16:44:29.659623+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:44:29.659623+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "29'1300", 'ondisk_log_start': "29'1300", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11564192, 'num_objects': 1412, 'num_object_clones': 0, 'num_object_copies': 5648, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1412, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11396, 'num_write_kb': 51230, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -833}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:44:32.552 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:44:32.552 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "39'11681", 'reported_seq': 11811, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:30.611233+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:44:30.611233+0000', 'last_peered': '2024-09-05T16:44:30.611233+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:44:30.611233+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "29'1600", 'ondisk_log_start': "29'1600", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10081, 'log_dups_size': 0, 'ondisk_log_size': 10081, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11808496, 'num_objects': 1442, 'num_object_clones': 0, 'num_object_copies': 5768, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2284, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1442, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11678, 'num_write_kb': 52476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -839}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:44:32.552 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:44:32.552 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "39'11327", 'reported_seq': 11430, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:30.773385+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:44:30.773385+0000', 'last_peered': '2024-09-05T16:44:30.773385+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:44:30.773385+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "29'1300", 'ondisk_log_start': "29'1300", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10027, 'log_dups_size': 0, 'ondisk_log_size': 10027, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11729488, 'num_objects': 1432, 'num_object_clones': 0, 'num_object_copies': 5728, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1432, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11326, 'num_write_kb': 51031, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -806}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:44:32.552 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:44:32.552 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "39'11702", 'reported_seq': 11813, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:30.707448+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:44:30.707448+0000', 'last_peered': '2024-09-05T16:44:30.707448+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:44:30.707448+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "29'1700", 'ondisk_log_start': "29'1700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10002, 'log_dups_size': 0, 'ondisk_log_size': 10002, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11662496, 'num_objects': 1424, 'num_object_clones': 0, 'num_object_copies': 5696, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1424, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11700, 'num_write_kb': 52500, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -910}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:44:32.552 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:44:32.552 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "39'11435", 'reported_seq': 11551, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:29.715022+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:44:29.715022+0000', 'last_peered': '2024-09-05T16:44:29.715022+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:44:29.715022+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "29'1400", 'ondisk_log_start': "29'1400", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10035, 'log_dups_size': 0, 'ondisk_log_size': 10035, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11449504, 'num_objects': 1398, 'num_object_clones': 0, 'num_object_copies': 5592, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2310, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1398, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11433, 'num_write_kb': 51325, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -910}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:44:32.552 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:44:32.552 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "39'11410", 'reported_seq': 11523, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:29.866955+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:44:29.866955+0000', 'last_peered': '2024-09-05T16:44:29.866955+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:44:29.866955+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "29'1400", 'ondisk_log_start': "29'1400", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10010, 'log_dups_size': 0, 'ondisk_log_size': 10010, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11179168, 'num_objects': 1365, 'num_object_clones': 0, 'num_object_copies': 5460, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1365, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11408, 'num_write_kb': 51092, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -855}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:44:32.553 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:44:33.298 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:44:33.299 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 28693/89064 objects degraded (32.216%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":676,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":39,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":22266,"data_bytes":182361416,"bytes_used":2887397376,"bytes_avail":383659659264,"bytes_total":386547056640,"degraded_objects":28693,"degraded_total":89064,"degraded_ratio":0.3221615916644211,"recovering_objects_per_sec":6,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":76,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":167},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:44:33.323 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:44:33.323 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:44:33.915 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:44:33.915 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:44:33.960 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":368,"stamp":"2024-09-05T16:44:33.044858+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":182361416,"num_objects":22266,"num_object_clones":0,"num_object_copies":89064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28693,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22266,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":157884,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":673,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160693,"ondisk_log_size":160693,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2819724,"kb_used_data":358392,"kb_used_omap":14,"kb_used_meta":2461233,"kb_avail":374667636,"statfs":{"total":386547056640,"available":383659659264,"internally_reserved":0,"allocated":366993408,"data_stored":366151084,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2520303159},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,3,2,3,19,5,14,12,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16482304,"num_objects":-2012,"num_object_clones":0,"num_object_copies":-8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-76,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2012,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2012,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":76,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007419"},"pg_stats":[{"pgid":"1.f","version":"39'11492","reported_seq":12389,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:30.770940+0000","last_change":"2024-09-05T16:43:43.436935+0000","last_active":"2024-09-05T16:44:30.770940+0000","last_peered":"2024-09-05T16:44:30.770940+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:44:30.770940+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"29'1400","ondisk_log_start":"29'1400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11567104,"num_objects":1412,"num_object_clones":0,"num_object_copies":5648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1842,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1412,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11492,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":420,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-430}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'11234","reported_seq":15890,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:44:30.345431+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:44:30.345431+0000","last_peered":"2024-09-05T16:44:30.345431+0000","last_clean":"2024-09-05T16:44:30.345431+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:44:30.345431+0000","last_undegraded":"2024-09-05T16:44:30.345431+0000","last_fullsized":"2024-09-05T16:44:30.345431+0000","mapping_epoch":38,"log_start":"29'1200","ondisk_log_start":"29'1200","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10797056,"num_objects":1318,"num_object_clones":0,"num_object_copies":5272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1318,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":547,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'11359","reported_seq":11486,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:29.628713+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:44:29.628713+0000","last_peered":"2024-09-05T16:44:29.628713+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:44:29.628713+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"29'1300","ondisk_log_start":"29'1300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11410024,"num_objects":1394,"num_object_clones":0,"num_object_copies":5576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1394,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11358,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-838}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'11533","reported_seq":16267,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:44:30.241026+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:44:30.241026+0000","last_peered":"2024-09-05T16:44:30.241026+0000","last_clean":"2024-09-05T16:44:30.241026+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:44:30.241026+0000","last_undegraded":"2024-09-05T16:44:30.241026+0000","last_fullsized":"2024-09-05T16:44:30.241026+0000","mapping_epoch":34,"log_start":"29'1500","ondisk_log_start":"29'1500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11476992,"num_objects":1401,"num_object_clones":0,"num_object_copies":5604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5833,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'11239","reported_seq":11343,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:30.707854+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:44:30.707854+0000","last_peered":"2024-09-05T16:44:30.707854+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:44:30.707854+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"29'1200","ondisk_log_start":"29'1200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11105440,"num_objects":1356,"num_object_clones":0,"num_object_copies":5424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1356,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11237,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-789}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'11417","reported_seq":11536,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:30.638533+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:44:30.638533+0000","last_peered":"2024-09-05T16:44:30.638533+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:44:30.638533+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"29'1400","ondisk_log_start":"29'1400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11100160,"num_objects":1355,"num_object_clones":0,"num_object_copies":5420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1355,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11417,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-864}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'11205","reported_seq":11317,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:29.721977+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:44:29.721977+0000","last_peered":"2024-09-05T16:44:29.721977+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:44:29.721977+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"29'1200","ondisk_log_start":"29'1200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11932832,"num_objects":1457,"num_object_clones":0,"num_object_copies":5828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1457,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11203,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-674}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'11375","reported_seq":11468,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:29.657989+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:44:29.657989+0000","last_peered":"2024-09-05T16:44:29.657989+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:44:29.657989+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"29'1300","ondisk_log_start":"29'1300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11498656,"num_objects":1404,"num_object_clones":0,"num_object_copies":5616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1404,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11373,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-828}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"39'11430","reported_seq":11535,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:30.679336+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:44:30.679336+0000","last_peered":"2024-09-05T16:44:30.679336+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:44:30.679336+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"29'1400","ondisk_log_start":"29'1400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11004224,"num_objects":1344,"num_object_clones":0,"num_object_copies":5376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1344,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11427,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-900}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'11398","reported_seq":11525,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:29.659623+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:44:29.659623+0000","last_peered":"2024-09-05T16:44:29.659623+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:44:29.659623+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"29'1300","ondisk_log_start":"29'1300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11564192,"num_objects":1412,"num_object_clones":0,"num_object_copies":5648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1412,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11396,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-833}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'11681","reported_seq":11811,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:30.611233+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:44:30.611233+0000","last_peered":"2024-09-05T16:44:30.611233+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:44:30.611233+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"29'1600","ondisk_log_start":"29'1600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11808496,"num_objects":1442,"num_object_clones":0,"num_object_copies":5768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1442,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-839}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'11327","reported_seq":11430,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:30.773385+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:44:30.773385+0000","last_peered":"2024-09-05T16:44:30.773385+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:44:30.773385+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"29'1300","ondisk_log_start":"29'1300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11729488,"num_objects":1432,"num_object_clones":0,"num_object_copies":5728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1432,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11326,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-806}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'11456","reported_seq":16195,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:44:30.464210+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:44:30.464210+0000","last_peered":"2024-09-05T16:44:30.464210+0000","last_clean":"2024-09-05T16:44:30.464210+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:44:30.464210+0000","last_undegraded":"2024-09-05T16:44:30.464210+0000","last_fullsized":"2024-09-05T16:44:30.464210+0000","mapping_epoch":36,"log_start":"29'1400","ondisk_log_start":"29'1400","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11075584,"num_objects":1352,"num_object_clones":0,"num_object_copies":5408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1352,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3056,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'11702","reported_seq":11813,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:30.707448+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:44:30.707448+0000","last_peered":"2024-09-05T16:44:30.707448+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:44:30.707448+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"29'1700","ondisk_log_start":"29'1700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11662496,"num_objects":1424,"num_object_clones":0,"num_object_copies":5696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1424,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11700,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-910}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'11435","reported_seq":11551,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:29.715022+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:44:29.715022+0000","last_peered":"2024-09-05T16:44:29.715022+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:44:29.715022+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"29'1400","ondisk_log_start":"29'1400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11449504,"num_objects":1398,"num_object_clones":0,"num_object_copies":5592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1398,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11433,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-910}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'11410","reported_seq":11523,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:29.866955+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:44:29.866955+0000","last_peered":"2024-09-05T16:44:29.866955+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:44:29.866955+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"29'1400","ondisk_log_start":"29'1400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11179168,"num_objects":1365,"num_object_clones":0,"num_object_copies":5460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11408,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-855}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":182361416,"num_objects":22266,"num_object_clones":0,"num_object_copies":89064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28693,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22266,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":157884,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":673,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":365060096,"data_stored":365060096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160693,"ondisk_log_size":160693,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084414,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":738712,"kb_used_data":89220,"kb_used_omap":4,"kb_used_meta":649467,"kb_avail":93633128,"statfs":{"total":96636764160,"available":95880323072,"internally_reserved":0,"allocated":91361280,"data_stored":91150699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":665054931},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1,3,0,3,5,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986291,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":594492,"kb_used_data":89388,"kb_used_omap":4,"kb_used_meta":505083,"kb_avail":93777348,"statfs":{"total":96636764160,"available":96028004352,"internally_reserved":0,"allocated":91533312,"data_stored":91322731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":517205714},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,6,1,4,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673097,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":743212,"kb_used_data":90068,"kb_used_omap":1,"kb_used_meta":653118,"kb_avail":93628628,"statfs":{"total":96636764160,"available":95875715072,"internally_reserved":0,"allocated":92229632,"data_stored":92019051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":668793285},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,0,2,4,7,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182528,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":743308,"kb_used_data":89716,"kb_used_omap":4,"kb_used_meta":653563,"kb_avail":93628532,"statfs":{"total":96636764160,"available":95875616768,"internally_reserved":0,"allocated":91869184,"data_stored":91658603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":669249229},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,8,0,0,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":91475968,"data_stored":91475968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":91131904,"data_stored":91131904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":91705344,"data_stored":91705344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":90746880,"data_stored":90746880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:44:36.963 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:44:37.568 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:44:37.568 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:44:37.588 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":370,"stamp":"2024-09-05T16:44:37.046394+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":174005576,"num_objects":21246,"num_object_clones":0,"num_object_copies":84984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28650,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21246,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":158904,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":716,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160813,"ondisk_log_size":160813,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2812100,"kb_used_data":342456,"kb_used_omap":14,"kb_used_meta":2469553,"kb_avail":374675260,"statfs":{"total":386547056640,"available":383667466240,"internally_reserved":0,"allocated":350674944,"data_stored":349832620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2528822839},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,0,3,4,18,10,11,7,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21823488,"num_objects":-2664,"num_object_clones":0,"num_object_copies":-10656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2664,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2664,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007397"},"pg_stats":[{"pgid":"1.f","version":"39'11547","reported_seq":12528,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:35.982021+0000","last_change":"2024-09-05T16:43:43.436935+0000","last_active":"2024-09-05T16:44:35.982021+0000","last_peered":"2024-09-05T16:44:35.982021+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:44:35.982021+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"29'1500","ondisk_log_start":"29'1500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11116544,"num_objects":1357,"num_object_clones":0,"num_object_copies":5428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1799,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1357,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11547,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":463,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-442}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'11295","reported_seq":15951,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:44:35.280975+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:44:35.280975+0000","last_peered":"2024-09-05T16:44:35.280975+0000","last_clean":"2024-09-05T16:44:35.280975+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:44:35.280975+0000","last_undegraded":"2024-09-05T16:44:35.280975+0000","last_fullsized":"2024-09-05T16:44:35.280975+0000","mapping_epoch":38,"log_start":"29'1200","ondisk_log_start":"29'1200","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10297344,"num_objects":1257,"num_object_clones":0,"num_object_copies":5028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1257,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":608,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'11409","reported_seq":11536,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:34.641847+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:44:34.641847+0000","last_peered":"2024-09-05T16:44:34.641847+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:44:34.641847+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"29'1400","ondisk_log_start":"29'1400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11000424,"num_objects":1344,"num_object_clones":0,"num_object_copies":5376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1344,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11408,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-888}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'11582","reported_seq":16316,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:44:35.417082+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:44:35.417082+0000","last_peered":"2024-09-05T16:44:35.417082+0000","last_clean":"2024-09-05T16:44:35.417082+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:44:35.417082+0000","last_undegraded":"2024-09-05T16:44:35.417082+0000","last_fullsized":"2024-09-05T16:44:35.417082+0000","mapping_epoch":34,"log_start":"29'1500","ondisk_log_start":"29'1500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11075584,"num_objects":1352,"num_object_clones":0,"num_object_copies":5408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1352,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5882,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'11308","reported_seq":11412,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:35.905134+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:44:35.905134+0000","last_peered":"2024-09-05T16:44:35.905134+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:44:35.905134+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"29'1300","ondisk_log_start":"29'1300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10540192,"num_objects":1287,"num_object_clones":0,"num_object_copies":5148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1287,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11306,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-858}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'11479","reported_seq":11598,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:35.712961+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:44:35.712961+0000","last_peered":"2024-09-05T16:44:35.712961+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:44:35.712961+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"29'1400","ondisk_log_start":"29'1400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10592256,"num_objects":1293,"num_object_clones":0,"num_object_copies":5172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1293,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11479,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-926}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'11283","reported_seq":11395,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:34.862413+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:44:34.862413+0000","last_peered":"2024-09-05T16:44:34.862413+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:44:34.862413+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"29'1200","ondisk_log_start":"29'1200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11293856,"num_objects":1379,"num_object_clones":0,"num_object_copies":5516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1379,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11281,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-752}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'11440","reported_seq":11531,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:34.936118+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:44:34.936118+0000","last_peered":"2024-09-05T16:44:34.936118+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:44:34.936118+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"29'1400","ondisk_log_start":"29'1400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10966176,"num_objects":1339,"num_object_clones":0,"num_object_copies":5356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1339,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11438,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-893}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"39'11502","reported_seq":11607,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:35.972891+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:44:35.972891+0000","last_peered":"2024-09-05T16:44:35.972891+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:44:35.972891+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"29'1500","ondisk_log_start":"29'1500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10414400,"num_objects":1272,"num_object_clones":0,"num_object_copies":5088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1272,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11499,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-972}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'11468","reported_seq":11594,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:34.692841+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:44:34.692841+0000","last_peered":"2024-09-05T16:44:34.692841+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:44:34.692841+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"29'1400","ondisk_log_start":"29'1400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10990752,"num_objects":1342,"num_object_clones":0,"num_object_copies":5368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1342,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11466,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-903}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'11743","reported_seq":11874,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:35.870119+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:44:35.870119+0000","last_peered":"2024-09-05T16:44:35.870119+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:44:35.870119+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"29'1700","ondisk_log_start":"29'1700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11300592,"num_objects":1380,"num_object_clones":0,"num_object_copies":5520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1380,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11740,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-901}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'11396","reported_seq":11499,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:35.787153+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:44:35.787153+0000","last_peered":"2024-09-05T16:44:35.787153+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:44:35.787153+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"29'1300","ondisk_log_start":"29'1300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11164240,"num_objects":1363,"num_object_clones":0,"num_object_copies":5452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1363,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11395,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-875}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'11516","reported_seq":16256,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:44:35.278226+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:44:35.278226+0000","last_peered":"2024-09-05T16:44:35.278226+0000","last_clean":"2024-09-05T16:44:35.278226+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:44:35.278226+0000","last_undegraded":"2024-09-05T16:44:35.278226+0000","last_fullsized":"2024-09-05T16:44:35.278226+0000","mapping_epoch":36,"log_start":"29'1500","ondisk_log_start":"29'1500","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10584064,"num_objects":1292,"num_object_clones":0,"num_object_copies":5168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3116,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'11767","reported_seq":11880,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:35.725212+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:44:35.725212+0000","last_peered":"2024-09-05T16:44:35.725212+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:44:35.725212+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"29'1700","ondisk_log_start":"29'1700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11130016,"num_objects":1359,"num_object_clones":0,"num_object_copies":5436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1359,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11765,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-975}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'11518","reported_seq":11635,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:34.718016+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:44:34.718016+0000","last_peered":"2024-09-05T16:44:34.718016+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:44:34.718016+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"29'1500","ondisk_log_start":"29'1500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10769568,"num_objects":1315,"num_object_clones":0,"num_object_copies":5260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1315,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11516,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-993}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'11460","reported_seq":11573,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:34.832975+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:44:34.832975+0000","last_peered":"2024-09-05T16:44:34.832975+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:44:34.832975+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"29'1400","ondisk_log_start":"29'1400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10769568,"num_objects":1315,"num_object_clones":0,"num_object_copies":5260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1315,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11458,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-905}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":174005576,"num_objects":21246,"num_object_clones":0,"num_object_copies":84984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28650,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21246,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":158904,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":716,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":348401664,"data_stored":348401664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160813,"ondisk_log_size":160813,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084415,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":734756,"kb_used_data":85256,"kb_used_omap":4,"kb_used_meta":649467,"kb_avail":93637084,"statfs":{"total":96636764160,"available":95884374016,"internally_reserved":0,"allocated":87302144,"data_stored":87091563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":665054931},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,3,7,4,2,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986292,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":594724,"kb_used_data":85460,"kb_used_omap":4,"kb_used_meta":509243,"kb_avail":93777116,"statfs":{"total":96636764160,"available":96027766784,"internally_reserved":0,"allocated":87511040,"data_stored":87300459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":521465554},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,0,0,0,5,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673098,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":743324,"kb_used_data":86028,"kb_used_omap":1,"kb_used_meta":657278,"kb_avail":93628516,"statfs":{"total":96636764160,"available":95875600384,"internally_reserved":0,"allocated":88092672,"data_stored":87882091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":673053125},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,0,3,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182529,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":739296,"kb_used_data":85712,"kb_used_omap":4,"kb_used_meta":653563,"kb_avail":93632544,"statfs":{"total":96636764160,"available":95879725056,"internally_reserved":0,"allocated":87769088,"data_stored":87558507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":669249229},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,4,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":87347200,"data_stored":87347200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86958080,"data_stored":86958080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":87564288,"data_stored":87564288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":86532096,"data_stored":86532096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:44:37.589 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:44:38.186 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:44:38.186 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:44:38.212 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":370,"stamp":"2024-09-05T16:44:37.046394+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":174005576,"num_objects":21246,"num_object_clones":0,"num_object_copies":84984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28650,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21246,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":158904,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":716,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160813,"ondisk_log_size":160813,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2812100,"kb_used_data":342456,"kb_used_omap":14,"kb_used_meta":2469553,"kb_avail":374675260,"statfs":{"total":386547056640,"available":383667466240,"internally_reserved":0,"allocated":350674944,"data_stored":349832620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2528822839},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,0,3,4,18,10,11,7,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21823488,"num_objects":-2664,"num_object_clones":0,"num_object_copies":-10656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2664,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2664,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007397"},"pg_stats":[{"pgid":"1.f","version":"39'11547","reported_seq":12528,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:35.982021+0000","last_change":"2024-09-05T16:43:43.436935+0000","last_active":"2024-09-05T16:44:35.982021+0000","last_peered":"2024-09-05T16:44:35.982021+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:44:35.982021+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"29'1500","ondisk_log_start":"29'1500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11116544,"num_objects":1357,"num_object_clones":0,"num_object_copies":5428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1799,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1357,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11547,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":463,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-442}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'11295","reported_seq":15951,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:44:35.280975+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:44:35.280975+0000","last_peered":"2024-09-05T16:44:35.280975+0000","last_clean":"2024-09-05T16:44:35.280975+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:44:35.280975+0000","last_undegraded":"2024-09-05T16:44:35.280975+0000","last_fullsized":"2024-09-05T16:44:35.280975+0000","mapping_epoch":38,"log_start":"29'1200","ondisk_log_start":"29'1200","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10297344,"num_objects":1257,"num_object_clones":0,"num_object_copies":5028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1257,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":608,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'11409","reported_seq":11536,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:34.641847+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:44:34.641847+0000","last_peered":"2024-09-05T16:44:34.641847+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:44:34.641847+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"29'1400","ondisk_log_start":"29'1400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11000424,"num_objects":1344,"num_object_clones":0,"num_object_copies":5376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1344,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11408,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-888}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'11582","reported_seq":16316,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:44:35.417082+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:44:35.417082+0000","last_peered":"2024-09-05T16:44:35.417082+0000","last_clean":"2024-09-05T16:44:35.417082+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:44:35.417082+0000","last_undegraded":"2024-09-05T16:44:35.417082+0000","last_fullsized":"2024-09-05T16:44:35.417082+0000","mapping_epoch":34,"log_start":"29'1500","ondisk_log_start":"29'1500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11075584,"num_objects":1352,"num_object_clones":0,"num_object_copies":5408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1352,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5882,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'11308","reported_seq":11412,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:35.905134+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:44:35.905134+0000","last_peered":"2024-09-05T16:44:35.905134+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:44:35.905134+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"29'1300","ondisk_log_start":"29'1300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10540192,"num_objects":1287,"num_object_clones":0,"num_object_copies":5148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1287,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11306,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-858}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'11479","reported_seq":11598,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:35.712961+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:44:35.712961+0000","last_peered":"2024-09-05T16:44:35.712961+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:44:35.712961+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"29'1400","ondisk_log_start":"29'1400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10592256,"num_objects":1293,"num_object_clones":0,"num_object_copies":5172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1293,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11479,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-926}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'11283","reported_seq":11395,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:34.862413+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:44:34.862413+0000","last_peered":"2024-09-05T16:44:34.862413+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:44:34.862413+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"29'1200","ondisk_log_start":"29'1200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11293856,"num_objects":1379,"num_object_clones":0,"num_object_copies":5516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1379,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11281,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-752}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'11440","reported_seq":11531,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:34.936118+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:44:34.936118+0000","last_peered":"2024-09-05T16:44:34.936118+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:44:34.936118+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"29'1400","ondisk_log_start":"29'1400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10966176,"num_objects":1339,"num_object_clones":0,"num_object_copies":5356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1339,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11438,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-893}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"39'11502","reported_seq":11607,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:35.972891+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:44:35.972891+0000","last_peered":"2024-09-05T16:44:35.972891+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:44:35.972891+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"29'1500","ondisk_log_start":"29'1500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10414400,"num_objects":1272,"num_object_clones":0,"num_object_copies":5088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1272,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11499,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-972}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'11468","reported_seq":11594,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:34.692841+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:44:34.692841+0000","last_peered":"2024-09-05T16:44:34.692841+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:44:34.692841+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"29'1400","ondisk_log_start":"29'1400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10990752,"num_objects":1342,"num_object_clones":0,"num_object_copies":5368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1342,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11466,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-903}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'11743","reported_seq":11874,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:35.870119+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:44:35.870119+0000","last_peered":"2024-09-05T16:44:35.870119+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:44:35.870119+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"29'1700","ondisk_log_start":"29'1700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11300592,"num_objects":1380,"num_object_clones":0,"num_object_copies":5520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1380,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11740,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-901}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'11396","reported_seq":11499,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:35.787153+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:44:35.787153+0000","last_peered":"2024-09-05T16:44:35.787153+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:44:35.787153+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"29'1300","ondisk_log_start":"29'1300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11164240,"num_objects":1363,"num_object_clones":0,"num_object_copies":5452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1363,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11395,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-875}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'11516","reported_seq":16256,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:44:35.278226+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:44:35.278226+0000","last_peered":"2024-09-05T16:44:35.278226+0000","last_clean":"2024-09-05T16:44:35.278226+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:44:35.278226+0000","last_undegraded":"2024-09-05T16:44:35.278226+0000","last_fullsized":"2024-09-05T16:44:35.278226+0000","mapping_epoch":36,"log_start":"29'1500","ondisk_log_start":"29'1500","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10584064,"num_objects":1292,"num_object_clones":0,"num_object_copies":5168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3116,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'11767","reported_seq":11880,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:35.725212+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:44:35.725212+0000","last_peered":"2024-09-05T16:44:35.725212+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:44:35.725212+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"29'1700","ondisk_log_start":"29'1700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11130016,"num_objects":1359,"num_object_clones":0,"num_object_copies":5436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1359,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11765,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-975}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'11518","reported_seq":11635,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:34.718016+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:44:34.718016+0000","last_peered":"2024-09-05T16:44:34.718016+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:44:34.718016+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"29'1500","ondisk_log_start":"29'1500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10769568,"num_objects":1315,"num_object_clones":0,"num_object_copies":5260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1315,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11516,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-993}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'11460","reported_seq":11573,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:34.832975+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:44:34.832975+0000","last_peered":"2024-09-05T16:44:34.832975+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:44:34.832975+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"29'1400","ondisk_log_start":"29'1400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10769568,"num_objects":1315,"num_object_clones":0,"num_object_copies":5260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1315,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11458,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-905}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":174005576,"num_objects":21246,"num_object_clones":0,"num_object_copies":84984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28650,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21246,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":158904,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":716,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":348401664,"data_stored":348401664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160813,"ondisk_log_size":160813,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084415,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":734756,"kb_used_data":85256,"kb_used_omap":4,"kb_used_meta":649467,"kb_avail":93637084,"statfs":{"total":96636764160,"available":95884374016,"internally_reserved":0,"allocated":87302144,"data_stored":87091563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":665054931},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,3,7,4,2,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986292,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":594724,"kb_used_data":85460,"kb_used_omap":4,"kb_used_meta":509243,"kb_avail":93777116,"statfs":{"total":96636764160,"available":96027766784,"internally_reserved":0,"allocated":87511040,"data_stored":87300459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":521465554},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,0,0,0,5,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673098,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":743324,"kb_used_data":86028,"kb_used_omap":1,"kb_used_meta":657278,"kb_avail":93628516,"statfs":{"total":96636764160,"available":95875600384,"internally_reserved":0,"allocated":88092672,"data_stored":87882091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":673053125},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,0,3,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182529,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":739296,"kb_used_data":85712,"kb_used_omap":4,"kb_used_meta":653563,"kb_avail":93632544,"statfs":{"total":96636764160,"available":95879725056,"internally_reserved":0,"allocated":87769088,"data_stored":87558507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":669249229},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,4,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":87347200,"data_stored":87347200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86958080,"data_stored":86958080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":87564288,"data_stored":87564288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":86532096,"data_stored":86532096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:44:38.213 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-05T16:44:38.213 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "39'11547", 'reported_seq': 12528, 'reported_epoch': 39, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:35.982021+0000', 'last_change': '2024-09-05T16:43:43.436935+0000', 'last_active': '2024-09-05T16:44:35.982021+0000', 'last_peered': '2024-09-05T16:44:35.982021+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.828289+0000', 'last_became_peered': '2024-09-05T16:34:56.828289+0000', 'last_unstale': '2024-09-05T16:44:35.982021+0000', 'last_undegraded': '2024-09-05T16:34:56.612160+0000', 'last_fullsized': '2024-09-05T16:34:56.557632+0000', 'mapping_epoch': 32, 'log_start': "29'1500", 'ondisk_log_start': "29'1500", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10047, 'log_dups_size': 0, 'ondisk_log_size': 10047, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11116544, 'num_objects': 1357, 'num_object_clones': 0, 'num_object_copies': 5428, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1799, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1357, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11547, 'num_write_kb': 51616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 463, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -442}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:44:38.214 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:44:38.214 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "39'11409", 'reported_seq': 11536, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:34.641847+0000', 'last_change': '2024-09-05T16:34:56.867900+0000', 'last_active': '2024-09-05T16:44:34.641847+0000', 'last_peered': '2024-09-05T16:44:34.641847+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:44:34.641847+0000', 'last_undegraded': '2024-09-05T16:34:56.590972+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 32, 'log_start': "29'1400", 'ondisk_log_start': "29'1400", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10009, 'log_dups_size': 0, 'ondisk_log_size': 10009, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11000424, 'num_objects': 1344, 'num_object_clones': 0, 'num_object_copies': 5376, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2233, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1344, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 11408, 'num_write_kb': 50999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -888}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:44:38.214 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:44:38.214 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "39'11308", 'reported_seq': 11412, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:35.905134+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:44:35.905134+0000', 'last_peered': '2024-09-05T16:44:35.905134+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:44:35.905134+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "29'1300", 'ondisk_log_start': "29'1300", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10008, 'log_dups_size': 0, 'ondisk_log_size': 10008, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10540192, 'num_objects': 1287, 'num_object_clones': 0, 'num_object_copies': 5148, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2147, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1287, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11306, 'num_write_kb': 50373, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -858}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:44:38.214 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:44:38.214 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "39'11479", 'reported_seq': 11598, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:35.712961+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:44:35.712961+0000', 'last_peered': '2024-09-05T16:44:35.712961+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:44:35.712961+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "29'1400", 'ondisk_log_start': "29'1400", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10079, 'log_dups_size': 0, 'ondisk_log_size': 10079, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10592256, 'num_objects': 1293, 'num_object_clones': 0, 'num_object_copies': 5172, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1293, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11479, 'num_write_kb': 51088, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -926}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:44:38.214 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:44:38.214 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "39'11283", 'reported_seq': 11395, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:34.862413+0000', 'last_change': '2024-09-05T16:34:56.903436+0000', 'last_active': '2024-09-05T16:44:34.862413+0000', 'last_peered': '2024-09-05T16:44:34.862413+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:44:34.862413+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "29'1200", 'ondisk_log_start': "29'1200", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10083, 'log_dups_size': 0, 'ondisk_log_size': 10083, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11293856, 'num_objects': 1379, 'num_object_clones': 0, 'num_object_copies': 5516, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2133, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1379, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11281, 'num_write_kb': 50636, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -752}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:44:38.214 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-05T16:44:38.215 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "39'11440", 'reported_seq': 11531, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:34.936118+0000', 'last_change': '2024-09-05T16:34:56.883319+0000', 'last_active': '2024-09-05T16:44:34.936118+0000', 'last_peered': '2024-09-05T16:44:34.936118+0000', 'last_clean': '2024-09-05T16:33:53.505305+0000', 'last_became_active': '2024-09-05T16:34:56.873389+0000', 'last_became_peered': '2024-09-05T16:34:56.873389+0000', 'last_unstale': '2024-09-05T16:44:34.936118+0000', 'last_undegraded': '2024-09-05T16:34:56.602567+0000', 'last_fullsized': '2024-09-05T16:34:56.559958+0000', 'mapping_epoch': 32, 'log_start': "29'1400", 'ondisk_log_start': "29'1400", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10966176, 'num_objects': 1339, 'num_object_clones': 0, 'num_object_copies': 5356, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2234, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1339, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11438, 'num_write_kb': 51108, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -893}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:44:38.215 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:44:38.215 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "39'11502", 'reported_seq': 11607, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:35.972891+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:44:35.972891+0000', 'last_peered': '2024-09-05T16:44:35.972891+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:44:35.972891+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "29'1500", 'ondisk_log_start': "29'1500", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10002, 'log_dups_size': 0, 'ondisk_log_size': 10002, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10414400, 'num_objects': 1272, 'num_object_clones': 0, 'num_object_copies': 5088, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1272, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11499, 'num_write_kb': 51081, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -972}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:44:38.215 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:44:38.215 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "39'11468", 'reported_seq': 11594, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:34.692841+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:44:34.692841+0000', 'last_peered': '2024-09-05T16:44:34.692841+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:44:34.692841+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "29'1400", 'ondisk_log_start': "29'1400", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10068, 'log_dups_size': 0, 'ondisk_log_size': 10068, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10990752, 'num_objects': 1342, 'num_object_clones': 0, 'num_object_copies': 5368, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1342, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11466, 'num_write_kb': 51230, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -903}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:44:38.215 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:44:38.215 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "39'11743", 'reported_seq': 11874, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:35.870119+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:44:35.870119+0000', 'last_peered': '2024-09-05T16:44:35.870119+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:44:35.870119+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "29'1700", 'ondisk_log_start': "29'1700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10043, 'log_dups_size': 0, 'ondisk_log_size': 10043, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11300592, 'num_objects': 1380, 'num_object_clones': 0, 'num_object_copies': 5520, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2284, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1380, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11740, 'num_write_kb': 52476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -901}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:44:38.215 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:44:38.215 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "39'11396", 'reported_seq': 11499, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:35.787153+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:44:35.787153+0000', 'last_peered': '2024-09-05T16:44:35.787153+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:44:35.787153+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "29'1300", 'ondisk_log_start': "29'1300", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10096, 'log_dups_size': 0, 'ondisk_log_size': 10096, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11164240, 'num_objects': 1363, 'num_object_clones': 0, 'num_object_copies': 5452, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1363, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11395, 'num_write_kb': 51031, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -875}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:44:38.215 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:44:38.215 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "39'11767", 'reported_seq': 11880, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:35.725212+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:44:35.725212+0000', 'last_peered': '2024-09-05T16:44:35.725212+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:44:35.725212+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "29'1700", 'ondisk_log_start': "29'1700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10067, 'log_dups_size': 0, 'ondisk_log_size': 10067, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11130016, 'num_objects': 1359, 'num_object_clones': 0, 'num_object_copies': 5436, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1359, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11765, 'num_write_kb': 52500, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -975}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:44:38.216 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:44:38.216 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "39'11518", 'reported_seq': 11635, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:34.718016+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:44:34.718016+0000', 'last_peered': '2024-09-05T16:44:34.718016+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:44:34.718016+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "29'1500", 'ondisk_log_start': "29'1500", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10018, 'log_dups_size': 0, 'ondisk_log_size': 10018, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10769568, 'num_objects': 1315, 'num_object_clones': 0, 'num_object_copies': 5260, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2310, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1315, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11516, 'num_write_kb': 51325, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -993}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:44:38.216 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:44:38.216 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "39'11460", 'reported_seq': 11573, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:34.832975+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:44:34.832975+0000', 'last_peered': '2024-09-05T16:44:34.832975+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:44:34.832975+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "29'1400", 'ondisk_log_start': "29'1400", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10060, 'log_dups_size': 0, 'ondisk_log_size': 10060, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10769568, 'num_objects': 1315, 'num_object_clones': 0, 'num_object_copies': 5260, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1315, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11458, 'num_write_kb': 51092, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -905}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:44:38.216 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:44:38.958 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:44:38.958 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 28650/84984 objects degraded (33.712%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":682,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":39,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":21246,"data_bytes":174005576,"bytes_used":2879590400,"bytes_avail":383667466240,"bytes_total":386547056640,"degraded_objects":28650,"degraded_total":84984,"degraded_ratio":0.33712228184128779,"recovering_objects_per_sec":9,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":221},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:44:38.983 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:44:38.983 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:44:39.593 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:44:39.594 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:44:39.620 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":371,"stamp":"2024-09-05T16:44:39.047271+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":174005576,"num_objects":21246,"num_object_clones":0,"num_object_copies":84984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28650,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21246,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":158904,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":716,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160813,"ondisk_log_size":160813,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2812100,"kb_used_data":342456,"kb_used_omap":14,"kb_used_meta":2469553,"kb_avail":374675260,"statfs":{"total":386547056640,"available":383667466240,"internally_reserved":0,"allocated":350674944,"data_stored":349832620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2528822839},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,0,3,4,18,10,11,7,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16457728,"num_objects":-2009,"num_object_clones":0,"num_object_copies":-8036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-82,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2009,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2009,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":82,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007530"},"pg_stats":[{"pgid":"1.f","version":"39'11547","reported_seq":12528,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:35.982021+0000","last_change":"2024-09-05T16:43:43.436935+0000","last_active":"2024-09-05T16:44:35.982021+0000","last_peered":"2024-09-05T16:44:35.982021+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:44:35.982021+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"29'1500","ondisk_log_start":"29'1500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11116544,"num_objects":1357,"num_object_clones":0,"num_object_copies":5428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1799,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1357,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11547,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":463,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-442}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'11295","reported_seq":15951,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:44:35.280975+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:44:35.280975+0000","last_peered":"2024-09-05T16:44:35.280975+0000","last_clean":"2024-09-05T16:44:35.280975+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:44:35.280975+0000","last_undegraded":"2024-09-05T16:44:35.280975+0000","last_fullsized":"2024-09-05T16:44:35.280975+0000","mapping_epoch":38,"log_start":"29'1200","ondisk_log_start":"29'1200","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10297344,"num_objects":1257,"num_object_clones":0,"num_object_copies":5028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1257,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":608,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'11409","reported_seq":11536,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:34.641847+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:44:34.641847+0000","last_peered":"2024-09-05T16:44:34.641847+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:44:34.641847+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"29'1400","ondisk_log_start":"29'1400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11000424,"num_objects":1344,"num_object_clones":0,"num_object_copies":5376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1344,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11408,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-888}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'11582","reported_seq":16316,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:44:35.417082+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:44:35.417082+0000","last_peered":"2024-09-05T16:44:35.417082+0000","last_clean":"2024-09-05T16:44:35.417082+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:44:35.417082+0000","last_undegraded":"2024-09-05T16:44:35.417082+0000","last_fullsized":"2024-09-05T16:44:35.417082+0000","mapping_epoch":34,"log_start":"29'1500","ondisk_log_start":"29'1500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11075584,"num_objects":1352,"num_object_clones":0,"num_object_copies":5408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1352,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5882,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'11308","reported_seq":11412,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:35.905134+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:44:35.905134+0000","last_peered":"2024-09-05T16:44:35.905134+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:44:35.905134+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"29'1300","ondisk_log_start":"29'1300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10540192,"num_objects":1287,"num_object_clones":0,"num_object_copies":5148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1287,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11306,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-858}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'11479","reported_seq":11598,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:35.712961+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:44:35.712961+0000","last_peered":"2024-09-05T16:44:35.712961+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:44:35.712961+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"29'1400","ondisk_log_start":"29'1400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10592256,"num_objects":1293,"num_object_clones":0,"num_object_copies":5172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1293,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11479,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-926}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'11283","reported_seq":11395,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:34.862413+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:44:34.862413+0000","last_peered":"2024-09-05T16:44:34.862413+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:44:34.862413+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"29'1200","ondisk_log_start":"29'1200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11293856,"num_objects":1379,"num_object_clones":0,"num_object_copies":5516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1379,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11281,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-752}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'11440","reported_seq":11531,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:34.936118+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:44:34.936118+0000","last_peered":"2024-09-05T16:44:34.936118+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:44:34.936118+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"29'1400","ondisk_log_start":"29'1400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10966176,"num_objects":1339,"num_object_clones":0,"num_object_copies":5356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1339,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11438,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-893}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"39'11502","reported_seq":11607,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:35.972891+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:44:35.972891+0000","last_peered":"2024-09-05T16:44:35.972891+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:44:35.972891+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"29'1500","ondisk_log_start":"29'1500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10414400,"num_objects":1272,"num_object_clones":0,"num_object_copies":5088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1272,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11499,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-972}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'11468","reported_seq":11594,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:34.692841+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:44:34.692841+0000","last_peered":"2024-09-05T16:44:34.692841+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:44:34.692841+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"29'1400","ondisk_log_start":"29'1400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10990752,"num_objects":1342,"num_object_clones":0,"num_object_copies":5368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1342,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11466,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-903}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'11743","reported_seq":11874,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:35.870119+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:44:35.870119+0000","last_peered":"2024-09-05T16:44:35.870119+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:44:35.870119+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"29'1700","ondisk_log_start":"29'1700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11300592,"num_objects":1380,"num_object_clones":0,"num_object_copies":5520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1380,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11740,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-901}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'11396","reported_seq":11499,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:35.787153+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:44:35.787153+0000","last_peered":"2024-09-05T16:44:35.787153+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:44:35.787153+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"29'1300","ondisk_log_start":"29'1300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11164240,"num_objects":1363,"num_object_clones":0,"num_object_copies":5452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1363,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11395,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-875}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'11516","reported_seq":16256,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:44:35.278226+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:44:35.278226+0000","last_peered":"2024-09-05T16:44:35.278226+0000","last_clean":"2024-09-05T16:44:35.278226+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:44:35.278226+0000","last_undegraded":"2024-09-05T16:44:35.278226+0000","last_fullsized":"2024-09-05T16:44:35.278226+0000","mapping_epoch":36,"log_start":"29'1500","ondisk_log_start":"29'1500","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10584064,"num_objects":1292,"num_object_clones":0,"num_object_copies":5168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3116,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'11767","reported_seq":11880,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:35.725212+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:44:35.725212+0000","last_peered":"2024-09-05T16:44:35.725212+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:44:35.725212+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"29'1700","ondisk_log_start":"29'1700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11130016,"num_objects":1359,"num_object_clones":0,"num_object_copies":5436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1359,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11765,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-975}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'11518","reported_seq":11635,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:34.718016+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:44:34.718016+0000","last_peered":"2024-09-05T16:44:34.718016+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:44:34.718016+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"29'1500","ondisk_log_start":"29'1500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10769568,"num_objects":1315,"num_object_clones":0,"num_object_copies":5260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1315,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11516,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-993}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'11460","reported_seq":11573,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:34.832975+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:44:34.832975+0000","last_peered":"2024-09-05T16:44:34.832975+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:44:34.832975+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"29'1400","ondisk_log_start":"29'1400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10769568,"num_objects":1315,"num_object_clones":0,"num_object_copies":5260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1315,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11458,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-905}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":174005576,"num_objects":21246,"num_object_clones":0,"num_object_copies":84984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28650,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21246,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":158904,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":716,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":348401664,"data_stored":348401664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160813,"ondisk_log_size":160813,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084415,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":734756,"kb_used_data":85256,"kb_used_omap":4,"kb_used_meta":649467,"kb_avail":93637084,"statfs":{"total":96636764160,"available":95884374016,"internally_reserved":0,"allocated":87302144,"data_stored":87091563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":665054931},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,3,7,4,2,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986292,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":594724,"kb_used_data":85460,"kb_used_omap":4,"kb_used_meta":509243,"kb_avail":93777116,"statfs":{"total":96636764160,"available":96027766784,"internally_reserved":0,"allocated":87511040,"data_stored":87300459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":521465554},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,0,0,0,5,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673098,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":743324,"kb_used_data":86028,"kb_used_omap":1,"kb_used_meta":657278,"kb_avail":93628516,"statfs":{"total":96636764160,"available":95875600384,"internally_reserved":0,"allocated":88092672,"data_stored":87882091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":673053125},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,0,3,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182529,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":739296,"kb_used_data":85712,"kb_used_omap":4,"kb_used_meta":653563,"kb_avail":93632544,"statfs":{"total":96636764160,"available":95879725056,"internally_reserved":0,"allocated":87769088,"data_stored":87558507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":669249229},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,4,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":87347200,"data_stored":87347200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86958080,"data_stored":86958080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":87564288,"data_stored":87564288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":86532096,"data_stored":86532096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:44:42.622 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:44:43.234 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:44:43.234 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:44:43.260 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":373,"stamp":"2024-09-05T16:44:43.048715+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":165756232,"num_objects":20239,"num_object_clones":0,"num_object_copies":80956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28614,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20239,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":159911,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160820,"ondisk_log_size":160820,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2781028,"kb_used_data":326600,"kb_used_omap":14,"kb_used_meta":2454321,"kb_avail":374706332,"statfs":{"total":386547056640,"available":383699283968,"internally_reserved":0,"allocated":334438400,"data_stored":333596076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2513225271},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,7,6,26,17,7,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16605184,"num_objects":-2027,"num_object_clones":0,"num_object_copies":-8108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-79,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2027,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2027,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004601"},"pg_stats":[{"pgid":"1.f","version":"39'11615","reported_seq":12669,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:41.050519+0000","last_change":"2024-09-05T16:43:43.436935+0000","last_active":"2024-09-05T16:44:41.050519+0000","last_peered":"2024-09-05T16:44:41.050519+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:44:41.050519+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"29'1600","ondisk_log_start":"29'1600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10559488,"num_objects":1289,"num_object_clones":0,"num_object_copies":5156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1763,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1289,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11615,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":499,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-474}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'11353","reported_seq":16009,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:44:40.615068+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:44:40.615068+0000","last_peered":"2024-09-05T16:44:40.615068+0000","last_clean":"2024-09-05T16:44:40.615068+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:44:40.615068+0000","last_undegraded":"2024-09-05T16:44:40.615068+0000","last_fullsized":"2024-09-05T16:44:40.615068+0000","mapping_epoch":38,"log_start":"29'1300","ondisk_log_start":"29'1300","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9822208,"num_objects":1199,"num_object_clones":0,"num_object_copies":4796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1199,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":666,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'11470","reported_seq":11597,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:39.876749+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:44:39.876749+0000","last_peered":"2024-09-05T16:44:39.876749+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:44:39.876749+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"29'1400","ondisk_log_start":"29'1400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10500712,"num_objects":1283,"num_object_clones":0,"num_object_copies":5132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1283,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11469,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-949}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'11645","reported_seq":16379,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:44:40.584336+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:44:40.584336+0000","last_peered":"2024-09-05T16:44:40.584336+0000","last_clean":"2024-09-05T16:44:40.584336+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:44:40.584336+0000","last_undegraded":"2024-09-05T16:44:40.584336+0000","last_fullsized":"2024-09-05T16:44:40.584336+0000","mapping_epoch":34,"log_start":"29'1600","ondisk_log_start":"29'1600","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10559488,"num_objects":1289,"num_object_clones":0,"num_object_copies":5156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1289,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5945,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'11361","reported_seq":11465,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:40.938213+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:44:40.938213+0000","last_peered":"2024-09-05T16:44:40.938213+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:44:40.938213+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"29'1300","ondisk_log_start":"29'1300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10106016,"num_objects":1234,"num_object_clones":0,"num_object_copies":4936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1234,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11359,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-911}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'11539","reported_seq":11658,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:40.945883+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:44:40.945883+0000","last_peered":"2024-09-05T16:44:40.945883+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:44:40.945883+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"29'1500","ondisk_log_start":"29'1500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10100736,"num_objects":1233,"num_object_clones":0,"num_object_copies":4932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1233,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11539,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-986}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'11355","reported_seq":11465,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:40.033609+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:44:40.033609+0000","last_peered":"2024-09-05T16:44:40.033609+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:44:40.033609+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"29'1300","ondisk_log_start":"29'1300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10704032,"num_objects":1307,"num_object_clones":0,"num_object_copies":5228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1307,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11353,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-824}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'11493","reported_seq":11586,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:40.089465+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:44:40.089465+0000","last_peered":"2024-09-05T16:44:40.089465+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:44:40.089465+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"29'1400","ondisk_log_start":"29'1400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10532000,"num_objects":1286,"num_object_clones":0,"num_object_copies":5144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1286,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11491,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-946}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"39'11569","reported_seq":11674,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:40.947732+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:44:40.947732+0000","last_peered":"2024-09-05T16:44:40.947732+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:44:40.947732+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"29'1500","ondisk_log_start":"29'1500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9865536,"num_objects":1205,"num_object_clones":0,"num_object_copies":4820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1205,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11566,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1039}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'11541","reported_seq":11668,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:39.873235+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:44:39.873235+0000","last_peered":"2024-09-05T16:44:39.873235+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:44:39.873235+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"29'1500","ondisk_log_start":"29'1500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10392736,"num_objects":1269,"num_object_clones":0,"num_object_copies":5076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1269,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11539,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-976}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'11802","reported_seq":11933,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:40.717141+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:44:40.717141+0000","last_peered":"2024-09-05T16:44:40.717141+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:44:40.717141+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"29'1800","ondisk_log_start":"29'1800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10817264,"num_objects":1321,"num_object_clones":0,"num_object_copies":5284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1321,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11799,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-960}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'11460","reported_seq":11563,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:40.930845+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:44:40.930845+0000","last_peered":"2024-09-05T16:44:40.930845+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:44:40.930845+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"29'1400","ondisk_log_start":"29'1400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10639952,"num_objects":1299,"num_object_clones":0,"num_object_copies":5196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1299,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11459,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-939}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'11580","reported_seq":16320,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:44:40.585150+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:44:40.585150+0000","last_peered":"2024-09-05T16:44:40.585150+0000","last_clean":"2024-09-05T16:44:40.585150+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:44:40.585150+0000","last_undegraded":"2024-09-05T16:44:40.585150+0000","last_fullsized":"2024-09-05T16:44:40.585150+0000","mapping_epoch":36,"log_start":"29'1500","ondisk_log_start":"29'1500","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10059776,"num_objects":1228,"num_object_clones":0,"num_object_copies":4912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1228,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3180,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'11830","reported_seq":11943,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:40.969569+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:44:40.969569+0000","last_peered":"2024-09-05T16:44:40.969569+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:44:40.969569+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"29'1800","ondisk_log_start":"29'1800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10613920,"num_objects":1296,"num_object_clones":0,"num_object_copies":5184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1296,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1038}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'11585","reported_seq":11702,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:39.785918+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:44:39.785918+0000","last_peered":"2024-09-05T16:44:39.785918+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:44:39.785918+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"29'1500","ondisk_log_start":"29'1500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10220704,"num_objects":1248,"num_object_clones":0,"num_object_copies":4992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1248,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11583,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1060}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'11522","reported_seq":11635,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:39.903110+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:44:39.903110+0000","last_peered":"2024-09-05T16:44:39.903110+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:44:39.903110+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"29'1500","ondisk_log_start":"29'1500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10261664,"num_objects":1253,"num_object_clones":0,"num_object_copies":5012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1253,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11520,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-967}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":165756232,"num_objects":20239,"num_object_clones":0,"num_object_copies":80956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28614,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20239,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":159911,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":331956224,"data_stored":331956224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160820,"ondisk_log_size":160820,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084416,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":734936,"kb_used_data":81288,"kb_used_omap":4,"kb_used_meta":653627,"kb_avail":93636904,"statfs":{"total":96636764160,"available":95884189696,"internally_reserved":0,"allocated":83238912,"data_stored":83028331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":669314771},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,5,7,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986293,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":596320,"kb_used_data":81488,"kb_used_omap":4,"kb_used_meta":514811,"kb_avail":93775520,"statfs":{"total":96636764160,"available":96026132480,"internally_reserved":0,"allocated":83443712,"data_stored":83233131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":527167186},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,10,4,5,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673099,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":726916,"kb_used_data":82092,"kb_used_omap":1,"kb_used_meta":644798,"kb_avail":93644924,"statfs":{"total":96636764160,"available":95892402176,"internally_reserved":0,"allocated":84062208,"data_stored":83851627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":660273605},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,7,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182530,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722856,"kb_used_data":81732,"kb_used_omap":4,"kb_used_meta":641083,"kb_avail":93648984,"statfs":{"total":96636764160,"available":95896559616,"internally_reserved":0,"allocated":83693568,"data_stored":83482987,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":656469709},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,3,4,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":83275776,"data_stored":83275776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":82853888,"data_stored":82853888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":83382272,"data_stored":83382272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":82444288,"data_stored":82444288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:44:43.261 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:44:43.853 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:44:43.854 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:44:43.874 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":373,"stamp":"2024-09-05T16:44:43.048715+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":165756232,"num_objects":20239,"num_object_clones":0,"num_object_copies":80956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28614,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20239,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":159911,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160820,"ondisk_log_size":160820,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2781028,"kb_used_data":326600,"kb_used_omap":14,"kb_used_meta":2454321,"kb_avail":374706332,"statfs":{"total":386547056640,"available":383699283968,"internally_reserved":0,"allocated":334438400,"data_stored":333596076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2513225271},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,7,6,26,17,7,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16605184,"num_objects":-2027,"num_object_clones":0,"num_object_copies":-8108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-79,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2027,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2027,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004601"},"pg_stats":[{"pgid":"1.f","version":"39'11615","reported_seq":12669,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:41.050519+0000","last_change":"2024-09-05T16:43:43.436935+0000","last_active":"2024-09-05T16:44:41.050519+0000","last_peered":"2024-09-05T16:44:41.050519+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:44:41.050519+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"29'1600","ondisk_log_start":"29'1600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10559488,"num_objects":1289,"num_object_clones":0,"num_object_copies":5156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1763,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1289,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11615,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":499,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-474}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'11353","reported_seq":16009,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:44:40.615068+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:44:40.615068+0000","last_peered":"2024-09-05T16:44:40.615068+0000","last_clean":"2024-09-05T16:44:40.615068+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:44:40.615068+0000","last_undegraded":"2024-09-05T16:44:40.615068+0000","last_fullsized":"2024-09-05T16:44:40.615068+0000","mapping_epoch":38,"log_start":"29'1300","ondisk_log_start":"29'1300","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9822208,"num_objects":1199,"num_object_clones":0,"num_object_copies":4796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1199,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":666,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'11470","reported_seq":11597,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:39.876749+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:44:39.876749+0000","last_peered":"2024-09-05T16:44:39.876749+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:44:39.876749+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"29'1400","ondisk_log_start":"29'1400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10500712,"num_objects":1283,"num_object_clones":0,"num_object_copies":5132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1283,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11469,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-949}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'11645","reported_seq":16379,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:44:40.584336+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:44:40.584336+0000","last_peered":"2024-09-05T16:44:40.584336+0000","last_clean":"2024-09-05T16:44:40.584336+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:44:40.584336+0000","last_undegraded":"2024-09-05T16:44:40.584336+0000","last_fullsized":"2024-09-05T16:44:40.584336+0000","mapping_epoch":34,"log_start":"29'1600","ondisk_log_start":"29'1600","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10559488,"num_objects":1289,"num_object_clones":0,"num_object_copies":5156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1289,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5945,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'11361","reported_seq":11465,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:40.938213+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:44:40.938213+0000","last_peered":"2024-09-05T16:44:40.938213+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:44:40.938213+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"29'1300","ondisk_log_start":"29'1300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10106016,"num_objects":1234,"num_object_clones":0,"num_object_copies":4936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1234,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11359,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-911}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'11539","reported_seq":11658,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:40.945883+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:44:40.945883+0000","last_peered":"2024-09-05T16:44:40.945883+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:44:40.945883+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"29'1500","ondisk_log_start":"29'1500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10100736,"num_objects":1233,"num_object_clones":0,"num_object_copies":4932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1233,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11539,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-986}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'11355","reported_seq":11465,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:40.033609+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:44:40.033609+0000","last_peered":"2024-09-05T16:44:40.033609+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:44:40.033609+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"29'1300","ondisk_log_start":"29'1300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10704032,"num_objects":1307,"num_object_clones":0,"num_object_copies":5228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1307,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11353,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-824}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'11493","reported_seq":11586,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:40.089465+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:44:40.089465+0000","last_peered":"2024-09-05T16:44:40.089465+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:44:40.089465+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"29'1400","ondisk_log_start":"29'1400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10532000,"num_objects":1286,"num_object_clones":0,"num_object_copies":5144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1286,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11491,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-946}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"39'11569","reported_seq":11674,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:40.947732+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:44:40.947732+0000","last_peered":"2024-09-05T16:44:40.947732+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:44:40.947732+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"29'1500","ondisk_log_start":"29'1500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9865536,"num_objects":1205,"num_object_clones":0,"num_object_copies":4820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1205,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11566,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1039}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'11541","reported_seq":11668,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:39.873235+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:44:39.873235+0000","last_peered":"2024-09-05T16:44:39.873235+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:44:39.873235+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"29'1500","ondisk_log_start":"29'1500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10392736,"num_objects":1269,"num_object_clones":0,"num_object_copies":5076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1269,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11539,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-976}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'11802","reported_seq":11933,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:40.717141+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:44:40.717141+0000","last_peered":"2024-09-05T16:44:40.717141+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:44:40.717141+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"29'1800","ondisk_log_start":"29'1800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10817264,"num_objects":1321,"num_object_clones":0,"num_object_copies":5284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1321,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11799,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-960}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'11460","reported_seq":11563,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:40.930845+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:44:40.930845+0000","last_peered":"2024-09-05T16:44:40.930845+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:44:40.930845+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"29'1400","ondisk_log_start":"29'1400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10639952,"num_objects":1299,"num_object_clones":0,"num_object_copies":5196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1299,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11459,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-939}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'11580","reported_seq":16320,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:44:40.585150+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:44:40.585150+0000","last_peered":"2024-09-05T16:44:40.585150+0000","last_clean":"2024-09-05T16:44:40.585150+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:44:40.585150+0000","last_undegraded":"2024-09-05T16:44:40.585150+0000","last_fullsized":"2024-09-05T16:44:40.585150+0000","mapping_epoch":36,"log_start":"29'1500","ondisk_log_start":"29'1500","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10059776,"num_objects":1228,"num_object_clones":0,"num_object_copies":4912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1228,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3180,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'11830","reported_seq":11943,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:40.969569+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:44:40.969569+0000","last_peered":"2024-09-05T16:44:40.969569+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:44:40.969569+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"29'1800","ondisk_log_start":"29'1800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10613920,"num_objects":1296,"num_object_clones":0,"num_object_copies":5184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1296,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1038}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'11585","reported_seq":11702,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:39.785918+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:44:39.785918+0000","last_peered":"2024-09-05T16:44:39.785918+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:44:39.785918+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"29'1500","ondisk_log_start":"29'1500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10220704,"num_objects":1248,"num_object_clones":0,"num_object_copies":4992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1248,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11583,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1060}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'11522","reported_seq":11635,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:39.903110+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:44:39.903110+0000","last_peered":"2024-09-05T16:44:39.903110+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:44:39.903110+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"29'1500","ondisk_log_start":"29'1500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10261664,"num_objects":1253,"num_object_clones":0,"num_object_copies":5012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1253,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11520,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-967}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":165756232,"num_objects":20239,"num_object_clones":0,"num_object_copies":80956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28614,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20239,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":159911,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":331956224,"data_stored":331956224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160820,"ondisk_log_size":160820,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084416,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":734936,"kb_used_data":81288,"kb_used_omap":4,"kb_used_meta":653627,"kb_avail":93636904,"statfs":{"total":96636764160,"available":95884189696,"internally_reserved":0,"allocated":83238912,"data_stored":83028331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":669314771},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,5,7,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986293,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":596320,"kb_used_data":81488,"kb_used_omap":4,"kb_used_meta":514811,"kb_avail":93775520,"statfs":{"total":96636764160,"available":96026132480,"internally_reserved":0,"allocated":83443712,"data_stored":83233131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":527167186},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,10,4,5,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673099,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":726916,"kb_used_data":82092,"kb_used_omap":1,"kb_used_meta":644798,"kb_avail":93644924,"statfs":{"total":96636764160,"available":95892402176,"internally_reserved":0,"allocated":84062208,"data_stored":83851627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":660273605},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,7,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182530,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722856,"kb_used_data":81732,"kb_used_omap":4,"kb_used_meta":641083,"kb_avail":93648984,"statfs":{"total":96636764160,"available":95896559616,"internally_reserved":0,"allocated":83693568,"data_stored":83482987,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":656469709},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,3,4,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":83275776,"data_stored":83275776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":82853888,"data_stored":82853888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":83382272,"data_stored":83382272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":82444288,"data_stored":82444288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:44:43.875 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-05T16:44:43.875 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "39'11615", 'reported_seq': 12669, 'reported_epoch': 39, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:41.050519+0000', 'last_change': '2024-09-05T16:43:43.436935+0000', 'last_active': '2024-09-05T16:44:41.050519+0000', 'last_peered': '2024-09-05T16:44:41.050519+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.828289+0000', 'last_became_peered': '2024-09-05T16:34:56.828289+0000', 'last_unstale': '2024-09-05T16:44:41.050519+0000', 'last_undegraded': '2024-09-05T16:34:56.612160+0000', 'last_fullsized': '2024-09-05T16:34:56.557632+0000', 'mapping_epoch': 32, 'log_start': "29'1600", 'ondisk_log_start': "29'1600", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10015, 'log_dups_size': 0, 'ondisk_log_size': 10015, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10559488, 'num_objects': 1289, 'num_object_clones': 0, 'num_object_copies': 5156, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1763, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1289, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11615, 'num_write_kb': 51616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 499, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -474}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:44:43.876 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:44:43.876 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "39'11470", 'reported_seq': 11597, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:39.876749+0000', 'last_change': '2024-09-05T16:34:56.867900+0000', 'last_active': '2024-09-05T16:44:39.876749+0000', 'last_peered': '2024-09-05T16:44:39.876749+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:44:39.876749+0000', 'last_undegraded': '2024-09-05T16:34:56.590972+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 32, 'log_start': "29'1400", 'ondisk_log_start': "29'1400", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10070, 'log_dups_size': 0, 'ondisk_log_size': 10070, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10500712, 'num_objects': 1283, 'num_object_clones': 0, 'num_object_copies': 5132, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2233, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1283, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 11469, 'num_write_kb': 50999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -949}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:44:43.876 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:44:43.876 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "39'11361", 'reported_seq': 11465, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:40.938213+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:44:40.938213+0000', 'last_peered': '2024-09-05T16:44:40.938213+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:44:40.938213+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "29'1300", 'ondisk_log_start': "29'1300", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10061, 'log_dups_size': 0, 'ondisk_log_size': 10061, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10106016, 'num_objects': 1234, 'num_object_clones': 0, 'num_object_copies': 4936, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2147, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1234, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11359, 'num_write_kb': 50373, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -911}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:44:43.876 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:44:43.876 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "39'11539", 'reported_seq': 11658, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:40.945883+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:44:40.945883+0000', 'last_peered': '2024-09-05T16:44:40.945883+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:44:40.945883+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "29'1500", 'ondisk_log_start': "29'1500", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10039, 'log_dups_size': 0, 'ondisk_log_size': 10039, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10100736, 'num_objects': 1233, 'num_object_clones': 0, 'num_object_copies': 4932, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1233, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11539, 'num_write_kb': 51088, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -986}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:44:43.876 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:44:43.876 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "39'11355", 'reported_seq': 11465, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:40.033609+0000', 'last_change': '2024-09-05T16:34:56.903436+0000', 'last_active': '2024-09-05T16:44:40.033609+0000', 'last_peered': '2024-09-05T16:44:40.033609+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:44:40.033609+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "29'1300", 'ondisk_log_start': "29'1300", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10055, 'log_dups_size': 0, 'ondisk_log_size': 10055, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10704032, 'num_objects': 1307, 'num_object_clones': 0, 'num_object_copies': 5228, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2133, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1307, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11353, 'num_write_kb': 50636, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -824}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:44:43.876 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-05T16:44:43.876 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "39'11493", 'reported_seq': 11586, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:40.089465+0000', 'last_change': '2024-09-05T16:34:56.883319+0000', 'last_active': '2024-09-05T16:44:40.089465+0000', 'last_peered': '2024-09-05T16:44:40.089465+0000', 'last_clean': '2024-09-05T16:33:53.505305+0000', 'last_became_active': '2024-09-05T16:34:56.873389+0000', 'last_became_peered': '2024-09-05T16:34:56.873389+0000', 'last_unstale': '2024-09-05T16:44:40.089465+0000', 'last_undegraded': '2024-09-05T16:34:56.602567+0000', 'last_fullsized': '2024-09-05T16:34:56.559958+0000', 'mapping_epoch': 32, 'log_start': "29'1400", 'ondisk_log_start': "29'1400", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10093, 'log_dups_size': 0, 'ondisk_log_size': 10093, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10532000, 'num_objects': 1286, 'num_object_clones': 0, 'num_object_copies': 5144, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2234, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1286, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11491, 'num_write_kb': 51108, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -946}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:44:43.877 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:44:43.877 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "39'11569", 'reported_seq': 11674, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:40.947732+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:44:40.947732+0000', 'last_peered': '2024-09-05T16:44:40.947732+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:44:40.947732+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "29'1500", 'ondisk_log_start': "29'1500", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10069, 'log_dups_size': 0, 'ondisk_log_size': 10069, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9865536, 'num_objects': 1205, 'num_object_clones': 0, 'num_object_copies': 4820, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1205, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11566, 'num_write_kb': 51081, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1039}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:44:43.877 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:44:43.877 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "39'11541", 'reported_seq': 11668, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:39.873235+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:44:39.873235+0000', 'last_peered': '2024-09-05T16:44:39.873235+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:44:39.873235+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "29'1500", 'ondisk_log_start': "29'1500", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10041, 'log_dups_size': 0, 'ondisk_log_size': 10041, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10392736, 'num_objects': 1269, 'num_object_clones': 0, 'num_object_copies': 5076, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1269, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11539, 'num_write_kb': 51230, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -976}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:44:43.877 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:44:43.877 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "39'11802", 'reported_seq': 11933, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:40.717141+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:44:40.717141+0000', 'last_peered': '2024-09-05T16:44:40.717141+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:44:40.717141+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "29'1800", 'ondisk_log_start': "29'1800", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10002, 'log_dups_size': 0, 'ondisk_log_size': 10002, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10817264, 'num_objects': 1321, 'num_object_clones': 0, 'num_object_copies': 5284, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2284, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1321, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11799, 'num_write_kb': 52476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -960}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:44:43.877 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:44:43.877 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "39'11460", 'reported_seq': 11563, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:40.930845+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:44:40.930845+0000', 'last_peered': '2024-09-05T16:44:40.930845+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:44:40.930845+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "29'1400", 'ondisk_log_start': "29'1400", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10060, 'log_dups_size': 0, 'ondisk_log_size': 10060, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10639952, 'num_objects': 1299, 'num_object_clones': 0, 'num_object_copies': 5196, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1299, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11459, 'num_write_kb': 51031, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -939}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:44:43.877 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:44:43.877 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "39'11830", 'reported_seq': 11943, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:40.969569+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:44:40.969569+0000', 'last_peered': '2024-09-05T16:44:40.969569+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:44:40.969569+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "29'1800", 'ondisk_log_start': "29'1800", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10613920, 'num_objects': 1296, 'num_object_clones': 0, 'num_object_copies': 5184, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1296, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11828, 'num_write_kb': 52500, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1038}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:44:43.877 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:44:43.878 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "39'11585", 'reported_seq': 11702, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:39.785918+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:44:39.785918+0000', 'last_peered': '2024-09-05T16:44:39.785918+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:44:39.785918+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "29'1500", 'ondisk_log_start': "29'1500", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10085, 'log_dups_size': 0, 'ondisk_log_size': 10085, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10220704, 'num_objects': 1248, 'num_object_clones': 0, 'num_object_copies': 4992, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2310, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1248, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11583, 'num_write_kb': 51325, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1060}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:44:43.878 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:44:43.878 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "39'11522", 'reported_seq': 11635, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:39.903110+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:44:39.903110+0000', 'last_peered': '2024-09-05T16:44:39.903110+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:44:39.903110+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "29'1500", 'ondisk_log_start': "29'1500", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10022, 'log_dups_size': 0, 'ondisk_log_size': 10022, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10261664, 'num_objects': 1253, 'num_object_clones': 0, 'num_object_copies': 5012, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1253, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11520, 'num_write_kb': 51092, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -967}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:44:43.878 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:44:44.665 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:44:44.666 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 28614/80956 objects degraded (35.345%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":687,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":39,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":20239,"data_bytes":165756232,"bytes_used":2847772672,"bytes_avail":383699283968,"bytes_total":386547056640,"degraded_objects":28614,"degraded_total":80956,"degraded_ratio":0.35345125747319533,"recovering_objects_per_sec":6,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":168},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:44:44.688 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:44:44.689 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:44:45.288 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:44:45.288 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:44:45.318 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":374,"stamp":"2024-09-05T16:44:45.049391+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":164101448,"num_objects":20037,"num_object_clones":0,"num_object_copies":80148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28614,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20037,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":160113,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160722,"ondisk_log_size":160722,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2781136,"kb_used_data":322556,"kb_used_omap":14,"kb_used_meta":2458481,"kb_avail":374706224,"statfs":{"total":386547056640,"available":383699173376,"internally_reserved":0,"allocated":330297344,"data_stored":329455020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2517485111},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,7,10,21,22,7,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18259968,"num_objects":-2229,"num_object_clones":0,"num_object_copies":-8916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-79,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2229,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2229,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004532"},"pg_stats":[{"pgid":"1.f","version":"39'11615","reported_seq":12669,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:41.050519+0000","last_change":"2024-09-05T16:43:43.436935+0000","last_active":"2024-09-05T16:44:41.050519+0000","last_peered":"2024-09-05T16:44:41.050519+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:44:41.050519+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"29'1600","ondisk_log_start":"29'1600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10559488,"num_objects":1289,"num_object_clones":0,"num_object_copies":5156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1763,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1289,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11615,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":499,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-474}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'11353","reported_seq":16009,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:44:40.615068+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:44:40.615068+0000","last_peered":"2024-09-05T16:44:40.615068+0000","last_clean":"2024-09-05T16:44:40.615068+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:44:40.615068+0000","last_undegraded":"2024-09-05T16:44:40.615068+0000","last_fullsized":"2024-09-05T16:44:40.615068+0000","mapping_epoch":38,"log_start":"29'1300","ondisk_log_start":"29'1300","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9822208,"num_objects":1199,"num_object_clones":0,"num_object_copies":4796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1199,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":666,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'11537","reported_seq":11664,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:44.920574+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:44:44.920574+0000","last_peered":"2024-09-05T16:44:44.920574+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:44:44.920574+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"29'1500","ondisk_log_start":"29'1500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9951848,"num_objects":1216,"num_object_clones":0,"num_object_copies":4864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1216,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11536,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1016}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'11645","reported_seq":16379,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:44:40.584336+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:44:40.584336+0000","last_peered":"2024-09-05T16:44:40.584336+0000","last_clean":"2024-09-05T16:44:40.584336+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:44:40.584336+0000","last_undegraded":"2024-09-05T16:44:40.584336+0000","last_fullsized":"2024-09-05T16:44:40.584336+0000","mapping_epoch":34,"log_start":"29'1600","ondisk_log_start":"29'1600","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10559488,"num_objects":1289,"num_object_clones":0,"num_object_copies":5156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1289,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5945,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'11361","reported_seq":11465,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:40.938213+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:44:40.938213+0000","last_peered":"2024-09-05T16:44:40.938213+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:44:40.938213+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"29'1300","ondisk_log_start":"29'1300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10106016,"num_objects":1234,"num_object_clones":0,"num_object_copies":4936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1234,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11359,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-911}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'11539","reported_seq":11658,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:40.945883+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:44:40.945883+0000","last_peered":"2024-09-05T16:44:40.945883+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:44:40.945883+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"29'1500","ondisk_log_start":"29'1500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10100736,"num_objects":1233,"num_object_clones":0,"num_object_copies":4932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1233,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11539,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-986}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'11355","reported_seq":11465,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:40.033609+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:44:40.033609+0000","last_peered":"2024-09-05T16:44:40.033609+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:44:40.033609+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"29'1300","ondisk_log_start":"29'1300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10704032,"num_objects":1307,"num_object_clones":0,"num_object_copies":5228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1307,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11353,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-824}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'11493","reported_seq":11586,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:40.089465+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:44:40.089465+0000","last_peered":"2024-09-05T16:44:40.089465+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:44:40.089465+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"29'1400","ondisk_log_start":"29'1400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10532000,"num_objects":1286,"num_object_clones":0,"num_object_copies":5144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1286,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11491,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-946}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"39'11569","reported_seq":11674,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:40.947732+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:44:40.947732+0000","last_peered":"2024-09-05T16:44:40.947732+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:44:40.947732+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"29'1500","ondisk_log_start":"29'1500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9865536,"num_objects":1205,"num_object_clones":0,"num_object_copies":4820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1205,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11566,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1039}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'11614","reported_seq":11741,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:44.771579+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:44:44.771579+0000","last_peered":"2024-09-05T16:44:44.771579+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:44:44.771579+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"29'1600","ondisk_log_start":"29'1600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9794720,"num_objects":1196,"num_object_clones":0,"num_object_copies":4784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1196,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11612,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1049}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'11802","reported_seq":11933,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:40.717141+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:44:40.717141+0000","last_peered":"2024-09-05T16:44:40.717141+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:44:40.717141+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"29'1800","ondisk_log_start":"29'1800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10817264,"num_objects":1321,"num_object_clones":0,"num_object_copies":5284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1321,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11799,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-960}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'11460","reported_seq":11563,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:40.930845+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:44:40.930845+0000","last_peered":"2024-09-05T16:44:40.930845+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:44:40.930845+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"29'1400","ondisk_log_start":"29'1400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10639952,"num_objects":1299,"num_object_clones":0,"num_object_copies":5196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1299,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11459,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-939}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'11580","reported_seq":16320,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:44:40.585150+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:44:40.585150+0000","last_peered":"2024-09-05T16:44:40.585150+0000","last_clean":"2024-09-05T16:44:40.585150+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:44:40.585150+0000","last_undegraded":"2024-09-05T16:44:40.585150+0000","last_fullsized":"2024-09-05T16:44:40.585150+0000","mapping_epoch":36,"log_start":"29'1500","ondisk_log_start":"29'1500","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10059776,"num_objects":1228,"num_object_clones":0,"num_object_copies":4912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1228,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3180,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'11830","reported_seq":11943,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:40.969569+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:44:40.969569+0000","last_peered":"2024-09-05T16:44:40.969569+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:44:40.969569+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"29'1800","ondisk_log_start":"29'1800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10613920,"num_objects":1296,"num_object_clones":0,"num_object_copies":5184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1296,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1038}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'11647","reported_seq":11762,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:44.944929+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:44:44.944929+0000","last_peered":"2024-09-05T16:44:44.944929+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:44:44.944929+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"29'1600","ondisk_log_start":"29'1600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9712800,"num_objects":1186,"num_object_clones":0,"num_object_copies":4744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1186,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11645,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1122}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'11522","reported_seq":11635,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:39.903110+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:44:39.903110+0000","last_peered":"2024-09-05T16:44:39.903110+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:44:39.903110+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"29'1500","ondisk_log_start":"29'1500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10261664,"num_objects":1253,"num_object_clones":0,"num_object_copies":5012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1253,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11520,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-967}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":164101448,"num_objects":20037,"num_object_clones":0,"num_object_copies":80148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28614,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20037,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":160113,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":327852032,"data_stored":327852032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160722,"ondisk_log_size":160722,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084416,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":734936,"kb_used_data":81288,"kb_used_omap":4,"kb_used_meta":653627,"kb_avail":93636904,"statfs":{"total":96636764160,"available":95884189696,"internally_reserved":0,"allocated":83238912,"data_stored":83028331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":669314771},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,5,7,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986293,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":596320,"kb_used_data":81488,"kb_used_omap":4,"kb_used_meta":514811,"kb_avail":93775520,"statfs":{"total":96636764160,"available":96026132480,"internally_reserved":0,"allocated":83443712,"data_stored":83233131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":527167186},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,10,4,5,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673100,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":727024,"kb_used_data":78048,"kb_used_omap":1,"kb_used_meta":648958,"kb_avail":93644816,"statfs":{"total":96636764160,"available":95892291584,"internally_reserved":0,"allocated":79921152,"data_stored":79710571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":664533445},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,5,2,8],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182530,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722856,"kb_used_data":81732,"kb_used_omap":4,"kb_used_meta":641083,"kb_avail":93648984,"statfs":{"total":96636764160,"available":95896559616,"internally_reserved":0,"allocated":83693568,"data_stored":83482987,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":656469709},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,3,4,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":83275776,"data_stored":83275776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":82853888,"data_stored":82853888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":79278080,"data_stored":79278080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":82444288,"data_stored":82444288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:44:48.320 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:44:48.903 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:44:48.903 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:44:48.927 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":375,"stamp":"2024-09-05T16:44:47.050226+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":157408584,"num_objects":19220,"num_object_clones":0,"num_object_copies":76880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28583,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19220,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":160930,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":783,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160739,"ondisk_log_size":160739,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2764980,"kb_used_data":310568,"kb_used_omap":14,"kb_used_meta":2454321,"kb_avail":374722380,"statfs":{"total":386547056640,"available":383715717120,"internally_reserved":0,"allocated":318021632,"data_stored":317179308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2513225271},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,7,11,14,16,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-23289856,"num_objects":-2843,"num_object_clones":0,"num_object_copies":-11372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2843,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2843,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":110,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004586"},"pg_stats":[{"pgid":"1.f","version":"39'11694","reported_seq":12808,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:46.214142+0000","last_change":"2024-09-05T16:43:43.436935+0000","last_active":"2024-09-05T16:44:46.214142+0000","last_peered":"2024-09-05T16:44:46.214142+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:44:46.214142+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"29'1600","ondisk_log_start":"29'1600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9912320,"num_objects":1210,"num_object_clones":0,"num_object_copies":4840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1732,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1210,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11694,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":530,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-522}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'11428","reported_seq":16084,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:44:45.603900+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:44:45.603900+0000","last_peered":"2024-09-05T16:44:45.603900+0000","last_clean":"2024-09-05T16:44:45.603900+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:44:45.603900+0000","last_undegraded":"2024-09-05T16:44:45.603900+0000","last_fullsized":"2024-09-05T16:44:45.603900+0000","mapping_epoch":38,"log_start":"29'1400","ondisk_log_start":"29'1400","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9207808,"num_objects":1124,"num_object_clones":0,"num_object_copies":4496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1124,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":741,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'11537","reported_seq":11664,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:44.920574+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:44:44.920574+0000","last_peered":"2024-09-05T16:44:44.920574+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:44:44.920574+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"29'1500","ondisk_log_start":"29'1500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9951848,"num_objects":1216,"num_object_clones":0,"num_object_copies":4864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1216,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11536,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1016}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'11707","reported_seq":16441,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:44:45.414218+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:44:45.414218+0000","last_peered":"2024-09-05T16:44:45.414218+0000","last_clean":"2024-09-05T16:44:45.414218+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:44:45.414218+0000","last_undegraded":"2024-09-05T16:44:45.414218+0000","last_fullsized":"2024-09-05T16:44:45.414218+0000","mapping_epoch":34,"log_start":"29'1700","ondisk_log_start":"29'1700","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10051584,"num_objects":1227,"num_object_clones":0,"num_object_copies":4908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1227,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6007,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'11416","reported_seq":11520,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:45.948158+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:44:45.948158+0000","last_peered":"2024-09-05T16:44:45.948158+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:44:45.948158+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"29'1400","ondisk_log_start":"29'1400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9655456,"num_objects":1179,"num_object_clones":0,"num_object_copies":4716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1179,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11414,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-966}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'11594","reported_seq":11713,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:46.070574+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:44:46.070574+0000","last_peered":"2024-09-05T16:44:46.070574+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:44:46.070574+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"29'1500","ondisk_log_start":"29'1500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9650176,"num_objects":1178,"num_object_clones":0,"num_object_copies":4712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1178,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11594,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1041}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'11418","reported_seq":11531,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:45.160023+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:44:45.160023+0000","last_peered":"2024-09-05T16:44:45.160023+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:44:45.160023+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"29'1400","ondisk_log_start":"29'1400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10187936,"num_objects":1244,"num_object_clones":0,"num_object_copies":4976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1244,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11416,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-887}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'11555","reported_seq":11649,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:45.129835+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:44:45.129835+0000","last_peered":"2024-09-05T16:44:45.129835+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:44:45.129835+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"29'1500","ondisk_log_start":"29'1500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10024096,"num_objects":1224,"num_object_clones":0,"num_object_copies":4896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1224,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11553,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1008}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"39'11632","reported_seq":11737,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:46.167592+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:44:46.167592+0000","last_peered":"2024-09-05T16:44:46.167592+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:44:46.167592+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"29'1600","ondisk_log_start":"29'1600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9349440,"num_objects":1142,"num_object_clones":0,"num_object_copies":4568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1142,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11629,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1102}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'11614","reported_seq":11741,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:44.771579+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:44:44.771579+0000","last_peered":"2024-09-05T16:44:44.771579+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:44:44.771579+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"29'1600","ondisk_log_start":"29'1600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9794720,"num_objects":1196,"num_object_clones":0,"num_object_copies":4784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1196,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11612,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1049}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'11866","reported_seq":11997,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:45.945448+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:44:45.945448+0000","last_peered":"2024-09-05T16:44:45.945448+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:44:45.945448+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"29'1800","ondisk_log_start":"29'1800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10292976,"num_objects":1257,"num_object_clones":0,"num_object_copies":5028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1257,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11863,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1024}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'11532","reported_seq":11635,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:46.214186+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:44:46.214186+0000","last_peered":"2024-09-05T16:44:46.214186+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:44:46.214186+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"29'1500","ondisk_log_start":"29'1500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10050128,"num_objects":1227,"num_object_clones":0,"num_object_copies":4908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1227,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11531,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1011}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'11621","reported_seq":16361,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:44:45.577601+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:44:45.577601+0000","last_peered":"2024-09-05T16:44:45.577601+0000","last_clean":"2024-09-05T16:44:45.577601+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:44:45.577601+0000","last_undegraded":"2024-09-05T16:44:45.577601+0000","last_fullsized":"2024-09-05T16:44:45.577601+0000","mapping_epoch":36,"log_start":"29'1600","ondisk_log_start":"29'1600","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9723904,"num_objects":1187,"num_object_clones":0,"num_object_copies":4748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1187,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3221,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'11896","reported_seq":12009,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:45.931927+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:44:45.931927+0000","last_peered":"2024-09-05T16:44:45.931927+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:44:45.931927+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"29'1800","ondisk_log_start":"29'1800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10073248,"num_objects":1230,"num_object_clones":0,"num_object_copies":4920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1230,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11894,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1104}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'11647","reported_seq":11762,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:44.944929+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:44:44.944929+0000","last_peered":"2024-09-05T16:44:44.944929+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:44:44.944929+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"29'1600","ondisk_log_start":"29'1600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9712800,"num_objects":1186,"num_object_clones":0,"num_object_copies":4744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1186,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11645,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1122}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'11582","reported_seq":11695,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:45.057221+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:44:45.057221+0000","last_peered":"2024-09-05T16:44:45.057221+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:44:45.057221+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"29'1500","ondisk_log_start":"29'1500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9770144,"num_objects":1193,"num_object_clones":0,"num_object_copies":4772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11580,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1027}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":157408584,"num_objects":19220,"num_object_clones":0,"num_object_copies":76880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28583,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19220,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":160930,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":783,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":315396096,"data_stored":315396096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160739,"ondisk_log_size":160739,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084417,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":718452,"kb_used_data":77280,"kb_used_omap":4,"kb_used_meta":641147,"kb_avail":93653388,"statfs":{"total":96636764160,"available":95901069312,"internally_reserved":0,"allocated":79134720,"data_stored":78924139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":656535251},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,2,2,2,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986294,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":596564,"kb_used_data":77572,"kb_used_omap":4,"kb_used_meta":518971,"kb_avail":93775276,"statfs":{"total":96636764160,"available":96025882624,"internally_reserved":0,"allocated":79433728,"data_stored":79223147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":531427026},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,6,6,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673100,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":727024,"kb_used_data":78048,"kb_used_omap":1,"kb_used_meta":648958,"kb_avail":93644816,"statfs":{"total":96636764160,"available":95892291584,"internally_reserved":0,"allocated":79921152,"data_stored":79710571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":664533445},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,5,2,8],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182531,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722940,"kb_used_data":77668,"kb_used_omap":4,"kb_used_meta":645243,"kb_avail":93648900,"statfs":{"total":96636764160,"available":95896473600,"internally_reserved":0,"allocated":79532032,"data_stored":79321451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":660729549},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,2,4,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":79122432,"data_stored":79122432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":78766080,"data_stored":78766080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":79278080,"data_stored":79278080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":78229504,"data_stored":78229504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:44:48.929 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:44:49.572 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:44:49.573 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:44:49.595 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":376,"stamp":"2024-09-05T16:44:49.051223+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":157408584,"num_objects":19220,"num_object_clones":0,"num_object_copies":76880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28583,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19220,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":160930,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":783,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160739,"ondisk_log_size":160739,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2764980,"kb_used_data":310568,"kb_used_omap":14,"kb_used_meta":2454321,"kb_avail":374722380,"statfs":{"total":386547056640,"available":383715717120,"internally_reserved":0,"allocated":318021632,"data_stored":317179308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2513225271},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,7,11,14,16,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16596992,"num_objects":-2026,"num_object_clones":0,"num_object_copies":-8104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-67,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2026,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2026,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004829"},"pg_stats":[{"pgid":"1.f","version":"39'11694","reported_seq":12808,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:46.214142+0000","last_change":"2024-09-05T16:43:43.436935+0000","last_active":"2024-09-05T16:44:46.214142+0000","last_peered":"2024-09-05T16:44:46.214142+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:44:46.214142+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"29'1600","ondisk_log_start":"29'1600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9912320,"num_objects":1210,"num_object_clones":0,"num_object_copies":4840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1732,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1210,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11694,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":530,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-522}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'11428","reported_seq":16084,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:44:45.603900+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:44:45.603900+0000","last_peered":"2024-09-05T16:44:45.603900+0000","last_clean":"2024-09-05T16:44:45.603900+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:44:45.603900+0000","last_undegraded":"2024-09-05T16:44:45.603900+0000","last_fullsized":"2024-09-05T16:44:45.603900+0000","mapping_epoch":38,"log_start":"29'1400","ondisk_log_start":"29'1400","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9207808,"num_objects":1124,"num_object_clones":0,"num_object_copies":4496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1124,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":741,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'11537","reported_seq":11664,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:44.920574+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:44:44.920574+0000","last_peered":"2024-09-05T16:44:44.920574+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:44:44.920574+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"29'1500","ondisk_log_start":"29'1500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9951848,"num_objects":1216,"num_object_clones":0,"num_object_copies":4864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1216,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11536,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1016}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'11707","reported_seq":16441,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:44:45.414218+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:44:45.414218+0000","last_peered":"2024-09-05T16:44:45.414218+0000","last_clean":"2024-09-05T16:44:45.414218+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:44:45.414218+0000","last_undegraded":"2024-09-05T16:44:45.414218+0000","last_fullsized":"2024-09-05T16:44:45.414218+0000","mapping_epoch":34,"log_start":"29'1700","ondisk_log_start":"29'1700","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10051584,"num_objects":1227,"num_object_clones":0,"num_object_copies":4908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1227,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6007,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'11416","reported_seq":11520,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:45.948158+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:44:45.948158+0000","last_peered":"2024-09-05T16:44:45.948158+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:44:45.948158+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"29'1400","ondisk_log_start":"29'1400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9655456,"num_objects":1179,"num_object_clones":0,"num_object_copies":4716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1179,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11414,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-966}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'11594","reported_seq":11713,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:46.070574+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:44:46.070574+0000","last_peered":"2024-09-05T16:44:46.070574+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:44:46.070574+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"29'1500","ondisk_log_start":"29'1500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9650176,"num_objects":1178,"num_object_clones":0,"num_object_copies":4712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1178,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11594,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1041}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'11418","reported_seq":11531,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:45.160023+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:44:45.160023+0000","last_peered":"2024-09-05T16:44:45.160023+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:44:45.160023+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"29'1400","ondisk_log_start":"29'1400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10187936,"num_objects":1244,"num_object_clones":0,"num_object_copies":4976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1244,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11416,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-887}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'11555","reported_seq":11649,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:45.129835+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:44:45.129835+0000","last_peered":"2024-09-05T16:44:45.129835+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:44:45.129835+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"29'1500","ondisk_log_start":"29'1500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10024096,"num_objects":1224,"num_object_clones":0,"num_object_copies":4896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1224,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11553,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1008}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"39'11632","reported_seq":11737,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:46.167592+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:44:46.167592+0000","last_peered":"2024-09-05T16:44:46.167592+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:44:46.167592+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"29'1600","ondisk_log_start":"29'1600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9349440,"num_objects":1142,"num_object_clones":0,"num_object_copies":4568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1142,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11629,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1102}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'11614","reported_seq":11741,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:44.771579+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:44:44.771579+0000","last_peered":"2024-09-05T16:44:44.771579+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:44:44.771579+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"29'1600","ondisk_log_start":"29'1600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9794720,"num_objects":1196,"num_object_clones":0,"num_object_copies":4784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1196,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11612,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1049}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'11866","reported_seq":11997,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:45.945448+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:44:45.945448+0000","last_peered":"2024-09-05T16:44:45.945448+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:44:45.945448+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"29'1800","ondisk_log_start":"29'1800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10292976,"num_objects":1257,"num_object_clones":0,"num_object_copies":5028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1257,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11863,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1024}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'11532","reported_seq":11635,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:46.214186+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:44:46.214186+0000","last_peered":"2024-09-05T16:44:46.214186+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:44:46.214186+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"29'1500","ondisk_log_start":"29'1500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10050128,"num_objects":1227,"num_object_clones":0,"num_object_copies":4908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1227,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11531,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1011}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'11621","reported_seq":16361,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:44:45.577601+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:44:45.577601+0000","last_peered":"2024-09-05T16:44:45.577601+0000","last_clean":"2024-09-05T16:44:45.577601+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:44:45.577601+0000","last_undegraded":"2024-09-05T16:44:45.577601+0000","last_fullsized":"2024-09-05T16:44:45.577601+0000","mapping_epoch":36,"log_start":"29'1600","ondisk_log_start":"29'1600","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9723904,"num_objects":1187,"num_object_clones":0,"num_object_copies":4748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1187,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3221,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'11896","reported_seq":12009,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:45.931927+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:44:45.931927+0000","last_peered":"2024-09-05T16:44:45.931927+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:44:45.931927+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"29'1800","ondisk_log_start":"29'1800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10073248,"num_objects":1230,"num_object_clones":0,"num_object_copies":4920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1230,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11894,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1104}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'11647","reported_seq":11762,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:44.944929+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:44:44.944929+0000","last_peered":"2024-09-05T16:44:44.944929+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:44:44.944929+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"29'1600","ondisk_log_start":"29'1600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9712800,"num_objects":1186,"num_object_clones":0,"num_object_copies":4744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1186,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11645,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1122}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'11582","reported_seq":11695,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:45.057221+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:44:45.057221+0000","last_peered":"2024-09-05T16:44:45.057221+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:44:45.057221+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"29'1500","ondisk_log_start":"29'1500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9770144,"num_objects":1193,"num_object_clones":0,"num_object_copies":4772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11580,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1027}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":157408584,"num_objects":19220,"num_object_clones":0,"num_object_copies":76880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28583,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19220,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":160930,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":783,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":315396096,"data_stored":315396096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160739,"ondisk_log_size":160739,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084417,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":718452,"kb_used_data":77280,"kb_used_omap":4,"kb_used_meta":641147,"kb_avail":93653388,"statfs":{"total":96636764160,"available":95901069312,"internally_reserved":0,"allocated":79134720,"data_stored":78924139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":656535251},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,2,2,2,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986294,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":596564,"kb_used_data":77572,"kb_used_omap":4,"kb_used_meta":518971,"kb_avail":93775276,"statfs":{"total":96636764160,"available":96025882624,"internally_reserved":0,"allocated":79433728,"data_stored":79223147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":531427026},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,6,6,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673100,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":727024,"kb_used_data":78048,"kb_used_omap":1,"kb_used_meta":648958,"kb_avail":93644816,"statfs":{"total":96636764160,"available":95892291584,"internally_reserved":0,"allocated":79921152,"data_stored":79710571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":664533445},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,5,2,8],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182531,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722940,"kb_used_data":77668,"kb_used_omap":4,"kb_used_meta":645243,"kb_avail":93648900,"statfs":{"total":96636764160,"available":95896473600,"internally_reserved":0,"allocated":79532032,"data_stored":79321451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":660729549},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,2,4,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":79122432,"data_stored":79122432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":78766080,"data_stored":78766080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":79278080,"data_stored":79278080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":78229504,"data_stored":78229504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:44:49.596 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-05T16:44:49.596 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "39'11694", 'reported_seq': 12808, 'reported_epoch': 39, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:46.214142+0000', 'last_change': '2024-09-05T16:43:43.436935+0000', 'last_active': '2024-09-05T16:44:46.214142+0000', 'last_peered': '2024-09-05T16:44:46.214142+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.828289+0000', 'last_became_peered': '2024-09-05T16:34:56.828289+0000', 'last_unstale': '2024-09-05T16:44:46.214142+0000', 'last_undegraded': '2024-09-05T16:34:56.612160+0000', 'last_fullsized': '2024-09-05T16:34:56.557632+0000', 'mapping_epoch': 32, 'log_start': "29'1600", 'ondisk_log_start': "29'1600", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10094, 'log_dups_size': 0, 'ondisk_log_size': 10094, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9912320, 'num_objects': 1210, 'num_object_clones': 0, 'num_object_copies': 4840, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1732, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1210, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11694, 'num_write_kb': 51616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 530, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -522}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:44:49.596 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:44:49.596 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "39'11537", 'reported_seq': 11664, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:44.920574+0000', 'last_change': '2024-09-05T16:34:56.867900+0000', 'last_active': '2024-09-05T16:44:44.920574+0000', 'last_peered': '2024-09-05T16:44:44.920574+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:44:44.920574+0000', 'last_undegraded': '2024-09-05T16:34:56.590972+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 32, 'log_start': "29'1500", 'ondisk_log_start': "29'1500", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10037, 'log_dups_size': 0, 'ondisk_log_size': 10037, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9951848, 'num_objects': 1216, 'num_object_clones': 0, 'num_object_copies': 4864, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2233, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1216, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 11536, 'num_write_kb': 50999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1016}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:44:49.596 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:44:49.596 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "39'11416", 'reported_seq': 11520, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:45.948158+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:44:45.948158+0000', 'last_peered': '2024-09-05T16:44:45.948158+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:44:45.948158+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "29'1400", 'ondisk_log_start': "29'1400", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10016, 'log_dups_size': 0, 'ondisk_log_size': 10016, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9655456, 'num_objects': 1179, 'num_object_clones': 0, 'num_object_copies': 4716, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2147, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1179, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11414, 'num_write_kb': 50373, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -966}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:44:49.597 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:44:49.597 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "39'11594", 'reported_seq': 11713, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:46.070574+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:44:46.070574+0000', 'last_peered': '2024-09-05T16:44:46.070574+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:44:46.070574+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "29'1500", 'ondisk_log_start': "29'1500", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10094, 'log_dups_size': 0, 'ondisk_log_size': 10094, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9650176, 'num_objects': 1178, 'num_object_clones': 0, 'num_object_copies': 4712, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1178, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11594, 'num_write_kb': 51088, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1041}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:44:49.597 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:44:49.597 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "39'11418", 'reported_seq': 11531, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:45.160023+0000', 'last_change': '2024-09-05T16:34:56.903436+0000', 'last_active': '2024-09-05T16:44:45.160023+0000', 'last_peered': '2024-09-05T16:44:45.160023+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:44:45.160023+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "29'1400", 'ondisk_log_start': "29'1400", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10018, 'log_dups_size': 0, 'ondisk_log_size': 10018, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10187936, 'num_objects': 1244, 'num_object_clones': 0, 'num_object_copies': 4976, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2133, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1244, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11416, 'num_write_kb': 50636, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -887}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:44:49.597 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-05T16:44:49.597 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "39'11555", 'reported_seq': 11649, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:45.129835+0000', 'last_change': '2024-09-05T16:34:56.883319+0000', 'last_active': '2024-09-05T16:44:45.129835+0000', 'last_peered': '2024-09-05T16:44:45.129835+0000', 'last_clean': '2024-09-05T16:33:53.505305+0000', 'last_became_active': '2024-09-05T16:34:56.873389+0000', 'last_became_peered': '2024-09-05T16:34:56.873389+0000', 'last_unstale': '2024-09-05T16:44:45.129835+0000', 'last_undegraded': '2024-09-05T16:34:56.602567+0000', 'last_fullsized': '2024-09-05T16:34:56.559958+0000', 'mapping_epoch': 32, 'log_start': "29'1500", 'ondisk_log_start': "29'1500", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10055, 'log_dups_size': 0, 'ondisk_log_size': 10055, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10024096, 'num_objects': 1224, 'num_object_clones': 0, 'num_object_copies': 4896, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2234, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1224, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11553, 'num_write_kb': 51108, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1008}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:44:49.597 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:44:49.597 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "39'11632", 'reported_seq': 11737, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:46.167592+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:44:46.167592+0000', 'last_peered': '2024-09-05T16:44:46.167592+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:44:46.167592+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "29'1600", 'ondisk_log_start': "29'1600", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10032, 'log_dups_size': 0, 'ondisk_log_size': 10032, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9349440, 'num_objects': 1142, 'num_object_clones': 0, 'num_object_copies': 4568, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1142, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11629, 'num_write_kb': 51081, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1102}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:44:49.597 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:44:49.597 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "39'11614", 'reported_seq': 11741, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:44.771579+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:44:44.771579+0000', 'last_peered': '2024-09-05T16:44:44.771579+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:44:44.771579+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "29'1600", 'ondisk_log_start': "29'1600", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10014, 'log_dups_size': 0, 'ondisk_log_size': 10014, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9794720, 'num_objects': 1196, 'num_object_clones': 0, 'num_object_copies': 4784, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1196, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11612, 'num_write_kb': 51230, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1049}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:44:49.598 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:44:49.598 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "39'11866", 'reported_seq': 11997, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:45.945448+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:44:45.945448+0000', 'last_peered': '2024-09-05T16:44:45.945448+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:44:45.945448+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "29'1800", 'ondisk_log_start': "29'1800", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10066, 'log_dups_size': 0, 'ondisk_log_size': 10066, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10292976, 'num_objects': 1257, 'num_object_clones': 0, 'num_object_copies': 5028, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2284, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1257, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11863, 'num_write_kb': 52476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1024}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:44:49.598 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:44:49.598 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "39'11532", 'reported_seq': 11635, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:46.214186+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:44:46.214186+0000', 'last_peered': '2024-09-05T16:44:46.214186+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:44:46.214186+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "29'1500", 'ondisk_log_start': "29'1500", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10032, 'log_dups_size': 0, 'ondisk_log_size': 10032, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10050128, 'num_objects': 1227, 'num_object_clones': 0, 'num_object_copies': 4908, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1227, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11531, 'num_write_kb': 51031, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1011}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:44:49.598 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:44:49.598 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "39'11896", 'reported_seq': 12009, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:45.931927+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:44:45.931927+0000', 'last_peered': '2024-09-05T16:44:45.931927+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:44:45.931927+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "29'1800", 'ondisk_log_start': "29'1800", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10096, 'log_dups_size': 0, 'ondisk_log_size': 10096, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10073248, 'num_objects': 1230, 'num_object_clones': 0, 'num_object_copies': 4920, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1230, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11894, 'num_write_kb': 52500, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1104}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:44:49.598 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:44:49.598 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "39'11647", 'reported_seq': 11762, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:44.944929+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:44:44.944929+0000', 'last_peered': '2024-09-05T16:44:44.944929+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:44:44.944929+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "29'1600", 'ondisk_log_start': "29'1600", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10047, 'log_dups_size': 0, 'ondisk_log_size': 10047, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9712800, 'num_objects': 1186, 'num_object_clones': 0, 'num_object_copies': 4744, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2310, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1186, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11645, 'num_write_kb': 51325, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1122}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:44:49.598 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:44:49.598 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "39'11582", 'reported_seq': 11695, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:45.057221+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:44:45.057221+0000', 'last_peered': '2024-09-05T16:44:45.057221+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:44:45.057221+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "29'1500", 'ondisk_log_start': "29'1500", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10082, 'log_dups_size': 0, 'ondisk_log_size': 10082, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9770144, 'num_objects': 1193, 'num_object_clones': 0, 'num_object_copies': 4772, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1193, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11580, 'num_write_kb': 51092, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1027}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:44:49.600 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:44:50.349 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:44:50.350 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 28583/76880 objects degraded (37.179%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":693,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":39,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":19220,"data_bytes":157408584,"bytes_used":2831339520,"bytes_avail":383715717120,"bytes_total":386547056640,"degraded_objects":28583,"degraded_total":76880,"degraded_ratio":0.37178720083246619,"recovering_objects_per_sec":5,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":168},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:44:50.378 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:44:50.378 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:44:51.025 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:44:51.026 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:44:51.045 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":376,"stamp":"2024-09-05T16:44:49.051223+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":157408584,"num_objects":19220,"num_object_clones":0,"num_object_copies":76880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28583,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19220,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":160930,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":783,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160739,"ondisk_log_size":160739,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2764980,"kb_used_data":310568,"kb_used_omap":14,"kb_used_meta":2454321,"kb_avail":374722380,"statfs":{"total":386547056640,"available":383715717120,"internally_reserved":0,"allocated":318021632,"data_stored":317179308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2513225271},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,7,11,14,16,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16596992,"num_objects":-2026,"num_object_clones":0,"num_object_copies":-8104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-67,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2026,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2026,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004829"},"pg_stats":[{"pgid":"1.f","version":"39'11694","reported_seq":12808,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:46.214142+0000","last_change":"2024-09-05T16:43:43.436935+0000","last_active":"2024-09-05T16:44:46.214142+0000","last_peered":"2024-09-05T16:44:46.214142+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:44:46.214142+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"29'1600","ondisk_log_start":"29'1600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9912320,"num_objects":1210,"num_object_clones":0,"num_object_copies":4840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1732,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1210,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11694,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":530,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-522}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'11428","reported_seq":16084,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:44:45.603900+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:44:45.603900+0000","last_peered":"2024-09-05T16:44:45.603900+0000","last_clean":"2024-09-05T16:44:45.603900+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:44:45.603900+0000","last_undegraded":"2024-09-05T16:44:45.603900+0000","last_fullsized":"2024-09-05T16:44:45.603900+0000","mapping_epoch":38,"log_start":"29'1400","ondisk_log_start":"29'1400","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9207808,"num_objects":1124,"num_object_clones":0,"num_object_copies":4496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1124,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":741,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'11537","reported_seq":11664,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:44.920574+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:44:44.920574+0000","last_peered":"2024-09-05T16:44:44.920574+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:44:44.920574+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"29'1500","ondisk_log_start":"29'1500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9951848,"num_objects":1216,"num_object_clones":0,"num_object_copies":4864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1216,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11536,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1016}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'11707","reported_seq":16441,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:44:45.414218+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:44:45.414218+0000","last_peered":"2024-09-05T16:44:45.414218+0000","last_clean":"2024-09-05T16:44:45.414218+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:44:45.414218+0000","last_undegraded":"2024-09-05T16:44:45.414218+0000","last_fullsized":"2024-09-05T16:44:45.414218+0000","mapping_epoch":34,"log_start":"29'1700","ondisk_log_start":"29'1700","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10051584,"num_objects":1227,"num_object_clones":0,"num_object_copies":4908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1227,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6007,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'11416","reported_seq":11520,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:45.948158+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:44:45.948158+0000","last_peered":"2024-09-05T16:44:45.948158+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:44:45.948158+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"29'1400","ondisk_log_start":"29'1400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9655456,"num_objects":1179,"num_object_clones":0,"num_object_copies":4716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1179,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11414,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-966}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'11594","reported_seq":11713,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:46.070574+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:44:46.070574+0000","last_peered":"2024-09-05T16:44:46.070574+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:44:46.070574+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"29'1500","ondisk_log_start":"29'1500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9650176,"num_objects":1178,"num_object_clones":0,"num_object_copies":4712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1178,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11594,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1041}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'11418","reported_seq":11531,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:45.160023+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:44:45.160023+0000","last_peered":"2024-09-05T16:44:45.160023+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:44:45.160023+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"29'1400","ondisk_log_start":"29'1400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10187936,"num_objects":1244,"num_object_clones":0,"num_object_copies":4976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1244,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11416,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-887}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'11555","reported_seq":11649,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:45.129835+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:44:45.129835+0000","last_peered":"2024-09-05T16:44:45.129835+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:44:45.129835+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"29'1500","ondisk_log_start":"29'1500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10024096,"num_objects":1224,"num_object_clones":0,"num_object_copies":4896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1224,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11553,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1008}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"39'11632","reported_seq":11737,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:46.167592+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:44:46.167592+0000","last_peered":"2024-09-05T16:44:46.167592+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:44:46.167592+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"29'1600","ondisk_log_start":"29'1600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9349440,"num_objects":1142,"num_object_clones":0,"num_object_copies":4568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1142,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11629,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1102}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'11614","reported_seq":11741,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:44.771579+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:44:44.771579+0000","last_peered":"2024-09-05T16:44:44.771579+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:44:44.771579+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"29'1600","ondisk_log_start":"29'1600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9794720,"num_objects":1196,"num_object_clones":0,"num_object_copies":4784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1196,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11612,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1049}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'11866","reported_seq":11997,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:45.945448+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:44:45.945448+0000","last_peered":"2024-09-05T16:44:45.945448+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:44:45.945448+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"29'1800","ondisk_log_start":"29'1800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10292976,"num_objects":1257,"num_object_clones":0,"num_object_copies":5028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1257,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11863,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1024}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'11532","reported_seq":11635,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:46.214186+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:44:46.214186+0000","last_peered":"2024-09-05T16:44:46.214186+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:44:46.214186+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"29'1500","ondisk_log_start":"29'1500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10050128,"num_objects":1227,"num_object_clones":0,"num_object_copies":4908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1227,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11531,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1011}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'11621","reported_seq":16361,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:44:45.577601+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:44:45.577601+0000","last_peered":"2024-09-05T16:44:45.577601+0000","last_clean":"2024-09-05T16:44:45.577601+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:44:45.577601+0000","last_undegraded":"2024-09-05T16:44:45.577601+0000","last_fullsized":"2024-09-05T16:44:45.577601+0000","mapping_epoch":36,"log_start":"29'1600","ondisk_log_start":"29'1600","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9723904,"num_objects":1187,"num_object_clones":0,"num_object_copies":4748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1187,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3221,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'11896","reported_seq":12009,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:45.931927+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:44:45.931927+0000","last_peered":"2024-09-05T16:44:45.931927+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:44:45.931927+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"29'1800","ondisk_log_start":"29'1800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10073248,"num_objects":1230,"num_object_clones":0,"num_object_copies":4920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1230,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11894,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1104}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'11647","reported_seq":11762,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:44.944929+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:44:44.944929+0000","last_peered":"2024-09-05T16:44:44.944929+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:44:44.944929+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"29'1600","ondisk_log_start":"29'1600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9712800,"num_objects":1186,"num_object_clones":0,"num_object_copies":4744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1186,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11645,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1122}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'11582","reported_seq":11695,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:45.057221+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:44:45.057221+0000","last_peered":"2024-09-05T16:44:45.057221+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:44:45.057221+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"29'1500","ondisk_log_start":"29'1500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9770144,"num_objects":1193,"num_object_clones":0,"num_object_copies":4772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11580,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1027}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":157408584,"num_objects":19220,"num_object_clones":0,"num_object_copies":76880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28583,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19220,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":160930,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":783,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":315396096,"data_stored":315396096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160739,"ondisk_log_size":160739,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084417,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":718452,"kb_used_data":77280,"kb_used_omap":4,"kb_used_meta":641147,"kb_avail":93653388,"statfs":{"total":96636764160,"available":95901069312,"internally_reserved":0,"allocated":79134720,"data_stored":78924139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":656535251},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,2,2,2,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986294,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":596564,"kb_used_data":77572,"kb_used_omap":4,"kb_used_meta":518971,"kb_avail":93775276,"statfs":{"total":96636764160,"available":96025882624,"internally_reserved":0,"allocated":79433728,"data_stored":79223147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":531427026},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,6,6,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673100,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":727024,"kb_used_data":78048,"kb_used_omap":1,"kb_used_meta":648958,"kb_avail":93644816,"statfs":{"total":96636764160,"available":95892291584,"internally_reserved":0,"allocated":79921152,"data_stored":79710571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":664533445},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,5,2,8],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182531,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722940,"kb_used_data":77668,"kb_used_omap":4,"kb_used_meta":645243,"kb_avail":93648900,"statfs":{"total":96636764160,"available":95896473600,"internally_reserved":0,"allocated":79532032,"data_stored":79321451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":660729549},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,2,4,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":79122432,"data_stored":79122432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":78766080,"data_stored":78766080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":79278080,"data_stored":79278080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":78229504,"data_stored":78229504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:44:54.047 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:44:54.650 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:44:54.650 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:44:54.679 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":378,"stamp":"2024-09-05T16:44:53.052890+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":149331272,"num_objects":18234,"num_object_clones":0,"num_object_copies":72936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18234,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":161916,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":832,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161025,"ondisk_log_size":161025,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2815576,"kb_used_data":294536,"kb_used_omap":14,"kb_used_meta":2520945,"kb_avail":374671784,"statfs":{"total":386547056640,"available":383663906816,"internally_reserved":0,"allocated":301604864,"data_stored":300762540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2581448247},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,10,8,6,13,9,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19980288,"num_objects":-2439,"num_object_clones":0,"num_object_copies":-9756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2439,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2439,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004867"},"pg_stats":[{"pgid":"1.f","version":"39'11747","reported_seq":12963,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:51.229036+0000","last_change":"2024-09-05T16:43:43.436935+0000","last_active":"2024-09-05T16:44:51.229036+0000","last_peered":"2024-09-05T16:44:51.229036+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:44:51.229036+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"29'1700","ondisk_log_start":"29'1700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9478144,"num_objects":1157,"num_object_clones":0,"num_object_copies":4628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1683,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1157,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11747,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":579,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-526}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'11479","reported_seq":16135,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:44:50.688484+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:44:50.688484+0000","last_peered":"2024-09-05T16:44:50.688484+0000","last_clean":"2024-09-05T16:44:50.688484+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:44:50.688484+0000","last_undegraded":"2024-09-05T16:44:50.688484+0000","last_fullsized":"2024-09-05T16:44:50.688484+0000","mapping_epoch":38,"log_start":"29'1400","ondisk_log_start":"29'1400","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8790016,"num_objects":1073,"num_object_clones":0,"num_object_copies":4292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1073,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":792,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'11596","reported_seq":11723,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:49.847173+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:44:49.847173+0000","last_peered":"2024-09-05T16:44:49.847173+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:44:49.847173+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"29'1500","ondisk_log_start":"29'1500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9468520,"num_objects":1157,"num_object_clones":0,"num_object_copies":4628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1157,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11595,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1075}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'11783","reported_seq":16516,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:44:50.713803+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:44:50.713803+0000","last_peered":"2024-09-05T16:44:50.713803+0000","last_clean":"2024-09-05T16:44:50.713803+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:44:50.713803+0000","last_undegraded":"2024-09-05T16:44:50.713803+0000","last_fullsized":"2024-09-05T16:44:50.713803+0000","mapping_epoch":34,"log_start":"29'1700","ondisk_log_start":"29'1700","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9428992,"num_objects":1151,"num_object_clones":0,"num_object_copies":4604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1151,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6083,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'11484","reported_seq":11588,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:51.031032+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:44:51.031032+0000","last_peered":"2024-09-05T16:44:51.031032+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:44:51.031032+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"29'1400","ondisk_log_start":"29'1400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9098400,"num_objects":1111,"num_object_clones":0,"num_object_copies":4444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1111,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11482,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1034}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'11653","reported_seq":11772,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:51.099569+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:44:51.099569+0000","last_peered":"2024-09-05T16:44:51.099569+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:44:51.099569+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"29'1600","ondisk_log_start":"29'1600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9166848,"num_objects":1119,"num_object_clones":0,"num_object_copies":4476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1119,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11653,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1100}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'11481","reported_seq":11594,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:50.036882+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:44:50.036882+0000","last_peered":"2024-09-05T16:44:50.036882+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:44:50.036882+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"29'1400","ondisk_log_start":"29'1400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9671840,"num_objects":1181,"num_object_clones":0,"num_object_copies":4724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1181,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11479,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-950}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'11615","reported_seq":11709,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:50.165739+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:44:50.165739+0000","last_peered":"2024-09-05T16:44:50.165739+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:44:50.165739+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"29'1600","ondisk_log_start":"29'1600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9532576,"num_objects":1164,"num_object_clones":0,"num_object_copies":4656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1164,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11613,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1068}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"39'11691","reported_seq":11797,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:50.965818+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:44:50.965818+0000","last_peered":"2024-09-05T16:44:50.965818+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:44:50.965818+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"29'1600","ondisk_log_start":"29'1600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8866112,"num_objects":1083,"num_object_clones":0,"num_object_copies":4332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1083,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11688,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1161}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'11687","reported_seq":11814,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:49.983568+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:44:49.983568+0000","last_peered":"2024-09-05T16:44:49.983568+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:44:49.983568+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"29'1600","ondisk_log_start":"29'1600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9196704,"num_objects":1123,"num_object_clones":0,"num_object_copies":4492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1123,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11685,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1122}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'11933","reported_seq":12065,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:51.241477+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:44:51.241477+0000","last_peered":"2024-09-05T16:44:51.241477+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:44:51.241477+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"29'1900","ondisk_log_start":"29'1900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9744112,"num_objects":1190,"num_object_clones":0,"num_object_copies":4760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1190,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11930,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1091}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'11580","reported_seq":11683,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:50.668438+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:44:50.668438+0000","last_peered":"2024-09-05T16:44:50.668438+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:44:50.668438+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"29'1500","ondisk_log_start":"29'1500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9656912,"num_objects":1179,"num_object_clones":0,"num_object_copies":4716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1179,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11579,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1059}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'11687","reported_seq":16427,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:44:50.775566+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:44:50.775566+0000","last_peered":"2024-09-05T16:44:50.775566+0000","last_clean":"2024-09-05T16:44:50.775566+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:44:50.775566+0000","last_undegraded":"2024-09-05T16:44:50.775566+0000","last_fullsized":"2024-09-05T16:44:50.775566+0000","mapping_epoch":36,"log_start":"29'1600","ondisk_log_start":"29'1600","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9183232,"num_objects":1121,"num_object_clones":0,"num_object_copies":4484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1121,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3287,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'11965","reported_seq":12078,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:51.160513+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:44:51.160513+0000","last_peered":"2024-09-05T16:44:51.160513+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:44:51.160513+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"29'1900","ondisk_log_start":"29'1900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9508000,"num_objects":1161,"num_object_clones":0,"num_object_copies":4644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1161,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11963,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1173}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'11710","reported_seq":11827,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:49.750097+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:44:49.750097+0000","last_peered":"2024-09-05T16:44:49.750097+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:44:49.750097+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"29'1700","ondisk_log_start":"29'1700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9196704,"num_objects":1123,"num_object_clones":0,"num_object_copies":4492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1123,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11708,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1185}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'11634","reported_seq":11747,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:50.186116+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:44:50.186116+0000","last_peered":"2024-09-05T16:44:50.186116+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:44:50.186116+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"29'1600","ondisk_log_start":"29'1600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9344160,"num_objects":1141,"num_object_clones":0,"num_object_copies":4564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1141,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11632,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1079}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":149331272,"num_objects":18234,"num_object_clones":0,"num_object_copies":72936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18234,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":161916,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":832,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":299102208,"data_stored":299102208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161025,"ondisk_log_size":161025,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084418,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":736656,"kb_used_data":73272,"kb_used_omap":4,"kb_used_meta":663355,"kb_avail":93635184,"statfs":{"total":96636764160,"available":95882428416,"internally_reserved":0,"allocated":75030528,"data_stored":74819947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":679276243},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,2,2,2,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986295,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":596756,"kb_used_data":73604,"kb_used_omap":4,"kb_used_meta":523131,"kb_avail":93775084,"statfs":{"total":96636764160,"available":96025686016,"internally_reserved":0,"allocated":75370496,"data_stored":75159915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":535686866},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,3,1,2,6,2,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673101,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":741024,"kb_used_data":73996,"kb_used_omap":1,"kb_used_meta":667006,"kb_avail":93630816,"statfs":{"total":96636764160,"available":95877955584,"internally_reserved":0,"allocated":75771904,"data_stored":75561323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":683014597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,1,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182532,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":741140,"kb_used_data":73664,"kb_used_omap":4,"kb_used_meta":667451,"kb_avail":93630700,"statfs":{"total":96636764160,"available":95877836800,"internally_reserved":0,"allocated":75431936,"data_stored":75221355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":683470541},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,0,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":75022336,"data_stored":75022336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":74661888,"data_stored":74661888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":75194368,"data_stored":75194368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":74223616,"data_stored":74223616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:44:54.680 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:44:55.285 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:44:55.286 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:44:55.310 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":379,"stamp":"2024-09-05T16:44:55.053729+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":149331272,"num_objects":18234,"num_object_clones":0,"num_object_copies":72936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18234,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":161916,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":832,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161025,"ondisk_log_size":161025,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2815576,"kb_used_data":294536,"kb_used_omap":14,"kb_used_meta":2520945,"kb_avail":374671784,"statfs":{"total":386547056640,"available":383663906816,"internally_reserved":0,"allocated":301604864,"data_stored":300762540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2581448247},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,10,8,6,13,9,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16424960,"num_objects":-2005,"num_object_clones":0,"num_object_copies":-8020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-80,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2005,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2005,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":80,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005013"},"pg_stats":[{"pgid":"1.f","version":"39'11747","reported_seq":12963,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:51.229036+0000","last_change":"2024-09-05T16:43:43.436935+0000","last_active":"2024-09-05T16:44:51.229036+0000","last_peered":"2024-09-05T16:44:51.229036+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:44:51.229036+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"29'1700","ondisk_log_start":"29'1700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9478144,"num_objects":1157,"num_object_clones":0,"num_object_copies":4628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1683,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1157,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11747,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":579,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-526}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'11479","reported_seq":16135,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:44:50.688484+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:44:50.688484+0000","last_peered":"2024-09-05T16:44:50.688484+0000","last_clean":"2024-09-05T16:44:50.688484+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:44:50.688484+0000","last_undegraded":"2024-09-05T16:44:50.688484+0000","last_fullsized":"2024-09-05T16:44:50.688484+0000","mapping_epoch":38,"log_start":"29'1400","ondisk_log_start":"29'1400","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8790016,"num_objects":1073,"num_object_clones":0,"num_object_copies":4292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1073,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":792,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'11596","reported_seq":11723,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:49.847173+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:44:49.847173+0000","last_peered":"2024-09-05T16:44:49.847173+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:44:49.847173+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"29'1500","ondisk_log_start":"29'1500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9468520,"num_objects":1157,"num_object_clones":0,"num_object_copies":4628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1157,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11595,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1075}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'11783","reported_seq":16516,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:44:50.713803+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:44:50.713803+0000","last_peered":"2024-09-05T16:44:50.713803+0000","last_clean":"2024-09-05T16:44:50.713803+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:44:50.713803+0000","last_undegraded":"2024-09-05T16:44:50.713803+0000","last_fullsized":"2024-09-05T16:44:50.713803+0000","mapping_epoch":34,"log_start":"29'1700","ondisk_log_start":"29'1700","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9428992,"num_objects":1151,"num_object_clones":0,"num_object_copies":4604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1151,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6083,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'11484","reported_seq":11588,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:51.031032+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:44:51.031032+0000","last_peered":"2024-09-05T16:44:51.031032+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:44:51.031032+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"29'1400","ondisk_log_start":"29'1400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9098400,"num_objects":1111,"num_object_clones":0,"num_object_copies":4444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1111,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11482,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1034}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'11653","reported_seq":11772,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:51.099569+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:44:51.099569+0000","last_peered":"2024-09-05T16:44:51.099569+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:44:51.099569+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"29'1600","ondisk_log_start":"29'1600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9166848,"num_objects":1119,"num_object_clones":0,"num_object_copies":4476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1119,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11653,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1100}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'11481","reported_seq":11594,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:50.036882+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:44:50.036882+0000","last_peered":"2024-09-05T16:44:50.036882+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:44:50.036882+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"29'1400","ondisk_log_start":"29'1400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9671840,"num_objects":1181,"num_object_clones":0,"num_object_copies":4724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1181,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11479,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-950}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'11615","reported_seq":11709,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:50.165739+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:44:50.165739+0000","last_peered":"2024-09-05T16:44:50.165739+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:44:50.165739+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"29'1600","ondisk_log_start":"29'1600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9532576,"num_objects":1164,"num_object_clones":0,"num_object_copies":4656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1164,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11613,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1068}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"39'11691","reported_seq":11797,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:50.965818+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:44:50.965818+0000","last_peered":"2024-09-05T16:44:50.965818+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:44:50.965818+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"29'1600","ondisk_log_start":"29'1600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8866112,"num_objects":1083,"num_object_clones":0,"num_object_copies":4332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1083,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11688,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1161}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'11687","reported_seq":11814,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:49.983568+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:44:49.983568+0000","last_peered":"2024-09-05T16:44:49.983568+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:44:49.983568+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"29'1600","ondisk_log_start":"29'1600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9196704,"num_objects":1123,"num_object_clones":0,"num_object_copies":4492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1123,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11685,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1122}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'11933","reported_seq":12065,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:51.241477+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:44:51.241477+0000","last_peered":"2024-09-05T16:44:51.241477+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:44:51.241477+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"29'1900","ondisk_log_start":"29'1900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9744112,"num_objects":1190,"num_object_clones":0,"num_object_copies":4760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1190,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11930,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1091}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'11580","reported_seq":11683,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:50.668438+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:44:50.668438+0000","last_peered":"2024-09-05T16:44:50.668438+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:44:50.668438+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"29'1500","ondisk_log_start":"29'1500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9656912,"num_objects":1179,"num_object_clones":0,"num_object_copies":4716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1179,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11579,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1059}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'11687","reported_seq":16427,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:44:50.775566+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:44:50.775566+0000","last_peered":"2024-09-05T16:44:50.775566+0000","last_clean":"2024-09-05T16:44:50.775566+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:44:50.775566+0000","last_undegraded":"2024-09-05T16:44:50.775566+0000","last_fullsized":"2024-09-05T16:44:50.775566+0000","mapping_epoch":36,"log_start":"29'1600","ondisk_log_start":"29'1600","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9183232,"num_objects":1121,"num_object_clones":0,"num_object_copies":4484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1121,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3287,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'11965","reported_seq":12078,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:51.160513+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:44:51.160513+0000","last_peered":"2024-09-05T16:44:51.160513+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:44:51.160513+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"29'1900","ondisk_log_start":"29'1900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9508000,"num_objects":1161,"num_object_clones":0,"num_object_copies":4644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1161,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11963,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1173}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'11710","reported_seq":11827,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:49.750097+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:44:49.750097+0000","last_peered":"2024-09-05T16:44:49.750097+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:44:49.750097+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"29'1700","ondisk_log_start":"29'1700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9196704,"num_objects":1123,"num_object_clones":0,"num_object_copies":4492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1123,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11708,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1185}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'11634","reported_seq":11747,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:50.186116+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:44:50.186116+0000","last_peered":"2024-09-05T16:44:50.186116+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:44:50.186116+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"29'1600","ondisk_log_start":"29'1600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9344160,"num_objects":1141,"num_object_clones":0,"num_object_copies":4564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1141,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11632,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1079}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":149331272,"num_objects":18234,"num_object_clones":0,"num_object_copies":72936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18234,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":161916,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":832,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":299102208,"data_stored":299102208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161025,"ondisk_log_size":161025,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084418,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":736656,"kb_used_data":73272,"kb_used_omap":4,"kb_used_meta":663355,"kb_avail":93635184,"statfs":{"total":96636764160,"available":95882428416,"internally_reserved":0,"allocated":75030528,"data_stored":74819947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":679276243},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,2,2,2,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986295,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":596756,"kb_used_data":73604,"kb_used_omap":4,"kb_used_meta":523131,"kb_avail":93775084,"statfs":{"total":96636764160,"available":96025686016,"internally_reserved":0,"allocated":75370496,"data_stored":75159915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":535686866},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,3,1,2,6,2,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673101,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":741024,"kb_used_data":73996,"kb_used_omap":1,"kb_used_meta":667006,"kb_avail":93630816,"statfs":{"total":96636764160,"available":95877955584,"internally_reserved":0,"allocated":75771904,"data_stored":75561323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":683014597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,1,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182532,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":741140,"kb_used_data":73664,"kb_used_omap":4,"kb_used_meta":667451,"kb_avail":93630700,"statfs":{"total":96636764160,"available":95877836800,"internally_reserved":0,"allocated":75431936,"data_stored":75221355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":683470541},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,0,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":75022336,"data_stored":75022336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":74661888,"data_stored":74661888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":75194368,"data_stored":75194368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":74223616,"data_stored":74223616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:44:55.311 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-05T16:44:55.311 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "39'11747", 'reported_seq': 12963, 'reported_epoch': 39, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:51.229036+0000', 'last_change': '2024-09-05T16:43:43.436935+0000', 'last_active': '2024-09-05T16:44:51.229036+0000', 'last_peered': '2024-09-05T16:44:51.229036+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.828289+0000', 'last_became_peered': '2024-09-05T16:34:56.828289+0000', 'last_unstale': '2024-09-05T16:44:51.229036+0000', 'last_undegraded': '2024-09-05T16:34:56.612160+0000', 'last_fullsized': '2024-09-05T16:34:56.557632+0000', 'mapping_epoch': 32, 'log_start': "29'1700", 'ondisk_log_start': "29'1700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10047, 'log_dups_size': 0, 'ondisk_log_size': 10047, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9478144, 'num_objects': 1157, 'num_object_clones': 0, 'num_object_copies': 4628, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1683, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1157, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11747, 'num_write_kb': 51616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 579, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -526}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:44:55.311 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:44:55.311 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "39'11596", 'reported_seq': 11723, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:49.847173+0000', 'last_change': '2024-09-05T16:34:56.867900+0000', 'last_active': '2024-09-05T16:44:49.847173+0000', 'last_peered': '2024-09-05T16:44:49.847173+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:44:49.847173+0000', 'last_undegraded': '2024-09-05T16:34:56.590972+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 32, 'log_start': "29'1500", 'ondisk_log_start': "29'1500", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10096, 'log_dups_size': 0, 'ondisk_log_size': 10096, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9468520, 'num_objects': 1157, 'num_object_clones': 0, 'num_object_copies': 4628, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2233, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1157, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 11595, 'num_write_kb': 50999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1075}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:44:55.311 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:44:55.311 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "39'11484", 'reported_seq': 11588, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:51.031032+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:44:51.031032+0000', 'last_peered': '2024-09-05T16:44:51.031032+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:44:51.031032+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "29'1400", 'ondisk_log_start': "29'1400", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10084, 'log_dups_size': 0, 'ondisk_log_size': 10084, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9098400, 'num_objects': 1111, 'num_object_clones': 0, 'num_object_copies': 4444, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2147, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1111, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11482, 'num_write_kb': 50373, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1034}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:44:55.312 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:44:55.312 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "39'11653", 'reported_seq': 11772, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:51.099569+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:44:51.099569+0000', 'last_peered': '2024-09-05T16:44:51.099569+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:44:51.099569+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "29'1600", 'ondisk_log_start': "29'1600", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10053, 'log_dups_size': 0, 'ondisk_log_size': 10053, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9166848, 'num_objects': 1119, 'num_object_clones': 0, 'num_object_copies': 4476, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1119, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11653, 'num_write_kb': 51088, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1100}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:44:55.312 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:44:55.312 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "39'11481", 'reported_seq': 11594, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:50.036882+0000', 'last_change': '2024-09-05T16:34:56.903436+0000', 'last_active': '2024-09-05T16:44:50.036882+0000', 'last_peered': '2024-09-05T16:44:50.036882+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:44:50.036882+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "29'1400", 'ondisk_log_start': "29'1400", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10081, 'log_dups_size': 0, 'ondisk_log_size': 10081, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9671840, 'num_objects': 1181, 'num_object_clones': 0, 'num_object_copies': 4724, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2133, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1181, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11479, 'num_write_kb': 50636, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -950}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:44:55.312 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-05T16:44:55.312 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "39'11615", 'reported_seq': 11709, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:50.165739+0000', 'last_change': '2024-09-05T16:34:56.883319+0000', 'last_active': '2024-09-05T16:44:50.165739+0000', 'last_peered': '2024-09-05T16:44:50.165739+0000', 'last_clean': '2024-09-05T16:33:53.505305+0000', 'last_became_active': '2024-09-05T16:34:56.873389+0000', 'last_became_peered': '2024-09-05T16:34:56.873389+0000', 'last_unstale': '2024-09-05T16:44:50.165739+0000', 'last_undegraded': '2024-09-05T16:34:56.602567+0000', 'last_fullsized': '2024-09-05T16:34:56.559958+0000', 'mapping_epoch': 32, 'log_start': "29'1600", 'ondisk_log_start': "29'1600", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10015, 'log_dups_size': 0, 'ondisk_log_size': 10015, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9532576, 'num_objects': 1164, 'num_object_clones': 0, 'num_object_copies': 4656, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2234, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1164, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11613, 'num_write_kb': 51108, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1068}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:44:55.312 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:44:55.312 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "39'11691", 'reported_seq': 11797, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:50.965818+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:44:50.965818+0000', 'last_peered': '2024-09-05T16:44:50.965818+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:44:50.965818+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "29'1600", 'ondisk_log_start': "29'1600", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10091, 'log_dups_size': 0, 'ondisk_log_size': 10091, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8866112, 'num_objects': 1083, 'num_object_clones': 0, 'num_object_copies': 4332, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1083, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11688, 'num_write_kb': 51081, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1161}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:44:55.312 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:44:55.312 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "39'11687", 'reported_seq': 11814, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:49.983568+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:44:49.983568+0000', 'last_peered': '2024-09-05T16:44:49.983568+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:44:49.983568+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "29'1600", 'ondisk_log_start': "29'1600", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10087, 'log_dups_size': 0, 'ondisk_log_size': 10087, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9196704, 'num_objects': 1123, 'num_object_clones': 0, 'num_object_copies': 4492, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1123, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11685, 'num_write_kb': 51230, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1122}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:44:55.312 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:44:55.312 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "39'11933", 'reported_seq': 12065, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:51.241477+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:44:51.241477+0000', 'last_peered': '2024-09-05T16:44:51.241477+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:44:51.241477+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "29'1900", 'ondisk_log_start': "29'1900", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10033, 'log_dups_size': 0, 'ondisk_log_size': 10033, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9744112, 'num_objects': 1190, 'num_object_clones': 0, 'num_object_copies': 4760, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2284, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1190, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11930, 'num_write_kb': 52476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1091}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:44:55.313 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:44:55.313 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "39'11580", 'reported_seq': 11683, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:50.668438+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:44:50.668438+0000', 'last_peered': '2024-09-05T16:44:50.668438+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:44:50.668438+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "29'1500", 'ondisk_log_start': "29'1500", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9656912, 'num_objects': 1179, 'num_object_clones': 0, 'num_object_copies': 4716, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1179, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11579, 'num_write_kb': 51031, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1059}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:44:55.313 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:44:55.313 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "39'11965", 'reported_seq': 12078, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:51.160513+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:44:51.160513+0000', 'last_peered': '2024-09-05T16:44:51.160513+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:44:51.160513+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "29'1900", 'ondisk_log_start': "29'1900", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10065, 'log_dups_size': 0, 'ondisk_log_size': 10065, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9508000, 'num_objects': 1161, 'num_object_clones': 0, 'num_object_copies': 4644, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1161, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11963, 'num_write_kb': 52500, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1173}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:44:55.313 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:44:55.313 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "39'11710", 'reported_seq': 11827, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:49.750097+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:44:49.750097+0000', 'last_peered': '2024-09-05T16:44:49.750097+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:44:49.750097+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "29'1700", 'ondisk_log_start': "29'1700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10010, 'log_dups_size': 0, 'ondisk_log_size': 10010, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9196704, 'num_objects': 1123, 'num_object_clones': 0, 'num_object_copies': 4492, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2310, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1123, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11708, 'num_write_kb': 51325, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1185}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:44:55.313 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:44:55.313 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "39'11634", 'reported_seq': 11747, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:50.186116+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:44:50.186116+0000', 'last_peered': '2024-09-05T16:44:50.186116+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:44:50.186116+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "29'1600", 'ondisk_log_start': "29'1600", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10034, 'log_dups_size': 0, 'ondisk_log_size': 10034, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9344160, 'num_objects': 1141, 'num_object_clones': 0, 'num_object_copies': 4564, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1141, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11632, 'num_write_kb': 51092, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1079}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:44:55.313 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:44:56.098 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:44:56.099 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 28534/72936 objects degraded (39.122%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":699,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":39,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":18234,"data_bytes":149331272,"bytes_used":2883149824,"bytes_avail":383663906816,"bytes_total":386547056640,"degraded_objects":28534,"degraded_total":72936,"degraded_ratio":0.39121969946254248,"recovering_objects_per_sec":6,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":80,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":167},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:44:56.135 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:44:56.135 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:44:56.723 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:44:56.723 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:44:56.767 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":379,"stamp":"2024-09-05T16:44:55.053729+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":149331272,"num_objects":18234,"num_object_clones":0,"num_object_copies":72936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18234,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":161916,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":832,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161025,"ondisk_log_size":161025,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2815576,"kb_used_data":294536,"kb_used_omap":14,"kb_used_meta":2520945,"kb_avail":374671784,"statfs":{"total":386547056640,"available":383663906816,"internally_reserved":0,"allocated":301604864,"data_stored":300762540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2581448247},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,10,8,6,13,9,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16424960,"num_objects":-2005,"num_object_clones":0,"num_object_copies":-8020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-80,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2005,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2005,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":80,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005013"},"pg_stats":[{"pgid":"1.f","version":"39'11747","reported_seq":12963,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:51.229036+0000","last_change":"2024-09-05T16:43:43.436935+0000","last_active":"2024-09-05T16:44:51.229036+0000","last_peered":"2024-09-05T16:44:51.229036+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:44:51.229036+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"29'1700","ondisk_log_start":"29'1700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9478144,"num_objects":1157,"num_object_clones":0,"num_object_copies":4628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1683,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1157,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11747,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":579,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-526}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'11479","reported_seq":16135,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:44:50.688484+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:44:50.688484+0000","last_peered":"2024-09-05T16:44:50.688484+0000","last_clean":"2024-09-05T16:44:50.688484+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:44:50.688484+0000","last_undegraded":"2024-09-05T16:44:50.688484+0000","last_fullsized":"2024-09-05T16:44:50.688484+0000","mapping_epoch":38,"log_start":"29'1400","ondisk_log_start":"29'1400","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8790016,"num_objects":1073,"num_object_clones":0,"num_object_copies":4292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1073,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":792,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'11596","reported_seq":11723,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:49.847173+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:44:49.847173+0000","last_peered":"2024-09-05T16:44:49.847173+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:44:49.847173+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"29'1500","ondisk_log_start":"29'1500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9468520,"num_objects":1157,"num_object_clones":0,"num_object_copies":4628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1157,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11595,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1075}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'11783","reported_seq":16516,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:44:50.713803+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:44:50.713803+0000","last_peered":"2024-09-05T16:44:50.713803+0000","last_clean":"2024-09-05T16:44:50.713803+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:44:50.713803+0000","last_undegraded":"2024-09-05T16:44:50.713803+0000","last_fullsized":"2024-09-05T16:44:50.713803+0000","mapping_epoch":34,"log_start":"29'1700","ondisk_log_start":"29'1700","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9428992,"num_objects":1151,"num_object_clones":0,"num_object_copies":4604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1151,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6083,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'11484","reported_seq":11588,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:51.031032+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:44:51.031032+0000","last_peered":"2024-09-05T16:44:51.031032+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:44:51.031032+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"29'1400","ondisk_log_start":"29'1400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9098400,"num_objects":1111,"num_object_clones":0,"num_object_copies":4444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1111,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11482,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1034}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'11653","reported_seq":11772,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:51.099569+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:44:51.099569+0000","last_peered":"2024-09-05T16:44:51.099569+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:44:51.099569+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"29'1600","ondisk_log_start":"29'1600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9166848,"num_objects":1119,"num_object_clones":0,"num_object_copies":4476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1119,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11653,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1100}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'11481","reported_seq":11594,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:50.036882+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:44:50.036882+0000","last_peered":"2024-09-05T16:44:50.036882+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:44:50.036882+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"29'1400","ondisk_log_start":"29'1400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9671840,"num_objects":1181,"num_object_clones":0,"num_object_copies":4724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1181,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11479,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-950}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'11615","reported_seq":11709,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:50.165739+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:44:50.165739+0000","last_peered":"2024-09-05T16:44:50.165739+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:44:50.165739+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"29'1600","ondisk_log_start":"29'1600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9532576,"num_objects":1164,"num_object_clones":0,"num_object_copies":4656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1164,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11613,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1068}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"39'11691","reported_seq":11797,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:50.965818+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:44:50.965818+0000","last_peered":"2024-09-05T16:44:50.965818+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:44:50.965818+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"29'1600","ondisk_log_start":"29'1600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8866112,"num_objects":1083,"num_object_clones":0,"num_object_copies":4332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1083,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11688,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1161}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'11687","reported_seq":11814,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:49.983568+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:44:49.983568+0000","last_peered":"2024-09-05T16:44:49.983568+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:44:49.983568+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"29'1600","ondisk_log_start":"29'1600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9196704,"num_objects":1123,"num_object_clones":0,"num_object_copies":4492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1123,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11685,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1122}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'11933","reported_seq":12065,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:51.241477+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:44:51.241477+0000","last_peered":"2024-09-05T16:44:51.241477+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:44:51.241477+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"29'1900","ondisk_log_start":"29'1900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9744112,"num_objects":1190,"num_object_clones":0,"num_object_copies":4760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1190,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11930,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1091}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'11580","reported_seq":11683,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:50.668438+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:44:50.668438+0000","last_peered":"2024-09-05T16:44:50.668438+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:44:50.668438+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"29'1500","ondisk_log_start":"29'1500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9656912,"num_objects":1179,"num_object_clones":0,"num_object_copies":4716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1179,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11579,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1059}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'11687","reported_seq":16427,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:44:50.775566+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:44:50.775566+0000","last_peered":"2024-09-05T16:44:50.775566+0000","last_clean":"2024-09-05T16:44:50.775566+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:44:50.775566+0000","last_undegraded":"2024-09-05T16:44:50.775566+0000","last_fullsized":"2024-09-05T16:44:50.775566+0000","mapping_epoch":36,"log_start":"29'1600","ondisk_log_start":"29'1600","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9183232,"num_objects":1121,"num_object_clones":0,"num_object_copies":4484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1121,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3287,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'11965","reported_seq":12078,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:51.160513+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:44:51.160513+0000","last_peered":"2024-09-05T16:44:51.160513+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:44:51.160513+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"29'1900","ondisk_log_start":"29'1900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9508000,"num_objects":1161,"num_object_clones":0,"num_object_copies":4644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1161,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11963,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1173}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'11710","reported_seq":11827,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:49.750097+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:44:49.750097+0000","last_peered":"2024-09-05T16:44:49.750097+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:44:49.750097+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"29'1700","ondisk_log_start":"29'1700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9196704,"num_objects":1123,"num_object_clones":0,"num_object_copies":4492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1123,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11708,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1185}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'11634","reported_seq":11747,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:50.186116+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:44:50.186116+0000","last_peered":"2024-09-05T16:44:50.186116+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:44:50.186116+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"29'1600","ondisk_log_start":"29'1600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9344160,"num_objects":1141,"num_object_clones":0,"num_object_copies":4564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1141,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11632,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1079}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":149331272,"num_objects":18234,"num_object_clones":0,"num_object_copies":72936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18234,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":161916,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":832,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":299102208,"data_stored":299102208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161025,"ondisk_log_size":161025,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084418,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":736656,"kb_used_data":73272,"kb_used_omap":4,"kb_used_meta":663355,"kb_avail":93635184,"statfs":{"total":96636764160,"available":95882428416,"internally_reserved":0,"allocated":75030528,"data_stored":74819947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":679276243},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,2,2,2,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986295,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":596756,"kb_used_data":73604,"kb_used_omap":4,"kb_used_meta":523131,"kb_avail":93775084,"statfs":{"total":96636764160,"available":96025686016,"internally_reserved":0,"allocated":75370496,"data_stored":75159915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":535686866},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,3,1,2,6,2,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673101,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":741024,"kb_used_data":73996,"kb_used_omap":1,"kb_used_meta":667006,"kb_avail":93630816,"statfs":{"total":96636764160,"available":95877955584,"internally_reserved":0,"allocated":75771904,"data_stored":75561323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":683014597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,1,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182532,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":741140,"kb_used_data":73664,"kb_used_omap":4,"kb_used_meta":667451,"kb_avail":93630700,"statfs":{"total":96636764160,"available":95877836800,"internally_reserved":0,"allocated":75431936,"data_stored":75221355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":683470541},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,0,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":75022336,"data_stored":75022336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":74661888,"data_stored":74661888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":75194368,"data_stored":75194368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":74223616,"data_stored":74223616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:44:59.769 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:45:00.398 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:45:00.399 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:45:00.439 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":381,"stamp":"2024-09-05T16:44:59.055451+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":141090120,"num_objects":17228,"num_object_clones":0,"num_object_copies":68912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28498,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17228,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":162922,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":868,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160831,"ondisk_log_size":160831,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2816272,"kb_used_data":278608,"kb_used_omap":14,"kb_used_meta":2537585,"kb_avail":374671088,"statfs":{"total":386547056640,"available":383663194112,"internally_reserved":0,"allocated":285294592,"data_stored":284452268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2598487607},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,15,2,11,10,7,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16318464,"num_objects":-1992,"num_object_clones":0,"num_object_copies":-7968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-85,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1992,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1992,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005225"},"pg_stats":[{"pgid":"1.f","version":"39'11819","reported_seq":13106,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:56.208591+0000","last_change":"2024-09-05T16:43:43.436935+0000","last_active":"2024-09-05T16:44:56.208591+0000","last_peered":"2024-09-05T16:44:56.208591+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:44:56.208591+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"29'1800","ondisk_log_start":"29'1800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8888320,"num_objects":1085,"num_object_clones":0,"num_object_copies":4340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1647,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1085,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11819,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":615,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-562}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'11557","reported_seq":16213,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:44:55.883996+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:44:55.883996+0000","last_peered":"2024-09-05T16:44:55.883996+0000","last_clean":"2024-09-05T16:44:55.883996+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:44:55.883996+0000","last_undegraded":"2024-09-05T16:44:55.883996+0000","last_fullsized":"2024-09-05T16:44:55.883996+0000","mapping_epoch":38,"log_start":"29'1500","ondisk_log_start":"29'1500","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8151040,"num_objects":995,"num_object_clones":0,"num_object_copies":3980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":995,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":870,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'11662","reported_seq":11789,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:54.929941+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:44:54.929941+0000","last_peered":"2024-09-05T16:44:54.929941+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:44:54.929941+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"29'1600","ondisk_log_start":"29'1600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8927848,"num_objects":1091,"num_object_clones":0,"num_object_copies":4364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1091,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11661,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1141}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'11859","reported_seq":16593,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:44:56.029319+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:44:56.029319+0000","last_peered":"2024-09-05T16:44:56.029319+0000","last_clean":"2024-09-05T16:44:56.029319+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:44:56.029319+0000","last_undegraded":"2024-09-05T16:44:56.029319+0000","last_fullsized":"2024-09-05T16:44:56.029319+0000","mapping_epoch":34,"log_start":"29'1800","ondisk_log_start":"29'1800","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8806400,"num_objects":1075,"num_object_clones":0,"num_object_copies":4300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1075,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6159,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'11542","reported_seq":11646,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:56.229366+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:44:56.229366+0000","last_peered":"2024-09-05T16:44:56.229366+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:44:56.229366+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"29'1500","ondisk_log_start":"29'1500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8623264,"num_objects":1053,"num_object_clones":0,"num_object_copies":4212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1053,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11540,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1092}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'11711","reported_seq":11830,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:56.134609+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:44:56.134609+0000","last_peered":"2024-09-05T16:44:56.134609+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:44:56.134609+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"29'1700","ondisk_log_start":"29'1700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8691712,"num_objects":1061,"num_object_clones":0,"num_object_copies":4244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1061,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11711,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1158}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'11527","reported_seq":11640,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:55.100499+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:44:55.100499+0000","last_peered":"2024-09-05T16:44:55.100499+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:44:55.100499+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"29'1500","ondisk_log_start":"29'1500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9295008,"num_objects":1135,"num_object_clones":0,"num_object_copies":4540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1135,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11525,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-996}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'11668","reported_seq":11762,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:55.109381+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:44:55.109381+0000","last_peered":"2024-09-05T16:44:55.109381+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:44:55.109381+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"29'1600","ondisk_log_start":"29'1600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9098400,"num_objects":1111,"num_object_clones":0,"num_object_copies":4444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1111,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11666,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1121}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"39'11745","reported_seq":11848,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:56.157164+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:44:56.157164+0000","last_peered":"2024-09-05T16:44:56.157164+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:44:56.157164+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"29'1700","ondisk_log_start":"29'1700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8423744,"num_objects":1029,"num_object_clones":0,"num_object_copies":4116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1029,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11742,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1215}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'11744","reported_seq":11871,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:54.924945+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:44:54.924945+0000","last_peered":"2024-09-05T16:44:54.924945+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:44:54.924945+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"29'1700","ondisk_log_start":"29'1700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8729760,"num_objects":1066,"num_object_clones":0,"num_object_copies":4264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1066,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11742,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1179}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'11997","reported_seq":12129,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:56.232325+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:44:56.232325+0000","last_peered":"2024-09-05T16:44:56.232325+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:44:56.232325+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"29'1900","ondisk_log_start":"29'1900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9219824,"num_objects":1126,"num_object_clones":0,"num_object_copies":4504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1126,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11994,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1155}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'11645","reported_seq":11748,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:56.175947+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:44:56.175947+0000","last_peered":"2024-09-05T16:44:56.175947+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:44:56.175947+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"29'1600","ondisk_log_start":"29'1600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9124432,"num_objects":1114,"num_object_clones":0,"num_object_copies":4456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1114,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11644,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1124}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'11751","reported_seq":16492,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:44:55.790110+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:44:55.790110+0000","last_peered":"2024-09-05T16:44:55.790110+0000","last_clean":"2024-09-05T16:44:55.790110+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:44:55.790110+0000","last_undegraded":"2024-09-05T16:44:55.790110+0000","last_fullsized":"2024-09-05T16:44:55.790110+0000","mapping_epoch":36,"log_start":"29'1700","ondisk_log_start":"29'1700","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8658944,"num_objects":1057,"num_object_clones":0,"num_object_copies":4228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1057,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3351,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'12035","reported_seq":12148,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:55.871891+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:44:55.871891+0000","last_peered":"2024-09-05T16:44:55.871891+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:44:55.871891+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"29'2000","ondisk_log_start":"29'2000","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8934560,"num_objects":1091,"num_object_clones":0,"num_object_copies":4364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1091,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12033,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1243}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'11775","reported_seq":11893,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:54.935949+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:44:54.935949+0000","last_peered":"2024-09-05T16:44:54.935949+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:44:54.935949+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"29'1700","ondisk_log_start":"29'1700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8664224,"num_objects":1058,"num_object_clones":0,"num_object_copies":4232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1058,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11773,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1250}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'11694","reported_seq":11807,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:55.166388+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:44:55.166388+0000","last_peered":"2024-09-05T16:44:55.166388+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:44:55.166388+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"29'1600","ondisk_log_start":"29'1600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8852640,"num_objects":1081,"num_object_clones":0,"num_object_copies":4324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1081,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11692,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1139}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":141090120,"num_objects":17228,"num_object_clones":0,"num_object_copies":68912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28498,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17228,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":162922,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":868,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":282562560,"data_stored":282562560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160831,"ondisk_log_size":160831,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084419,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":736860,"kb_used_data":69324,"kb_used_omap":4,"kb_used_meta":667515,"kb_avail":93634980,"statfs":{"total":96636764160,"available":95882219520,"internally_reserved":0,"allocated":70987776,"data_stored":70777195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":683536083},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,1,1,1,5,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986296,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":596944,"kb_used_data":69632,"kb_used_omap":4,"kb_used_meta":527291,"kb_avail":93774896,"statfs":{"total":96636764160,"available":96025493504,"internally_reserved":0,"allocated":71303168,"data_stored":71092587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":539946706},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,1,3,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673102,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":741212,"kb_used_data":70028,"kb_used_omap":1,"kb_used_meta":671166,"kb_avail":93630628,"statfs":{"total":96636764160,"available":95877763072,"internally_reserved":0,"allocated":71708672,"data_stored":71498091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":687274437},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,4,0,6,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182533,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":741256,"kb_used_data":69624,"kb_used_omap":4,"kb_used_meta":671611,"kb_avail":93630584,"statfs":{"total":96636764160,"available":95877718016,"internally_reserved":0,"allocated":71294976,"data_stored":71084395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":687730381},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,7,0,1,2,0,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":70893568,"data_stored":70893568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":70447104,"data_stored":70447104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":71069696,"data_stored":71069696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":70152192,"data_stored":70152192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:45:00.441 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:45:01.063 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:45:01.064 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:45:01.098 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":382,"stamp":"2024-09-05T16:45:01.062113+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":138059080,"num_objects":16858,"num_object_clones":0,"num_object_copies":67432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28498,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16858,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":163292,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":868,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160801,"ondisk_log_size":160801,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2801320,"kb_used_data":270568,"kb_used_omap":14,"kb_used_meta":2530673,"kb_avail":374686040,"statfs":{"total":386547056640,"available":383678504960,"internally_reserved":0,"allocated":277061632,"data_stored":276219308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2591409719},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,1,4,7,10,16,9,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19349504,"num_objects":-2362,"num_object_clones":0,"num_object_copies":-9448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-85,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2362,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2362,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.010889"},"pg_stats":[{"pgid":"1.f","version":"39'11819","reported_seq":13106,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:56.208591+0000","last_change":"2024-09-05T16:43:43.436935+0000","last_active":"2024-09-05T16:44:56.208591+0000","last_peered":"2024-09-05T16:44:56.208591+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:44:56.208591+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"29'1800","ondisk_log_start":"29'1800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8888320,"num_objects":1085,"num_object_clones":0,"num_object_copies":4340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1647,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1085,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11819,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":615,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-562}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'11557","reported_seq":16213,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:44:55.883996+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:44:55.883996+0000","last_peered":"2024-09-05T16:44:55.883996+0000","last_clean":"2024-09-05T16:44:55.883996+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:44:55.883996+0000","last_undegraded":"2024-09-05T16:44:55.883996+0000","last_fullsized":"2024-09-05T16:44:55.883996+0000","mapping_epoch":38,"log_start":"29'1500","ondisk_log_start":"29'1500","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8151040,"num_objects":995,"num_object_clones":0,"num_object_copies":3980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":995,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":870,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'11724","reported_seq":11851,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:59.820954+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:44:59.820954+0000","last_peered":"2024-09-05T16:44:59.820954+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:44:59.820954+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"29'1700","ondisk_log_start":"29'1700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8419944,"num_objects":1029,"num_object_clones":0,"num_object_copies":4116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1029,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11723,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1203}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'11859","reported_seq":16593,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:44:56.029319+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:44:56.029319+0000","last_peered":"2024-09-05T16:44:56.029319+0000","last_clean":"2024-09-05T16:44:56.029319+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:44:56.029319+0000","last_undegraded":"2024-09-05T16:44:56.029319+0000","last_fullsized":"2024-09-05T16:44:56.029319+0000","mapping_epoch":34,"log_start":"29'1800","ondisk_log_start":"29'1800","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8806400,"num_objects":1075,"num_object_clones":0,"num_object_copies":4300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1075,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6159,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'11542","reported_seq":11646,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:56.229366+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:44:56.229366+0000","last_peered":"2024-09-05T16:44:56.229366+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:44:56.229366+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"29'1500","ondisk_log_start":"29'1500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8623264,"num_objects":1053,"num_object_clones":0,"num_object_copies":4212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1053,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11540,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1092}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'11711","reported_seq":11830,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:56.134609+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:44:56.134609+0000","last_peered":"2024-09-05T16:44:56.134609+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:44:56.134609+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"29'1700","ondisk_log_start":"29'1700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8691712,"num_objects":1061,"num_object_clones":0,"num_object_copies":4244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1061,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11711,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1158}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'11582","reported_seq":11695,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:00.325909+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:45:00.325909+0000","last_peered":"2024-09-05T16:45:00.325909+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:45:00.325909+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"29'1500","ondisk_log_start":"29'1500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8844448,"num_objects":1080,"num_object_clones":0,"num_object_copies":4320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1080,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11580,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1051}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'11735","reported_seq":11827,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:00.293961+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:45:00.293961+0000","last_peered":"2024-09-05T16:45:00.293961+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:45:00.293961+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"29'1700","ondisk_log_start":"29'1700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8549536,"num_objects":1044,"num_object_clones":0,"num_object_copies":4176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1044,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1188}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"39'11745","reported_seq":11848,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:56.157164+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:44:56.157164+0000","last_peered":"2024-09-05T16:44:56.157164+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:44:56.157164+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"29'1700","ondisk_log_start":"29'1700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8423744,"num_objects":1029,"num_object_clones":0,"num_object_copies":4116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1029,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11742,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1215}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'11800","reported_seq":11927,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:59.960293+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:44:59.960293+0000","last_peered":"2024-09-05T16:44:59.960293+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:44:59.960293+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"29'1700","ondisk_log_start":"29'1700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8271008,"num_objects":1010,"num_object_clones":0,"num_object_copies":4040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1010,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11798,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1235}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'11997","reported_seq":12129,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:56.232325+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:44:56.232325+0000","last_peered":"2024-09-05T16:44:56.232325+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:44:56.232325+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"29'1900","ondisk_log_start":"29'1900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9219824,"num_objects":1126,"num_object_clones":0,"num_object_copies":4504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1126,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11994,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1155}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'11645","reported_seq":11748,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:56.175947+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:44:56.175947+0000","last_peered":"2024-09-05T16:44:56.175947+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:44:56.175947+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"29'1600","ondisk_log_start":"29'1600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9124432,"num_objects":1114,"num_object_clones":0,"num_object_copies":4456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1114,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11644,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1124}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'11751","reported_seq":16492,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:44:55.790110+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:44:55.790110+0000","last_peered":"2024-09-05T16:44:55.790110+0000","last_clean":"2024-09-05T16:44:55.790110+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:44:55.790110+0000","last_undegraded":"2024-09-05T16:44:55.790110+0000","last_fullsized":"2024-09-05T16:44:55.790110+0000","mapping_epoch":36,"log_start":"29'1700","ondisk_log_start":"29'1700","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8658944,"num_objects":1057,"num_object_clones":0,"num_object_copies":4228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1057,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3351,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'12035","reported_seq":12148,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:55.871891+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:44:55.871891+0000","last_peered":"2024-09-05T16:44:55.871891+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:44:55.871891+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"29'2000","ondisk_log_start":"29'2000","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8934560,"num_objects":1091,"num_object_clones":0,"num_object_copies":4364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1091,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12033,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1243}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'11834","reported_seq":11952,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:59.969129+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:44:59.969129+0000","last_peered":"2024-09-05T16:44:59.969129+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:44:59.969129+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"29'1800","ondisk_log_start":"29'1800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8180896,"num_objects":999,"num_object_clones":0,"num_object_copies":3996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":999,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11832,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1309}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'11765","reported_seq":11878,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:00.052042+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:45:00.052042+0000","last_peered":"2024-09-05T16:45:00.052042+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:45:00.052042+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"29'1700","ondisk_log_start":"29'1700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8271008,"num_objects":1010,"num_object_clones":0,"num_object_copies":4040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1010,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11763,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1210}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":138059080,"num_objects":16858,"num_object_clones":0,"num_object_copies":67432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28498,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16858,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":163292,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":868,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":274178048,"data_stored":274178048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160801,"ondisk_log_size":160801,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084419,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":736860,"kb_used_data":69324,"kb_used_omap":4,"kb_used_meta":667515,"kb_avail":93634980,"statfs":{"total":96636764160,"available":95882219520,"internally_reserved":0,"allocated":70987776,"data_stored":70777195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":683536083},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,1,1,1,5,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986296,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":596944,"kb_used_data":69632,"kb_used_omap":4,"kb_used_meta":527291,"kb_avail":93774896,"statfs":{"total":96636764160,"available":96025493504,"internally_reserved":0,"allocated":71303168,"data_stored":71092587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":539946706},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,1,3,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673103,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":742808,"kb_used_data":66056,"kb_used_omap":1,"kb_used_meta":676734,"kb_avail":93629032,"statfs":{"total":96636764160,"available":95876128768,"internally_reserved":0,"allocated":67641344,"data_stored":67430763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":692976069},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,0,0,1,4,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182534,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":724708,"kb_used_data":65556,"kb_used_omap":4,"kb_used_meta":659131,"kb_avail":93647132,"statfs":{"total":96636764160,"available":95894663168,"internally_reserved":0,"allocated":67129344,"data_stored":66918763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":674950861},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,5,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":66703360,"data_stored":66703360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":70447104,"data_stored":70447104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":66875392,"data_stored":66875392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":70152192,"data_stored":70152192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:45:01.099 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-05T16:45:01.100 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "39'11819", 'reported_seq': 13106, 'reported_epoch': 39, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:56.208591+0000', 'last_change': '2024-09-05T16:43:43.436935+0000', 'last_active': '2024-09-05T16:44:56.208591+0000', 'last_peered': '2024-09-05T16:44:56.208591+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.828289+0000', 'last_became_peered': '2024-09-05T16:34:56.828289+0000', 'last_unstale': '2024-09-05T16:44:56.208591+0000', 'last_undegraded': '2024-09-05T16:34:56.612160+0000', 'last_fullsized': '2024-09-05T16:34:56.557632+0000', 'mapping_epoch': 32, 'log_start': "29'1800", 'ondisk_log_start': "29'1800", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10019, 'log_dups_size': 0, 'ondisk_log_size': 10019, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8888320, 'num_objects': 1085, 'num_object_clones': 0, 'num_object_copies': 4340, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1647, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1085, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11819, 'num_write_kb': 51616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 615, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -562}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:45:01.100 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:45:01.100 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "39'11724", 'reported_seq': 11851, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:59.820954+0000', 'last_change': '2024-09-05T16:34:56.867900+0000', 'last_active': '2024-09-05T16:44:59.820954+0000', 'last_peered': '2024-09-05T16:44:59.820954+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:44:59.820954+0000', 'last_undegraded': '2024-09-05T16:34:56.590972+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 32, 'log_start': "29'1700", 'ondisk_log_start': "29'1700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10024, 'log_dups_size': 0, 'ondisk_log_size': 10024, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8419944, 'num_objects': 1029, 'num_object_clones': 0, 'num_object_copies': 4116, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2233, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1029, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 11723, 'num_write_kb': 50999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1203}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:45:01.100 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:45:01.100 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "39'11542", 'reported_seq': 11646, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:56.229366+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:44:56.229366+0000', 'last_peered': '2024-09-05T16:44:56.229366+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:44:56.229366+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "29'1500", 'ondisk_log_start': "29'1500", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10042, 'log_dups_size': 0, 'ondisk_log_size': 10042, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8623264, 'num_objects': 1053, 'num_object_clones': 0, 'num_object_copies': 4212, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2147, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1053, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11540, 'num_write_kb': 50373, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1092}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:45:01.100 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:45:01.100 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "39'11711", 'reported_seq': 11830, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:56.134609+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:44:56.134609+0000', 'last_peered': '2024-09-05T16:44:56.134609+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:44:56.134609+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "29'1700", 'ondisk_log_start': "29'1700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10011, 'log_dups_size': 0, 'ondisk_log_size': 10011, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8691712, 'num_objects': 1061, 'num_object_clones': 0, 'num_object_copies': 4244, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1061, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11711, 'num_write_kb': 51088, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1158}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:45:01.100 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:45:01.100 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "39'11582", 'reported_seq': 11695, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:00.325909+0000', 'last_change': '2024-09-05T16:34:56.903436+0000', 'last_active': '2024-09-05T16:45:00.325909+0000', 'last_peered': '2024-09-05T16:45:00.325909+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:45:00.325909+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "29'1500", 'ondisk_log_start': "29'1500", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10082, 'log_dups_size': 0, 'ondisk_log_size': 10082, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8844448, 'num_objects': 1080, 'num_object_clones': 0, 'num_object_copies': 4320, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2133, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1080, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11580, 'num_write_kb': 50636, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -1051}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:45:01.101 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-05T16:45:01.101 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "39'11735", 'reported_seq': 11827, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:00.293961+0000', 'last_change': '2024-09-05T16:34:56.883319+0000', 'last_active': '2024-09-05T16:45:00.293961+0000', 'last_peered': '2024-09-05T16:45:00.293961+0000', 'last_clean': '2024-09-05T16:33:53.505305+0000', 'last_became_active': '2024-09-05T16:34:56.873389+0000', 'last_became_peered': '2024-09-05T16:34:56.873389+0000', 'last_unstale': '2024-09-05T16:45:00.293961+0000', 'last_undegraded': '2024-09-05T16:34:56.602567+0000', 'last_fullsized': '2024-09-05T16:34:56.559958+0000', 'mapping_epoch': 32, 'log_start': "29'1700", 'ondisk_log_start': "29'1700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10035, 'log_dups_size': 0, 'ondisk_log_size': 10035, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8549536, 'num_objects': 1044, 'num_object_clones': 0, 'num_object_copies': 4176, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2234, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1044, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11733, 'num_write_kb': 51108, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1188}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:45:01.101 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:45:01.101 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "39'11745", 'reported_seq': 11848, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:56.157164+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:44:56.157164+0000', 'last_peered': '2024-09-05T16:44:56.157164+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:44:56.157164+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "29'1700", 'ondisk_log_start': "29'1700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10045, 'log_dups_size': 0, 'ondisk_log_size': 10045, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8423744, 'num_objects': 1029, 'num_object_clones': 0, 'num_object_copies': 4116, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1029, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11742, 'num_write_kb': 51081, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1215}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:45:01.101 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:45:01.101 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "39'11800", 'reported_seq': 11927, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:59.960293+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:44:59.960293+0000', 'last_peered': '2024-09-05T16:44:59.960293+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:44:59.960293+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "29'1700", 'ondisk_log_start': "29'1700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10100, 'log_dups_size': 0, 'ondisk_log_size': 10100, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8271008, 'num_objects': 1010, 'num_object_clones': 0, 'num_object_copies': 4040, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1010, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11798, 'num_write_kb': 51230, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1235}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:45:01.101 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:45:01.101 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "39'11997", 'reported_seq': 12129, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:56.232325+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:44:56.232325+0000', 'last_peered': '2024-09-05T16:44:56.232325+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:44:56.232325+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "29'1900", 'ondisk_log_start': "29'1900", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10097, 'log_dups_size': 0, 'ondisk_log_size': 10097, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9219824, 'num_objects': 1126, 'num_object_clones': 0, 'num_object_copies': 4504, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2284, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1126, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11994, 'num_write_kb': 52476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1155}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:45:01.101 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:45:01.101 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "39'11645", 'reported_seq': 11748, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:56.175947+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:44:56.175947+0000', 'last_peered': '2024-09-05T16:44:56.175947+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:44:56.175947+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "29'1600", 'ondisk_log_start': "29'1600", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10045, 'log_dups_size': 0, 'ondisk_log_size': 10045, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9124432, 'num_objects': 1114, 'num_object_clones': 0, 'num_object_copies': 4456, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1114, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11644, 'num_write_kb': 51031, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1124}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:45:01.101 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:45:01.102 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "39'12035", 'reported_seq': 12148, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:55.871891+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:44:55.871891+0000', 'last_peered': '2024-09-05T16:44:55.871891+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:44:55.871891+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "29'2000", 'ondisk_log_start': "29'2000", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10035, 'log_dups_size': 0, 'ondisk_log_size': 10035, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8934560, 'num_objects': 1091, 'num_object_clones': 0, 'num_object_copies': 4364, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1091, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12033, 'num_write_kb': 52500, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1243}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:45:01.102 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:45:01.102 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "39'11834", 'reported_seq': 11952, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:59.969129+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:44:59.969129+0000', 'last_peered': '2024-09-05T16:44:59.969129+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:44:59.969129+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "29'1800", 'ondisk_log_start': "29'1800", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10034, 'log_dups_size': 0, 'ondisk_log_size': 10034, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8180896, 'num_objects': 999, 'num_object_clones': 0, 'num_object_copies': 3996, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2310, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 999, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11832, 'num_write_kb': 51325, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1309}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:45:01.102 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:45:01.102 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "39'11765", 'reported_seq': 11878, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:00.052042+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:45:00.052042+0000', 'last_peered': '2024-09-05T16:45:00.052042+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:45:00.052042+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "29'1700", 'ondisk_log_start': "29'1700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10065, 'log_dups_size': 0, 'ondisk_log_size': 10065, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8271008, 'num_objects': 1010, 'num_object_clones': 0, 'num_object_copies': 4040, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1010, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11763, 'num_write_kb': 51092, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1210}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:45:01.102 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:45:01.858 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:45:01.858 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 28498/67432 objects degraded (42.262%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":705,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":39,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":16858,"data_bytes":138059080,"bytes_used":2868551680,"bytes_avail":383678504960,"bytes_total":386547056640,"degraded_objects":28498,"degraded_total":67432,"degraded_ratio":0.42261834144026578,"recovering_objects_per_sec":7,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":196},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:45:01.892 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:45:01.892 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:45:02.519 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:45:02.520 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:45:02.542 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":382,"stamp":"2024-09-05T16:45:01.062113+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":138059080,"num_objects":16858,"num_object_clones":0,"num_object_copies":67432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28498,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16858,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":163292,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":868,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160801,"ondisk_log_size":160801,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2801320,"kb_used_data":270568,"kb_used_omap":14,"kb_used_meta":2530673,"kb_avail":374686040,"statfs":{"total":386547056640,"available":383678504960,"internally_reserved":0,"allocated":277061632,"data_stored":276219308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2591409719},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,1,4,7,10,16,9,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19349504,"num_objects":-2362,"num_object_clones":0,"num_object_copies":-9448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-85,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2362,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2362,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.010889"},"pg_stats":[{"pgid":"1.f","version":"39'11819","reported_seq":13106,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:56.208591+0000","last_change":"2024-09-05T16:43:43.436935+0000","last_active":"2024-09-05T16:44:56.208591+0000","last_peered":"2024-09-05T16:44:56.208591+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:44:56.208591+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"29'1800","ondisk_log_start":"29'1800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8888320,"num_objects":1085,"num_object_clones":0,"num_object_copies":4340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1647,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1085,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11819,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":615,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-562}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'11557","reported_seq":16213,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:44:55.883996+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:44:55.883996+0000","last_peered":"2024-09-05T16:44:55.883996+0000","last_clean":"2024-09-05T16:44:55.883996+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:44:55.883996+0000","last_undegraded":"2024-09-05T16:44:55.883996+0000","last_fullsized":"2024-09-05T16:44:55.883996+0000","mapping_epoch":38,"log_start":"29'1500","ondisk_log_start":"29'1500","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8151040,"num_objects":995,"num_object_clones":0,"num_object_copies":3980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":995,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":870,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'11724","reported_seq":11851,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:59.820954+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:44:59.820954+0000","last_peered":"2024-09-05T16:44:59.820954+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:44:59.820954+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"29'1700","ondisk_log_start":"29'1700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8419944,"num_objects":1029,"num_object_clones":0,"num_object_copies":4116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1029,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11723,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1203}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'11859","reported_seq":16593,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:44:56.029319+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:44:56.029319+0000","last_peered":"2024-09-05T16:44:56.029319+0000","last_clean":"2024-09-05T16:44:56.029319+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:44:56.029319+0000","last_undegraded":"2024-09-05T16:44:56.029319+0000","last_fullsized":"2024-09-05T16:44:56.029319+0000","mapping_epoch":34,"log_start":"29'1800","ondisk_log_start":"29'1800","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8806400,"num_objects":1075,"num_object_clones":0,"num_object_copies":4300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1075,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6159,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'11542","reported_seq":11646,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:56.229366+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:44:56.229366+0000","last_peered":"2024-09-05T16:44:56.229366+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:44:56.229366+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"29'1500","ondisk_log_start":"29'1500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8623264,"num_objects":1053,"num_object_clones":0,"num_object_copies":4212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1053,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11540,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1092}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'11711","reported_seq":11830,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:56.134609+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:44:56.134609+0000","last_peered":"2024-09-05T16:44:56.134609+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:44:56.134609+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"29'1700","ondisk_log_start":"29'1700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8691712,"num_objects":1061,"num_object_clones":0,"num_object_copies":4244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1061,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11711,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1158}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'11582","reported_seq":11695,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:00.325909+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:45:00.325909+0000","last_peered":"2024-09-05T16:45:00.325909+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:45:00.325909+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"29'1500","ondisk_log_start":"29'1500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8844448,"num_objects":1080,"num_object_clones":0,"num_object_copies":4320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1080,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11580,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1051}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'11735","reported_seq":11827,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:00.293961+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:45:00.293961+0000","last_peered":"2024-09-05T16:45:00.293961+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:45:00.293961+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"29'1700","ondisk_log_start":"29'1700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8549536,"num_objects":1044,"num_object_clones":0,"num_object_copies":4176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1044,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1188}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"39'11745","reported_seq":11848,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:56.157164+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:44:56.157164+0000","last_peered":"2024-09-05T16:44:56.157164+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:44:56.157164+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"29'1700","ondisk_log_start":"29'1700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8423744,"num_objects":1029,"num_object_clones":0,"num_object_copies":4116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1029,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11742,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1215}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'11800","reported_seq":11927,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:59.960293+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:44:59.960293+0000","last_peered":"2024-09-05T16:44:59.960293+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:44:59.960293+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"29'1700","ondisk_log_start":"29'1700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8271008,"num_objects":1010,"num_object_clones":0,"num_object_copies":4040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1010,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11798,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1235}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'11997","reported_seq":12129,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:56.232325+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:44:56.232325+0000","last_peered":"2024-09-05T16:44:56.232325+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:44:56.232325+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"29'1900","ondisk_log_start":"29'1900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9219824,"num_objects":1126,"num_object_clones":0,"num_object_copies":4504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1126,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11994,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1155}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'11645","reported_seq":11748,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:56.175947+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:44:56.175947+0000","last_peered":"2024-09-05T16:44:56.175947+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:44:56.175947+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"29'1600","ondisk_log_start":"29'1600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9124432,"num_objects":1114,"num_object_clones":0,"num_object_copies":4456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1114,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11644,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1124}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'11751","reported_seq":16492,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:44:55.790110+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:44:55.790110+0000","last_peered":"2024-09-05T16:44:55.790110+0000","last_clean":"2024-09-05T16:44:55.790110+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:44:55.790110+0000","last_undegraded":"2024-09-05T16:44:55.790110+0000","last_fullsized":"2024-09-05T16:44:55.790110+0000","mapping_epoch":36,"log_start":"29'1700","ondisk_log_start":"29'1700","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8658944,"num_objects":1057,"num_object_clones":0,"num_object_copies":4228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1057,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3351,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'12035","reported_seq":12148,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:55.871891+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:44:55.871891+0000","last_peered":"2024-09-05T16:44:55.871891+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:44:55.871891+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"29'2000","ondisk_log_start":"29'2000","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8934560,"num_objects":1091,"num_object_clones":0,"num_object_copies":4364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1091,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12033,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1243}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'11834","reported_seq":11952,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:59.969129+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:44:59.969129+0000","last_peered":"2024-09-05T16:44:59.969129+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:44:59.969129+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"29'1800","ondisk_log_start":"29'1800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8180896,"num_objects":999,"num_object_clones":0,"num_object_copies":3996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":999,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11832,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1309}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'11765","reported_seq":11878,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:00.052042+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:45:00.052042+0000","last_peered":"2024-09-05T16:45:00.052042+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:45:00.052042+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"29'1700","ondisk_log_start":"29'1700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8271008,"num_objects":1010,"num_object_clones":0,"num_object_copies":4040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1010,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11763,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1210}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":138059080,"num_objects":16858,"num_object_clones":0,"num_object_copies":67432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28498,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16858,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":163292,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":868,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":274178048,"data_stored":274178048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160801,"ondisk_log_size":160801,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084419,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":736860,"kb_used_data":69324,"kb_used_omap":4,"kb_used_meta":667515,"kb_avail":93634980,"statfs":{"total":96636764160,"available":95882219520,"internally_reserved":0,"allocated":70987776,"data_stored":70777195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":683536083},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,1,1,1,5,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986296,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":596944,"kb_used_data":69632,"kb_used_omap":4,"kb_used_meta":527291,"kb_avail":93774896,"statfs":{"total":96636764160,"available":96025493504,"internally_reserved":0,"allocated":71303168,"data_stored":71092587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":539946706},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,1,3,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673103,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":742808,"kb_used_data":66056,"kb_used_omap":1,"kb_used_meta":676734,"kb_avail":93629032,"statfs":{"total":96636764160,"available":95876128768,"internally_reserved":0,"allocated":67641344,"data_stored":67430763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":692976069},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,0,0,1,4,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182534,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":724708,"kb_used_data":65556,"kb_used_omap":4,"kb_used_meta":659131,"kb_avail":93647132,"statfs":{"total":96636764160,"available":95894663168,"internally_reserved":0,"allocated":67129344,"data_stored":66918763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":674950861},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,5,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":66703360,"data_stored":66703360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":70447104,"data_stored":70447104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":66875392,"data_stored":66875392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":70152192,"data_stored":70152192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:45:05.545 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:45:06.163 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:45:06.163 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:45:06.183 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":384,"stamp":"2024-09-05T16:45:05.063837+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":132521288,"num_objects":16182,"num_object_clones":0,"num_object_copies":64728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28460,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16182,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":163968,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":906,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160777,"ondisk_log_size":160777,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2819376,"kb_used_data":262248,"kb_used_omap":14,"kb_used_meta":2557041,"kb_avail":374667984,"statfs":{"total":386547056640,"available":383660015616,"internally_reserved":0,"allocated":268541952,"data_stored":267699628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2618410551},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,4,0,12,9,17,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16809984,"num_objects":-2052,"num_object_clones":0,"num_object_copies":-8208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-74,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2052,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2052,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":74,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.010947"},"pg_stats":[{"pgid":"1.f","version":"39'11880","reported_seq":13244,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:01.374442+0000","last_change":"2024-09-05T16:43:43.436935+0000","last_active":"2024-09-05T16:45:01.374442+0000","last_peered":"2024-09-05T16:45:01.374442+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:45:01.374442+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"29'1800","ondisk_log_start":"29'1800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":1024,"num_object_clones":0,"num_object_copies":4096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1609,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11880,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":653,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-585}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'11631","reported_seq":16287,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:45:01.011041+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:45:01.011041+0000","last_peered":"2024-09-05T16:45:01.011041+0000","last_clean":"2024-09-05T16:45:01.011041+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:45:01.011041+0000","last_undegraded":"2024-09-05T16:45:01.011041+0000","last_fullsized":"2024-09-05T16:45:01.011041+0000","mapping_epoch":38,"log_start":"29'1600","ondisk_log_start":"29'1600","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7544832,"num_objects":921,"num_object_clones":0,"num_object_copies":3684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":921,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":944,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'11724","reported_seq":11851,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:59.820954+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:44:59.820954+0000","last_peered":"2024-09-05T16:44:59.820954+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:44:59.820954+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"29'1700","ondisk_log_start":"29'1700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8419944,"num_objects":1029,"num_object_clones":0,"num_object_copies":4116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1029,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11723,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1203}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'11925","reported_seq":16659,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:45:01.040170+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:45:01.040170+0000","last_peered":"2024-09-05T16:45:01.040170+0000","last_clean":"2024-09-05T16:45:01.040170+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:45:01.040170+0000","last_undegraded":"2024-09-05T16:45:01.040170+0000","last_fullsized":"2024-09-05T16:45:01.040170+0000","mapping_epoch":34,"log_start":"29'1900","ondisk_log_start":"29'1900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8265728,"num_objects":1009,"num_object_clones":0,"num_object_copies":4036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1009,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6225,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'11611","reported_seq":11715,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:01.134553+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:45:01.134553+0000","last_peered":"2024-09-05T16:45:01.134553+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:45:01.134553+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"29'1600","ondisk_log_start":"29'1600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8058016,"num_objects":984,"num_object_clones":0,"num_object_copies":3936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":984,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11609,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1161}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'11771","reported_seq":11888,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:01.368734+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:45:01.368734+0000","last_peered":"2024-09-05T16:45:01.368734+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:45:01.368734+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"29'1700","ondisk_log_start":"29'1700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8200192,"num_objects":1001,"num_object_clones":0,"num_object_copies":4004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1001,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11771,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1218}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'11582","reported_seq":11695,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:00.325909+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:45:00.325909+0000","last_peered":"2024-09-05T16:45:00.325909+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:45:00.325909+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"29'1500","ondisk_log_start":"29'1500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8844448,"num_objects":1080,"num_object_clones":0,"num_object_copies":4320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1080,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11580,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1051}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'11735","reported_seq":11827,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:00.293961+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:45:00.293961+0000","last_peered":"2024-09-05T16:45:00.293961+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:45:00.293961+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"29'1700","ondisk_log_start":"29'1700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8549536,"num_objects":1044,"num_object_clones":0,"num_object_copies":4176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1044,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1188}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"39'11816","reported_seq":11922,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:01.273827+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:45:01.273827+0000","last_peered":"2024-09-05T16:45:01.273827+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:45:01.273827+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"29'1800","ondisk_log_start":"29'1800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7842112,"num_objects":958,"num_object_clones":0,"num_object_copies":3832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":958,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11813,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1286}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'11800","reported_seq":11927,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:59.960293+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:44:59.960293+0000","last_peered":"2024-09-05T16:44:59.960293+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:44:59.960293+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"29'1700","ondisk_log_start":"29'1700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8271008,"num_objects":1010,"num_object_clones":0,"num_object_copies":4040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1010,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11798,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1235}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'12064","reported_seq":12196,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:01.306044+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:45:01.306044+0000","last_peered":"2024-09-05T16:45:01.306044+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:45:01.306044+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"29'2000","ondisk_log_start":"29'2000","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8670960,"num_objects":1059,"num_object_clones":0,"num_object_copies":4236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1059,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12061,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1222}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'11716","reported_seq":11819,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:01.034821+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:45:01.034821+0000","last_peered":"2024-09-05T16:45:01.034821+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:45:01.034821+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"29'1700","ondisk_log_start":"29'1700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8542800,"num_objects":1043,"num_object_clones":0,"num_object_copies":4172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1043,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11715,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1195}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'11828","reported_seq":16569,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:45:01.144409+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:45:01.144409+0000","last_peered":"2024-09-05T16:45:01.144409+0000","last_clean":"2024-09-05T16:45:01.144409+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:45:01.144409+0000","last_undegraded":"2024-09-05T16:45:01.144409+0000","last_fullsized":"2024-09-05T16:45:01.144409+0000","mapping_epoch":36,"log_start":"29'1800","ondisk_log_start":"29'1800","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8028160,"num_objects":980,"num_object_clones":0,"num_object_copies":3920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":980,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3428,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'12095","reported_seq":12208,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:01.306422+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:45:01.306422+0000","last_peered":"2024-09-05T16:45:01.306422+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:45:01.306422+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"29'2000","ondisk_log_start":"29'2000","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8443040,"num_objects":1031,"num_object_clones":0,"num_object_copies":4124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1031,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12093,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1303}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'11834","reported_seq":11952,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:59.969129+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:44:59.969129+0000","last_peered":"2024-09-05T16:44:59.969129+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:44:59.969129+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"29'1800","ondisk_log_start":"29'1800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8180896,"num_objects":999,"num_object_clones":0,"num_object_copies":3996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":999,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11832,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1309}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'11765","reported_seq":11878,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:00.052042+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:45:00.052042+0000","last_peered":"2024-09-05T16:45:00.052042+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:45:00.052042+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"29'1700","ondisk_log_start":"29'1700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8271008,"num_objects":1010,"num_object_clones":0,"num_object_copies":4040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1010,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11763,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1210}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":132521288,"num_objects":16182,"num_object_clones":0,"num_object_copies":64728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28460,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16182,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":163968,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":906,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":265633792,"data_stored":265633792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160777,"ondisk_log_size":160777,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084420,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":754924,"kb_used_data":65172,"kb_used_omap":4,"kb_used_meta":689723,"kb_avail":93616916,"statfs":{"total":96636764160,"available":95863721984,"internally_reserved":0,"allocated":66736128,"data_stored":66525547,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":706277075},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,7,1,2,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986297,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":596936,"kb_used_data":65464,"kb_used_omap":4,"kb_used_meta":531451,"kb_avail":93774904,"statfs":{"total":96636764160,"available":96025501696,"internally_reserved":0,"allocated":67035136,"data_stored":66824555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":544206546},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,2,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673103,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":742808,"kb_used_data":66056,"kb_used_omap":1,"kb_used_meta":676734,"kb_avail":93629032,"statfs":{"total":96636764160,"available":95876128768,"internally_reserved":0,"allocated":67641344,"data_stored":67430763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":692976069},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,0,0,1,4,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182534,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":724708,"kb_used_data":65556,"kb_used_omap":4,"kb_used_meta":659131,"kb_avail":93647132,"statfs":{"total":96636764160,"available":95894663168,"internally_reserved":0,"allocated":67129344,"data_stored":66918763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":674950861},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,5,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":66703360,"data_stored":66703360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":66150400,"data_stored":66150400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":66875392,"data_stored":66875392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":65904640,"data_stored":65904640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:45:06.184 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:45:06.782 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:45:06.783 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:45:06.811 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":384,"stamp":"2024-09-05T16:45:05.063837+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":132521288,"num_objects":16182,"num_object_clones":0,"num_object_copies":64728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28460,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16182,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":163968,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":906,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160777,"ondisk_log_size":160777,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2819376,"kb_used_data":262248,"kb_used_omap":14,"kb_used_meta":2557041,"kb_avail":374667984,"statfs":{"total":386547056640,"available":383660015616,"internally_reserved":0,"allocated":268541952,"data_stored":267699628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2618410551},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,4,0,12,9,17,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16809984,"num_objects":-2052,"num_object_clones":0,"num_object_copies":-8208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-74,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2052,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2052,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":74,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.010947"},"pg_stats":[{"pgid":"1.f","version":"39'11880","reported_seq":13244,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:01.374442+0000","last_change":"2024-09-05T16:43:43.436935+0000","last_active":"2024-09-05T16:45:01.374442+0000","last_peered":"2024-09-05T16:45:01.374442+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:45:01.374442+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"29'1800","ondisk_log_start":"29'1800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":1024,"num_object_clones":0,"num_object_copies":4096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1609,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11880,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":653,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-585}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'11631","reported_seq":16287,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:45:01.011041+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:45:01.011041+0000","last_peered":"2024-09-05T16:45:01.011041+0000","last_clean":"2024-09-05T16:45:01.011041+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:45:01.011041+0000","last_undegraded":"2024-09-05T16:45:01.011041+0000","last_fullsized":"2024-09-05T16:45:01.011041+0000","mapping_epoch":38,"log_start":"29'1600","ondisk_log_start":"29'1600","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7544832,"num_objects":921,"num_object_clones":0,"num_object_copies":3684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":921,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":944,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'11724","reported_seq":11851,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:59.820954+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:44:59.820954+0000","last_peered":"2024-09-05T16:44:59.820954+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:44:59.820954+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"29'1700","ondisk_log_start":"29'1700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8419944,"num_objects":1029,"num_object_clones":0,"num_object_copies":4116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1029,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11723,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1203}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'11925","reported_seq":16659,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:45:01.040170+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:45:01.040170+0000","last_peered":"2024-09-05T16:45:01.040170+0000","last_clean":"2024-09-05T16:45:01.040170+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:45:01.040170+0000","last_undegraded":"2024-09-05T16:45:01.040170+0000","last_fullsized":"2024-09-05T16:45:01.040170+0000","mapping_epoch":34,"log_start":"29'1900","ondisk_log_start":"29'1900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8265728,"num_objects":1009,"num_object_clones":0,"num_object_copies":4036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1009,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6225,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'11611","reported_seq":11715,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:01.134553+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:45:01.134553+0000","last_peered":"2024-09-05T16:45:01.134553+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:45:01.134553+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"29'1600","ondisk_log_start":"29'1600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8058016,"num_objects":984,"num_object_clones":0,"num_object_copies":3936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":984,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11609,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1161}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'11771","reported_seq":11888,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:01.368734+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:45:01.368734+0000","last_peered":"2024-09-05T16:45:01.368734+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:45:01.368734+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"29'1700","ondisk_log_start":"29'1700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8200192,"num_objects":1001,"num_object_clones":0,"num_object_copies":4004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1001,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11771,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1218}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'11582","reported_seq":11695,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:00.325909+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:45:00.325909+0000","last_peered":"2024-09-05T16:45:00.325909+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:45:00.325909+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"29'1500","ondisk_log_start":"29'1500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8844448,"num_objects":1080,"num_object_clones":0,"num_object_copies":4320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1080,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11580,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1051}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'11735","reported_seq":11827,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:00.293961+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:45:00.293961+0000","last_peered":"2024-09-05T16:45:00.293961+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:45:00.293961+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"29'1700","ondisk_log_start":"29'1700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8549536,"num_objects":1044,"num_object_clones":0,"num_object_copies":4176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1044,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1188}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"39'11816","reported_seq":11922,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:01.273827+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:45:01.273827+0000","last_peered":"2024-09-05T16:45:01.273827+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:45:01.273827+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"29'1800","ondisk_log_start":"29'1800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7842112,"num_objects":958,"num_object_clones":0,"num_object_copies":3832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":958,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11813,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1286}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'11800","reported_seq":11927,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:59.960293+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:44:59.960293+0000","last_peered":"2024-09-05T16:44:59.960293+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:44:59.960293+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"29'1700","ondisk_log_start":"29'1700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8271008,"num_objects":1010,"num_object_clones":0,"num_object_copies":4040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1010,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11798,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1235}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'12064","reported_seq":12196,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:01.306044+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:45:01.306044+0000","last_peered":"2024-09-05T16:45:01.306044+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:45:01.306044+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"29'2000","ondisk_log_start":"29'2000","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8670960,"num_objects":1059,"num_object_clones":0,"num_object_copies":4236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1059,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12061,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1222}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'11716","reported_seq":11819,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:01.034821+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:45:01.034821+0000","last_peered":"2024-09-05T16:45:01.034821+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:45:01.034821+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"29'1700","ondisk_log_start":"29'1700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8542800,"num_objects":1043,"num_object_clones":0,"num_object_copies":4172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1043,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11715,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1195}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'11828","reported_seq":16569,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:45:01.144409+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:45:01.144409+0000","last_peered":"2024-09-05T16:45:01.144409+0000","last_clean":"2024-09-05T16:45:01.144409+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:45:01.144409+0000","last_undegraded":"2024-09-05T16:45:01.144409+0000","last_fullsized":"2024-09-05T16:45:01.144409+0000","mapping_epoch":36,"log_start":"29'1800","ondisk_log_start":"29'1800","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8028160,"num_objects":980,"num_object_clones":0,"num_object_copies":3920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":980,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3428,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'12095","reported_seq":12208,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:01.306422+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:45:01.306422+0000","last_peered":"2024-09-05T16:45:01.306422+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:45:01.306422+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"29'2000","ondisk_log_start":"29'2000","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8443040,"num_objects":1031,"num_object_clones":0,"num_object_copies":4124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1031,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12093,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1303}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'11834","reported_seq":11952,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:44:59.969129+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:44:59.969129+0000","last_peered":"2024-09-05T16:44:59.969129+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:44:59.969129+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"29'1800","ondisk_log_start":"29'1800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8180896,"num_objects":999,"num_object_clones":0,"num_object_copies":3996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":999,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11832,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1309}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'11765","reported_seq":11878,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:00.052042+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:45:00.052042+0000","last_peered":"2024-09-05T16:45:00.052042+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:45:00.052042+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"29'1700","ondisk_log_start":"29'1700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8271008,"num_objects":1010,"num_object_clones":0,"num_object_copies":4040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1010,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11763,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1210}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":132521288,"num_objects":16182,"num_object_clones":0,"num_object_copies":64728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28460,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16182,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":163968,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":906,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":265633792,"data_stored":265633792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160777,"ondisk_log_size":160777,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084420,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":754924,"kb_used_data":65172,"kb_used_omap":4,"kb_used_meta":689723,"kb_avail":93616916,"statfs":{"total":96636764160,"available":95863721984,"internally_reserved":0,"allocated":66736128,"data_stored":66525547,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":706277075},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,7,1,2,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986297,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":596936,"kb_used_data":65464,"kb_used_omap":4,"kb_used_meta":531451,"kb_avail":93774904,"statfs":{"total":96636764160,"available":96025501696,"internally_reserved":0,"allocated":67035136,"data_stored":66824555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":544206546},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,2,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673103,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":742808,"kb_used_data":66056,"kb_used_omap":1,"kb_used_meta":676734,"kb_avail":93629032,"statfs":{"total":96636764160,"available":95876128768,"internally_reserved":0,"allocated":67641344,"data_stored":67430763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":692976069},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,0,0,1,4,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182534,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":724708,"kb_used_data":65556,"kb_used_omap":4,"kb_used_meta":659131,"kb_avail":93647132,"statfs":{"total":96636764160,"available":95894663168,"internally_reserved":0,"allocated":67129344,"data_stored":66918763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":674950861},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,5,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":66703360,"data_stored":66703360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":66150400,"data_stored":66150400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":66875392,"data_stored":66875392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":65904640,"data_stored":65904640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:45:06.812 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-05T16:45:06.812 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "39'11880", 'reported_seq': 13244, 'reported_epoch': 39, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:01.374442+0000', 'last_change': '2024-09-05T16:43:43.436935+0000', 'last_active': '2024-09-05T16:45:01.374442+0000', 'last_peered': '2024-09-05T16:45:01.374442+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.828289+0000', 'last_became_peered': '2024-09-05T16:34:56.828289+0000', 'last_unstale': '2024-09-05T16:45:01.374442+0000', 'last_undegraded': '2024-09-05T16:34:56.612160+0000', 'last_fullsized': '2024-09-05T16:34:56.557632+0000', 'mapping_epoch': 32, 'log_start': "29'1800", 'ondisk_log_start': "29'1800", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8388608, 'num_objects': 1024, 'num_object_clones': 0, 'num_object_copies': 4096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1609, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1024, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11880, 'num_write_kb': 51616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 653, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -585}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:45:06.812 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:45:06.812 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "39'11724", 'reported_seq': 11851, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:59.820954+0000', 'last_change': '2024-09-05T16:34:56.867900+0000', 'last_active': '2024-09-05T16:44:59.820954+0000', 'last_peered': '2024-09-05T16:44:59.820954+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:44:59.820954+0000', 'last_undegraded': '2024-09-05T16:34:56.590972+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 32, 'log_start': "29'1700", 'ondisk_log_start': "29'1700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10024, 'log_dups_size': 0, 'ondisk_log_size': 10024, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8419944, 'num_objects': 1029, 'num_object_clones': 0, 'num_object_copies': 4116, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2233, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1029, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 11723, 'num_write_kb': 50999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1203}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:45:06.813 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:45:06.813 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "39'11611", 'reported_seq': 11715, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:01.134553+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:45:01.134553+0000', 'last_peered': '2024-09-05T16:45:01.134553+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:45:01.134553+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "29'1600", 'ondisk_log_start': "29'1600", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10011, 'log_dups_size': 0, 'ondisk_log_size': 10011, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8058016, 'num_objects': 984, 'num_object_clones': 0, 'num_object_copies': 3936, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2147, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 984, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11609, 'num_write_kb': 50373, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1161}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:45:06.813 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:45:06.813 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "39'11771", 'reported_seq': 11888, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:01.368734+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:45:01.368734+0000', 'last_peered': '2024-09-05T16:45:01.368734+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:45:01.368734+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "29'1700", 'ondisk_log_start': "29'1700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10071, 'log_dups_size': 0, 'ondisk_log_size': 10071, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8200192, 'num_objects': 1001, 'num_object_clones': 0, 'num_object_copies': 4004, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1001, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11771, 'num_write_kb': 51088, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1218}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:45:06.813 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:45:06.813 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "39'11582", 'reported_seq': 11695, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:00.325909+0000', 'last_change': '2024-09-05T16:34:56.903436+0000', 'last_active': '2024-09-05T16:45:00.325909+0000', 'last_peered': '2024-09-05T16:45:00.325909+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:45:00.325909+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "29'1500", 'ondisk_log_start': "29'1500", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10082, 'log_dups_size': 0, 'ondisk_log_size': 10082, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8844448, 'num_objects': 1080, 'num_object_clones': 0, 'num_object_copies': 4320, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2133, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1080, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11580, 'num_write_kb': 50636, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -1051}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:45:06.813 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-05T16:45:06.813 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "39'11735", 'reported_seq': 11827, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:00.293961+0000', 'last_change': '2024-09-05T16:34:56.883319+0000', 'last_active': '2024-09-05T16:45:00.293961+0000', 'last_peered': '2024-09-05T16:45:00.293961+0000', 'last_clean': '2024-09-05T16:33:53.505305+0000', 'last_became_active': '2024-09-05T16:34:56.873389+0000', 'last_became_peered': '2024-09-05T16:34:56.873389+0000', 'last_unstale': '2024-09-05T16:45:00.293961+0000', 'last_undegraded': '2024-09-05T16:34:56.602567+0000', 'last_fullsized': '2024-09-05T16:34:56.559958+0000', 'mapping_epoch': 32, 'log_start': "29'1700", 'ondisk_log_start': "29'1700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10035, 'log_dups_size': 0, 'ondisk_log_size': 10035, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8549536, 'num_objects': 1044, 'num_object_clones': 0, 'num_object_copies': 4176, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2234, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1044, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11733, 'num_write_kb': 51108, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1188}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:45:06.813 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:45:06.814 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "39'11816", 'reported_seq': 11922, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:01.273827+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:45:01.273827+0000', 'last_peered': '2024-09-05T16:45:01.273827+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:45:01.273827+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "29'1800", 'ondisk_log_start': "29'1800", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10016, 'log_dups_size': 0, 'ondisk_log_size': 10016, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7842112, 'num_objects': 958, 'num_object_clones': 0, 'num_object_copies': 3832, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 958, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11813, 'num_write_kb': 51081, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1286}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:45:06.814 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:45:06.814 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "39'11800", 'reported_seq': 11927, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:59.960293+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:44:59.960293+0000', 'last_peered': '2024-09-05T16:44:59.960293+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:44:59.960293+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "29'1700", 'ondisk_log_start': "29'1700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10100, 'log_dups_size': 0, 'ondisk_log_size': 10100, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8271008, 'num_objects': 1010, 'num_object_clones': 0, 'num_object_copies': 4040, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1010, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11798, 'num_write_kb': 51230, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1235}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:45:06.814 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:45:06.814 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "39'12064", 'reported_seq': 12196, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:01.306044+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:45:01.306044+0000', 'last_peered': '2024-09-05T16:45:01.306044+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:45:01.306044+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "29'2000", 'ondisk_log_start': "29'2000", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10064, 'log_dups_size': 0, 'ondisk_log_size': 10064, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8670960, 'num_objects': 1059, 'num_object_clones': 0, 'num_object_copies': 4236, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2284, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1059, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12061, 'num_write_kb': 52476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1222}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:45:06.814 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:45:06.814 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "39'11716", 'reported_seq': 11819, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:01.034821+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:45:01.034821+0000', 'last_peered': '2024-09-05T16:45:01.034821+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:45:01.034821+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "29'1700", 'ondisk_log_start': "29'1700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10016, 'log_dups_size': 0, 'ondisk_log_size': 10016, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8542800, 'num_objects': 1043, 'num_object_clones': 0, 'num_object_copies': 4172, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1043, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11715, 'num_write_kb': 51031, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1195}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:45:06.814 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:45:06.814 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "39'12095", 'reported_seq': 12208, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:01.306422+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:45:01.306422+0000', 'last_peered': '2024-09-05T16:45:01.306422+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:45:01.306422+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "29'2000", 'ondisk_log_start': "29'2000", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10095, 'log_dups_size': 0, 'ondisk_log_size': 10095, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8443040, 'num_objects': 1031, 'num_object_clones': 0, 'num_object_copies': 4124, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1031, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12093, 'num_write_kb': 52500, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1303}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:45:06.814 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:45:06.814 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "39'11834", 'reported_seq': 11952, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:44:59.969129+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:44:59.969129+0000', 'last_peered': '2024-09-05T16:44:59.969129+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:44:59.969129+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "29'1800", 'ondisk_log_start': "29'1800", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10034, 'log_dups_size': 0, 'ondisk_log_size': 10034, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8180896, 'num_objects': 999, 'num_object_clones': 0, 'num_object_copies': 3996, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2310, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 999, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11832, 'num_write_kb': 51325, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1309}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:45:06.814 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:45:06.814 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "39'11765", 'reported_seq': 11878, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:00.052042+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:45:00.052042+0000', 'last_peered': '2024-09-05T16:45:00.052042+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:45:00.052042+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "29'1700", 'ondisk_log_start': "29'1700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10065, 'log_dups_size': 0, 'ondisk_log_size': 10065, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8271008, 'num_objects': 1010, 'num_object_clones': 0, 'num_object_copies': 4040, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1010, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11763, 'num_write_kb': 51092, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1210}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:45:06.815 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:45:07.555 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:45:07.555 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 28422/60692 objects degraded (46.830%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":710,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":39,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":15173,"data_bytes":124255560,"bytes_used":2872160256,"bytes_avail":383674896384,"bytes_total":386547056640,"degraded_objects":28422,"degraded_total":60692,"degraded_ratio":0.46829895208594219,"recovering_objects_per_sec":9,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":112,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":254},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:45:07.584 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:45:07.584 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:45:08.196 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:45:08.197 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:45:08.219 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":385,"stamp":"2024-09-05T16:45:07.064845+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":124255560,"num_objects":15173,"num_object_clones":0,"num_object_copies":60692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28422,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15173,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":164977,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":944,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160986,"ondisk_log_size":160986,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2804844,"kb_used_data":246312,"kb_used_omap":14,"kb_used_meta":2558449,"kb_avail":374682516,"statfs":{"total":386547056640,"available":383674896384,"internally_reserved":0,"allocated":252223488,"data_stored":251381164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2619852343},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,8,7,17,11,4,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25075712,"num_objects":-3061,"num_object_clones":0,"num_object_copies":-12244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3061,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3061,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":112,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.011116"},"pg_stats":[{"pgid":"1.f","version":"39'11949","reported_seq":13389,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:06.475957+0000","last_change":"2024-09-05T16:43:43.436935+0000","last_active":"2024-09-05T16:45:06.475957+0000","last_peered":"2024-09-05T16:45:06.475957+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:45:06.475957+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"29'1900","ondisk_log_start":"29'1900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7823360,"num_objects":955,"num_object_clones":0,"num_object_copies":3820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1571,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11949,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":691,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-616}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'11694","reported_seq":16350,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:45:06.105005+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:45:06.105005+0000","last_peered":"2024-09-05T16:45:06.105005+0000","last_clean":"2024-09-05T16:45:06.105005+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:45:06.105005+0000","last_undegraded":"2024-09-05T16:45:06.105005+0000","last_fullsized":"2024-09-05T16:45:06.105005+0000","mapping_epoch":38,"log_start":"29'1600","ondisk_log_start":"29'1600","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7028736,"num_objects":858,"num_object_clones":0,"num_object_copies":3432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":858,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1007,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'11782","reported_seq":11909,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:05.170688+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:45:05.170688+0000","last_peered":"2024-09-05T16:45:05.170688+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:45:05.170688+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"29'1700","ondisk_log_start":"29'1700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7944808,"num_objects":971,"num_object_clones":0,"num_object_copies":3884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":971,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11781,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1261}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'11992","reported_seq":16726,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:45:06.093448+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:45:06.093448+0000","last_peered":"2024-09-05T16:45:06.093448+0000","last_clean":"2024-09-05T16:45:06.093448+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:45:06.093448+0000","last_undegraded":"2024-09-05T16:45:06.093448+0000","last_fullsized":"2024-09-05T16:45:06.093448+0000","mapping_epoch":34,"log_start":"29'1900","ondisk_log_start":"29'1900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7716864,"num_objects":942,"num_object_clones":0,"num_object_copies":3768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":942,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6292,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'11658","reported_seq":11762,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:05.951556+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:45:05.951556+0000","last_peered":"2024-09-05T16:45:05.951556+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:45:05.951556+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"29'1600","ondisk_log_start":"29'1600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7672992,"num_objects":937,"num_object_clones":0,"num_object_copies":3748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":937,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11656,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1208}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'11839","reported_seq":11958,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:06.222844+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:45:06.222844+0000","last_peered":"2024-09-05T16:45:06.222844+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:45:06.222844+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"29'1800","ondisk_log_start":"29'1800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7643136,"num_objects":933,"num_object_clones":0,"num_object_copies":3732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":933,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11839,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1286}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'11654","reported_seq":11767,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:05.418537+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:45:05.418537+0000","last_peered":"2024-09-05T16:45:05.418537+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:45:05.418537+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"29'1600","ondisk_log_start":"29'1600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8254624,"num_objects":1008,"num_object_clones":0,"num_object_copies":4032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1008,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11652,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1123}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'11804","reported_seq":11898,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:05.404006+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:45:05.404006+0000","last_peered":"2024-09-05T16:45:05.404006+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:45:05.404006+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"29'1800","ondisk_log_start":"29'1800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7984288,"num_objects":975,"num_object_clones":0,"num_object_copies":3900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":975,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11802,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1257}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"39'11883","reported_seq":11989,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:06.463278+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:45:06.463278+0000","last_peered":"2024-09-05T16:45:06.463278+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:45:06.463278+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"29'1800","ondisk_log_start":"29'1800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7293248,"num_objects":891,"num_object_clones":0,"num_object_copies":3564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":891,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11880,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1353}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'11855","reported_seq":11981,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:05.023825+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:45:05.023825+0000","last_peered":"2024-09-05T16:45:05.023825+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:45:05.023825+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"29'1800","ondisk_log_start":"29'1800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7820448,"num_objects":955,"num_object_clones":0,"num_object_copies":3820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11853,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1290}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'12129","reported_seq":12261,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:06.505302+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:45:06.505302+0000","last_peered":"2024-09-05T16:45:06.505302+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:45:06.505302+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"29'2100","ondisk_log_start":"29'2100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8138480,"num_objects":994,"num_object_clones":0,"num_object_copies":3976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":994,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12126,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1287}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'11786","reported_seq":11889,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:06.303033+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:45:06.303033+0000","last_peered":"2024-09-05T16:45:06.303033+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:45:06.303033+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"29'1700","ondisk_log_start":"29'1700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7969360,"num_objects":973,"num_object_clones":0,"num_object_copies":3892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":973,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11785,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1265}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'11886","reported_seq":16627,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:45:06.195006+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:45:06.195006+0000","last_peered":"2024-09-05T16:45:06.195006+0000","last_clean":"2024-09-05T16:45:06.195006+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:45:06.195006+0000","last_undegraded":"2024-09-05T16:45:06.195006+0000","last_fullsized":"2024-09-05T16:45:06.195006+0000","mapping_epoch":36,"log_start":"29'1800","ondisk_log_start":"29'1800","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7553024,"num_objects":922,"num_object_clones":0,"num_object_copies":3688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":922,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3486,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'12155","reported_seq":12268,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:06.418223+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:45:06.418223+0000","last_peered":"2024-09-05T16:45:06.418223+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:45:06.418223+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"29'2100","ondisk_log_start":"29'2100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7951520,"num_objects":971,"num_object_clones":0,"num_object_copies":3884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12153,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1363}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'11888","reported_seq":12005,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:05.129569+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:45:05.129569+0000","last_peered":"2024-09-05T16:45:05.129569+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:45:05.129569+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"29'1800","ondisk_log_start":"29'1800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7738528,"num_objects":945,"num_object_clones":0,"num_object_copies":3780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":945,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11886,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1363}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'11832","reported_seq":11943,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:05.439038+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:45:05.439038+0000","last_peered":"2024-09-05T16:45:05.439038+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:45:05.439038+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"29'1800","ondisk_log_start":"29'1800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7722144,"num_objects":943,"num_object_clones":0,"num_object_copies":3772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":943,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11830,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1277}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":124255560,"num_objects":15173,"num_object_clones":0,"num_object_copies":60692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28422,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15173,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":164977,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":944,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":249020416,"data_stored":249020416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160986,"ondisk_log_size":160986,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084421,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":738444,"kb_used_data":61180,"kb_used_omap":4,"kb_used_meta":677243,"kb_avail":93633396,"statfs":{"total":96636764160,"available":95880597504,"internally_reserved":0,"allocated":62648320,"data_stored":62437739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":693497555},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,5,3,6,3,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986298,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":580404,"kb_used_data":61408,"kb_used_omap":4,"kb_used_meta":518971,"kb_avail":93791436,"statfs":{"total":96636764160,"available":96042430464,"internally_reserved":0,"allocated":62881792,"data_stored":62671211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":531427026},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,4,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673104,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":743012,"kb_used_data":62100,"kb_used_omap":1,"kb_used_meta":680894,"kb_avail":93628828,"statfs":{"total":96636764160,"available":95875919872,"internally_reserved":0,"allocated":63590400,"data_stored":63379819,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":697235909},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1,4,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182535,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":742984,"kb_used_data":61624,"kb_used_omap":4,"kb_used_meta":681339,"kb_avail":93628856,"statfs":{"total":96636764160,"available":95875948544,"internally_reserved":0,"allocated":63102976,"data_stored":62892395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":697691853},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,3,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62541824,"data_stored":62541824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61988864,"data_stored":61988864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":62795776,"data_stored":62795776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61693952,"data_stored":61693952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:45:11.221 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:45:11.839 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:45:11.840 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:45:11.874 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":387,"stamp":"2024-09-05T16:45:11.066381+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":121249096,"num_objects":14806,"num_object_clones":0,"num_object_copies":59224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28422,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14806,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":165344,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":944,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160953,"ondisk_log_size":160953,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2805136,"kb_used_data":238276,"kb_used_omap":14,"kb_used_meta":2566769,"kb_avail":374682224,"statfs":{"total":386547056640,"available":383674597376,"internally_reserved":0,"allocated":243994624,"data_stored":243152300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2628372023},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,4,9,7,16,9,4,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19841024,"num_objects":-2422,"num_object_clones":0,"num_object_copies":-9688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-76,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2422,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2422,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":76,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.010929"},"pg_stats":[{"pgid":"1.f","version":"39'11949","reported_seq":13389,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:06.475957+0000","last_change":"2024-09-05T16:43:43.436935+0000","last_active":"2024-09-05T16:45:06.475957+0000","last_peered":"2024-09-05T16:45:06.475957+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:45:06.475957+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"29'1900","ondisk_log_start":"29'1900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7823360,"num_objects":955,"num_object_clones":0,"num_object_copies":3820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1571,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11949,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":691,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-616}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'11694","reported_seq":16350,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:45:06.105005+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:45:06.105005+0000","last_peered":"2024-09-05T16:45:06.105005+0000","last_clean":"2024-09-05T16:45:06.105005+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:45:06.105005+0000","last_undegraded":"2024-09-05T16:45:06.105005+0000","last_fullsized":"2024-09-05T16:45:06.105005+0000","mapping_epoch":38,"log_start":"29'1600","ondisk_log_start":"29'1600","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7028736,"num_objects":858,"num_object_clones":0,"num_object_copies":3432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":858,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1007,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'11837","reported_seq":11965,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:09.921725+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:45:09.921725+0000","last_peered":"2024-09-05T16:45:09.921725+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:45:09.921725+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"29'1800","ondisk_log_start":"29'1800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7494248,"num_objects":916,"num_object_clones":0,"num_object_copies":3664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":916,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11836,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1316}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'11992","reported_seq":16726,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:45:06.093448+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:45:06.093448+0000","last_peered":"2024-09-05T16:45:06.093448+0000","last_clean":"2024-09-05T16:45:06.093448+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:45:06.093448+0000","last_undegraded":"2024-09-05T16:45:06.093448+0000","last_fullsized":"2024-09-05T16:45:06.093448+0000","mapping_epoch":34,"log_start":"29'1900","ondisk_log_start":"29'1900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7716864,"num_objects":942,"num_object_clones":0,"num_object_copies":3768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":942,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6292,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'11658","reported_seq":11762,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:05.951556+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:45:05.951556+0000","last_peered":"2024-09-05T16:45:05.951556+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:45:05.951556+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"29'1600","ondisk_log_start":"29'1600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7672992,"num_objects":937,"num_object_clones":0,"num_object_copies":3748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":937,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11656,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1208}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'11839","reported_seq":11958,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:06.222844+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:45:06.222844+0000","last_peered":"2024-09-05T16:45:06.222844+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:45:06.222844+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"29'1800","ondisk_log_start":"29'1800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7643136,"num_objects":933,"num_object_clones":0,"num_object_copies":3732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":933,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11839,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1286}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'11722","reported_seq":11835,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:10.245187+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:45:10.245187+0000","last_peered":"2024-09-05T16:45:10.245187+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:45:10.245187+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"29'1700","ondisk_log_start":"29'1700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7697568,"num_objects":940,"num_object_clones":0,"num_object_copies":3760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":940,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11720,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1191}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'11875","reported_seq":11968,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:10.519924+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:45:10.519924+0000","last_peered":"2024-09-05T16:45:10.519924+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:45:10.519924+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"29'1800","ondisk_log_start":"29'1800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7402656,"num_objects":904,"num_object_clones":0,"num_object_copies":3616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":904,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11873,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1328}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"39'11883","reported_seq":11989,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:06.463278+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:45:06.463278+0000","last_peered":"2024-09-05T16:45:06.463278+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:45:06.463278+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"29'1800","ondisk_log_start":"29'1800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7293248,"num_objects":891,"num_object_clones":0,"num_object_copies":3564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":891,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11880,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1353}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'11925","reported_seq":12051,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:10.192933+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:45:10.192933+0000","last_peered":"2024-09-05T16:45:10.192933+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:45:10.192933+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"29'1900","ondisk_log_start":"29'1900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7247008,"num_objects":885,"num_object_clones":0,"num_object_copies":3540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":885,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11923,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1360}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'12129","reported_seq":12261,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:06.505302+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:45:06.505302+0000","last_peered":"2024-09-05T16:45:06.505302+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:45:06.505302+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"29'2100","ondisk_log_start":"29'2100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8138480,"num_objects":994,"num_object_clones":0,"num_object_copies":3976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":994,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12126,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1287}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'11786","reported_seq":11889,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:06.303033+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:45:06.303033+0000","last_peered":"2024-09-05T16:45:06.303033+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:45:06.303033+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"29'1700","ondisk_log_start":"29'1700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7969360,"num_objects":973,"num_object_clones":0,"num_object_copies":3892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":973,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11785,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1265}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'11886","reported_seq":16627,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:45:06.195006+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:45:06.195006+0000","last_peered":"2024-09-05T16:45:06.195006+0000","last_clean":"2024-09-05T16:45:06.195006+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:45:06.195006+0000","last_undegraded":"2024-09-05T16:45:06.195006+0000","last_fullsized":"2024-09-05T16:45:06.195006+0000","mapping_epoch":36,"log_start":"29'1800","ondisk_log_start":"29'1800","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7553024,"num_objects":922,"num_object_clones":0,"num_object_copies":3688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":922,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3486,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'12155","reported_seq":12268,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:06.418223+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:45:06.418223+0000","last_peered":"2024-09-05T16:45:06.418223+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:45:06.418223+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"29'2100","ondisk_log_start":"29'2100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7951520,"num_objects":971,"num_object_clones":0,"num_object_copies":3884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12153,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1363}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'11934","reported_seq":12052,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:10.100059+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:45:10.100059+0000","last_peered":"2024-09-05T16:45:10.100059+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:45:10.100059+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"29'1900","ondisk_log_start":"29'1900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7361696,"num_objects":899,"num_object_clones":0,"num_object_copies":3596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":899,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11932,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1409}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'11889","reported_seq":12002,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:10.317162+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:45:10.317162+0000","last_peered":"2024-09-05T16:45:10.317162+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:45:10.317162+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"29'1800","ondisk_log_start":"29'1800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7255200,"num_objects":886,"num_object_clones":0,"num_object_copies":3544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":886,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11887,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1334}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":121249096,"num_objects":14806,"num_object_clones":0,"num_object_copies":59224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28422,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14806,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":165344,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":944,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":240775168,"data_stored":240775168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160953,"ondisk_log_size":160953,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084421,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":738444,"kb_used_data":61180,"kb_used_omap":4,"kb_used_meta":677243,"kb_avail":93633396,"statfs":{"total":96636764160,"available":95880597504,"internally_reserved":0,"allocated":62648320,"data_stored":62437739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":693497555},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,5,3,6,3,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986298,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":580404,"kb_used_data":61408,"kb_used_omap":4,"kb_used_meta":518971,"kb_avail":93791436,"statfs":{"total":96636764160,"available":96042430464,"internally_reserved":0,"allocated":62881792,"data_stored":62671211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":531427026},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,4,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673105,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":743192,"kb_used_data":58120,"kb_used_omap":1,"kb_used_meta":685054,"kb_avail":93628648,"statfs":{"total":96636764160,"available":95875735552,"internally_reserved":0,"allocated":59514880,"data_stored":59304299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":701495749},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,0,0,0,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182536,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":743096,"kb_used_data":57568,"kb_used_omap":4,"kb_used_meta":685499,"kb_avail":93628744,"statfs":{"total":96636764160,"available":95875833856,"internally_reserved":0,"allocated":58949632,"data_stored":58739051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":701951693},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,1,6,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":58408960,"data_stored":58408960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61988864,"data_stored":61988864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":58683392,"data_stored":58683392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61693952,"data_stored":61693952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:45:11.875 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:45:12.465 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:45:12.466 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:45:12.490 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":387,"stamp":"2024-09-05T16:45:11.066381+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":121249096,"num_objects":14806,"num_object_clones":0,"num_object_copies":59224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28422,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14806,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":165344,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":944,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160953,"ondisk_log_size":160953,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2805136,"kb_used_data":238276,"kb_used_omap":14,"kb_used_meta":2566769,"kb_avail":374682224,"statfs":{"total":386547056640,"available":383674597376,"internally_reserved":0,"allocated":243994624,"data_stored":243152300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2628372023},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,4,9,7,16,9,4,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19841024,"num_objects":-2422,"num_object_clones":0,"num_object_copies":-9688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-76,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2422,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2422,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":76,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.010929"},"pg_stats":[{"pgid":"1.f","version":"39'11949","reported_seq":13389,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:06.475957+0000","last_change":"2024-09-05T16:43:43.436935+0000","last_active":"2024-09-05T16:45:06.475957+0000","last_peered":"2024-09-05T16:45:06.475957+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:45:06.475957+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"29'1900","ondisk_log_start":"29'1900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7823360,"num_objects":955,"num_object_clones":0,"num_object_copies":3820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1571,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11949,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":691,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-616}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'11694","reported_seq":16350,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:45:06.105005+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:45:06.105005+0000","last_peered":"2024-09-05T16:45:06.105005+0000","last_clean":"2024-09-05T16:45:06.105005+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:45:06.105005+0000","last_undegraded":"2024-09-05T16:45:06.105005+0000","last_fullsized":"2024-09-05T16:45:06.105005+0000","mapping_epoch":38,"log_start":"29'1600","ondisk_log_start":"29'1600","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7028736,"num_objects":858,"num_object_clones":0,"num_object_copies":3432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":858,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1007,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'11837","reported_seq":11965,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:09.921725+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:45:09.921725+0000","last_peered":"2024-09-05T16:45:09.921725+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:45:09.921725+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"29'1800","ondisk_log_start":"29'1800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7494248,"num_objects":916,"num_object_clones":0,"num_object_copies":3664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":916,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11836,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1316}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'11992","reported_seq":16726,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:45:06.093448+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:45:06.093448+0000","last_peered":"2024-09-05T16:45:06.093448+0000","last_clean":"2024-09-05T16:45:06.093448+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:45:06.093448+0000","last_undegraded":"2024-09-05T16:45:06.093448+0000","last_fullsized":"2024-09-05T16:45:06.093448+0000","mapping_epoch":34,"log_start":"29'1900","ondisk_log_start":"29'1900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7716864,"num_objects":942,"num_object_clones":0,"num_object_copies":3768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":942,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6292,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'11658","reported_seq":11762,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:05.951556+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:45:05.951556+0000","last_peered":"2024-09-05T16:45:05.951556+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:45:05.951556+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"29'1600","ondisk_log_start":"29'1600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7672992,"num_objects":937,"num_object_clones":0,"num_object_copies":3748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":937,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11656,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1208}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'11839","reported_seq":11958,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:06.222844+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:45:06.222844+0000","last_peered":"2024-09-05T16:45:06.222844+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:45:06.222844+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"29'1800","ondisk_log_start":"29'1800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7643136,"num_objects":933,"num_object_clones":0,"num_object_copies":3732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":933,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11839,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1286}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'11722","reported_seq":11835,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:10.245187+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:45:10.245187+0000","last_peered":"2024-09-05T16:45:10.245187+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:45:10.245187+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"29'1700","ondisk_log_start":"29'1700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7697568,"num_objects":940,"num_object_clones":0,"num_object_copies":3760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":940,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11720,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1191}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'11875","reported_seq":11968,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:10.519924+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:45:10.519924+0000","last_peered":"2024-09-05T16:45:10.519924+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:45:10.519924+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"29'1800","ondisk_log_start":"29'1800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7402656,"num_objects":904,"num_object_clones":0,"num_object_copies":3616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":904,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11873,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1328}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"39'11883","reported_seq":11989,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:06.463278+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:45:06.463278+0000","last_peered":"2024-09-05T16:45:06.463278+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:45:06.463278+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"29'1800","ondisk_log_start":"29'1800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7293248,"num_objects":891,"num_object_clones":0,"num_object_copies":3564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":891,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11880,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1353}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'11925","reported_seq":12051,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:10.192933+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:45:10.192933+0000","last_peered":"2024-09-05T16:45:10.192933+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:45:10.192933+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"29'1900","ondisk_log_start":"29'1900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7247008,"num_objects":885,"num_object_clones":0,"num_object_copies":3540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":885,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11923,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1360}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'12129","reported_seq":12261,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:06.505302+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:45:06.505302+0000","last_peered":"2024-09-05T16:45:06.505302+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:45:06.505302+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"29'2100","ondisk_log_start":"29'2100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8138480,"num_objects":994,"num_object_clones":0,"num_object_copies":3976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":994,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12126,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1287}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'11786","reported_seq":11889,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:06.303033+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:45:06.303033+0000","last_peered":"2024-09-05T16:45:06.303033+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:45:06.303033+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"29'1700","ondisk_log_start":"29'1700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7969360,"num_objects":973,"num_object_clones":0,"num_object_copies":3892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":973,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11785,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1265}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'11886","reported_seq":16627,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:45:06.195006+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:45:06.195006+0000","last_peered":"2024-09-05T16:45:06.195006+0000","last_clean":"2024-09-05T16:45:06.195006+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:45:06.195006+0000","last_undegraded":"2024-09-05T16:45:06.195006+0000","last_fullsized":"2024-09-05T16:45:06.195006+0000","mapping_epoch":36,"log_start":"29'1800","ondisk_log_start":"29'1800","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7553024,"num_objects":922,"num_object_clones":0,"num_object_copies":3688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":922,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3486,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'12155","reported_seq":12268,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:06.418223+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:45:06.418223+0000","last_peered":"2024-09-05T16:45:06.418223+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:45:06.418223+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"29'2100","ondisk_log_start":"29'2100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7951520,"num_objects":971,"num_object_clones":0,"num_object_copies":3884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12153,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1363}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'11934","reported_seq":12052,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:10.100059+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:45:10.100059+0000","last_peered":"2024-09-05T16:45:10.100059+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:45:10.100059+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"29'1900","ondisk_log_start":"29'1900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7361696,"num_objects":899,"num_object_clones":0,"num_object_copies":3596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":899,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11932,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1409}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'11889","reported_seq":12002,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:10.317162+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:45:10.317162+0000","last_peered":"2024-09-05T16:45:10.317162+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:45:10.317162+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"29'1800","ondisk_log_start":"29'1800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7255200,"num_objects":886,"num_object_clones":0,"num_object_copies":3544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":886,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11887,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1334}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":121249096,"num_objects":14806,"num_object_clones":0,"num_object_copies":59224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28422,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14806,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":165344,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":944,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":240775168,"data_stored":240775168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160953,"ondisk_log_size":160953,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084421,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":738444,"kb_used_data":61180,"kb_used_omap":4,"kb_used_meta":677243,"kb_avail":93633396,"statfs":{"total":96636764160,"available":95880597504,"internally_reserved":0,"allocated":62648320,"data_stored":62437739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":693497555},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,5,3,6,3,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986298,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":580404,"kb_used_data":61408,"kb_used_omap":4,"kb_used_meta":518971,"kb_avail":93791436,"statfs":{"total":96636764160,"available":96042430464,"internally_reserved":0,"allocated":62881792,"data_stored":62671211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":531427026},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,4,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673105,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":743192,"kb_used_data":58120,"kb_used_omap":1,"kb_used_meta":685054,"kb_avail":93628648,"statfs":{"total":96636764160,"available":95875735552,"internally_reserved":0,"allocated":59514880,"data_stored":59304299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":701495749},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,0,0,0,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182536,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":743096,"kb_used_data":57568,"kb_used_omap":4,"kb_used_meta":685499,"kb_avail":93628744,"statfs":{"total":96636764160,"available":95875833856,"internally_reserved":0,"allocated":58949632,"data_stored":58739051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":701951693},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,1,6,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":58408960,"data_stored":58408960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61988864,"data_stored":61988864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":58683392,"data_stored":58683392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61693952,"data_stored":61693952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:45:12.491 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-05T16:45:12.492 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "39'11949", 'reported_seq': 13389, 'reported_epoch': 39, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:06.475957+0000', 'last_change': '2024-09-05T16:43:43.436935+0000', 'last_active': '2024-09-05T16:45:06.475957+0000', 'last_peered': '2024-09-05T16:45:06.475957+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.828289+0000', 'last_became_peered': '2024-09-05T16:34:56.828289+0000', 'last_unstale': '2024-09-05T16:45:06.475957+0000', 'last_undegraded': '2024-09-05T16:34:56.612160+0000', 'last_fullsized': '2024-09-05T16:34:56.557632+0000', 'mapping_epoch': 32, 'log_start': "29'1900", 'ondisk_log_start': "29'1900", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10049, 'log_dups_size': 0, 'ondisk_log_size': 10049, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7823360, 'num_objects': 955, 'num_object_clones': 0, 'num_object_copies': 3820, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1571, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 955, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11949, 'num_write_kb': 51616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 691, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -616}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:45:12.492 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:45:12.492 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "39'11837", 'reported_seq': 11965, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:09.921725+0000', 'last_change': '2024-09-05T16:34:56.867900+0000', 'last_active': '2024-09-05T16:45:09.921725+0000', 'last_peered': '2024-09-05T16:45:09.921725+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:45:09.921725+0000', 'last_undegraded': '2024-09-05T16:34:56.590972+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 32, 'log_start': "29'1800", 'ondisk_log_start': "29'1800", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10037, 'log_dups_size': 0, 'ondisk_log_size': 10037, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7494248, 'num_objects': 916, 'num_object_clones': 0, 'num_object_copies': 3664, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2233, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 916, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 11836, 'num_write_kb': 50999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1316}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:45:12.492 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:45:12.492 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "39'11658", 'reported_seq': 11762, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:05.951556+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:45:05.951556+0000', 'last_peered': '2024-09-05T16:45:05.951556+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:45:05.951556+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "29'1600", 'ondisk_log_start': "29'1600", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10058, 'log_dups_size': 0, 'ondisk_log_size': 10058, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7672992, 'num_objects': 937, 'num_object_clones': 0, 'num_object_copies': 3748, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2147, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 937, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11656, 'num_write_kb': 50373, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1208}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:45:12.492 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:45:12.492 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "39'11839", 'reported_seq': 11958, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:06.222844+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:45:06.222844+0000', 'last_peered': '2024-09-05T16:45:06.222844+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:45:06.222844+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "29'1800", 'ondisk_log_start': "29'1800", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10039, 'log_dups_size': 0, 'ondisk_log_size': 10039, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7643136, 'num_objects': 933, 'num_object_clones': 0, 'num_object_copies': 3732, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 933, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11839, 'num_write_kb': 51088, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1286}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:45:12.492 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:45:12.492 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "39'11722", 'reported_seq': 11835, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:10.245187+0000', 'last_change': '2024-09-05T16:34:56.903436+0000', 'last_active': '2024-09-05T16:45:10.245187+0000', 'last_peered': '2024-09-05T16:45:10.245187+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:45:10.245187+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "29'1700", 'ondisk_log_start': "29'1700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10022, 'log_dups_size': 0, 'ondisk_log_size': 10022, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7697568, 'num_objects': 940, 'num_object_clones': 0, 'num_object_copies': 3760, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2133, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 940, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11720, 'num_write_kb': 50636, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -1191}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:45:12.493 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-05T16:45:12.493 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "39'11875", 'reported_seq': 11968, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:10.519924+0000', 'last_change': '2024-09-05T16:34:56.883319+0000', 'last_active': '2024-09-05T16:45:10.519924+0000', 'last_peered': '2024-09-05T16:45:10.519924+0000', 'last_clean': '2024-09-05T16:33:53.505305+0000', 'last_became_active': '2024-09-05T16:34:56.873389+0000', 'last_became_peered': '2024-09-05T16:34:56.873389+0000', 'last_unstale': '2024-09-05T16:45:10.519924+0000', 'last_undegraded': '2024-09-05T16:34:56.602567+0000', 'last_fullsized': '2024-09-05T16:34:56.559958+0000', 'mapping_epoch': 32, 'log_start': "29'1800", 'ondisk_log_start': "29'1800", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10075, 'log_dups_size': 0, 'ondisk_log_size': 10075, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7402656, 'num_objects': 904, 'num_object_clones': 0, 'num_object_copies': 3616, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2234, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 904, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11873, 'num_write_kb': 51108, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1328}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:45:12.493 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:45:12.493 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "39'11883", 'reported_seq': 11989, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:06.463278+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:45:06.463278+0000', 'last_peered': '2024-09-05T16:45:06.463278+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:45:06.463278+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "29'1800", 'ondisk_log_start': "29'1800", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10083, 'log_dups_size': 0, 'ondisk_log_size': 10083, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7293248, 'num_objects': 891, 'num_object_clones': 0, 'num_object_copies': 3564, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 891, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11880, 'num_write_kb': 51081, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1353}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:45:12.493 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:45:12.493 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "39'11925", 'reported_seq': 12051, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:10.192933+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:45:10.192933+0000', 'last_peered': '2024-09-05T16:45:10.192933+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:45:10.192933+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "29'1900", 'ondisk_log_start': "29'1900", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10025, 'log_dups_size': 0, 'ondisk_log_size': 10025, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7247008, 'num_objects': 885, 'num_object_clones': 0, 'num_object_copies': 3540, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 885, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11923, 'num_write_kb': 51230, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1360}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:45:12.493 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:45:12.493 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "39'12129", 'reported_seq': 12261, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:06.505302+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:45:06.505302+0000', 'last_peered': '2024-09-05T16:45:06.505302+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:45:06.505302+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "29'2100", 'ondisk_log_start': "29'2100", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10029, 'log_dups_size': 0, 'ondisk_log_size': 10029, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8138480, 'num_objects': 994, 'num_object_clones': 0, 'num_object_copies': 3976, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2284, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 994, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12126, 'num_write_kb': 52476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1287}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:45:12.493 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:45:12.493 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "39'11786", 'reported_seq': 11889, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:06.303033+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:45:06.303033+0000', 'last_peered': '2024-09-05T16:45:06.303033+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:45:06.303033+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "29'1700", 'ondisk_log_start': "29'1700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10086, 'log_dups_size': 0, 'ondisk_log_size': 10086, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7969360, 'num_objects': 973, 'num_object_clones': 0, 'num_object_copies': 3892, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 973, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11785, 'num_write_kb': 51031, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1265}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:45:12.493 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:45:12.494 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "39'12155", 'reported_seq': 12268, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:06.418223+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:45:06.418223+0000', 'last_peered': '2024-09-05T16:45:06.418223+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:45:06.418223+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "29'2100", 'ondisk_log_start': "29'2100", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10055, 'log_dups_size': 0, 'ondisk_log_size': 10055, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7951520, 'num_objects': 971, 'num_object_clones': 0, 'num_object_copies': 3884, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 971, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12153, 'num_write_kb': 52500, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1363}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:45:12.494 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:45:12.494 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "39'11934", 'reported_seq': 12052, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:10.100059+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:45:10.100059+0000', 'last_peered': '2024-09-05T16:45:10.100059+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:45:10.100059+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "29'1900", 'ondisk_log_start': "29'1900", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10034, 'log_dups_size': 0, 'ondisk_log_size': 10034, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7361696, 'num_objects': 899, 'num_object_clones': 0, 'num_object_copies': 3596, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2310, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 899, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11932, 'num_write_kb': 51325, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1409}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:45:12.494 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:45:12.494 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "39'11889", 'reported_seq': 12002, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:10.317162+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:45:10.317162+0000', 'last_peered': '2024-09-05T16:45:10.317162+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:45:10.317162+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "29'1800", 'ondisk_log_start': "29'1800", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10089, 'log_dups_size': 0, 'ondisk_log_size': 10089, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7255200, 'num_objects': 886, 'num_object_clones': 0, 'num_object_copies': 3544, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 886, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11887, 'num_write_kb': 51092, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1334}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:45:12.494 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:45:13.276 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:45:13.276 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 28422/59224 objects degraded (47.991%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":716,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":39,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":14806,"data_bytes":121249096,"bytes_used":2872459264,"bytes_avail":383674597376,"bytes_total":386547056640,"degraded_objects":28422,"degraded_total":59224,"degraded_ratio":0.47990679454275292,"recovering_objects_per_sec":6,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":76,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":201},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:45:13.308 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:45:13.308 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:45:13.909 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:45:13.910 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:45:13.936 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":388,"stamp":"2024-09-05T16:45:13.067214+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":115891528,"num_objects":14152,"num_object_clones":0,"num_object_copies":56608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28385,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14152,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":165998,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":981,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160707,"ondisk_log_size":160707,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2805444,"kb_used_data":230256,"kb_used_omap":14,"kb_used_meta":2575089,"kb_avail":374681916,"statfs":{"total":386547056640,"available":383674281984,"internally_reserved":0,"allocated":235782144,"data_stored":234939820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2636891703},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,4,15,9,8,14,7,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22167552,"num_objects":-2706,"num_object_clones":0,"num_object_copies":-10824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-113,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2706,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2706,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":113,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005101"},"pg_stats":[{"pgid":"1.f","version":"39'12015","reported_seq":13529,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:11.554995+0000","last_change":"2024-09-05T16:43:43.436935+0000","last_active":"2024-09-05T16:45:11.554995+0000","last_peered":"2024-09-05T16:45:11.554995+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:45:11.554995+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"29'2000","ondisk_log_start":"29'2000","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7282688,"num_objects":889,"num_object_clones":0,"num_object_copies":3556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":889,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12015,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":728,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-645}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'11756","reported_seq":16412,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:45:11.068278+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:45:11.068278+0000","last_peered":"2024-09-05T16:45:11.068278+0000","last_clean":"2024-09-05T16:45:11.068278+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:45:11.068278+0000","last_undegraded":"2024-09-05T16:45:11.068278+0000","last_fullsized":"2024-09-05T16:45:11.068278+0000","mapping_epoch":38,"log_start":"29'1700","ondisk_log_start":"29'1700","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6520832,"num_objects":796,"num_object_clones":0,"num_object_copies":3184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":796,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1069,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'11837","reported_seq":11965,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:09.921725+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:45:09.921725+0000","last_peered":"2024-09-05T16:45:09.921725+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:45:09.921725+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"29'1800","ondisk_log_start":"29'1800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7494248,"num_objects":916,"num_object_clones":0,"num_object_copies":3664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":916,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11836,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1316}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12067","reported_seq":16801,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:45:11.220377+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:45:11.220377+0000","last_peered":"2024-09-05T16:45:11.220377+0000","last_clean":"2024-09-05T16:45:11.220377+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:45:11.220377+0000","last_undegraded":"2024-09-05T16:45:11.220377+0000","last_fullsized":"2024-09-05T16:45:11.220377+0000","mapping_epoch":34,"log_start":"29'2000","ondisk_log_start":"29'2000","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7102464,"num_objects":867,"num_object_clones":0,"num_object_copies":3468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":867,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6367,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'11730","reported_seq":11834,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:11.375154+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:45:11.375154+0000","last_peered":"2024-09-05T16:45:11.375154+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:45:11.375154+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"29'1700","ondisk_log_start":"29'1700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7083168,"num_objects":865,"num_object_clones":0,"num_object_copies":3460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":865,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11728,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1280}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'11903","reported_seq":12022,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:11.598216+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:45:11.598216+0000","last_peered":"2024-09-05T16:45:11.598216+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:45:11.598216+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"29'1900","ondisk_log_start":"29'1900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7118848,"num_objects":869,"num_object_clones":0,"num_object_copies":3476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":869,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11903,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1350}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'11722","reported_seq":11835,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:10.245187+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:45:10.245187+0000","last_peered":"2024-09-05T16:45:10.245187+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:45:10.245187+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"29'1700","ondisk_log_start":"29'1700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7697568,"num_objects":940,"num_object_clones":0,"num_object_copies":3760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":940,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11720,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1191}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'11875","reported_seq":11968,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:10.519924+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:45:10.519924+0000","last_peered":"2024-09-05T16:45:10.519924+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:45:10.519924+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"29'1800","ondisk_log_start":"29'1800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7402656,"num_objects":904,"num_object_clones":0,"num_object_copies":3616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":904,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11873,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1328}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"39'11953","reported_seq":12059,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:11.566512+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:45:11.566512+0000","last_peered":"2024-09-05T16:45:11.566512+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:45:11.566512+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"29'1900","ondisk_log_start":"29'1900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6719808,"num_objects":821,"num_object_clones":0,"num_object_copies":3284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":821,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11950,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1423}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'11925","reported_seq":12051,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:10.192933+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:45:10.192933+0000","last_peered":"2024-09-05T16:45:10.192933+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:45:10.192933+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"29'1900","ondisk_log_start":"29'1900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7247008,"num_objects":885,"num_object_clones":0,"num_object_copies":3540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":885,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11923,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1360}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'12193","reported_seq":12324,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:11.536574+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:45:11.536574+0000","last_peered":"2024-09-05T16:45:11.536574+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:45:11.536574+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"29'2100","ondisk_log_start":"29'2100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7614192,"num_objects":930,"num_object_clones":0,"num_object_copies":3720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":930,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12190,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1351}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'11851","reported_seq":11953,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:11.712656+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:45:11.712656+0000","last_peered":"2024-09-05T16:45:11.712656+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:45:11.712656+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"29'1800","ondisk_log_start":"29'1800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7436880,"num_objects":908,"num_object_clones":0,"num_object_copies":3632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":908,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11850,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1330}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'11945","reported_seq":16686,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:45:11.072858+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:45:11.072858+0000","last_peered":"2024-09-05T16:45:11.072858+0000","last_clean":"2024-09-05T16:45:11.072858+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:45:11.072858+0000","last_undegraded":"2024-09-05T16:45:11.072858+0000","last_fullsized":"2024-09-05T16:45:11.072858+0000","mapping_epoch":36,"log_start":"29'1900","ondisk_log_start":"29'1900","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7069696,"num_objects":863,"num_object_clones":0,"num_object_copies":3452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":863,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3545,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'12212","reported_seq":12325,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:11.459782+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:45:11.459782+0000","last_peered":"2024-09-05T16:45:11.459782+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:45:11.459782+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"29'2200","ondisk_log_start":"29'2200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7484576,"num_objects":914,"num_object_clones":0,"num_object_copies":3656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":914,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12210,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1420}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'11934","reported_seq":12052,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:10.100059+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:45:10.100059+0000","last_peered":"2024-09-05T16:45:10.100059+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:45:10.100059+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"29'1900","ondisk_log_start":"29'1900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7361696,"num_objects":899,"num_object_clones":0,"num_object_copies":3596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":899,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11932,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1409}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'11889","reported_seq":12002,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:10.317162+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:45:10.317162+0000","last_peered":"2024-09-05T16:45:10.317162+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:45:10.317162+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"29'1800","ondisk_log_start":"29'1800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7255200,"num_objects":886,"num_object_clones":0,"num_object_copies":3544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":886,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11887,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1334}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":115891528,"num_objects":14152,"num_object_clones":0,"num_object_copies":56608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28385,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14152,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":165998,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":981,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":232390656,"data_stored":232390656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160707,"ondisk_log_size":160707,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084422,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":738588,"kb_used_data":57160,"kb_used_omap":4,"kb_used_meta":681403,"kb_avail":93633252,"statfs":{"total":96636764160,"available":95880450048,"internally_reserved":0,"allocated":58531840,"data_stored":58321259,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":697757395},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,7,1,2,7,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986299,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":580568,"kb_used_data":57408,"kb_used_omap":4,"kb_used_meta":523131,"kb_avail":93791272,"statfs":{"total":96636764160,"available":96042262528,"internally_reserved":0,"allocated":58785792,"data_stored":58575211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":535686866},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,5,7,0,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673105,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":743192,"kb_used_data":58120,"kb_used_omap":1,"kb_used_meta":685054,"kb_avail":93628648,"statfs":{"total":96636764160,"available":95875735552,"internally_reserved":0,"allocated":59514880,"data_stored":59304299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":701495749},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,0,0,0,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182536,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":743096,"kb_used_data":57568,"kb_used_omap":4,"kb_used_meta":685499,"kb_avail":93628744,"statfs":{"total":96636764160,"available":95875833856,"internally_reserved":0,"allocated":58949632,"data_stored":58739051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":701951693},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,1,6,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":58408960,"data_stored":58408960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57839616,"data_stored":57839616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":58683392,"data_stored":58683392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":57458688,"data_stored":57458688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:45:16.938 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:45:17.604 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:45:17.604 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:45:17.626 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":390,"stamp":"2024-09-05T16:45:17.068440+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":107519304,"num_objects":13130,"num_object_clones":0,"num_object_copies":52520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28346,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13130,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":167020,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1020,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160729,"ondisk_log_size":160729,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2801204,"kb_used_data":213556,"kb_used_omap":14,"kb_used_meta":2587569,"kb_avail":374686156,"statfs":{"total":386547056640,"available":383678623744,"internally_reserved":0,"allocated":218681344,"data_stored":217839020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2649671223},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,17,12,10,11],"upper_bound":256},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25001984,"num_objects":-3052,"num_object_clones":0,"num_object_copies":-12208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-114,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3052,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3052,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":114,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004602"},"pg_stats":[{"pgid":"1.f","version":"39'12097","reported_seq":13689,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:16.865841+0000","last_change":"2024-09-05T16:43:43.436935+0000","last_active":"2024-09-05T16:45:16.865841+0000","last_peered":"2024-09-05T16:45:16.865841+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:45:16.865841+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"29'2000","ondisk_log_start":"29'2000","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6610944,"num_objects":807,"num_object_clones":0,"num_object_copies":3228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1495,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":807,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12097,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":767,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-688}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'11818","reported_seq":16474,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:45:16.268560+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:45:16.268560+0000","last_peered":"2024-09-05T16:45:16.268560+0000","last_clean":"2024-09-05T16:45:16.268560+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:45:16.268560+0000","last_undegraded":"2024-09-05T16:45:16.268560+0000","last_fullsized":"2024-09-05T16:45:16.268560+0000","mapping_epoch":38,"log_start":"29'1800","ondisk_log_start":"29'1800","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6012928,"num_objects":734,"num_object_clones":0,"num_object_copies":2936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":734,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1131,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'11901","reported_seq":12029,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:15.278453+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:45:15.278453+0000","last_peered":"2024-09-05T16:45:15.278453+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:45:15.278453+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"29'1900","ondisk_log_start":"29'1900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969960,"num_objects":852,"num_object_clones":0,"num_object_copies":3408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":852,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11900,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1380}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12122","reported_seq":16856,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:45:16.019609+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:45:16.019609+0000","last_peered":"2024-09-05T16:45:16.019609+0000","last_clean":"2024-09-05T16:45:16.019609+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:45:16.019609+0000","last_undegraded":"2024-09-05T16:45:16.019609+0000","last_fullsized":"2024-09-05T16:45:16.019609+0000","mapping_epoch":34,"log_start":"29'2100","ondisk_log_start":"29'2100","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6651904,"num_objects":812,"num_object_clones":0,"num_object_copies":3248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":812,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6422,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'11810","reported_seq":11914,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:16.685502+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:45:16.685502+0000","last_peered":"2024-09-05T16:45:16.685502+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:45:16.685502+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"29'1800","ondisk_log_start":"29'1800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6427808,"num_objects":785,"num_object_clones":0,"num_object_copies":3140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":785,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11808,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1360}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'11965","reported_seq":12083,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:16.831092+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:45:16.831092+0000","last_peered":"2024-09-05T16:45:16.831092+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:45:16.831092+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"29'1900","ondisk_log_start":"29'1900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6610944,"num_objects":807,"num_object_clones":0,"num_object_copies":3228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":807,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11965,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1412}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'11782","reported_seq":11894,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:15.548458+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:45:15.548458+0000","last_peered":"2024-09-05T16:45:15.548458+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:45:15.548458+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"29'1700","ondisk_log_start":"29'1700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7206048,"num_objects":880,"num_object_clones":0,"num_object_copies":3520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":880,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11780,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1251}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'11938","reported_seq":12032,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:15.540741+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:45:15.540741+0000","last_peered":"2024-09-05T16:45:15.540741+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:45:15.540741+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"29'1900","ondisk_log_start":"29'1900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6886560,"num_objects":841,"num_object_clones":0,"num_object_copies":3364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":841,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11936,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1391}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"39'12015","reported_seq":12121,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:16.714203+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:45:16.714203+0000","last_peered":"2024-09-05T16:45:16.714203+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:45:16.714203+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"29'2000","ondisk_log_start":"29'2000","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6211904,"num_objects":759,"num_object_clones":0,"num_object_copies":3036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":759,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12012,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1485}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'11980","reported_seq":12107,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:15.113482+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:45:15.113482+0000","last_peered":"2024-09-05T16:45:15.113482+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:45:15.113482+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"29'1900","ondisk_log_start":"29'1900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6796448,"num_objects":830,"num_object_clones":0,"num_object_copies":3320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":830,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11978,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1415}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'12256","reported_seq":12387,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:16.673664+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:45:16.673664+0000","last_peered":"2024-09-05T16:45:16.673664+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:45:16.673664+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"29'2200","ondisk_log_start":"29'2200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7098096,"num_objects":867,"num_object_clones":0,"num_object_copies":3468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":867,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12253,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1414}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'11923","reported_seq":12027,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:16.818748+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:45:16.818748+0000","last_peered":"2024-09-05T16:45:16.818748+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:45:16.818748+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"29'1900","ondisk_log_start":"29'1900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6847056,"num_objects":836,"num_object_clones":0,"num_object_copies":3344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":836,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11922,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1402}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12006","reported_seq":16747,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:45:15.952393+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:45:15.952393+0000","last_peered":"2024-09-05T16:45:15.952393+0000","last_clean":"2024-09-05T16:45:15.952393+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:45:15.952393+0000","last_undegraded":"2024-09-05T16:45:15.952393+0000","last_fullsized":"2024-09-05T16:45:15.952393+0000","mapping_epoch":36,"log_start":"29'2000","ondisk_log_start":"29'2000","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6569984,"num_objects":802,"num_object_clones":0,"num_object_copies":3208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":802,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3606,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'12280","reported_seq":12394,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:16.783568+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:45:16.783568+0000","last_peered":"2024-09-05T16:45:16.783568+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:45:16.783568+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"29'2200","ondisk_log_start":"29'2200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6927520,"num_objects":846,"num_object_clones":0,"num_object_copies":3384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12278,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1488}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'11989","reported_seq":12107,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:15.121404+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:45:15.121404+0000","last_peered":"2024-09-05T16:45:15.121404+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:45:15.121404+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"29'1900","ondisk_log_start":"29'1900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6911136,"num_objects":844,"num_object_clones":0,"num_object_copies":3376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":844,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11987,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1464}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'11947","reported_seq":12059,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:15.541758+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:45:15.541758+0000","last_peered":"2024-09-05T16:45:15.541758+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:45:15.541758+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"29'1900","ondisk_log_start":"29'1900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6780064,"num_objects":828,"num_object_clones":0,"num_object_copies":3312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":828,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11945,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1392}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":107519304,"num_objects":13130,"num_object_clones":0,"num_object_copies":52520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28346,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13130,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":167020,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1020,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":215982080,"data_stored":215982080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160729,"ondisk_log_size":160729,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084423,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":737952,"kb_used_data":52368,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93633888,"statfs":{"total":96636764160,"available":95881101312,"internally_reserved":0,"allocated":53624832,"data_stored":53414251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":702017235},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,7,0,3,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986300,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":576616,"kb_used_data":53464,"kb_used_omap":4,"kb_used_meta":523131,"kb_avail":93795224,"statfs":{"total":96636764160,"available":96046309376,"internally_reserved":0,"allocated":54747136,"data_stored":54536555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":535686866},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673106,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":743300,"kb_used_data":54068,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93628540,"statfs":{"total":96636764160,"available":95875624960,"internally_reserved":0,"allocated":55365632,"data_stored":55155051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4,3,6,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182537,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":743336,"kb_used_data":53656,"kb_used_omap":4,"kb_used_meta":689659,"kb_avail":93628504,"statfs":{"total":96636764160,"available":95875588096,"internally_reserved":0,"allocated":54943744,"data_stored":54733163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":706211533},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,6,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":54358016,"data_stored":54358016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53796864,"data_stored":53796864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54595584,"data_stored":54595584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53231616,"data_stored":53231616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:45:17.627 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:45:18.264 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:45:18.264 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:45:18.291 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":390,"stamp":"2024-09-05T16:45:17.068440+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":107519304,"num_objects":13130,"num_object_clones":0,"num_object_copies":52520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28346,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13130,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":167020,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1020,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160729,"ondisk_log_size":160729,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2801204,"kb_used_data":213556,"kb_used_omap":14,"kb_used_meta":2587569,"kb_avail":374686156,"statfs":{"total":386547056640,"available":383678623744,"internally_reserved":0,"allocated":218681344,"data_stored":217839020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2649671223},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,17,12,10,11],"upper_bound":256},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25001984,"num_objects":-3052,"num_object_clones":0,"num_object_copies":-12208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-114,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3052,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3052,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":114,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004602"},"pg_stats":[{"pgid":"1.f","version":"39'12097","reported_seq":13689,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:16.865841+0000","last_change":"2024-09-05T16:43:43.436935+0000","last_active":"2024-09-05T16:45:16.865841+0000","last_peered":"2024-09-05T16:45:16.865841+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:45:16.865841+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"29'2000","ondisk_log_start":"29'2000","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6610944,"num_objects":807,"num_object_clones":0,"num_object_copies":3228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1495,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":807,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12097,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":767,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-688}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'11818","reported_seq":16474,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:45:16.268560+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:45:16.268560+0000","last_peered":"2024-09-05T16:45:16.268560+0000","last_clean":"2024-09-05T16:45:16.268560+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:45:16.268560+0000","last_undegraded":"2024-09-05T16:45:16.268560+0000","last_fullsized":"2024-09-05T16:45:16.268560+0000","mapping_epoch":38,"log_start":"29'1800","ondisk_log_start":"29'1800","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6012928,"num_objects":734,"num_object_clones":0,"num_object_copies":2936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":734,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1131,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'11901","reported_seq":12029,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:15.278453+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:45:15.278453+0000","last_peered":"2024-09-05T16:45:15.278453+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:45:15.278453+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"29'1900","ondisk_log_start":"29'1900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969960,"num_objects":852,"num_object_clones":0,"num_object_copies":3408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":852,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11900,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1380}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12122","reported_seq":16856,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:45:16.019609+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:45:16.019609+0000","last_peered":"2024-09-05T16:45:16.019609+0000","last_clean":"2024-09-05T16:45:16.019609+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:45:16.019609+0000","last_undegraded":"2024-09-05T16:45:16.019609+0000","last_fullsized":"2024-09-05T16:45:16.019609+0000","mapping_epoch":34,"log_start":"29'2100","ondisk_log_start":"29'2100","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6651904,"num_objects":812,"num_object_clones":0,"num_object_copies":3248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":812,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6422,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'11810","reported_seq":11914,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:16.685502+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:45:16.685502+0000","last_peered":"2024-09-05T16:45:16.685502+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:45:16.685502+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"29'1800","ondisk_log_start":"29'1800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6427808,"num_objects":785,"num_object_clones":0,"num_object_copies":3140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":785,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11808,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1360}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'11965","reported_seq":12083,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:16.831092+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:45:16.831092+0000","last_peered":"2024-09-05T16:45:16.831092+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:45:16.831092+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"29'1900","ondisk_log_start":"29'1900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6610944,"num_objects":807,"num_object_clones":0,"num_object_copies":3228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":807,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11965,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1412}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'11782","reported_seq":11894,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:15.548458+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:45:15.548458+0000","last_peered":"2024-09-05T16:45:15.548458+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:45:15.548458+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"29'1700","ondisk_log_start":"29'1700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7206048,"num_objects":880,"num_object_clones":0,"num_object_copies":3520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":880,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11780,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1251}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'11938","reported_seq":12032,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:15.540741+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:45:15.540741+0000","last_peered":"2024-09-05T16:45:15.540741+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:45:15.540741+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"29'1900","ondisk_log_start":"29'1900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6886560,"num_objects":841,"num_object_clones":0,"num_object_copies":3364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":841,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11936,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1391}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"39'12015","reported_seq":12121,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:16.714203+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:45:16.714203+0000","last_peered":"2024-09-05T16:45:16.714203+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:45:16.714203+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"29'2000","ondisk_log_start":"29'2000","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6211904,"num_objects":759,"num_object_clones":0,"num_object_copies":3036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":759,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12012,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1485}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'11980","reported_seq":12107,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:15.113482+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:45:15.113482+0000","last_peered":"2024-09-05T16:45:15.113482+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:45:15.113482+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"29'1900","ondisk_log_start":"29'1900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6796448,"num_objects":830,"num_object_clones":0,"num_object_copies":3320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":830,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11978,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1415}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'12256","reported_seq":12387,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:16.673664+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:45:16.673664+0000","last_peered":"2024-09-05T16:45:16.673664+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:45:16.673664+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"29'2200","ondisk_log_start":"29'2200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7098096,"num_objects":867,"num_object_clones":0,"num_object_copies":3468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":867,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12253,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1414}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'11923","reported_seq":12027,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:16.818748+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:45:16.818748+0000","last_peered":"2024-09-05T16:45:16.818748+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:45:16.818748+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"29'1900","ondisk_log_start":"29'1900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6847056,"num_objects":836,"num_object_clones":0,"num_object_copies":3344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":836,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11922,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1402}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12006","reported_seq":16747,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:45:15.952393+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:45:15.952393+0000","last_peered":"2024-09-05T16:45:15.952393+0000","last_clean":"2024-09-05T16:45:15.952393+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:45:15.952393+0000","last_undegraded":"2024-09-05T16:45:15.952393+0000","last_fullsized":"2024-09-05T16:45:15.952393+0000","mapping_epoch":36,"log_start":"29'2000","ondisk_log_start":"29'2000","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6569984,"num_objects":802,"num_object_clones":0,"num_object_copies":3208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":802,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3606,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'12280","reported_seq":12394,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:16.783568+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:45:16.783568+0000","last_peered":"2024-09-05T16:45:16.783568+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:45:16.783568+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"29'2200","ondisk_log_start":"29'2200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6927520,"num_objects":846,"num_object_clones":0,"num_object_copies":3384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12278,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1488}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'11989","reported_seq":12107,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:15.121404+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:45:15.121404+0000","last_peered":"2024-09-05T16:45:15.121404+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:45:15.121404+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"29'1900","ondisk_log_start":"29'1900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6911136,"num_objects":844,"num_object_clones":0,"num_object_copies":3376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":844,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11987,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1464}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'11947","reported_seq":12059,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:15.541758+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:45:15.541758+0000","last_peered":"2024-09-05T16:45:15.541758+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:45:15.541758+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"29'1900","ondisk_log_start":"29'1900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6780064,"num_objects":828,"num_object_clones":0,"num_object_copies":3312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":828,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11945,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1392}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":107519304,"num_objects":13130,"num_object_clones":0,"num_object_copies":52520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28346,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13130,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":167020,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1020,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":215982080,"data_stored":215982080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160729,"ondisk_log_size":160729,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084423,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":737952,"kb_used_data":52368,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93633888,"statfs":{"total":96636764160,"available":95881101312,"internally_reserved":0,"allocated":53624832,"data_stored":53414251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":702017235},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,7,0,3,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986300,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":576616,"kb_used_data":53464,"kb_used_omap":4,"kb_used_meta":523131,"kb_avail":93795224,"statfs":{"total":96636764160,"available":96046309376,"internally_reserved":0,"allocated":54747136,"data_stored":54536555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":535686866},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673106,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":743300,"kb_used_data":54068,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93628540,"statfs":{"total":96636764160,"available":95875624960,"internally_reserved":0,"allocated":55365632,"data_stored":55155051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4,3,6,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182537,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":743336,"kb_used_data":53656,"kb_used_omap":4,"kb_used_meta":689659,"kb_avail":93628504,"statfs":{"total":96636764160,"available":95875588096,"internally_reserved":0,"allocated":54943744,"data_stored":54733163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":706211533},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,6,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":54358016,"data_stored":54358016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53796864,"data_stored":53796864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54595584,"data_stored":54595584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53231616,"data_stored":53231616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:45:18.292 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-05T16:45:18.292 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "39'12097", 'reported_seq': 13689, 'reported_epoch': 39, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:16.865841+0000', 'last_change': '2024-09-05T16:43:43.436935+0000', 'last_active': '2024-09-05T16:45:16.865841+0000', 'last_peered': '2024-09-05T16:45:16.865841+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.828289+0000', 'last_became_peered': '2024-09-05T16:34:56.828289+0000', 'last_unstale': '2024-09-05T16:45:16.865841+0000', 'last_undegraded': '2024-09-05T16:34:56.612160+0000', 'last_fullsized': '2024-09-05T16:34:56.557632+0000', 'mapping_epoch': 32, 'log_start': "29'2000", 'ondisk_log_start': "29'2000", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10097, 'log_dups_size': 0, 'ondisk_log_size': 10097, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6610944, 'num_objects': 807, 'num_object_clones': 0, 'num_object_copies': 3228, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1495, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 807, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12097, 'num_write_kb': 51616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 767, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -688}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:45:18.292 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:45:18.293 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "39'11901", 'reported_seq': 12029, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:15.278453+0000', 'last_change': '2024-09-05T16:34:56.867900+0000', 'last_active': '2024-09-05T16:45:15.278453+0000', 'last_peered': '2024-09-05T16:45:15.278453+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:45:15.278453+0000', 'last_undegraded': '2024-09-05T16:34:56.590972+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 32, 'log_start': "29'1900", 'ondisk_log_start': "29'1900", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10001, 'log_dups_size': 0, 'ondisk_log_size': 10001, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6969960, 'num_objects': 852, 'num_object_clones': 0, 'num_object_copies': 3408, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2233, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 852, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 11900, 'num_write_kb': 50999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1380}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:45:18.293 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:45:18.293 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "39'11810", 'reported_seq': 11914, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:16.685502+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:45:16.685502+0000', 'last_peered': '2024-09-05T16:45:16.685502+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:45:16.685502+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "29'1800", 'ondisk_log_start': "29'1800", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10010, 'log_dups_size': 0, 'ondisk_log_size': 10010, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6427808, 'num_objects': 785, 'num_object_clones': 0, 'num_object_copies': 3140, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2147, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 785, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11808, 'num_write_kb': 50373, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1360}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:45:18.293 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:45:18.293 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "39'11965", 'reported_seq': 12083, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:16.831092+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:45:16.831092+0000', 'last_peered': '2024-09-05T16:45:16.831092+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:45:16.831092+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "29'1900", 'ondisk_log_start': "29'1900", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10065, 'log_dups_size': 0, 'ondisk_log_size': 10065, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6610944, 'num_objects': 807, 'num_object_clones': 0, 'num_object_copies': 3228, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 807, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11965, 'num_write_kb': 51088, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1412}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:45:18.293 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:45:18.293 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "39'11782", 'reported_seq': 11894, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:15.548458+0000', 'last_change': '2024-09-05T16:34:56.903436+0000', 'last_active': '2024-09-05T16:45:15.548458+0000', 'last_peered': '2024-09-05T16:45:15.548458+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:45:15.548458+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "29'1700", 'ondisk_log_start': "29'1700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10082, 'log_dups_size': 0, 'ondisk_log_size': 10082, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7206048, 'num_objects': 880, 'num_object_clones': 0, 'num_object_copies': 3520, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2133, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 880, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11780, 'num_write_kb': 50636, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -1251}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:45:18.293 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-05T16:45:18.293 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "39'11938", 'reported_seq': 12032, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:15.540741+0000', 'last_change': '2024-09-05T16:34:56.883319+0000', 'last_active': '2024-09-05T16:45:15.540741+0000', 'last_peered': '2024-09-05T16:45:15.540741+0000', 'last_clean': '2024-09-05T16:33:53.505305+0000', 'last_became_active': '2024-09-05T16:34:56.873389+0000', 'last_became_peered': '2024-09-05T16:34:56.873389+0000', 'last_unstale': '2024-09-05T16:45:15.540741+0000', 'last_undegraded': '2024-09-05T16:34:56.602567+0000', 'last_fullsized': '2024-09-05T16:34:56.559958+0000', 'mapping_epoch': 32, 'log_start': "29'1900", 'ondisk_log_start': "29'1900", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10038, 'log_dups_size': 0, 'ondisk_log_size': 10038, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6886560, 'num_objects': 841, 'num_object_clones': 0, 'num_object_copies': 3364, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2234, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 841, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11936, 'num_write_kb': 51108, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1391}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:45:18.294 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:45:18.294 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "39'12015", 'reported_seq': 12121, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:16.714203+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:45:16.714203+0000', 'last_peered': '2024-09-05T16:45:16.714203+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:45:16.714203+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "29'2000", 'ondisk_log_start': "29'2000", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10015, 'log_dups_size': 0, 'ondisk_log_size': 10015, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6211904, 'num_objects': 759, 'num_object_clones': 0, 'num_object_copies': 3036, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 759, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12012, 'num_write_kb': 51081, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1485}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:45:18.294 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:45:18.294 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "39'11980", 'reported_seq': 12107, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:15.113482+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:45:15.113482+0000', 'last_peered': '2024-09-05T16:45:15.113482+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:45:15.113482+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "29'1900", 'ondisk_log_start': "29'1900", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6796448, 'num_objects': 830, 'num_object_clones': 0, 'num_object_copies': 3320, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 830, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11978, 'num_write_kb': 51230, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1415}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:45:18.294 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:45:18.294 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "39'12256", 'reported_seq': 12387, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:16.673664+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:45:16.673664+0000', 'last_peered': '2024-09-05T16:45:16.673664+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:45:16.673664+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "29'2200", 'ondisk_log_start': "29'2200", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10056, 'log_dups_size': 0, 'ondisk_log_size': 10056, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7098096, 'num_objects': 867, 'num_object_clones': 0, 'num_object_copies': 3468, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2284, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 867, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12253, 'num_write_kb': 52476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1414}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:45:18.294 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:45:18.294 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "39'11923", 'reported_seq': 12027, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:16.818748+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:45:16.818748+0000', 'last_peered': '2024-09-05T16:45:16.818748+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:45:16.818748+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "29'1900", 'ondisk_log_start': "29'1900", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10023, 'log_dups_size': 0, 'ondisk_log_size': 10023, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6847056, 'num_objects': 836, 'num_object_clones': 0, 'num_object_copies': 3344, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 836, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11922, 'num_write_kb': 51031, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1402}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:45:18.294 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:45:18.294 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "39'12280", 'reported_seq': 12394, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:16.783568+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:45:16.783568+0000', 'last_peered': '2024-09-05T16:45:16.783568+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:45:16.783568+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "29'2200", 'ondisk_log_start': "29'2200", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6927520, 'num_objects': 846, 'num_object_clones': 0, 'num_object_copies': 3384, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 846, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12278, 'num_write_kb': 52500, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1488}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:45:18.295 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:45:18.295 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "39'11989", 'reported_seq': 12107, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:15.121404+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:45:15.121404+0000', 'last_peered': '2024-09-05T16:45:15.121404+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:45:15.121404+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "29'1900", 'ondisk_log_start': "29'1900", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10089, 'log_dups_size': 0, 'ondisk_log_size': 10089, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6911136, 'num_objects': 844, 'num_object_clones': 0, 'num_object_copies': 3376, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2310, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 844, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11987, 'num_write_kb': 51325, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1464}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:45:18.295 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:45:18.295 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "39'11947", 'reported_seq': 12059, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:15.541758+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:45:15.541758+0000', 'last_peered': '2024-09-05T16:45:15.541758+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:45:15.541758+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "29'1900", 'ondisk_log_start': "29'1900", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10047, 'log_dups_size': 0, 'ondisk_log_size': 10047, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6780064, 'num_objects': 828, 'num_object_clones': 0, 'num_object_copies': 3312, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 828, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11945, 'num_write_kb': 51092, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1392}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:45:18.295 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:45:19.167 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:45:19.167 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 28346/52520 objects degraded (53.972%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":722,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":39,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":13130,"data_bytes":107519304,"bytes_used":2868432896,"bytes_avail":383678623744,"bytes_total":386547056640,"degraded_objects":28346,"degraded_total":52520,"degraded_ratio":0.53971820258948977,"recovering_objects_per_sec":9,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":114,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":254},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:45:19.197 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:45:19.197 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:45:19.847 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:45:19.848 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:45:19.878 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":391,"stamp":"2024-09-05T16:45:19.069165+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":107519304,"num_objects":13130,"num_object_clones":0,"num_object_copies":52520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28346,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13130,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":167020,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1020,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160729,"ondisk_log_size":160729,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2801204,"kb_used_data":213556,"kb_used_omap":14,"kb_used_meta":2587569,"kb_avail":374686156,"statfs":{"total":386547056640,"available":383678623744,"internally_reserved":0,"allocated":218681344,"data_stored":217839020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2649671223},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,17,12,10,11],"upper_bound":256},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16736256,"num_objects":-2043,"num_object_clones":0,"num_object_copies":-8172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-76,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2043,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2043,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":76,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004319"},"pg_stats":[{"pgid":"1.f","version":"39'12097","reported_seq":13689,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:16.865841+0000","last_change":"2024-09-05T16:43:43.436935+0000","last_active":"2024-09-05T16:45:16.865841+0000","last_peered":"2024-09-05T16:45:16.865841+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:45:16.865841+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"29'2000","ondisk_log_start":"29'2000","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6610944,"num_objects":807,"num_object_clones":0,"num_object_copies":3228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1495,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":807,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12097,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":767,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-688}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'11818","reported_seq":16474,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:45:16.268560+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:45:16.268560+0000","last_peered":"2024-09-05T16:45:16.268560+0000","last_clean":"2024-09-05T16:45:16.268560+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:45:16.268560+0000","last_undegraded":"2024-09-05T16:45:16.268560+0000","last_fullsized":"2024-09-05T16:45:16.268560+0000","mapping_epoch":38,"log_start":"29'1800","ondisk_log_start":"29'1800","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6012928,"num_objects":734,"num_object_clones":0,"num_object_copies":2936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":734,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1131,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'11901","reported_seq":12029,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:15.278453+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:45:15.278453+0000","last_peered":"2024-09-05T16:45:15.278453+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:45:15.278453+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"29'1900","ondisk_log_start":"29'1900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969960,"num_objects":852,"num_object_clones":0,"num_object_copies":3408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":852,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11900,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1380}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12122","reported_seq":16856,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:45:16.019609+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:45:16.019609+0000","last_peered":"2024-09-05T16:45:16.019609+0000","last_clean":"2024-09-05T16:45:16.019609+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:45:16.019609+0000","last_undegraded":"2024-09-05T16:45:16.019609+0000","last_fullsized":"2024-09-05T16:45:16.019609+0000","mapping_epoch":34,"log_start":"29'2100","ondisk_log_start":"29'2100","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6651904,"num_objects":812,"num_object_clones":0,"num_object_copies":3248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":812,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6422,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'11810","reported_seq":11914,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:16.685502+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:45:16.685502+0000","last_peered":"2024-09-05T16:45:16.685502+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:45:16.685502+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"29'1800","ondisk_log_start":"29'1800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6427808,"num_objects":785,"num_object_clones":0,"num_object_copies":3140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":785,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11808,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1360}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'11965","reported_seq":12083,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:16.831092+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:45:16.831092+0000","last_peered":"2024-09-05T16:45:16.831092+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:45:16.831092+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"29'1900","ondisk_log_start":"29'1900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6610944,"num_objects":807,"num_object_clones":0,"num_object_copies":3228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":807,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11965,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1412}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'11782","reported_seq":11894,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:15.548458+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:45:15.548458+0000","last_peered":"2024-09-05T16:45:15.548458+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:45:15.548458+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"29'1700","ondisk_log_start":"29'1700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7206048,"num_objects":880,"num_object_clones":0,"num_object_copies":3520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":880,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11780,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1251}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'11938","reported_seq":12032,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:15.540741+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:45:15.540741+0000","last_peered":"2024-09-05T16:45:15.540741+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:45:15.540741+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"29'1900","ondisk_log_start":"29'1900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6886560,"num_objects":841,"num_object_clones":0,"num_object_copies":3364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":841,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11936,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1391}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"39'12015","reported_seq":12121,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:16.714203+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:45:16.714203+0000","last_peered":"2024-09-05T16:45:16.714203+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:45:16.714203+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"29'2000","ondisk_log_start":"29'2000","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6211904,"num_objects":759,"num_object_clones":0,"num_object_copies":3036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":759,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12012,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1485}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'11980","reported_seq":12107,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:15.113482+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:45:15.113482+0000","last_peered":"2024-09-05T16:45:15.113482+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:45:15.113482+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"29'1900","ondisk_log_start":"29'1900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6796448,"num_objects":830,"num_object_clones":0,"num_object_copies":3320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":830,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11978,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1415}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'12256","reported_seq":12387,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:16.673664+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:45:16.673664+0000","last_peered":"2024-09-05T16:45:16.673664+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:45:16.673664+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"29'2200","ondisk_log_start":"29'2200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7098096,"num_objects":867,"num_object_clones":0,"num_object_copies":3468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":867,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12253,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1414}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'11923","reported_seq":12027,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:16.818748+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:45:16.818748+0000","last_peered":"2024-09-05T16:45:16.818748+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:45:16.818748+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"29'1900","ondisk_log_start":"29'1900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6847056,"num_objects":836,"num_object_clones":0,"num_object_copies":3344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":836,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11922,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1402}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12006","reported_seq":16747,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:45:15.952393+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:45:15.952393+0000","last_peered":"2024-09-05T16:45:15.952393+0000","last_clean":"2024-09-05T16:45:15.952393+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:45:15.952393+0000","last_undegraded":"2024-09-05T16:45:15.952393+0000","last_fullsized":"2024-09-05T16:45:15.952393+0000","mapping_epoch":36,"log_start":"29'2000","ondisk_log_start":"29'2000","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6569984,"num_objects":802,"num_object_clones":0,"num_object_copies":3208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":802,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3606,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'12280","reported_seq":12394,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:16.783568+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:45:16.783568+0000","last_peered":"2024-09-05T16:45:16.783568+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:45:16.783568+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"29'2200","ondisk_log_start":"29'2200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6927520,"num_objects":846,"num_object_clones":0,"num_object_copies":3384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12278,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1488}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'11989","reported_seq":12107,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:15.121404+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:45:15.121404+0000","last_peered":"2024-09-05T16:45:15.121404+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:45:15.121404+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"29'1900","ondisk_log_start":"29'1900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6911136,"num_objects":844,"num_object_clones":0,"num_object_copies":3376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":844,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11987,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1464}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'11947","reported_seq":12059,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:15.541758+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:45:15.541758+0000","last_peered":"2024-09-05T16:45:15.541758+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:45:15.541758+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"29'1900","ondisk_log_start":"29'1900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6780064,"num_objects":828,"num_object_clones":0,"num_object_copies":3312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":828,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11945,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1392}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":107519304,"num_objects":13130,"num_object_clones":0,"num_object_copies":52520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28346,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13130,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":167020,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1020,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":215982080,"data_stored":215982080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160729,"ondisk_log_size":160729,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084423,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":737952,"kb_used_data":52368,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93633888,"statfs":{"total":96636764160,"available":95881101312,"internally_reserved":0,"allocated":53624832,"data_stored":53414251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":702017235},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,7,0,3,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986300,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":576616,"kb_used_data":53464,"kb_used_omap":4,"kb_used_meta":523131,"kb_avail":93795224,"statfs":{"total":96636764160,"available":96046309376,"internally_reserved":0,"allocated":54747136,"data_stored":54536555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":535686866},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673106,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":743300,"kb_used_data":54068,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93628540,"statfs":{"total":96636764160,"available":95875624960,"internally_reserved":0,"allocated":55365632,"data_stored":55155051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4,3,6,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182537,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":743336,"kb_used_data":53656,"kb_used_omap":4,"kb_used_meta":689659,"kb_avail":93628504,"statfs":{"total":96636764160,"available":95875588096,"internally_reserved":0,"allocated":54943744,"data_stored":54733163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":706211533},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,6,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":54358016,"data_stored":54358016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53796864,"data_stored":53796864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54595584,"data_stored":54595584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53231616,"data_stored":53231616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:45:22.880 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:45:23.485 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:45:23.486 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:45:23.519 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":393,"stamp":"2024-09-05T16:45:23.072290+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":99556680,"num_objects":12158,"num_object_clones":0,"num_object_copies":48632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12158,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":167992,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1057,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161001,"ondisk_log_size":161001,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2769472,"kb_used_data":198440,"kb_used_omap":14,"kb_used_meta":2570929,"kb_avail":374717888,"statfs":{"total":386547056640,"available":383711117312,"internally_reserved":0,"allocated":203202560,"data_stored":202360236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2632631863},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,4,5,9,9,5,12],"upper_bound":256},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21692416,"num_objects":-2648,"num_object_clones":0,"num_object_copies":-10592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-113,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2648,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2648,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":113,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005909"},"pg_stats":[{"pgid":"1.f","version":"39'12157","reported_seq":13823,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:21.746151+0000","last_change":"2024-09-05T16:43:43.436935+0000","last_active":"2024-09-05T16:45:21.746151+0000","last_peered":"2024-09-05T16:45:21.746151+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:45:21.746151+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"29'2100","ondisk_log_start":"29'2100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6119424,"num_objects":747,"num_object_clones":0,"num_object_copies":2988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1458,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":747,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12157,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":804,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-711}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'11884","reported_seq":16540,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:45:21.271250+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:45:21.271250+0000","last_peered":"2024-09-05T16:45:21.271250+0000","last_clean":"2024-09-05T16:45:21.271250+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:45:21.271250+0000","last_undegraded":"2024-09-05T16:45:21.271250+0000","last_fullsized":"2024-09-05T16:45:21.271250+0000","mapping_epoch":38,"log_start":"29'1800","ondisk_log_start":"29'1800","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5472256,"num_objects":668,"num_object_clones":0,"num_object_copies":2672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":668,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1197,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'11977","reported_seq":12102,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:20.262690+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:45:20.262690+0000","last_peered":"2024-09-05T16:45:20.262690+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:45:20.262690+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"29'1900","ondisk_log_start":"29'1900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6347368,"num_objects":776,"num_object_clones":0,"num_object_copies":3104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":776,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11976,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1456}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12182","reported_seq":16916,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:45:21.226852+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:45:21.226852+0000","last_peered":"2024-09-05T16:45:21.226852+0000","last_clean":"2024-09-05T16:45:21.226852+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:45:21.226852+0000","last_undegraded":"2024-09-05T16:45:21.226852+0000","last_fullsized":"2024-09-05T16:45:21.226852+0000","mapping_epoch":34,"log_start":"29'2100","ondisk_log_start":"29'2100","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6160384,"num_objects":752,"num_object_clones":0,"num_object_copies":3008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6482,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'11870","reported_seq":11973,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:21.726494+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:45:21.726494+0000","last_peered":"2024-09-05T16:45:21.726494+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:45:21.726494+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"29'1800","ondisk_log_start":"29'1800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5936288,"num_objects":725,"num_object_clones":0,"num_object_copies":2900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":725,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11868,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1420}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12015","reported_seq":12134,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:21.760282+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:45:21.760282+0000","last_peered":"2024-09-05T16:45:21.760282+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:45:21.760282+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"29'2000","ondisk_log_start":"29'2000","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6201344,"num_objects":757,"num_object_clones":0,"num_object_copies":3028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":757,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12015,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1462}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'11841","reported_seq":11954,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:20.608060+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:45:20.608060+0000","last_peered":"2024-09-05T16:45:20.608060+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:45:20.608060+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"29'1800","ondisk_log_start":"29'1800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6722720,"num_objects":821,"num_object_clones":0,"num_object_copies":3284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":821,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11839,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1310}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'11991","reported_seq":12085,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:20.560149+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:45:20.560149+0000","last_peered":"2024-09-05T16:45:20.560149+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:45:20.560149+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"29'1900","ondisk_log_start":"29'1900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6452384,"num_objects":788,"num_object_clones":0,"num_object_copies":3152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11989,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1444}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"39'12075","reported_seq":12181,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:21.767768+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:45:21.767768+0000","last_peered":"2024-09-05T16:45:21.767768+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:45:21.767768+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"29'2000","ondisk_log_start":"29'2000","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5720384,"num_objects":699,"num_object_clones":0,"num_object_copies":2796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":699,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12072,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1545}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12042","reported_seq":12169,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:20.176445+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:45:20.176445+0000","last_peered":"2024-09-05T16:45:20.176445+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:45:20.176445+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"29'2000","ondisk_log_start":"29'2000","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6288544,"num_objects":768,"num_object_clones":0,"num_object_copies":3072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":768,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12040,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1477}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'12328","reported_seq":12460,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:21.789467+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:45:21.789467+0000","last_peered":"2024-09-05T16:45:21.789467+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:45:21.789467+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"30'2300","ondisk_log_start":"30'2300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6508272,"num_objects":795,"num_object_clones":0,"num_object_copies":3180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":795,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12325,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1486}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'11985","reported_seq":12089,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:21.705165+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:45:21.705165+0000","last_peered":"2024-09-05T16:45:21.705165+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:45:21.705165+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"29'1900","ondisk_log_start":"29'1900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6339152,"num_objects":774,"num_object_clones":0,"num_object_copies":3096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":774,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11984,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1464}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12068","reported_seq":16809,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:45:21.034230+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:45:21.034230+0000","last_peered":"2024-09-05T16:45:21.034230+0000","last_clean":"2024-09-05T16:45:21.034230+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:45:21.034230+0000","last_undegraded":"2024-09-05T16:45:21.034230+0000","last_fullsized":"2024-09-05T16:45:21.034230+0000","mapping_epoch":36,"log_start":"29'2000","ondisk_log_start":"29'2000","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6062080,"num_objects":740,"num_object_clones":0,"num_object_copies":2960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":740,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3668,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'12342","reported_seq":12456,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:21.721035+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:45:21.721035+0000","last_peered":"2024-09-05T16:45:21.721035+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:45:21.721035+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"29'2300","ondisk_log_start":"29'2300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6419616,"num_objects":784,"num_object_clones":0,"num_object_copies":3136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":784,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12340,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1550}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12050","reported_seq":12168,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:20.313420+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:45:20.313420+0000","last_peered":"2024-09-05T16:45:20.313420+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:45:20.313420+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"29'2000","ondisk_log_start":"29'2000","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6411424,"num_objects":783,"num_object_clones":0,"num_object_copies":3132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":783,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12048,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1525}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'11994","reported_seq":12108,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:20.560185+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:45:20.560185+0000","last_peered":"2024-09-05T16:45:20.560185+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:45:20.560185+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"29'1900","ondisk_log_start":"29'1900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6395040,"num_objects":781,"num_object_clones":0,"num_object_copies":3124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":781,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11992,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1439}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":99556680,"num_objects":12158,"num_object_clones":0,"num_object_copies":48632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12158,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":167992,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1057,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":199581696,"data_stored":199581696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161001,"ondisk_log_size":161001,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084424,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":738924,"kb_used_data":49176,"kb_used_omap":4,"kb_used_meta":689723,"kb_avail":93632916,"statfs":{"total":96636764160,"available":95880105984,"internally_reserved":0,"allocated":50356224,"data_stored":50145643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":706277075},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,2,5,2,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986301,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":576788,"kb_used_data":49472,"kb_used_omap":4,"kb_used_meta":527291,"kb_avail":93795052,"statfs":{"total":96636764160,"available":96046133248,"internally_reserved":0,"allocated":50659328,"data_stored":50448747,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":539946706},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,2,3,3,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673107,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":726924,"kb_used_data":50156,"kb_used_omap":1,"kb_used_meta":676734,"kb_avail":93644916,"statfs":{"total":96636764160,"available":95892393984,"internally_reserved":0,"allocated":51359744,"data_stored":51149163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":692976069},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182538,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":726836,"kb_used_data":49636,"kb_used_omap":4,"kb_used_meta":677179,"kb_avail":93645004,"statfs":{"total":96636764160,"available":95892484096,"internally_reserved":0,"allocated":50827264,"data_stored":50616683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":693432013},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":50225152,"data_stored":50225152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49655808,"data_stored":49655808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":50499584,"data_stored":50499584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49201152,"data_stored":49201152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:45:23.520 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:45:24.166 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:45:24.166 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:45:24.204 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":393,"stamp":"2024-09-05T16:45:23.072290+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":99556680,"num_objects":12158,"num_object_clones":0,"num_object_copies":48632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12158,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":167992,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1057,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161001,"ondisk_log_size":161001,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2769472,"kb_used_data":198440,"kb_used_omap":14,"kb_used_meta":2570929,"kb_avail":374717888,"statfs":{"total":386547056640,"available":383711117312,"internally_reserved":0,"allocated":203202560,"data_stored":202360236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2632631863},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,4,5,9,9,5,12],"upper_bound":256},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21692416,"num_objects":-2648,"num_object_clones":0,"num_object_copies":-10592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-113,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2648,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2648,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":113,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005909"},"pg_stats":[{"pgid":"1.f","version":"39'12157","reported_seq":13823,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:21.746151+0000","last_change":"2024-09-05T16:43:43.436935+0000","last_active":"2024-09-05T16:45:21.746151+0000","last_peered":"2024-09-05T16:45:21.746151+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:45:21.746151+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"29'2100","ondisk_log_start":"29'2100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6119424,"num_objects":747,"num_object_clones":0,"num_object_copies":2988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1458,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":747,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12157,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":804,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-711}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'11884","reported_seq":16540,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:45:21.271250+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:45:21.271250+0000","last_peered":"2024-09-05T16:45:21.271250+0000","last_clean":"2024-09-05T16:45:21.271250+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:45:21.271250+0000","last_undegraded":"2024-09-05T16:45:21.271250+0000","last_fullsized":"2024-09-05T16:45:21.271250+0000","mapping_epoch":38,"log_start":"29'1800","ondisk_log_start":"29'1800","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5472256,"num_objects":668,"num_object_clones":0,"num_object_copies":2672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":668,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1197,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'11977","reported_seq":12102,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:20.262690+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:45:20.262690+0000","last_peered":"2024-09-05T16:45:20.262690+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:45:20.262690+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"29'1900","ondisk_log_start":"29'1900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6347368,"num_objects":776,"num_object_clones":0,"num_object_copies":3104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":776,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11976,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1456}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12182","reported_seq":16916,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:45:21.226852+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:45:21.226852+0000","last_peered":"2024-09-05T16:45:21.226852+0000","last_clean":"2024-09-05T16:45:21.226852+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:45:21.226852+0000","last_undegraded":"2024-09-05T16:45:21.226852+0000","last_fullsized":"2024-09-05T16:45:21.226852+0000","mapping_epoch":34,"log_start":"29'2100","ondisk_log_start":"29'2100","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6160384,"num_objects":752,"num_object_clones":0,"num_object_copies":3008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6482,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'11870","reported_seq":11973,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:21.726494+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:45:21.726494+0000","last_peered":"2024-09-05T16:45:21.726494+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:45:21.726494+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"29'1800","ondisk_log_start":"29'1800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5936288,"num_objects":725,"num_object_clones":0,"num_object_copies":2900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":725,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11868,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1420}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12015","reported_seq":12134,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:21.760282+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:45:21.760282+0000","last_peered":"2024-09-05T16:45:21.760282+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:45:21.760282+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"29'2000","ondisk_log_start":"29'2000","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6201344,"num_objects":757,"num_object_clones":0,"num_object_copies":3028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":757,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12015,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1462}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'11841","reported_seq":11954,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:20.608060+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:45:20.608060+0000","last_peered":"2024-09-05T16:45:20.608060+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:45:20.608060+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"29'1800","ondisk_log_start":"29'1800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6722720,"num_objects":821,"num_object_clones":0,"num_object_copies":3284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":821,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11839,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1310}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'11991","reported_seq":12085,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:20.560149+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:45:20.560149+0000","last_peered":"2024-09-05T16:45:20.560149+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:45:20.560149+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"29'1900","ondisk_log_start":"29'1900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6452384,"num_objects":788,"num_object_clones":0,"num_object_copies":3152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11989,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1444}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"39'12075","reported_seq":12181,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:21.767768+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:45:21.767768+0000","last_peered":"2024-09-05T16:45:21.767768+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:45:21.767768+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"29'2000","ondisk_log_start":"29'2000","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5720384,"num_objects":699,"num_object_clones":0,"num_object_copies":2796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":699,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12072,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1545}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12042","reported_seq":12169,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:20.176445+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:45:20.176445+0000","last_peered":"2024-09-05T16:45:20.176445+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:45:20.176445+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"29'2000","ondisk_log_start":"29'2000","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6288544,"num_objects":768,"num_object_clones":0,"num_object_copies":3072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":768,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12040,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1477}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'12328","reported_seq":12460,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:21.789467+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:45:21.789467+0000","last_peered":"2024-09-05T16:45:21.789467+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:45:21.789467+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"30'2300","ondisk_log_start":"30'2300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6508272,"num_objects":795,"num_object_clones":0,"num_object_copies":3180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":795,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12325,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1486}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'11985","reported_seq":12089,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:21.705165+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:45:21.705165+0000","last_peered":"2024-09-05T16:45:21.705165+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:45:21.705165+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"29'1900","ondisk_log_start":"29'1900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6339152,"num_objects":774,"num_object_clones":0,"num_object_copies":3096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":774,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11984,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1464}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12068","reported_seq":16809,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:45:21.034230+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:45:21.034230+0000","last_peered":"2024-09-05T16:45:21.034230+0000","last_clean":"2024-09-05T16:45:21.034230+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:45:21.034230+0000","last_undegraded":"2024-09-05T16:45:21.034230+0000","last_fullsized":"2024-09-05T16:45:21.034230+0000","mapping_epoch":36,"log_start":"29'2000","ondisk_log_start":"29'2000","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6062080,"num_objects":740,"num_object_clones":0,"num_object_copies":2960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":740,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3668,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'12342","reported_seq":12456,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:21.721035+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:45:21.721035+0000","last_peered":"2024-09-05T16:45:21.721035+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:45:21.721035+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"29'2300","ondisk_log_start":"29'2300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6419616,"num_objects":784,"num_object_clones":0,"num_object_copies":3136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":784,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12340,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1550}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12050","reported_seq":12168,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:20.313420+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:45:20.313420+0000","last_peered":"2024-09-05T16:45:20.313420+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:45:20.313420+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"29'2000","ondisk_log_start":"29'2000","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6411424,"num_objects":783,"num_object_clones":0,"num_object_copies":3132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":783,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12048,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1525}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'11994","reported_seq":12108,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:20.560185+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:45:20.560185+0000","last_peered":"2024-09-05T16:45:20.560185+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:45:20.560185+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"29'1900","ondisk_log_start":"29'1900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6395040,"num_objects":781,"num_object_clones":0,"num_object_copies":3124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":781,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11992,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1439}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":99556680,"num_objects":12158,"num_object_clones":0,"num_object_copies":48632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12158,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":167992,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1057,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":199581696,"data_stored":199581696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161001,"ondisk_log_size":161001,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084424,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":738924,"kb_used_data":49176,"kb_used_omap":4,"kb_used_meta":689723,"kb_avail":93632916,"statfs":{"total":96636764160,"available":95880105984,"internally_reserved":0,"allocated":50356224,"data_stored":50145643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":706277075},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,2,5,2,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986301,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":576788,"kb_used_data":49472,"kb_used_omap":4,"kb_used_meta":527291,"kb_avail":93795052,"statfs":{"total":96636764160,"available":96046133248,"internally_reserved":0,"allocated":50659328,"data_stored":50448747,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":539946706},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,2,3,3,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673107,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":726924,"kb_used_data":50156,"kb_used_omap":1,"kb_used_meta":676734,"kb_avail":93644916,"statfs":{"total":96636764160,"available":95892393984,"internally_reserved":0,"allocated":51359744,"data_stored":51149163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":692976069},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182538,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":726836,"kb_used_data":49636,"kb_used_omap":4,"kb_used_meta":677179,"kb_avail":93645004,"statfs":{"total":96636764160,"available":95892484096,"internally_reserved":0,"allocated":50827264,"data_stored":50616683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":693432013},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":50225152,"data_stored":50225152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49655808,"data_stored":49655808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":50499584,"data_stored":50499584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49201152,"data_stored":49201152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:45:24.205 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-05T16:45:24.206 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "39'12157", 'reported_seq': 13823, 'reported_epoch': 39, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:21.746151+0000', 'last_change': '2024-09-05T16:43:43.436935+0000', 'last_active': '2024-09-05T16:45:21.746151+0000', 'last_peered': '2024-09-05T16:45:21.746151+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.828289+0000', 'last_became_peered': '2024-09-05T16:34:56.828289+0000', 'last_unstale': '2024-09-05T16:45:21.746151+0000', 'last_undegraded': '2024-09-05T16:34:56.612160+0000', 'last_fullsized': '2024-09-05T16:34:56.557632+0000', 'mapping_epoch': 32, 'log_start': "29'2100", 'ondisk_log_start': "29'2100", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10057, 'log_dups_size': 0, 'ondisk_log_size': 10057, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6119424, 'num_objects': 747, 'num_object_clones': 0, 'num_object_copies': 2988, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1458, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 747, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12157, 'num_write_kb': 51616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 804, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -711}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:45:24.206 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:45:24.206 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "39'11977", 'reported_seq': 12102, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:20.262690+0000', 'last_change': '2024-09-05T16:34:56.867900+0000', 'last_active': '2024-09-05T16:45:20.262690+0000', 'last_peered': '2024-09-05T16:45:20.262690+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:45:20.262690+0000', 'last_undegraded': '2024-09-05T16:34:56.590972+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 32, 'log_start': "29'1900", 'ondisk_log_start': "29'1900", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10077, 'log_dups_size': 0, 'ondisk_log_size': 10077, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6347368, 'num_objects': 776, 'num_object_clones': 0, 'num_object_copies': 3104, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2233, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 776, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 11976, 'num_write_kb': 50999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1456}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:45:24.206 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:45:24.206 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "39'11870", 'reported_seq': 11973, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:21.726494+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:45:21.726494+0000', 'last_peered': '2024-09-05T16:45:21.726494+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:45:21.726494+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "29'1800", 'ondisk_log_start': "29'1800", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10070, 'log_dups_size': 0, 'ondisk_log_size': 10070, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5936288, 'num_objects': 725, 'num_object_clones': 0, 'num_object_copies': 2900, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2147, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 725, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11868, 'num_write_kb': 50373, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1420}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:45:24.206 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:45:24.207 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "39'12015", 'reported_seq': 12134, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:21.760282+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:45:21.760282+0000', 'last_peered': '2024-09-05T16:45:21.760282+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:45:21.760282+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "29'2000", 'ondisk_log_start': "29'2000", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10015, 'log_dups_size': 0, 'ondisk_log_size': 10015, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6201344, 'num_objects': 757, 'num_object_clones': 0, 'num_object_copies': 3028, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 757, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12015, 'num_write_kb': 51088, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1462}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:45:24.207 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:45:24.207 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "39'11841", 'reported_seq': 11954, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:20.608060+0000', 'last_change': '2024-09-05T16:34:56.903436+0000', 'last_active': '2024-09-05T16:45:20.608060+0000', 'last_peered': '2024-09-05T16:45:20.608060+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:45:20.608060+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "29'1800", 'ondisk_log_start': "29'1800", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10041, 'log_dups_size': 0, 'ondisk_log_size': 10041, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6722720, 'num_objects': 821, 'num_object_clones': 0, 'num_object_copies': 3284, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2133, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 821, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11839, 'num_write_kb': 50636, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -1310}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:45:24.207 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-05T16:45:24.207 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "39'11991", 'reported_seq': 12085, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:20.560149+0000', 'last_change': '2024-09-05T16:34:56.883319+0000', 'last_active': '2024-09-05T16:45:20.560149+0000', 'last_peered': '2024-09-05T16:45:20.560149+0000', 'last_clean': '2024-09-05T16:33:53.505305+0000', 'last_became_active': '2024-09-05T16:34:56.873389+0000', 'last_became_peered': '2024-09-05T16:34:56.873389+0000', 'last_unstale': '2024-09-05T16:45:20.560149+0000', 'last_undegraded': '2024-09-05T16:34:56.602567+0000', 'last_fullsized': '2024-09-05T16:34:56.559958+0000', 'mapping_epoch': 32, 'log_start': "29'1900", 'ondisk_log_start': "29'1900", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10091, 'log_dups_size': 0, 'ondisk_log_size': 10091, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6452384, 'num_objects': 788, 'num_object_clones': 0, 'num_object_copies': 3152, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2234, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 788, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11989, 'num_write_kb': 51108, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1444}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:45:24.208 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:45:24.208 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "39'12075", 'reported_seq': 12181, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:21.767768+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:45:21.767768+0000', 'last_peered': '2024-09-05T16:45:21.767768+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:45:21.767768+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "29'2000", 'ondisk_log_start': "29'2000", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10075, 'log_dups_size': 0, 'ondisk_log_size': 10075, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5720384, 'num_objects': 699, 'num_object_clones': 0, 'num_object_copies': 2796, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 699, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12072, 'num_write_kb': 51081, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1545}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:45:24.208 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:45:24.208 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "39'12042", 'reported_seq': 12169, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:20.176445+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:45:20.176445+0000', 'last_peered': '2024-09-05T16:45:20.176445+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:45:20.176445+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "29'2000", 'ondisk_log_start': "29'2000", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10042, 'log_dups_size': 0, 'ondisk_log_size': 10042, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6288544, 'num_objects': 768, 'num_object_clones': 0, 'num_object_copies': 3072, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 768, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12040, 'num_write_kb': 51230, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1477}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:45:24.208 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:45:24.208 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "39'12328", 'reported_seq': 12460, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:21.789467+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:45:21.789467+0000', 'last_peered': '2024-09-05T16:45:21.789467+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:45:21.789467+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "30'2300", 'ondisk_log_start': "30'2300", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6508272, 'num_objects': 795, 'num_object_clones': 0, 'num_object_copies': 3180, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2284, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 795, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12325, 'num_write_kb': 52476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1486}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:45:24.208 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:45:24.209 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "39'11985", 'reported_seq': 12089, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:21.705165+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:45:21.705165+0000', 'last_peered': '2024-09-05T16:45:21.705165+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:45:21.705165+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "29'1900", 'ondisk_log_start': "29'1900", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10085, 'log_dups_size': 0, 'ondisk_log_size': 10085, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6339152, 'num_objects': 774, 'num_object_clones': 0, 'num_object_copies': 3096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 774, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11984, 'num_write_kb': 51031, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1464}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:45:24.209 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:45:24.209 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "39'12342", 'reported_seq': 12456, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:21.721035+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:45:21.721035+0000', 'last_peered': '2024-09-05T16:45:21.721035+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:45:21.721035+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "29'2300", 'ondisk_log_start': "29'2300", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10042, 'log_dups_size': 0, 'ondisk_log_size': 10042, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6419616, 'num_objects': 784, 'num_object_clones': 0, 'num_object_copies': 3136, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 784, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12340, 'num_write_kb': 52500, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1550}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:45:24.209 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:45:24.210 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "39'12050", 'reported_seq': 12168, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:20.313420+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:45:20.313420+0000', 'last_peered': '2024-09-05T16:45:20.313420+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:45:20.313420+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "29'2000", 'ondisk_log_start': "29'2000", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6411424, 'num_objects': 783, 'num_object_clones': 0, 'num_object_copies': 3132, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2310, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 783, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12048, 'num_write_kb': 51325, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1525}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:45:24.210 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:45:24.210 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "39'11994", 'reported_seq': 12108, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:20.560185+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:45:20.560185+0000', 'last_peered': '2024-09-05T16:45:20.560185+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:45:20.560185+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "29'1900", 'ondisk_log_start': "29'1900", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10094, 'log_dups_size': 0, 'ondisk_log_size': 10094, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6395040, 'num_objects': 781, 'num_object_clones': 0, 'num_object_copies': 3124, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 781, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11992, 'num_write_kb': 51092, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1439}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:45:24.211 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:45:24.943 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:45:24.943 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 28309/48632 objects degraded (58.211%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":728,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":39,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":12158,"data_bytes":99556680,"bytes_used":2835939328,"bytes_avail":383711117312,"bytes_total":386547056640,"degraded_objects":28309,"degraded_total":48632,"degraded_ratio":0.58210643197894385,"recovering_objects_per_sec":9,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":113,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":220},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:45:24.961 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:45:24.961 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:45:25.554 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:45:25.554 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:45:25.581 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":394,"stamp":"2024-09-05T16:45:25.073019+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":99556680,"num_objects":12158,"num_object_clones":0,"num_object_copies":48632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12158,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":167992,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1057,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161001,"ondisk_log_size":161001,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2769472,"kb_used_data":198440,"kb_used_omap":14,"kb_used_meta":2570929,"kb_avail":374717888,"statfs":{"total":386547056640,"available":383711117312,"internally_reserved":0,"allocated":203202560,"data_stored":202360236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2632631863},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,4,5,9,9,5,12],"upper_bound":256},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16334848,"num_objects":-1994,"num_object_clones":0,"num_object_copies":-7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-76,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1994,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1994,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":76,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005805"},"pg_stats":[{"pgid":"1.f","version":"39'12157","reported_seq":13823,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:21.746151+0000","last_change":"2024-09-05T16:43:43.436935+0000","last_active":"2024-09-05T16:45:21.746151+0000","last_peered":"2024-09-05T16:45:21.746151+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:45:21.746151+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"29'2100","ondisk_log_start":"29'2100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6119424,"num_objects":747,"num_object_clones":0,"num_object_copies":2988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1458,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":747,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12157,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":804,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-711}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'11884","reported_seq":16540,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:45:21.271250+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:45:21.271250+0000","last_peered":"2024-09-05T16:45:21.271250+0000","last_clean":"2024-09-05T16:45:21.271250+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:45:21.271250+0000","last_undegraded":"2024-09-05T16:45:21.271250+0000","last_fullsized":"2024-09-05T16:45:21.271250+0000","mapping_epoch":38,"log_start":"29'1800","ondisk_log_start":"29'1800","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5472256,"num_objects":668,"num_object_clones":0,"num_object_copies":2672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":668,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1197,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'11977","reported_seq":12102,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:20.262690+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:45:20.262690+0000","last_peered":"2024-09-05T16:45:20.262690+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:45:20.262690+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"29'1900","ondisk_log_start":"29'1900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6347368,"num_objects":776,"num_object_clones":0,"num_object_copies":3104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":776,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11976,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1456}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12182","reported_seq":16916,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:45:21.226852+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:45:21.226852+0000","last_peered":"2024-09-05T16:45:21.226852+0000","last_clean":"2024-09-05T16:45:21.226852+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:45:21.226852+0000","last_undegraded":"2024-09-05T16:45:21.226852+0000","last_fullsized":"2024-09-05T16:45:21.226852+0000","mapping_epoch":34,"log_start":"29'2100","ondisk_log_start":"29'2100","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6160384,"num_objects":752,"num_object_clones":0,"num_object_copies":3008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6482,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'11870","reported_seq":11973,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:21.726494+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:45:21.726494+0000","last_peered":"2024-09-05T16:45:21.726494+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:45:21.726494+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"29'1800","ondisk_log_start":"29'1800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5936288,"num_objects":725,"num_object_clones":0,"num_object_copies":2900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":725,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11868,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1420}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12015","reported_seq":12134,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:21.760282+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:45:21.760282+0000","last_peered":"2024-09-05T16:45:21.760282+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:45:21.760282+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"29'2000","ondisk_log_start":"29'2000","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6201344,"num_objects":757,"num_object_clones":0,"num_object_copies":3028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":757,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12015,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1462}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'11841","reported_seq":11954,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:20.608060+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:45:20.608060+0000","last_peered":"2024-09-05T16:45:20.608060+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:45:20.608060+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"29'1800","ondisk_log_start":"29'1800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6722720,"num_objects":821,"num_object_clones":0,"num_object_copies":3284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":821,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11839,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1310}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'11991","reported_seq":12085,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:20.560149+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:45:20.560149+0000","last_peered":"2024-09-05T16:45:20.560149+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:45:20.560149+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"29'1900","ondisk_log_start":"29'1900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6452384,"num_objects":788,"num_object_clones":0,"num_object_copies":3152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11989,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1444}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"39'12075","reported_seq":12181,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:21.767768+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:45:21.767768+0000","last_peered":"2024-09-05T16:45:21.767768+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:45:21.767768+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"29'2000","ondisk_log_start":"29'2000","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5720384,"num_objects":699,"num_object_clones":0,"num_object_copies":2796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":699,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12072,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1545}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12042","reported_seq":12169,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:20.176445+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:45:20.176445+0000","last_peered":"2024-09-05T16:45:20.176445+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:45:20.176445+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"29'2000","ondisk_log_start":"29'2000","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6288544,"num_objects":768,"num_object_clones":0,"num_object_copies":3072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":768,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12040,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1477}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'12328","reported_seq":12460,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:21.789467+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:45:21.789467+0000","last_peered":"2024-09-05T16:45:21.789467+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:45:21.789467+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"30'2300","ondisk_log_start":"30'2300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6508272,"num_objects":795,"num_object_clones":0,"num_object_copies":3180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":795,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12325,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1486}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'11985","reported_seq":12089,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:21.705165+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:45:21.705165+0000","last_peered":"2024-09-05T16:45:21.705165+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:45:21.705165+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"29'1900","ondisk_log_start":"29'1900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6339152,"num_objects":774,"num_object_clones":0,"num_object_copies":3096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":774,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11984,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1464}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12068","reported_seq":16809,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:45:21.034230+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:45:21.034230+0000","last_peered":"2024-09-05T16:45:21.034230+0000","last_clean":"2024-09-05T16:45:21.034230+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:45:21.034230+0000","last_undegraded":"2024-09-05T16:45:21.034230+0000","last_fullsized":"2024-09-05T16:45:21.034230+0000","mapping_epoch":36,"log_start":"29'2000","ondisk_log_start":"29'2000","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6062080,"num_objects":740,"num_object_clones":0,"num_object_copies":2960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":740,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3668,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'12342","reported_seq":12456,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:21.721035+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:45:21.721035+0000","last_peered":"2024-09-05T16:45:21.721035+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:45:21.721035+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"29'2300","ondisk_log_start":"29'2300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6419616,"num_objects":784,"num_object_clones":0,"num_object_copies":3136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":784,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12340,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1550}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12050","reported_seq":12168,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:20.313420+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:45:20.313420+0000","last_peered":"2024-09-05T16:45:20.313420+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:45:20.313420+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"29'2000","ondisk_log_start":"29'2000","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6411424,"num_objects":783,"num_object_clones":0,"num_object_copies":3132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":783,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12048,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1525}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'11994","reported_seq":12108,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:20.560185+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:45:20.560185+0000","last_peered":"2024-09-05T16:45:20.560185+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:45:20.560185+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"29'1900","ondisk_log_start":"29'1900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6395040,"num_objects":781,"num_object_clones":0,"num_object_copies":3124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":781,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11992,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1439}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":99556680,"num_objects":12158,"num_object_clones":0,"num_object_copies":48632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12158,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":167992,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1057,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":199581696,"data_stored":199581696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161001,"ondisk_log_size":161001,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084424,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":738924,"kb_used_data":49176,"kb_used_omap":4,"kb_used_meta":689723,"kb_avail":93632916,"statfs":{"total":96636764160,"available":95880105984,"internally_reserved":0,"allocated":50356224,"data_stored":50145643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":706277075},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,2,5,2,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986301,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":576788,"kb_used_data":49472,"kb_used_omap":4,"kb_used_meta":527291,"kb_avail":93795052,"statfs":{"total":96636764160,"available":96046133248,"internally_reserved":0,"allocated":50659328,"data_stored":50448747,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":539946706},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,2,3,3,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673107,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":726924,"kb_used_data":50156,"kb_used_omap":1,"kb_used_meta":676734,"kb_avail":93644916,"statfs":{"total":96636764160,"available":95892393984,"internally_reserved":0,"allocated":51359744,"data_stored":51149163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":692976069},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182538,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":726836,"kb_used_data":49636,"kb_used_omap":4,"kb_used_meta":677179,"kb_avail":93645004,"statfs":{"total":96636764160,"available":95892484096,"internally_reserved":0,"allocated":50827264,"data_stored":50616683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":693432013},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":50225152,"data_stored":50225152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49655808,"data_stored":49655808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":50499584,"data_stored":50499584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49201152,"data_stored":49201152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:45:28.582 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:45:29.366 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:45:29.367 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:45:29.391 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":396,"stamp":"2024-09-05T16:45:29.074513+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":91274568,"num_objects":11147,"num_object_clones":0,"num_object_copies":44588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28271,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11147,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":169003,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1095,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160612,"ondisk_log_size":160612,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2754800,"kb_used_data":182348,"kb_used_omap":14,"kb_used_meta":2572337,"kb_avail":374732560,"statfs":{"total":386547056640,"available":383726141440,"internally_reserved":0,"allocated":186724352,"data_stored":185882028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2634073655},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,14,10,9,9,15],"upper_bound":256},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16244736,"num_objects":-1983,"num_object_clones":0,"num_object_copies":-7932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-75,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1983,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1983,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":75,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006073"},"pg_stats":[{"pgid":"1.f","version":"39'12231","reported_seq":13972,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:26.908026+0000","last_change":"2024-09-05T16:43:43.436935+0000","last_active":"2024-09-05T16:45:26.908026+0000","last_peered":"2024-09-05T16:45:26.908026+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:45:26.908026+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"29'2200","ondisk_log_start":"29'2200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5513216,"num_objects":673,"num_object_clones":0,"num_object_copies":2692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1420,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":673,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12231,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":842,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-747}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'11934","reported_seq":16590,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:45:26.275010+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:45:26.275010+0000","last_peered":"2024-09-05T16:45:26.275010+0000","last_clean":"2024-09-05T16:45:26.275010+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:45:26.275010+0000","last_undegraded":"2024-09-05T16:45:26.275010+0000","last_fullsized":"2024-09-05T16:45:26.275010+0000","mapping_epoch":38,"log_start":"29'1900","ondisk_log_start":"29'1900","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5062656,"num_objects":618,"num_object_clones":0,"num_object_copies":2472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":618,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1247,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12036","reported_seq":12164,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:25.234862+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:45:25.234862+0000","last_peered":"2024-09-05T16:45:25.234862+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:45:25.234862+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"29'2000","ondisk_log_start":"29'2000","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5864040,"num_objects":717,"num_object_clones":0,"num_object_copies":2868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":717,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12035,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1515}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12253","reported_seq":16985,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:45:26.462080+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:45:26.462080+0000","last_peered":"2024-09-05T16:45:26.462080+0000","last_clean":"2024-09-05T16:45:26.462080+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:45:26.462080+0000","last_undegraded":"2024-09-05T16:45:26.462080+0000","last_fullsized":"2024-09-05T16:45:26.462080+0000","mapping_epoch":34,"log_start":"29'2200","ondisk_log_start":"29'2200","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5578752,"num_objects":681,"num_object_clones":0,"num_object_copies":2724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":681,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6553,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'11928","reported_seq":12031,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:26.656422+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:45:26.656422+0000","last_peered":"2024-09-05T16:45:26.656422+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:45:26.656422+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"29'1900","ondisk_log_start":"29'1900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5461152,"num_objects":667,"num_object_clones":0,"num_object_copies":2668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":667,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11926,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1478}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12078","reported_seq":12197,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:26.526096+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:45:26.526096+0000","last_peered":"2024-09-05T16:45:26.526096+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:45:26.526096+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"29'2000","ondisk_log_start":"29'2000","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5685248,"num_objects":694,"num_object_clones":0,"num_object_copies":2776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":694,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12078,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1525}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'11910","reported_seq":12023,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:25.547116+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:45:25.547116+0000","last_peered":"2024-09-05T16:45:25.547116+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:45:25.547116+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"29'1900","ondisk_log_start":"29'1900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6157472,"num_objects":752,"num_object_clones":0,"num_object_copies":3008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11908,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1379}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12049","reported_seq":12142,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:25.671397+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:45:25.671397+0000","last_peered":"2024-09-05T16:45:25.671397+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:45:25.671397+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"29'2000","ondisk_log_start":"29'2000","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5977248,"num_objects":730,"num_object_clones":0,"num_object_copies":2920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":730,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12047,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1502}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"39'12134","reported_seq":12240,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:26.766435+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:45:26.766435+0000","last_peered":"2024-09-05T16:45:26.766435+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:45:26.766435+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"29'2100","ondisk_log_start":"29'2100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5237056,"num_objects":640,"num_object_clones":0,"num_object_copies":2560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":640,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1604}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12109","reported_seq":12236,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:25.075906+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:45:25.075906+0000","last_peered":"2024-09-05T16:45:25.075906+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:45:25.075906+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"29'2100","ondisk_log_start":"29'2100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5739680,"num_objects":701,"num_object_clones":0,"num_object_copies":2804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":701,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12107,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1544}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'12391","reported_seq":12523,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:26.918819+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:45:26.918819+0000","last_peered":"2024-09-05T16:45:26.918819+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:45:26.918819+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"30'2300","ondisk_log_start":"30'2300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5992176,"num_objects":732,"num_object_clones":0,"num_object_copies":2928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":732,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12388,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1549}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12049","reported_seq":12151,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:26.936751+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:45:26.936751+0000","last_peered":"2024-09-05T16:45:26.936751+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:45:26.936751+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"29'2000","ondisk_log_start":"29'2000","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5814864,"num_objects":710,"num_object_clones":0,"num_object_copies":2840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":710,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12048,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1528}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12125","reported_seq":16866,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:45:26.328732+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:45:26.328732+0000","last_peered":"2024-09-05T16:45:26.328732+0000","last_clean":"2024-09-05T16:45:26.328732+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:45:26.328732+0000","last_undegraded":"2024-09-05T16:45:26.328732+0000","last_fullsized":"2024-09-05T16:45:26.328732+0000","mapping_epoch":36,"log_start":"29'2100","ondisk_log_start":"29'2100","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5595136,"num_objects":683,"num_object_clones":0,"num_object_copies":2732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":683,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3725,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'12408","reported_seq":12521,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:26.859885+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:45:26.859885+0000","last_peered":"2024-09-05T16:45:26.859885+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:45:26.859885+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"33'2400","ondisk_log_start":"33'2400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5878944,"num_objects":718,"num_object_clones":0,"num_object_copies":2872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":718,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12406,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1616}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12118","reported_seq":12236,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:25.215389+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:45:25.215389+0000","last_peered":"2024-09-05T16:45:25.215389+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:45:25.215389+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"29'2100","ondisk_log_start":"29'2100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5854368,"num_objects":715,"num_object_clones":0,"num_object_copies":2860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":715,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12116,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'12059","reported_seq":12172,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:25.657080+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:45:25.657080+0000","last_peered":"2024-09-05T16:45:25.657080+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:45:25.657080+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"29'2000","ondisk_log_start":"29'2000","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5862560,"num_objects":716,"num_object_clones":0,"num_object_copies":2864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":716,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12057,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1504}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":91274568,"num_objects":11147,"num_object_clones":0,"num_object_copies":44588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28271,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11147,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":169003,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1095,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":183173120,"data_stored":183173120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160612,"ondisk_log_size":160612,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084425,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722432,"kb_used_data":45168,"kb_used_omap":4,"kb_used_meta":677243,"kb_avail":93649408,"statfs":{"total":96636764160,"available":95896993792,"internally_reserved":0,"allocated":46252032,"data_stored":46041451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":693497555},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,7,5,5,4,11],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986302,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":578408,"kb_used_data":45528,"kb_used_omap":4,"kb_used_meta":532859,"kb_avail":93793432,"statfs":{"total":96636764160,"available":96044474368,"internally_reserved":0,"allocated":46620672,"data_stored":46410091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":545648338},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,1,2,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673108,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":726992,"kb_used_data":46068,"kb_used_omap":1,"kb_used_meta":680894,"kb_avail":93644848,"statfs":{"total":96636764160,"available":95892324352,"internally_reserved":0,"allocated":47173632,"data_stored":46963051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":697235909},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182539,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":726968,"kb_used_data":45584,"kb_used_omap":4,"kb_used_meta":681339,"kb_avail":93644872,"statfs":{"total":96636764160,"available":95892348928,"internally_reserved":0,"allocated":46678016,"data_stored":46467435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":697691853},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,0,3,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":46166016,"data_stored":46166016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":45543424,"data_stored":45543424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":46391296,"data_stored":46391296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":45072384,"data_stored":45072384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:45:29.392 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:45:30.040 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:45:30.041 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:45:30.066 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":396,"stamp":"2024-09-05T16:45:29.074513+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":91274568,"num_objects":11147,"num_object_clones":0,"num_object_copies":44588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28271,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11147,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":169003,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1095,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160612,"ondisk_log_size":160612,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2754800,"kb_used_data":182348,"kb_used_omap":14,"kb_used_meta":2572337,"kb_avail":374732560,"statfs":{"total":386547056640,"available":383726141440,"internally_reserved":0,"allocated":186724352,"data_stored":185882028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2634073655},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,14,10,9,9,15],"upper_bound":256},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16244736,"num_objects":-1983,"num_object_clones":0,"num_object_copies":-7932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-75,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1983,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1983,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":75,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006073"},"pg_stats":[{"pgid":"1.f","version":"39'12231","reported_seq":13972,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:26.908026+0000","last_change":"2024-09-05T16:43:43.436935+0000","last_active":"2024-09-05T16:45:26.908026+0000","last_peered":"2024-09-05T16:45:26.908026+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:45:26.908026+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"29'2200","ondisk_log_start":"29'2200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5513216,"num_objects":673,"num_object_clones":0,"num_object_copies":2692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1420,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":673,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12231,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":842,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-747}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'11934","reported_seq":16590,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:45:26.275010+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:45:26.275010+0000","last_peered":"2024-09-05T16:45:26.275010+0000","last_clean":"2024-09-05T16:45:26.275010+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:45:26.275010+0000","last_undegraded":"2024-09-05T16:45:26.275010+0000","last_fullsized":"2024-09-05T16:45:26.275010+0000","mapping_epoch":38,"log_start":"29'1900","ondisk_log_start":"29'1900","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5062656,"num_objects":618,"num_object_clones":0,"num_object_copies":2472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":618,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1247,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12036","reported_seq":12164,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:25.234862+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:45:25.234862+0000","last_peered":"2024-09-05T16:45:25.234862+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:45:25.234862+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"29'2000","ondisk_log_start":"29'2000","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5864040,"num_objects":717,"num_object_clones":0,"num_object_copies":2868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":717,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12035,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1515}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12253","reported_seq":16985,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:45:26.462080+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:45:26.462080+0000","last_peered":"2024-09-05T16:45:26.462080+0000","last_clean":"2024-09-05T16:45:26.462080+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:45:26.462080+0000","last_undegraded":"2024-09-05T16:45:26.462080+0000","last_fullsized":"2024-09-05T16:45:26.462080+0000","mapping_epoch":34,"log_start":"29'2200","ondisk_log_start":"29'2200","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5578752,"num_objects":681,"num_object_clones":0,"num_object_copies":2724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":681,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6553,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'11928","reported_seq":12031,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:26.656422+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:45:26.656422+0000","last_peered":"2024-09-05T16:45:26.656422+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:45:26.656422+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"29'1900","ondisk_log_start":"29'1900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5461152,"num_objects":667,"num_object_clones":0,"num_object_copies":2668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":667,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11926,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1478}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12078","reported_seq":12197,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:26.526096+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:45:26.526096+0000","last_peered":"2024-09-05T16:45:26.526096+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:45:26.526096+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"29'2000","ondisk_log_start":"29'2000","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5685248,"num_objects":694,"num_object_clones":0,"num_object_copies":2776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":694,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12078,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1525}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'11910","reported_seq":12023,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:25.547116+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:45:25.547116+0000","last_peered":"2024-09-05T16:45:25.547116+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:45:25.547116+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"29'1900","ondisk_log_start":"29'1900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6157472,"num_objects":752,"num_object_clones":0,"num_object_copies":3008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11908,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1379}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12049","reported_seq":12142,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:25.671397+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:45:25.671397+0000","last_peered":"2024-09-05T16:45:25.671397+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:45:25.671397+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"29'2000","ondisk_log_start":"29'2000","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5977248,"num_objects":730,"num_object_clones":0,"num_object_copies":2920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":730,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12047,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1502}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"39'12134","reported_seq":12240,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:26.766435+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:45:26.766435+0000","last_peered":"2024-09-05T16:45:26.766435+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:45:26.766435+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"29'2100","ondisk_log_start":"29'2100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5237056,"num_objects":640,"num_object_clones":0,"num_object_copies":2560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":640,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1604}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12109","reported_seq":12236,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:25.075906+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:45:25.075906+0000","last_peered":"2024-09-05T16:45:25.075906+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:45:25.075906+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"29'2100","ondisk_log_start":"29'2100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5739680,"num_objects":701,"num_object_clones":0,"num_object_copies":2804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":701,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12107,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1544}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'12391","reported_seq":12523,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:26.918819+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:45:26.918819+0000","last_peered":"2024-09-05T16:45:26.918819+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:45:26.918819+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"30'2300","ondisk_log_start":"30'2300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5992176,"num_objects":732,"num_object_clones":0,"num_object_copies":2928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":732,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12388,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1549}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12049","reported_seq":12151,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:26.936751+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:45:26.936751+0000","last_peered":"2024-09-05T16:45:26.936751+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:45:26.936751+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"29'2000","ondisk_log_start":"29'2000","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5814864,"num_objects":710,"num_object_clones":0,"num_object_copies":2840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":710,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12048,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1528}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12125","reported_seq":16866,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:45:26.328732+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:45:26.328732+0000","last_peered":"2024-09-05T16:45:26.328732+0000","last_clean":"2024-09-05T16:45:26.328732+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:45:26.328732+0000","last_undegraded":"2024-09-05T16:45:26.328732+0000","last_fullsized":"2024-09-05T16:45:26.328732+0000","mapping_epoch":36,"log_start":"29'2100","ondisk_log_start":"29'2100","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5595136,"num_objects":683,"num_object_clones":0,"num_object_copies":2732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":683,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3725,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'12408","reported_seq":12521,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:26.859885+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:45:26.859885+0000","last_peered":"2024-09-05T16:45:26.859885+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:45:26.859885+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"33'2400","ondisk_log_start":"33'2400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5878944,"num_objects":718,"num_object_clones":0,"num_object_copies":2872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":718,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12406,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1616}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12118","reported_seq":12236,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:25.215389+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:45:25.215389+0000","last_peered":"2024-09-05T16:45:25.215389+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:45:25.215389+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"29'2100","ondisk_log_start":"29'2100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5854368,"num_objects":715,"num_object_clones":0,"num_object_copies":2860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":715,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12116,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'12059","reported_seq":12172,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:25.657080+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:45:25.657080+0000","last_peered":"2024-09-05T16:45:25.657080+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:45:25.657080+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"29'2000","ondisk_log_start":"29'2000","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5862560,"num_objects":716,"num_object_clones":0,"num_object_copies":2864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":716,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12057,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1504}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":91274568,"num_objects":11147,"num_object_clones":0,"num_object_copies":44588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28271,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11147,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":169003,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1095,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":183173120,"data_stored":183173120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160612,"ondisk_log_size":160612,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084425,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722432,"kb_used_data":45168,"kb_used_omap":4,"kb_used_meta":677243,"kb_avail":93649408,"statfs":{"total":96636764160,"available":95896993792,"internally_reserved":0,"allocated":46252032,"data_stored":46041451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":693497555},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,7,5,5,4,11],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986302,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":578408,"kb_used_data":45528,"kb_used_omap":4,"kb_used_meta":532859,"kb_avail":93793432,"statfs":{"total":96636764160,"available":96044474368,"internally_reserved":0,"allocated":46620672,"data_stored":46410091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":545648338},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,1,2,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673108,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":726992,"kb_used_data":46068,"kb_used_omap":1,"kb_used_meta":680894,"kb_avail":93644848,"statfs":{"total":96636764160,"available":95892324352,"internally_reserved":0,"allocated":47173632,"data_stored":46963051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":697235909},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182539,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":726968,"kb_used_data":45584,"kb_used_omap":4,"kb_used_meta":681339,"kb_avail":93644872,"statfs":{"total":96636764160,"available":95892348928,"internally_reserved":0,"allocated":46678016,"data_stored":46467435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":697691853},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,0,3,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":46166016,"data_stored":46166016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":45543424,"data_stored":45543424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":46391296,"data_stored":46391296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":45072384,"data_stored":45072384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:45:30.067 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-05T16:45:30.067 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "39'12231", 'reported_seq': 13972, 'reported_epoch': 39, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:26.908026+0000', 'last_change': '2024-09-05T16:43:43.436935+0000', 'last_active': '2024-09-05T16:45:26.908026+0000', 'last_peered': '2024-09-05T16:45:26.908026+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.828289+0000', 'last_became_peered': '2024-09-05T16:34:56.828289+0000', 'last_unstale': '2024-09-05T16:45:26.908026+0000', 'last_undegraded': '2024-09-05T16:34:56.612160+0000', 'last_fullsized': '2024-09-05T16:34:56.557632+0000', 'mapping_epoch': 32, 'log_start': "29'2200", 'ondisk_log_start': "29'2200", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10031, 'log_dups_size': 0, 'ondisk_log_size': 10031, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5513216, 'num_objects': 673, 'num_object_clones': 0, 'num_object_copies': 2692, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1420, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 673, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12231, 'num_write_kb': 51616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 842, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -747}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:45:30.067 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:45:30.067 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "39'12036", 'reported_seq': 12164, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:25.234862+0000', 'last_change': '2024-09-05T16:34:56.867900+0000', 'last_active': '2024-09-05T16:45:25.234862+0000', 'last_peered': '2024-09-05T16:45:25.234862+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:45:25.234862+0000', 'last_undegraded': '2024-09-05T16:34:56.590972+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 32, 'log_start': "29'2000", 'ondisk_log_start': "29'2000", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10036, 'log_dups_size': 0, 'ondisk_log_size': 10036, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5864040, 'num_objects': 717, 'num_object_clones': 0, 'num_object_copies': 2868, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2233, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 717, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12035, 'num_write_kb': 50999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1515}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:45:30.068 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:45:30.068 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "39'11928", 'reported_seq': 12031, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:26.656422+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:45:26.656422+0000', 'last_peered': '2024-09-05T16:45:26.656422+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:45:26.656422+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "29'1900", 'ondisk_log_start': "29'1900", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5461152, 'num_objects': 667, 'num_object_clones': 0, 'num_object_copies': 2668, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2147, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 667, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11926, 'num_write_kb': 50373, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1478}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:45:30.068 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:45:30.068 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "39'12078", 'reported_seq': 12197, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:26.526096+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:45:26.526096+0000', 'last_peered': '2024-09-05T16:45:26.526096+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:45:26.526096+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "29'2000", 'ondisk_log_start': "29'2000", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10078, 'log_dups_size': 0, 'ondisk_log_size': 10078, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5685248, 'num_objects': 694, 'num_object_clones': 0, 'num_object_copies': 2776, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 694, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12078, 'num_write_kb': 51088, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1525}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:45:30.068 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:45:30.068 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "39'11910", 'reported_seq': 12023, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:25.547116+0000', 'last_change': '2024-09-05T16:34:56.903436+0000', 'last_active': '2024-09-05T16:45:25.547116+0000', 'last_peered': '2024-09-05T16:45:25.547116+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:45:25.547116+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "29'1900", 'ondisk_log_start': "29'1900", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10010, 'log_dups_size': 0, 'ondisk_log_size': 10010, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6157472, 'num_objects': 752, 'num_object_clones': 0, 'num_object_copies': 3008, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2133, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 752, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11908, 'num_write_kb': 50636, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -1379}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:45:30.068 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-05T16:45:30.068 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "39'12049", 'reported_seq': 12142, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:25.671397+0000', 'last_change': '2024-09-05T16:34:56.883319+0000', 'last_active': '2024-09-05T16:45:25.671397+0000', 'last_peered': '2024-09-05T16:45:25.671397+0000', 'last_clean': '2024-09-05T16:33:53.505305+0000', 'last_became_active': '2024-09-05T16:34:56.873389+0000', 'last_became_peered': '2024-09-05T16:34:56.873389+0000', 'last_unstale': '2024-09-05T16:45:25.671397+0000', 'last_undegraded': '2024-09-05T16:34:56.602567+0000', 'last_fullsized': '2024-09-05T16:34:56.559958+0000', 'mapping_epoch': 32, 'log_start': "29'2000", 'ondisk_log_start': "29'2000", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10049, 'log_dups_size': 0, 'ondisk_log_size': 10049, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5977248, 'num_objects': 730, 'num_object_clones': 0, 'num_object_copies': 2920, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2234, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 730, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12047, 'num_write_kb': 51108, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1502}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:45:30.068 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:45:30.068 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "39'12134", 'reported_seq': 12240, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:26.766435+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:45:26.766435+0000', 'last_peered': '2024-09-05T16:45:26.766435+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:45:26.766435+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "29'2100", 'ondisk_log_start': "29'2100", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10034, 'log_dups_size': 0, 'ondisk_log_size': 10034, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5237056, 'num_objects': 640, 'num_object_clones': 0, 'num_object_copies': 2560, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 640, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12131, 'num_write_kb': 51081, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1604}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:45:30.069 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:45:30.069 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "39'12109", 'reported_seq': 12236, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:25.075906+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:45:25.075906+0000', 'last_peered': '2024-09-05T16:45:25.075906+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:45:25.075906+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "29'2100", 'ondisk_log_start': "29'2100", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10009, 'log_dups_size': 0, 'ondisk_log_size': 10009, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5739680, 'num_objects': 701, 'num_object_clones': 0, 'num_object_copies': 2804, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 701, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12107, 'num_write_kb': 51230, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1544}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:45:30.069 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:45:30.069 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "39'12391", 'reported_seq': 12523, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:26.918819+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:45:26.918819+0000', 'last_peered': '2024-09-05T16:45:26.918819+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:45:26.918819+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "30'2300", 'ondisk_log_start': "30'2300", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10091, 'log_dups_size': 0, 'ondisk_log_size': 10091, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5992176, 'num_objects': 732, 'num_object_clones': 0, 'num_object_copies': 2928, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2284, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 732, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12388, 'num_write_kb': 52476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1549}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:45:30.069 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:45:30.069 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "39'12049", 'reported_seq': 12151, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:26.936751+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:45:26.936751+0000', 'last_peered': '2024-09-05T16:45:26.936751+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:45:26.936751+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "29'2000", 'ondisk_log_start': "29'2000", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10049, 'log_dups_size': 0, 'ondisk_log_size': 10049, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5814864, 'num_objects': 710, 'num_object_clones': 0, 'num_object_copies': 2840, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 710, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12048, 'num_write_kb': 51031, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1528}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:45:30.069 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:45:30.069 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "39'12408", 'reported_seq': 12521, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:26.859885+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:45:26.859885+0000', 'last_peered': '2024-09-05T16:45:26.859885+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:45:26.859885+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "33'2400", 'ondisk_log_start': "33'2400", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10008, 'log_dups_size': 0, 'ondisk_log_size': 10008, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5878944, 'num_objects': 718, 'num_object_clones': 0, 'num_object_copies': 2872, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 718, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12406, 'num_write_kb': 52500, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1616}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:45:30.069 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:45:30.069 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "39'12118", 'reported_seq': 12236, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:25.215389+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:45:25.215389+0000', 'last_peered': '2024-09-05T16:45:25.215389+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:45:25.215389+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "29'2100", 'ondisk_log_start': "29'2100", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10018, 'log_dups_size': 0, 'ondisk_log_size': 10018, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5854368, 'num_objects': 715, 'num_object_clones': 0, 'num_object_copies': 2860, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2310, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 715, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12116, 'num_write_kb': 51325, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:45:30.070 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:45:30.070 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "39'12059", 'reported_seq': 12172, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:25.657080+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:45:25.657080+0000', 'last_peered': '2024-09-05T16:45:25.657080+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:45:25.657080+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "29'2000", 'ondisk_log_start': "29'2000", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10059, 'log_dups_size': 0, 'ondisk_log_size': 10059, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5862560, 'num_objects': 716, 'num_object_clones': 0, 'num_object_copies': 2864, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 716, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12057, 'num_write_kb': 51092, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1504}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:45:30.070 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:45:30.811 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:45:30.811 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 28271/44588 objects degraded (63.405%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":734,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":39,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":11147,"data_bytes":91274568,"bytes_used":2820915200,"bytes_avail":383726141440,"bytes_total":386547056640,"degraded_objects":28271,"degraded_total":44588,"degraded_ratio":0.63404952005023774,"recovering_objects_per_sec":6,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":75,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":165},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:45:30.842 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:45:30.842 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:45:31.508 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:45:31.509 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:45:31.545 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":397,"stamp":"2024-09-05T16:45:31.074977+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":88112456,"num_objects":10761,"num_object_clones":0,"num_object_copies":43044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28271,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10761,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":169389,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1095,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160698,"ondisk_log_size":160698,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2755016,"kb_used_data":174268,"kb_used_omap":14,"kb_used_meta":2580657,"kb_avail":374732344,"statfs":{"total":386547056640,"available":383725920256,"internally_reserved":0,"allocated":178450432,"data_stored":177608108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2642593335},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,15,9,16,13,18,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19406848,"num_objects":-2369,"num_object_clones":0,"num_object_copies":-9476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-75,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2369,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2369,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":75,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005812"},"pg_stats":[{"pgid":"1.f","version":"39'12231","reported_seq":13972,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:26.908026+0000","last_change":"2024-09-05T16:43:43.436935+0000","last_active":"2024-09-05T16:45:26.908026+0000","last_peered":"2024-09-05T16:45:26.908026+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:45:26.908026+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"29'2200","ondisk_log_start":"29'2200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5513216,"num_objects":673,"num_object_clones":0,"num_object_copies":2692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1420,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":673,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12231,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":842,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-747}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'11934","reported_seq":16590,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:45:26.275010+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:45:26.275010+0000","last_peered":"2024-09-05T16:45:26.275010+0000","last_clean":"2024-09-05T16:45:26.275010+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:45:26.275010+0000","last_undegraded":"2024-09-05T16:45:26.275010+0000","last_fullsized":"2024-09-05T16:45:26.275010+0000","mapping_epoch":38,"log_start":"29'1900","ondisk_log_start":"29'1900","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5062656,"num_objects":618,"num_object_clones":0,"num_object_copies":2472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":618,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1247,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12101","reported_seq":12229,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:30.372403+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:45:30.372403+0000","last_peered":"2024-09-05T16:45:30.372403+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:45:30.372403+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"29'2100","ondisk_log_start":"29'2100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5331560,"num_objects":652,"num_object_clones":0,"num_object_copies":2608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":652,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12100,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1580}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12253","reported_seq":16985,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:45:26.462080+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:45:26.462080+0000","last_peered":"2024-09-05T16:45:26.462080+0000","last_clean":"2024-09-05T16:45:26.462080+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:45:26.462080+0000","last_undegraded":"2024-09-05T16:45:26.462080+0000","last_fullsized":"2024-09-05T16:45:26.462080+0000","mapping_epoch":34,"log_start":"29'2200","ondisk_log_start":"29'2200","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5578752,"num_objects":681,"num_object_clones":0,"num_object_copies":2724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":681,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6553,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'11928","reported_seq":12031,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:26.656422+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:45:26.656422+0000","last_peered":"2024-09-05T16:45:26.656422+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:45:26.656422+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"29'1900","ondisk_log_start":"29'1900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5461152,"num_objects":667,"num_object_clones":0,"num_object_copies":2668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":667,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11926,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1478}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12078","reported_seq":12197,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:26.526096+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:45:26.526096+0000","last_peered":"2024-09-05T16:45:26.526096+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:45:26.526096+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"29'2000","ondisk_log_start":"29'2000","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5685248,"num_objects":694,"num_object_clones":0,"num_object_copies":2776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":694,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12078,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1525}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'11972","reported_seq":12085,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:30.661159+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:45:30.661159+0000","last_peered":"2024-09-05T16:45:30.661159+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:45:30.661159+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"29'1900","ondisk_log_start":"29'1900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5649568,"num_objects":690,"num_object_clones":0,"num_object_copies":2760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":690,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11970,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1441}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12129","reported_seq":12223,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:30.468586+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:45:30.468586+0000","last_peered":"2024-09-05T16:45:30.468586+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:45:30.468586+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"29'2100","ondisk_log_start":"29'2100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5321888,"num_objects":650,"num_object_clones":0,"num_object_copies":2600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":650,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12127,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1582}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"39'12134","reported_seq":12240,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:26.766435+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:45:26.766435+0000","last_peered":"2024-09-05T16:45:26.766435+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:45:26.766435+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"29'2100","ondisk_log_start":"29'2100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5237056,"num_objects":640,"num_object_clones":0,"num_object_copies":2560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":640,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1604}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12168","reported_seq":12295,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:30.466791+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:45:30.466791+0000","last_peered":"2024-09-05T16:45:30.466791+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:45:30.466791+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"29'2100","ondisk_log_start":"29'2100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5256352,"num_objects":642,"num_object_clones":0,"num_object_copies":2568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":642,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12166,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1603}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'12391","reported_seq":12523,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:26.918819+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:45:26.918819+0000","last_peered":"2024-09-05T16:45:26.918819+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:45:26.918819+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"30'2300","ondisk_log_start":"30'2300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5992176,"num_objects":732,"num_object_clones":0,"num_object_copies":2928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":732,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12388,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1549}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12049","reported_seq":12151,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:26.936751+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:45:26.936751+0000","last_peered":"2024-09-05T16:45:26.936751+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:45:26.936751+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"29'2000","ondisk_log_start":"29'2000","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5814864,"num_objects":710,"num_object_clones":0,"num_object_copies":2840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":710,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12048,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1528}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12125","reported_seq":16866,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:45:26.328732+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:45:26.328732+0000","last_peered":"2024-09-05T16:45:26.328732+0000","last_clean":"2024-09-05T16:45:26.328732+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:45:26.328732+0000","last_undegraded":"2024-09-05T16:45:26.328732+0000","last_fullsized":"2024-09-05T16:45:26.328732+0000","mapping_epoch":36,"log_start":"29'2100","ondisk_log_start":"29'2100","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5595136,"num_objects":683,"num_object_clones":0,"num_object_copies":2732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":683,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3725,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'12408","reported_seq":12521,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:26.859885+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:45:26.859885+0000","last_peered":"2024-09-05T16:45:26.859885+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:45:26.859885+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"33'2400","ondisk_log_start":"33'2400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5878944,"num_objects":718,"num_object_clones":0,"num_object_copies":2872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":718,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12406,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1616}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12181","reported_seq":12298,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:30.435063+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:45:30.435063+0000","last_peered":"2024-09-05T16:45:30.435063+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:45:30.435063+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"29'2100","ondisk_log_start":"29'2100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5338272,"num_objects":652,"num_object_clones":0,"num_object_copies":2608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":652,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12179,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1656}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'12116","reported_seq":12227,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:30.542338+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:45:30.542338+0000","last_peered":"2024-09-05T16:45:30.542338+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:45:30.542338+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"29'2100","ondisk_log_start":"29'2100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5395616,"num_objects":659,"num_object_clones":0,"num_object_copies":2636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":659,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12114,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1561}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":88112456,"num_objects":10761,"num_object_clones":0,"num_object_copies":43044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28271,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10761,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":169389,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1095,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":174841856,"data_stored":174841856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160698,"ondisk_log_size":160698,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084425,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722432,"kb_used_data":45168,"kb_used_omap":4,"kb_used_meta":677243,"kb_avail":93649408,"statfs":{"total":96636764160,"available":95896993792,"internally_reserved":0,"allocated":46252032,"data_stored":46041451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":693497555},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,7,5,5,4,11],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986302,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":578408,"kb_used_data":45528,"kb_used_omap":4,"kb_used_meta":532859,"kb_avail":93793432,"statfs":{"total":96636764160,"available":96044474368,"internally_reserved":0,"allocated":46620672,"data_stored":46410091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":545648338},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,1,2,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673109,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":727112,"kb_used_data":42040,"kb_used_omap":1,"kb_used_meta":685054,"kb_avail":93644728,"statfs":{"total":96636764160,"available":95892201472,"internally_reserved":0,"allocated":43048960,"data_stored":42838379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":701495749},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,7,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182540,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":727064,"kb_used_data":41532,"kb_used_omap":4,"kb_used_meta":685499,"kb_avail":93644776,"statfs":{"total":96636764160,"available":95892250624,"internally_reserved":0,"allocated":42528768,"data_stored":42318187,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":701951693},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,1,2,0,1,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":42033152,"data_stored":42033152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":45543424,"data_stored":45543424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":42192896,"data_stored":42192896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":45072384,"data_stored":45072384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:45:34.547 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:45:35.198 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:45:35.199 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:45:35.221 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":399,"stamp":"2024-09-05T16:45:35.076559+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":83074376,"num_objects":10146,"num_object_clones":0,"num_object_copies":40584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10146,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":170004,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1138,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160913,"ondisk_log_size":160913,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2754636,"kb_used_data":165568,"kb_used_omap":14,"kb_used_meta":2588977,"kb_avail":374732724,"statfs":{"total":386547056640,"available":383726309376,"internally_reserved":0,"allocated":169541632,"data_stored":168699308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2651113015},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,6,8,11,24,6,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16482304,"num_objects":-2012,"num_object_clones":0,"num_object_copies":-8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-81,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2012,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2012,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004269"},"pg_stats":[{"pgid":"1.f","version":"39'12293","reported_seq":14121,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:32.030872+0000","last_change":"2024-09-05T16:43:43.436935+0000","last_active":"2024-09-05T16:45:32.030872+0000","last_peered":"2024-09-05T16:45:32.030872+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:45:32.030872+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"29'2200","ondisk_log_start":"29'2200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5005312,"num_objects":611,"num_object_clones":0,"num_object_copies":2444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1377,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":611,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12293,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":885,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-766}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'11990","reported_seq":16646,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:45:31.461066+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:45:31.461066+0000","last_peered":"2024-09-05T16:45:31.461066+0000","last_clean":"2024-09-05T16:45:31.461066+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:45:31.461066+0000","last_undegraded":"2024-09-05T16:45:31.461066+0000","last_fullsized":"2024-09-05T16:45:31.461066+0000","mapping_epoch":38,"log_start":"29'1900","ondisk_log_start":"29'1900","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4603904,"num_objects":562,"num_object_clones":0,"num_object_copies":2248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":562,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1303,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12101","reported_seq":12229,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:30.372403+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:45:30.372403+0000","last_peered":"2024-09-05T16:45:30.372403+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:45:30.372403+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"29'2100","ondisk_log_start":"29'2100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5331560,"num_objects":652,"num_object_clones":0,"num_object_copies":2608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":652,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12100,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1580}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12317","reported_seq":17051,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:45:31.528166+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:45:31.528166+0000","last_peered":"2024-09-05T16:45:31.528166+0000","last_clean":"2024-09-05T16:45:31.528166+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:45:31.528166+0000","last_undegraded":"2024-09-05T16:45:31.528166+0000","last_fullsized":"2024-09-05T16:45:31.528166+0000","mapping_epoch":34,"log_start":"33'2300","ondisk_log_start":"33'2300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5054464,"num_objects":617,"num_object_clones":0,"num_object_copies":2468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":617,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6617,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'11997","reported_seq":12101,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:31.965992+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:45:31.965992+0000","last_peered":"2024-09-05T16:45:31.965992+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:45:31.965992+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"29'1900","ondisk_log_start":"29'1900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4895904,"num_objects":598,"num_object_clones":0,"num_object_copies":2392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":598,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11995,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1547}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12137","reported_seq":12257,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:32.006263+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:45:32.006263+0000","last_peered":"2024-09-05T16:45:32.006263+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:45:32.006263+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"29'2100","ondisk_log_start":"29'2100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5201920,"num_objects":635,"num_object_clones":0,"num_object_copies":2540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":635,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12137,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1584}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'11972","reported_seq":12085,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:30.661159+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:45:30.661159+0000","last_peered":"2024-09-05T16:45:30.661159+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:45:30.661159+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"29'1900","ondisk_log_start":"29'1900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5649568,"num_objects":690,"num_object_clones":0,"num_object_copies":2760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":690,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11970,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1441}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12129","reported_seq":12223,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:30.468586+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:45:30.468586+0000","last_peered":"2024-09-05T16:45:30.468586+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:45:30.468586+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"29'2100","ondisk_log_start":"29'2100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5321888,"num_objects":650,"num_object_clones":0,"num_object_copies":2600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":650,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12127,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1582}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"39'12199","reported_seq":12305,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:31.979522+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:45:31.979522+0000","last_peered":"2024-09-05T16:45:31.979522+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:45:31.979522+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"29'2100","ondisk_log_start":"29'2100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4704576,"num_objects":575,"num_object_clones":0,"num_object_copies":2300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":575,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12196,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1669}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12168","reported_seq":12295,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:30.466791+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:45:30.466791+0000","last_peered":"2024-09-05T16:45:30.466791+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:45:30.466791+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"29'2100","ondisk_log_start":"29'2100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5256352,"num_objects":642,"num_object_clones":0,"num_object_copies":2568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":642,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12166,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1603}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'12462","reported_seq":12594,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:32.007866+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:45:32.007866+0000","last_peered":"2024-09-05T16:45:32.007866+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:45:32.007866+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"33'2400","ondisk_log_start":"33'2400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5410544,"num_objects":661,"num_object_clones":0,"num_object_copies":2644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":661,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12459,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1620}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12109","reported_seq":12213,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:32.029586+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:45:32.029586+0000","last_peered":"2024-09-05T16:45:32.029586+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:45:32.029586+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"29'2100","ondisk_log_start":"29'2100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5323344,"num_objects":650,"num_object_clones":0,"num_object_copies":2600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":650,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12108,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1588}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12184","reported_seq":16925,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:45:31.356190+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:45:31.356190+0000","last_peered":"2024-09-05T16:45:31.356190+0000","last_clean":"2024-09-05T16:45:31.356190+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:45:31.356190+0000","last_undegraded":"2024-09-05T16:45:31.356190+0000","last_fullsized":"2024-09-05T16:45:31.356190+0000","mapping_epoch":36,"log_start":"29'2100","ondisk_log_start":"29'2100","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5111808,"num_objects":624,"num_object_clones":0,"num_object_copies":2496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":624,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3784,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'12458","reported_seq":12572,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:31.833465+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:45:31.833465+0000","last_peered":"2024-09-05T16:45:31.833465+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:45:31.833465+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"33'2400","ondisk_log_start":"33'2400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5469344,"num_objects":668,"num_object_clones":0,"num_object_copies":2672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":668,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12456,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1666}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12181","reported_seq":12298,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:30.435063+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:45:30.435063+0000","last_peered":"2024-09-05T16:45:30.435063+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:45:30.435063+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"29'2100","ondisk_log_start":"29'2100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5338272,"num_objects":652,"num_object_clones":0,"num_object_copies":2608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":652,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12179,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1656}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'12116","reported_seq":12227,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:30.542338+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:45:30.542338+0000","last_peered":"2024-09-05T16:45:30.542338+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:45:30.542338+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"29'2100","ondisk_log_start":"29'2100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5395616,"num_objects":659,"num_object_clones":0,"num_object_copies":2636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":659,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12114,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1561}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":83074376,"num_objects":10146,"num_object_clones":0,"num_object_copies":40584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10146,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":170004,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1138,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":166690816,"data_stored":166690816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160913,"ondisk_log_size":160913,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084426,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721916,"kb_used_data":40492,"kb_used_omap":4,"kb_used_meta":681403,"kb_avail":93649924,"statfs":{"total":96636764160,"available":95897522176,"internally_reserved":0,"allocated":41463808,"data_stored":41253227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":697757395},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2,0,14,2,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986303,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":578544,"kb_used_data":41504,"kb_used_omap":4,"kb_used_meta":537019,"kb_avail":93793296,"statfs":{"total":96636764160,"available":96044335104,"internally_reserved":0,"allocated":42500096,"data_stored":42289515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":549908178},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,3,2,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673109,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":727112,"kb_used_data":42040,"kb_used_omap":1,"kb_used_meta":685054,"kb_avail":93644728,"statfs":{"total":96636764160,"available":95892201472,"internally_reserved":0,"allocated":43048960,"data_stored":42838379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":701495749},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,7,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182540,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":727064,"kb_used_data":41532,"kb_used_omap":4,"kb_used_meta":685499,"kb_avail":93644776,"statfs":{"total":96636764160,"available":95892250624,"internally_reserved":0,"allocated":42528768,"data_stored":42318187,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":701951693},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,1,2,0,1,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":42033152,"data_stored":42033152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":41476096,"data_stored":41476096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":42192896,"data_stored":42192896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40988672,"data_stored":40988672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:45:35.222 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:45:35.881 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:45:35.881 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:45:35.904 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":399,"stamp":"2024-09-05T16:45:35.076559+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":83074376,"num_objects":10146,"num_object_clones":0,"num_object_copies":40584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10146,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":170004,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1138,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160913,"ondisk_log_size":160913,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2754636,"kb_used_data":165568,"kb_used_omap":14,"kb_used_meta":2588977,"kb_avail":374732724,"statfs":{"total":386547056640,"available":383726309376,"internally_reserved":0,"allocated":169541632,"data_stored":168699308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2651113015},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,6,8,11,24,6,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16482304,"num_objects":-2012,"num_object_clones":0,"num_object_copies":-8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-81,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2012,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2012,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004269"},"pg_stats":[{"pgid":"1.f","version":"39'12293","reported_seq":14121,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:32.030872+0000","last_change":"2024-09-05T16:43:43.436935+0000","last_active":"2024-09-05T16:45:32.030872+0000","last_peered":"2024-09-05T16:45:32.030872+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:45:32.030872+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"29'2200","ondisk_log_start":"29'2200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5005312,"num_objects":611,"num_object_clones":0,"num_object_copies":2444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1377,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":611,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12293,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":885,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-766}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'11990","reported_seq":16646,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:45:31.461066+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:45:31.461066+0000","last_peered":"2024-09-05T16:45:31.461066+0000","last_clean":"2024-09-05T16:45:31.461066+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:45:31.461066+0000","last_undegraded":"2024-09-05T16:45:31.461066+0000","last_fullsized":"2024-09-05T16:45:31.461066+0000","mapping_epoch":38,"log_start":"29'1900","ondisk_log_start":"29'1900","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4603904,"num_objects":562,"num_object_clones":0,"num_object_copies":2248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":562,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1303,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12101","reported_seq":12229,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:30.372403+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:45:30.372403+0000","last_peered":"2024-09-05T16:45:30.372403+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:45:30.372403+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"29'2100","ondisk_log_start":"29'2100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5331560,"num_objects":652,"num_object_clones":0,"num_object_copies":2608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":652,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12100,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1580}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12317","reported_seq":17051,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:45:31.528166+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:45:31.528166+0000","last_peered":"2024-09-05T16:45:31.528166+0000","last_clean":"2024-09-05T16:45:31.528166+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:45:31.528166+0000","last_undegraded":"2024-09-05T16:45:31.528166+0000","last_fullsized":"2024-09-05T16:45:31.528166+0000","mapping_epoch":34,"log_start":"33'2300","ondisk_log_start":"33'2300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5054464,"num_objects":617,"num_object_clones":0,"num_object_copies":2468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":617,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6617,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'11997","reported_seq":12101,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:31.965992+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:45:31.965992+0000","last_peered":"2024-09-05T16:45:31.965992+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:45:31.965992+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"29'1900","ondisk_log_start":"29'1900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4895904,"num_objects":598,"num_object_clones":0,"num_object_copies":2392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":598,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11995,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1547}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12137","reported_seq":12257,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:32.006263+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:45:32.006263+0000","last_peered":"2024-09-05T16:45:32.006263+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:45:32.006263+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"29'2100","ondisk_log_start":"29'2100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5201920,"num_objects":635,"num_object_clones":0,"num_object_copies":2540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":635,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12137,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1584}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'11972","reported_seq":12085,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:30.661159+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:45:30.661159+0000","last_peered":"2024-09-05T16:45:30.661159+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:45:30.661159+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"29'1900","ondisk_log_start":"29'1900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5649568,"num_objects":690,"num_object_clones":0,"num_object_copies":2760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":690,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11970,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1441}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12129","reported_seq":12223,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:30.468586+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:45:30.468586+0000","last_peered":"2024-09-05T16:45:30.468586+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:45:30.468586+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"29'2100","ondisk_log_start":"29'2100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5321888,"num_objects":650,"num_object_clones":0,"num_object_copies":2600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":650,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12127,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1582}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"39'12199","reported_seq":12305,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:31.979522+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:45:31.979522+0000","last_peered":"2024-09-05T16:45:31.979522+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:45:31.979522+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"29'2100","ondisk_log_start":"29'2100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4704576,"num_objects":575,"num_object_clones":0,"num_object_copies":2300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":575,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12196,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1669}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12168","reported_seq":12295,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:30.466791+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:45:30.466791+0000","last_peered":"2024-09-05T16:45:30.466791+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:45:30.466791+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"29'2100","ondisk_log_start":"29'2100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5256352,"num_objects":642,"num_object_clones":0,"num_object_copies":2568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":642,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12166,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1603}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'12462","reported_seq":12594,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:32.007866+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:45:32.007866+0000","last_peered":"2024-09-05T16:45:32.007866+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:45:32.007866+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"33'2400","ondisk_log_start":"33'2400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5410544,"num_objects":661,"num_object_clones":0,"num_object_copies":2644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":661,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12459,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1620}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12109","reported_seq":12213,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:32.029586+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:45:32.029586+0000","last_peered":"2024-09-05T16:45:32.029586+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:45:32.029586+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"29'2100","ondisk_log_start":"29'2100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5323344,"num_objects":650,"num_object_clones":0,"num_object_copies":2600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":650,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12108,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1588}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12184","reported_seq":16925,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:45:31.356190+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:45:31.356190+0000","last_peered":"2024-09-05T16:45:31.356190+0000","last_clean":"2024-09-05T16:45:31.356190+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:45:31.356190+0000","last_undegraded":"2024-09-05T16:45:31.356190+0000","last_fullsized":"2024-09-05T16:45:31.356190+0000","mapping_epoch":36,"log_start":"29'2100","ondisk_log_start":"29'2100","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5111808,"num_objects":624,"num_object_clones":0,"num_object_copies":2496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":624,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3784,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'12458","reported_seq":12572,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:31.833465+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:45:31.833465+0000","last_peered":"2024-09-05T16:45:31.833465+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:45:31.833465+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"33'2400","ondisk_log_start":"33'2400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5469344,"num_objects":668,"num_object_clones":0,"num_object_copies":2672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":668,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12456,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1666}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12181","reported_seq":12298,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:30.435063+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:45:30.435063+0000","last_peered":"2024-09-05T16:45:30.435063+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:45:30.435063+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"29'2100","ondisk_log_start":"29'2100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5338272,"num_objects":652,"num_object_clones":0,"num_object_copies":2608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":652,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12179,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1656}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'12116","reported_seq":12227,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:30.542338+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:45:30.542338+0000","last_peered":"2024-09-05T16:45:30.542338+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:45:30.542338+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"29'2100","ondisk_log_start":"29'2100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5395616,"num_objects":659,"num_object_clones":0,"num_object_copies":2636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":659,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12114,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1561}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":83074376,"num_objects":10146,"num_object_clones":0,"num_object_copies":40584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10146,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":170004,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1138,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":166690816,"data_stored":166690816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160913,"ondisk_log_size":160913,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084426,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721916,"kb_used_data":40492,"kb_used_omap":4,"kb_used_meta":681403,"kb_avail":93649924,"statfs":{"total":96636764160,"available":95897522176,"internally_reserved":0,"allocated":41463808,"data_stored":41253227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":697757395},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2,0,14,2,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986303,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":578544,"kb_used_data":41504,"kb_used_omap":4,"kb_used_meta":537019,"kb_avail":93793296,"statfs":{"total":96636764160,"available":96044335104,"internally_reserved":0,"allocated":42500096,"data_stored":42289515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":549908178},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,3,2,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673109,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":727112,"kb_used_data":42040,"kb_used_omap":1,"kb_used_meta":685054,"kb_avail":93644728,"statfs":{"total":96636764160,"available":95892201472,"internally_reserved":0,"allocated":43048960,"data_stored":42838379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":701495749},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,7,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182540,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":727064,"kb_used_data":41532,"kb_used_omap":4,"kb_used_meta":685499,"kb_avail":93644776,"statfs":{"total":96636764160,"available":95892250624,"internally_reserved":0,"allocated":42528768,"data_stored":42318187,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":701951693},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,1,2,0,1,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":42033152,"data_stored":42033152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":41476096,"data_stored":41476096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":42192896,"data_stored":42192896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40988672,"data_stored":40988672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:45:35.906 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-05T16:45:35.906 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "39'12293", 'reported_seq': 14121, 'reported_epoch': 39, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:32.030872+0000', 'last_change': '2024-09-05T16:43:43.436935+0000', 'last_active': '2024-09-05T16:45:32.030872+0000', 'last_peered': '2024-09-05T16:45:32.030872+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.828289+0000', 'last_became_peered': '2024-09-05T16:34:56.828289+0000', 'last_unstale': '2024-09-05T16:45:32.030872+0000', 'last_undegraded': '2024-09-05T16:34:56.612160+0000', 'last_fullsized': '2024-09-05T16:34:56.557632+0000', 'mapping_epoch': 32, 'log_start': "29'2200", 'ondisk_log_start': "29'2200", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10093, 'log_dups_size': 0, 'ondisk_log_size': 10093, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5005312, 'num_objects': 611, 'num_object_clones': 0, 'num_object_copies': 2444, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1377, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 611, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12293, 'num_write_kb': 51616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 885, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -766}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:45:35.906 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:45:35.906 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "39'12101", 'reported_seq': 12229, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:30.372403+0000', 'last_change': '2024-09-05T16:34:56.867900+0000', 'last_active': '2024-09-05T16:45:30.372403+0000', 'last_peered': '2024-09-05T16:45:30.372403+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:45:30.372403+0000', 'last_undegraded': '2024-09-05T16:34:56.590972+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 32, 'log_start': "29'2100", 'ondisk_log_start': "29'2100", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10001, 'log_dups_size': 0, 'ondisk_log_size': 10001, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5331560, 'num_objects': 652, 'num_object_clones': 0, 'num_object_copies': 2608, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2233, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 652, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12100, 'num_write_kb': 50999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1580}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:45:35.906 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:45:35.906 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "39'11997", 'reported_seq': 12101, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:31.965992+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:45:31.965992+0000', 'last_peered': '2024-09-05T16:45:31.965992+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:45:31.965992+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "29'1900", 'ondisk_log_start': "29'1900", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10097, 'log_dups_size': 0, 'ondisk_log_size': 10097, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4895904, 'num_objects': 598, 'num_object_clones': 0, 'num_object_copies': 2392, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2147, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 598, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11995, 'num_write_kb': 50373, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1547}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:45:35.906 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:45:35.906 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "39'12137", 'reported_seq': 12257, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:32.006263+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:45:32.006263+0000', 'last_peered': '2024-09-05T16:45:32.006263+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:45:32.006263+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "29'2100", 'ondisk_log_start': "29'2100", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10037, 'log_dups_size': 0, 'ondisk_log_size': 10037, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5201920, 'num_objects': 635, 'num_object_clones': 0, 'num_object_copies': 2540, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 635, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12137, 'num_write_kb': 51088, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1584}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:45:35.906 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:45:35.907 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "39'11972", 'reported_seq': 12085, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:30.661159+0000', 'last_change': '2024-09-05T16:34:56.903436+0000', 'last_active': '2024-09-05T16:45:30.661159+0000', 'last_peered': '2024-09-05T16:45:30.661159+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:45:30.661159+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "29'1900", 'ondisk_log_start': "29'1900", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10072, 'log_dups_size': 0, 'ondisk_log_size': 10072, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5649568, 'num_objects': 690, 'num_object_clones': 0, 'num_object_copies': 2760, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2133, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 690, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11970, 'num_write_kb': 50636, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -1441}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:45:35.907 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-05T16:45:35.907 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "39'12129", 'reported_seq': 12223, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:30.468586+0000', 'last_change': '2024-09-05T16:34:56.883319+0000', 'last_active': '2024-09-05T16:45:30.468586+0000', 'last_peered': '2024-09-05T16:45:30.468586+0000', 'last_clean': '2024-09-05T16:33:53.505305+0000', 'last_became_active': '2024-09-05T16:34:56.873389+0000', 'last_became_peered': '2024-09-05T16:34:56.873389+0000', 'last_unstale': '2024-09-05T16:45:30.468586+0000', 'last_undegraded': '2024-09-05T16:34:56.602567+0000', 'last_fullsized': '2024-09-05T16:34:56.559958+0000', 'mapping_epoch': 32, 'log_start': "29'2100", 'ondisk_log_start': "29'2100", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10029, 'log_dups_size': 0, 'ondisk_log_size': 10029, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5321888, 'num_objects': 650, 'num_object_clones': 0, 'num_object_copies': 2600, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2234, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 650, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12127, 'num_write_kb': 51108, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1582}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:45:35.907 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:45:35.907 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "39'12199", 'reported_seq': 12305, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:31.979522+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:45:31.979522+0000', 'last_peered': '2024-09-05T16:45:31.979522+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:45:31.979522+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "29'2100", 'ondisk_log_start': "29'2100", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10099, 'log_dups_size': 0, 'ondisk_log_size': 10099, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4704576, 'num_objects': 575, 'num_object_clones': 0, 'num_object_copies': 2300, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 575, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12196, 'num_write_kb': 51081, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1669}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:45:35.907 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:45:35.907 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "39'12168", 'reported_seq': 12295, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:30.466791+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:45:30.466791+0000', 'last_peered': '2024-09-05T16:45:30.466791+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:45:30.466791+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "29'2100", 'ondisk_log_start': "29'2100", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10068, 'log_dups_size': 0, 'ondisk_log_size': 10068, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5256352, 'num_objects': 642, 'num_object_clones': 0, 'num_object_copies': 2568, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 642, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12166, 'num_write_kb': 51230, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1603}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:45:35.907 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:45:35.907 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "39'12462", 'reported_seq': 12594, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:32.007866+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:45:32.007866+0000', 'last_peered': '2024-09-05T16:45:32.007866+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:45:32.007866+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "33'2400", 'ondisk_log_start': "33'2400", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10062, 'log_dups_size': 0, 'ondisk_log_size': 10062, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5410544, 'num_objects': 661, 'num_object_clones': 0, 'num_object_copies': 2644, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2284, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 661, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12459, 'num_write_kb': 52476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1620}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:45:35.908 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:45:35.908 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "39'12109", 'reported_seq': 12213, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:32.029586+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:45:32.029586+0000', 'last_peered': '2024-09-05T16:45:32.029586+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:45:32.029586+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "29'2100", 'ondisk_log_start': "29'2100", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10009, 'log_dups_size': 0, 'ondisk_log_size': 10009, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5323344, 'num_objects': 650, 'num_object_clones': 0, 'num_object_copies': 2600, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 650, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12108, 'num_write_kb': 51031, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1588}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:45:35.908 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:45:35.908 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "39'12458", 'reported_seq': 12572, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:31.833465+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:45:31.833465+0000', 'last_peered': '2024-09-05T16:45:31.833465+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:45:31.833465+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "33'2400", 'ondisk_log_start': "33'2400", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10058, 'log_dups_size': 0, 'ondisk_log_size': 10058, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5469344, 'num_objects': 668, 'num_object_clones': 0, 'num_object_copies': 2672, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 668, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12456, 'num_write_kb': 52500, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1666}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:45:35.908 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:45:35.908 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "39'12181", 'reported_seq': 12298, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:30.435063+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:45:30.435063+0000', 'last_peered': '2024-09-05T16:45:30.435063+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:45:30.435063+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "29'2100", 'ondisk_log_start': "29'2100", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10081, 'log_dups_size': 0, 'ondisk_log_size': 10081, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5338272, 'num_objects': 652, 'num_object_clones': 0, 'num_object_copies': 2608, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2310, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 652, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12179, 'num_write_kb': 51325, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1656}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:45:35.908 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:45:35.908 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "39'12116", 'reported_seq': 12227, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:30.542338+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:45:30.542338+0000', 'last_peered': '2024-09-05T16:45:30.542338+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:45:30.542338+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "29'2100", 'ondisk_log_start': "29'2100", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10016, 'log_dups_size': 0, 'ondisk_log_size': 10016, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5395616, 'num_objects': 659, 'num_object_clones': 0, 'num_object_copies': 2636, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 659, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12114, 'num_write_kb': 51092, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1561}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:45:35.909 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:45:36.708 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:45:36.709 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 28228/40584 objects degraded (69.555%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":739,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":39,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":10146,"data_bytes":83074376,"bytes_used":2820747264,"bytes_avail":383726309376,"bytes_total":386547056640,"degraded_objects":28228,"degraded_total":40584,"degraded_ratio":0.69554504238123405,"recovering_objects_per_sec":6,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":167},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:45:36.739 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:45:36.739 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:45:37.359 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:45:37.359 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:45:37.383 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":400,"stamp":"2024-09-05T16:45:37.077409+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":78363976,"num_objects":9571,"num_object_clones":0,"num_object_copies":38284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9571,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":170579,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1138,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160988,"ondisk_log_size":160988,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2773072,"kb_used_data":153464,"kb_used_omap":14,"kb_used_meta":2619505,"kb_avail":374714288,"statfs":{"total":386547056640,"available":383707430912,"internally_reserved":0,"allocated":157147136,"data_stored":156304812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2682373687},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,10,5,8,20,7,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21192704,"num_objects":-2587,"num_object_clones":0,"num_object_copies":-10348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-81,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2587,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2587,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004390"},"pg_stats":[{"pgid":"1.f","version":"39'12293","reported_seq":14121,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:32.030872+0000","last_change":"2024-09-05T16:43:43.436935+0000","last_active":"2024-09-05T16:45:32.030872+0000","last_peered":"2024-09-05T16:45:32.030872+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:45:32.030872+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"29'2200","ondisk_log_start":"29'2200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5005312,"num_objects":611,"num_object_clones":0,"num_object_copies":2444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1377,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":611,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12293,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":885,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-766}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12047","reported_seq":16703,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:45:36.395123+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:45:36.395123+0000","last_peered":"2024-09-05T16:45:36.395123+0000","last_clean":"2024-09-05T16:45:36.395123+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:45:36.395123+0000","last_undegraded":"2024-09-05T16:45:36.395123+0000","last_fullsized":"2024-09-05T16:45:36.395123+0000","mapping_epoch":38,"log_start":"29'2000","ondisk_log_start":"29'2000","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4136960,"num_objects":505,"num_object_clones":0,"num_object_copies":2020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":505,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1360,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12167","reported_seq":12295,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:35.350879+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:45:35.350879+0000","last_peered":"2024-09-05T16:45:35.350879+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:45:35.350879+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"29'2100","ondisk_log_start":"29'2100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4790888,"num_objects":586,"num_object_clones":0,"num_object_copies":2344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":586,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12166,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1646}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12397","reported_seq":17131,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:45:36.528269+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:45:36.528269+0000","last_peered":"2024-09-05T16:45:36.528269+0000","last_clean":"2024-09-05T16:45:36.528269+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:45:36.528269+0000","last_undegraded":"2024-09-05T16:45:36.528269+0000","last_fullsized":"2024-09-05T16:45:36.528269+0000","mapping_epoch":34,"log_start":"33'2300","ondisk_log_start":"33'2300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4399104,"num_objects":537,"num_object_clones":0,"num_object_copies":2148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":537,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6697,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'11997","reported_seq":12101,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:31.965992+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:45:31.965992+0000","last_peered":"2024-09-05T16:45:31.965992+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:45:31.965992+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"29'1900","ondisk_log_start":"29'1900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4895904,"num_objects":598,"num_object_clones":0,"num_object_copies":2392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":598,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11995,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1547}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12137","reported_seq":12257,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:32.006263+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:45:32.006263+0000","last_peered":"2024-09-05T16:45:32.006263+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:45:32.006263+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"29'2100","ondisk_log_start":"29'2100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5201920,"num_objects":635,"num_object_clones":0,"num_object_copies":2540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":635,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12137,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1584}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12031","reported_seq":12144,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:35.812041+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:45:35.812041+0000","last_peered":"2024-09-05T16:45:35.812041+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:45:35.812041+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"29'2000","ondisk_log_start":"29'2000","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5166240,"num_objects":631,"num_object_clones":0,"num_object_copies":2524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":631,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12029,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1500}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12190","reported_seq":12283,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:35.704886+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:45:35.704886+0000","last_peered":"2024-09-05T16:45:35.704886+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:45:35.704886+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"29'2100","ondisk_log_start":"29'2100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4822176,"num_objects":589,"num_object_clones":0,"num_object_copies":2356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":589,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12188,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1643}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"39'12199","reported_seq":12305,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:31.979522+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:45:31.979522+0000","last_peered":"2024-09-05T16:45:31.979522+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:45:31.979522+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"29'2100","ondisk_log_start":"29'2100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4704576,"num_objects":575,"num_object_clones":0,"num_object_copies":2300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":575,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12196,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1669}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12223","reported_seq":12350,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:35.295828+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:45:35.295828+0000","last_peered":"2024-09-05T16:45:35.295828+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:45:35.295828+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"29'2200","ondisk_log_start":"29'2200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4805792,"num_objects":587,"num_object_clones":0,"num_object_copies":2348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":587,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12221,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1658}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'12462","reported_seq":12594,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:32.007866+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:45:32.007866+0000","last_peered":"2024-09-05T16:45:32.007866+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:45:32.007866+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"33'2400","ondisk_log_start":"33'2400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5410544,"num_objects":661,"num_object_clones":0,"num_object_copies":2644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":661,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12459,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1620}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12109","reported_seq":12213,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:32.029586+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:45:32.029586+0000","last_peered":"2024-09-05T16:45:32.029586+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:45:32.029586+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"29'2100","ondisk_log_start":"29'2100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5323344,"num_objects":650,"num_object_clones":0,"num_object_copies":2600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":650,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12108,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1588}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12240","reported_seq":16979,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:45:36.649108+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:45:36.649108+0000","last_peered":"2024-09-05T16:45:36.649108+0000","last_clean":"2024-09-05T16:45:36.649108+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:45:36.649108+0000","last_undegraded":"2024-09-05T16:45:36.649108+0000","last_fullsized":"2024-09-05T16:45:36.649108+0000","mapping_epoch":36,"log_start":"29'2200","ondisk_log_start":"29'2200","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4653056,"num_objects":568,"num_object_clones":0,"num_object_copies":2272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":568,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3840,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'12458","reported_seq":12572,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:31.833465+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:45:31.833465+0000","last_peered":"2024-09-05T16:45:31.833465+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:45:31.833465+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"33'2400","ondisk_log_start":"33'2400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5469344,"num_objects":668,"num_object_clones":0,"num_object_copies":2672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":668,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12456,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1666}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12260","reported_seq":12378,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:35.374221+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:45:35.374221+0000","last_peered":"2024-09-05T16:45:35.374221+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:45:35.374221+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"29'2200","ondisk_log_start":"29'2200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4691104,"num_objects":573,"num_object_clones":0,"num_object_copies":2292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":573,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12258,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1735}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'12178","reported_seq":12292,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:35.694367+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:45:35.694367+0000","last_peered":"2024-09-05T16:45:35.694367+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:45:35.694367+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"29'2100","ondisk_log_start":"29'2100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4887712,"num_objects":597,"num_object_clones":0,"num_object_copies":2388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":597,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12176,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1623}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":78363976,"num_objects":9571,"num_object_clones":0,"num_object_copies":38284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9571,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":170579,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1138,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":154054656,"data_stored":154054656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160988,"ondisk_log_size":160988,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084426,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721916,"kb_used_data":40492,"kb_used_omap":4,"kb_used_meta":681403,"kb_avail":93649924,"statfs":{"total":96636764160,"available":95897522176,"internally_reserved":0,"allocated":41463808,"data_stored":41253227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":697757395},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2,0,14,2,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986304,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":596652,"kb_used_data":37400,"kb_used_omap":4,"kb_used_meta":559227,"kb_avail":93775188,"statfs":{"total":96636764160,"available":96025792512,"internally_reserved":0,"allocated":38297600,"data_stored":38087019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":572649170},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,0,2,2,5,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673110,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":727272,"kb_used_data":38028,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93644568,"statfs":{"total":96636764160,"available":95892037632,"internally_reserved":0,"allocated":38940672,"data_stored":38730091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,2,5,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182541,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":727232,"kb_used_data":37544,"kb_used_omap":4,"kb_used_meta":689659,"kb_avail":93644608,"statfs":{"total":96636764160,"available":95892078592,"internally_reserved":0,"allocated":38445056,"data_stored":38234475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":706211533},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":37769216,"data_stored":37769216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":37265408,"data_stored":37265408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38031360,"data_stored":38031360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40988672,"data_stored":40988672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:45:40.385 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:45:41.435 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:45:41.435 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:45:41.467 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":402,"stamp":"2024-09-05T16:45:41.079134+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":71400776,"num_objects":8721,"num_object_clones":0,"num_object_copies":34884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8721,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":171429,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1174,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160938,"ondisk_log_size":160938,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2740212,"kb_used_data":141420,"kb_used_omap":14,"kb_used_meta":2598705,"kb_avail":374747148,"statfs":{"total":386547056640,"available":383741079552,"internally_reserved":0,"allocated":144814080,"data_stored":143971756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2661074487},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,8,6,10,13,12,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19873792,"num_objects":-2426,"num_object_clones":0,"num_object_copies":-9704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-79,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2426,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2426,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004620"},"pg_stats":[{"pgid":"1.f","version":"39'12364","reported_seq":14265,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:37.160954+0000","last_change":"2024-09-05T16:43:43.436935+0000","last_active":"2024-09-05T16:45:37.160954+0000","last_peered":"2024-09-05T16:45:37.160954+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:45:37.160954+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"33'2300","ondisk_log_start":"33'2300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4423680,"num_objects":540,"num_object_clones":0,"num_object_copies":2160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1341,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":540,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12364,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":921,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-801}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12047","reported_seq":16703,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:45:36.395123+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:45:36.395123+0000","last_peered":"2024-09-05T16:45:36.395123+0000","last_clean":"2024-09-05T16:45:36.395123+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:45:36.395123+0000","last_undegraded":"2024-09-05T16:45:36.395123+0000","last_fullsized":"2024-09-05T16:45:36.395123+0000","mapping_epoch":38,"log_start":"29'2000","ondisk_log_start":"29'2000","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4136960,"num_objects":505,"num_object_clones":0,"num_object_copies":2020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":505,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1360,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12231","reported_seq":12359,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:40.387074+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:45:40.387074+0000","last_peered":"2024-09-05T16:45:40.387074+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:45:40.387074+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"29'2200","ondisk_log_start":"29'2200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4266600,"num_objects":522,"num_object_clones":0,"num_object_copies":2088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":522,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12230,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1710}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12397","reported_seq":17131,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:45:36.528269+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:45:36.528269+0000","last_peered":"2024-09-05T16:45:36.528269+0000","last_clean":"2024-09-05T16:45:36.528269+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:45:36.528269+0000","last_undegraded":"2024-09-05T16:45:36.528269+0000","last_fullsized":"2024-09-05T16:45:36.528269+0000","mapping_epoch":34,"log_start":"33'2300","ondisk_log_start":"33'2300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4399104,"num_objects":537,"num_object_clones":0,"num_object_copies":2148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":537,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6697,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12056","reported_seq":12160,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:37.131612+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:45:37.131612+0000","last_peered":"2024-09-05T16:45:37.131612+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:45:37.131612+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"29'2000","ondisk_log_start":"29'2000","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4412576,"num_objects":539,"num_object_clones":0,"num_object_copies":2156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":539,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12054,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1606}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12194","reported_seq":12314,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:37.016846+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:45:37.016846+0000","last_peered":"2024-09-05T16:45:37.016846+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:45:37.016846+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"29'2100","ondisk_log_start":"29'2100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4734976,"num_objects":578,"num_object_clones":0,"num_object_copies":2312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":578,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12194,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1641}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12096","reported_seq":12209,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:40.667158+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:45:40.667158+0000","last_peered":"2024-09-05T16:45:40.667158+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:45:40.667158+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"29'2000","ondisk_log_start":"29'2000","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4633760,"num_objects":566,"num_object_clones":0,"num_object_copies":2264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":566,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12094,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1565}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12246","reported_seq":12338,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:40.854903+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:45:40.854903+0000","last_peered":"2024-09-05T16:45:40.854903+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:45:40.854903+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"30'2200","ondisk_log_start":"30'2200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4363424,"num_objects":533,"num_object_clones":0,"num_object_copies":2132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12244,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1699}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"39'12263","reported_seq":12369,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:37.118111+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:45:37.118111+0000","last_peered":"2024-09-05T16:45:37.118111+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:45:37.118111+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"29'2200","ondisk_log_start":"29'2200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4180288,"num_objects":511,"num_object_clones":0,"num_object_copies":2044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":511,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12260,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1733}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12299","reported_seq":12424,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:40.442235+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:45:40.442235+0000","last_peered":"2024-09-05T16:45:40.442235+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:45:40.442235+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"29'2200","ondisk_log_start":"29'2200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4183200,"num_objects":511,"num_object_clones":0,"num_object_copies":2044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":511,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12297,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1734}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'12534","reported_seq":12666,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:37.168294+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:45:37.168294+0000","last_peered":"2024-09-05T16:45:37.168294+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:45:37.168294+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4820720,"num_objects":589,"num_object_clones":0,"num_object_copies":2356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":589,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12531,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1692}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12178","reported_seq":12282,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:37.025679+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:45:37.025679+0000","last_peered":"2024-09-05T16:45:37.025679+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:45:37.025679+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"29'2100","ondisk_log_start":"29'2100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4758096,"num_objects":581,"num_object_clones":0,"num_object_copies":2324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":581,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12177,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1657}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12240","reported_seq":16979,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:45:36.649108+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:45:36.649108+0000","last_peered":"2024-09-05T16:45:36.649108+0000","last_clean":"2024-09-05T16:45:36.649108+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:45:36.649108+0000","last_undegraded":"2024-09-05T16:45:36.649108+0000","last_fullsized":"2024-09-05T16:45:36.649108+0000","mapping_epoch":36,"log_start":"29'2200","ondisk_log_start":"29'2200","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4653056,"num_objects":568,"num_object_clones":0,"num_object_copies":2272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":568,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3840,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'12523","reported_seq":12637,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:37.070949+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:45:37.070949+0000","last_peered":"2024-09-05T16:45:37.070949+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:45:37.070949+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4936864,"num_objects":603,"num_object_clones":0,"num_object_copies":2412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":603,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12521,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1731}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12322","reported_seq":12440,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:40.169933+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:45:40.169933+0000","last_peered":"2024-09-05T16:45:40.169933+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:45:40.169933+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"30'2300","ondisk_log_start":"30'2300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4183200,"num_objects":511,"num_object_clones":0,"num_object_copies":2044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":511,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12320,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1797}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'12248","reported_seq":12362,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:40.940992+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:45:40.940992+0000","last_peered":"2024-09-05T16:45:40.940992+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:45:40.940992+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"29'2200","ondisk_log_start":"29'2200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4314272,"num_objects":527,"num_object_clones":0,"num_object_copies":2108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":527,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12246,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1693}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":71400776,"num_objects":8721,"num_object_clones":0,"num_object_copies":34884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8721,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":171429,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1174,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":141647872,"data_stored":141647872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160938,"ondisk_log_size":160938,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084427,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721996,"kb_used_data":36412,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93649844,"statfs":{"total":96636764160,"available":95897440256,"internally_reserved":0,"allocated":37285888,"data_stored":37075307,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":702017235},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,3,4,1,5,5,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986304,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":596652,"kb_used_data":37400,"kb_used_omap":4,"kb_used_meta":559227,"kb_avail":93775188,"statfs":{"total":96636764160,"available":96025792512,"internally_reserved":0,"allocated":38297600,"data_stored":38087019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":572649170},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,0,2,2,5,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673111,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":710880,"kb_used_data":34128,"kb_used_omap":1,"kb_used_meta":676734,"kb_avail":93660960,"statfs":{"total":96636764160,"available":95908823040,"internally_reserved":0,"allocated":34947072,"data_stored":34736491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":692976069},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182542,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":710684,"kb_used_data":33480,"kb_used_omap":4,"kb_used_meta":677179,"kb_avail":93661156,"statfs":{"total":96636764160,"available":95909023744,"internally_reserved":0,"allocated":34283520,"data_stored":34072939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":693432013},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2,4,5,2,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":33636352,"data_stored":33636352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":37265408,"data_stored":37265408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":33959936,"data_stored":33959936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":36786176,"data_stored":36786176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:45:41.468 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:45:42.106 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:45:42.106 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:45:42.135 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":402,"stamp":"2024-09-05T16:45:41.079134+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":71400776,"num_objects":8721,"num_object_clones":0,"num_object_copies":34884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8721,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":171429,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1174,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160938,"ondisk_log_size":160938,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2740212,"kb_used_data":141420,"kb_used_omap":14,"kb_used_meta":2598705,"kb_avail":374747148,"statfs":{"total":386547056640,"available":383741079552,"internally_reserved":0,"allocated":144814080,"data_stored":143971756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2661074487},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,8,6,10,13,12,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19873792,"num_objects":-2426,"num_object_clones":0,"num_object_copies":-9704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-79,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2426,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2426,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004620"},"pg_stats":[{"pgid":"1.f","version":"39'12364","reported_seq":14265,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:37.160954+0000","last_change":"2024-09-05T16:43:43.436935+0000","last_active":"2024-09-05T16:45:37.160954+0000","last_peered":"2024-09-05T16:45:37.160954+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:45:37.160954+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"33'2300","ondisk_log_start":"33'2300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4423680,"num_objects":540,"num_object_clones":0,"num_object_copies":2160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1341,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":540,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12364,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":921,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-801}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12047","reported_seq":16703,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:45:36.395123+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:45:36.395123+0000","last_peered":"2024-09-05T16:45:36.395123+0000","last_clean":"2024-09-05T16:45:36.395123+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:45:36.395123+0000","last_undegraded":"2024-09-05T16:45:36.395123+0000","last_fullsized":"2024-09-05T16:45:36.395123+0000","mapping_epoch":38,"log_start":"29'2000","ondisk_log_start":"29'2000","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4136960,"num_objects":505,"num_object_clones":0,"num_object_copies":2020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":505,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1360,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12231","reported_seq":12359,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:40.387074+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:45:40.387074+0000","last_peered":"2024-09-05T16:45:40.387074+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:45:40.387074+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"29'2200","ondisk_log_start":"29'2200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4266600,"num_objects":522,"num_object_clones":0,"num_object_copies":2088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":522,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12230,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1710}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12397","reported_seq":17131,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:45:36.528269+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:45:36.528269+0000","last_peered":"2024-09-05T16:45:36.528269+0000","last_clean":"2024-09-05T16:45:36.528269+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:45:36.528269+0000","last_undegraded":"2024-09-05T16:45:36.528269+0000","last_fullsized":"2024-09-05T16:45:36.528269+0000","mapping_epoch":34,"log_start":"33'2300","ondisk_log_start":"33'2300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4399104,"num_objects":537,"num_object_clones":0,"num_object_copies":2148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":537,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6697,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12056","reported_seq":12160,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:37.131612+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:45:37.131612+0000","last_peered":"2024-09-05T16:45:37.131612+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:45:37.131612+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"29'2000","ondisk_log_start":"29'2000","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4412576,"num_objects":539,"num_object_clones":0,"num_object_copies":2156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":539,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12054,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1606}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12194","reported_seq":12314,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:37.016846+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:45:37.016846+0000","last_peered":"2024-09-05T16:45:37.016846+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:45:37.016846+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"29'2100","ondisk_log_start":"29'2100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4734976,"num_objects":578,"num_object_clones":0,"num_object_copies":2312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":578,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12194,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1641}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12096","reported_seq":12209,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:40.667158+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:45:40.667158+0000","last_peered":"2024-09-05T16:45:40.667158+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:45:40.667158+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"29'2000","ondisk_log_start":"29'2000","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4633760,"num_objects":566,"num_object_clones":0,"num_object_copies":2264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":566,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12094,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1565}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12246","reported_seq":12338,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:40.854903+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:45:40.854903+0000","last_peered":"2024-09-05T16:45:40.854903+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:45:40.854903+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"30'2200","ondisk_log_start":"30'2200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4363424,"num_objects":533,"num_object_clones":0,"num_object_copies":2132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12244,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1699}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"39'12263","reported_seq":12369,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:37.118111+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:45:37.118111+0000","last_peered":"2024-09-05T16:45:37.118111+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:45:37.118111+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"29'2200","ondisk_log_start":"29'2200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4180288,"num_objects":511,"num_object_clones":0,"num_object_copies":2044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":511,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12260,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1733}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12299","reported_seq":12424,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:40.442235+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:45:40.442235+0000","last_peered":"2024-09-05T16:45:40.442235+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:45:40.442235+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"29'2200","ondisk_log_start":"29'2200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4183200,"num_objects":511,"num_object_clones":0,"num_object_copies":2044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":511,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12297,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1734}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'12534","reported_seq":12666,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:37.168294+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:45:37.168294+0000","last_peered":"2024-09-05T16:45:37.168294+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:45:37.168294+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4820720,"num_objects":589,"num_object_clones":0,"num_object_copies":2356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":589,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12531,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1692}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12178","reported_seq":12282,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:37.025679+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:45:37.025679+0000","last_peered":"2024-09-05T16:45:37.025679+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:45:37.025679+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"29'2100","ondisk_log_start":"29'2100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4758096,"num_objects":581,"num_object_clones":0,"num_object_copies":2324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":581,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12177,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1657}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12240","reported_seq":16979,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:45:36.649108+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:45:36.649108+0000","last_peered":"2024-09-05T16:45:36.649108+0000","last_clean":"2024-09-05T16:45:36.649108+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:45:36.649108+0000","last_undegraded":"2024-09-05T16:45:36.649108+0000","last_fullsized":"2024-09-05T16:45:36.649108+0000","mapping_epoch":36,"log_start":"29'2200","ondisk_log_start":"29'2200","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4653056,"num_objects":568,"num_object_clones":0,"num_object_copies":2272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":568,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3840,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'12523","reported_seq":12637,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:37.070949+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:45:37.070949+0000","last_peered":"2024-09-05T16:45:37.070949+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:45:37.070949+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4936864,"num_objects":603,"num_object_clones":0,"num_object_copies":2412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":603,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12521,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1731}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12322","reported_seq":12440,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:40.169933+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:45:40.169933+0000","last_peered":"2024-09-05T16:45:40.169933+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:45:40.169933+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"30'2300","ondisk_log_start":"30'2300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4183200,"num_objects":511,"num_object_clones":0,"num_object_copies":2044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":511,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12320,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1797}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'12248","reported_seq":12362,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:40.940992+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:45:40.940992+0000","last_peered":"2024-09-05T16:45:40.940992+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:45:40.940992+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"29'2200","ondisk_log_start":"29'2200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4314272,"num_objects":527,"num_object_clones":0,"num_object_copies":2108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":527,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12246,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1693}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":71400776,"num_objects":8721,"num_object_clones":0,"num_object_copies":34884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8721,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":171429,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1174,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":141647872,"data_stored":141647872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160938,"ondisk_log_size":160938,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084427,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721996,"kb_used_data":36412,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93649844,"statfs":{"total":96636764160,"available":95897440256,"internally_reserved":0,"allocated":37285888,"data_stored":37075307,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":702017235},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,3,4,1,5,5,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986304,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":596652,"kb_used_data":37400,"kb_used_omap":4,"kb_used_meta":559227,"kb_avail":93775188,"statfs":{"total":96636764160,"available":96025792512,"internally_reserved":0,"allocated":38297600,"data_stored":38087019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":572649170},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,0,2,2,5,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673111,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":710880,"kb_used_data":34128,"kb_used_omap":1,"kb_used_meta":676734,"kb_avail":93660960,"statfs":{"total":96636764160,"available":95908823040,"internally_reserved":0,"allocated":34947072,"data_stored":34736491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":692976069},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182542,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":710684,"kb_used_data":33480,"kb_used_omap":4,"kb_used_meta":677179,"kb_avail":93661156,"statfs":{"total":96636764160,"available":95909023744,"internally_reserved":0,"allocated":34283520,"data_stored":34072939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":693432013},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2,4,5,2,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":33636352,"data_stored":33636352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":37265408,"data_stored":37265408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":33959936,"data_stored":33959936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":36786176,"data_stored":36786176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:45:42.136 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-05T16:45:42.136 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "39'12364", 'reported_seq': 14265, 'reported_epoch': 39, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:37.160954+0000', 'last_change': '2024-09-05T16:43:43.436935+0000', 'last_active': '2024-09-05T16:45:37.160954+0000', 'last_peered': '2024-09-05T16:45:37.160954+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.828289+0000', 'last_became_peered': '2024-09-05T16:34:56.828289+0000', 'last_unstale': '2024-09-05T16:45:37.160954+0000', 'last_undegraded': '2024-09-05T16:34:56.612160+0000', 'last_fullsized': '2024-09-05T16:34:56.557632+0000', 'mapping_epoch': 32, 'log_start': "33'2300", 'ondisk_log_start': "33'2300", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10064, 'log_dups_size': 0, 'ondisk_log_size': 10064, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4423680, 'num_objects': 540, 'num_object_clones': 0, 'num_object_copies': 2160, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1341, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 540, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12364, 'num_write_kb': 51616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 921, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -801}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:45:42.137 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:45:42.137 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "39'12231", 'reported_seq': 12359, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:40.387074+0000', 'last_change': '2024-09-05T16:34:56.867900+0000', 'last_active': '2024-09-05T16:45:40.387074+0000', 'last_peered': '2024-09-05T16:45:40.387074+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:45:40.387074+0000', 'last_undegraded': '2024-09-05T16:34:56.590972+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 32, 'log_start': "29'2200", 'ondisk_log_start': "29'2200", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10031, 'log_dups_size': 0, 'ondisk_log_size': 10031, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4266600, 'num_objects': 522, 'num_object_clones': 0, 'num_object_copies': 2088, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2233, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 522, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12230, 'num_write_kb': 50999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1710}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:45:42.137 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:45:42.137 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "39'12056", 'reported_seq': 12160, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:37.131612+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:45:37.131612+0000', 'last_peered': '2024-09-05T16:45:37.131612+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:45:37.131612+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "29'2000", 'ondisk_log_start': "29'2000", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10056, 'log_dups_size': 0, 'ondisk_log_size': 10056, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4412576, 'num_objects': 539, 'num_object_clones': 0, 'num_object_copies': 2156, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2147, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 539, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12054, 'num_write_kb': 50373, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1606}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:45:42.137 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:45:42.137 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "39'12194", 'reported_seq': 12314, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:37.016846+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:45:37.016846+0000', 'last_peered': '2024-09-05T16:45:37.016846+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:45:37.016846+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "29'2100", 'ondisk_log_start': "29'2100", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10094, 'log_dups_size': 0, 'ondisk_log_size': 10094, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4734976, 'num_objects': 578, 'num_object_clones': 0, 'num_object_copies': 2312, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 578, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12194, 'num_write_kb': 51088, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1641}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:45:42.137 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:45:42.137 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "39'12096", 'reported_seq': 12209, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:40.667158+0000', 'last_change': '2024-09-05T16:34:56.903436+0000', 'last_active': '2024-09-05T16:45:40.667158+0000', 'last_peered': '2024-09-05T16:45:40.667158+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:45:40.667158+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "29'2000", 'ondisk_log_start': "29'2000", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10096, 'log_dups_size': 0, 'ondisk_log_size': 10096, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4633760, 'num_objects': 566, 'num_object_clones': 0, 'num_object_copies': 2264, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2133, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 566, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12094, 'num_write_kb': 50636, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -1565}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:45:42.137 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-05T16:45:42.137 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "39'12246", 'reported_seq': 12338, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:40.854903+0000', 'last_change': '2024-09-05T16:34:56.883319+0000', 'last_active': '2024-09-05T16:45:40.854903+0000', 'last_peered': '2024-09-05T16:45:40.854903+0000', 'last_clean': '2024-09-05T16:33:53.505305+0000', 'last_became_active': '2024-09-05T16:34:56.873389+0000', 'last_became_peered': '2024-09-05T16:34:56.873389+0000', 'last_unstale': '2024-09-05T16:45:40.854903+0000', 'last_undegraded': '2024-09-05T16:34:56.602567+0000', 'last_fullsized': '2024-09-05T16:34:56.559958+0000', 'mapping_epoch': 32, 'log_start': "30'2200", 'ondisk_log_start': "30'2200", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10046, 'log_dups_size': 0, 'ondisk_log_size': 10046, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4363424, 'num_objects': 533, 'num_object_clones': 0, 'num_object_copies': 2132, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2234, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 533, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12244, 'num_write_kb': 51108, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1699}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:45:42.138 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:45:42.138 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "39'12263", 'reported_seq': 12369, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:37.118111+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:45:37.118111+0000', 'last_peered': '2024-09-05T16:45:37.118111+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:45:37.118111+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "29'2200", 'ondisk_log_start': "29'2200", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10063, 'log_dups_size': 0, 'ondisk_log_size': 10063, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4180288, 'num_objects': 511, 'num_object_clones': 0, 'num_object_copies': 2044, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 511, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12260, 'num_write_kb': 51081, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1733}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:45:42.138 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:45:42.138 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "39'12299", 'reported_seq': 12424, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:40.442235+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:45:40.442235+0000', 'last_peered': '2024-09-05T16:45:40.442235+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:45:40.442235+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "29'2200", 'ondisk_log_start': "29'2200", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10099, 'log_dups_size': 0, 'ondisk_log_size': 10099, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4183200, 'num_objects': 511, 'num_object_clones': 0, 'num_object_copies': 2044, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 511, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12297, 'num_write_kb': 51230, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1734}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:45:42.138 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:45:42.138 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "39'12534", 'reported_seq': 12666, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:37.168294+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:45:37.168294+0000', 'last_peered': '2024-09-05T16:45:37.168294+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:45:37.168294+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "33'2500", 'ondisk_log_start': "33'2500", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10034, 'log_dups_size': 0, 'ondisk_log_size': 10034, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4820720, 'num_objects': 589, 'num_object_clones': 0, 'num_object_copies': 2356, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2284, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 589, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12531, 'num_write_kb': 52476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1692}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:45:42.138 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:45:42.138 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "39'12178", 'reported_seq': 12282, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:37.025679+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:45:37.025679+0000', 'last_peered': '2024-09-05T16:45:37.025679+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:45:37.025679+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "29'2100", 'ondisk_log_start': "29'2100", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10078, 'log_dups_size': 0, 'ondisk_log_size': 10078, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4758096, 'num_objects': 581, 'num_object_clones': 0, 'num_object_copies': 2324, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 581, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12177, 'num_write_kb': 51031, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1657}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:45:42.138 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:45:42.138 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "39'12523", 'reported_seq': 12637, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:37.070949+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:45:37.070949+0000', 'last_peered': '2024-09-05T16:45:37.070949+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:45:37.070949+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "33'2500", 'ondisk_log_start': "33'2500", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10023, 'log_dups_size': 0, 'ondisk_log_size': 10023, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4936864, 'num_objects': 603, 'num_object_clones': 0, 'num_object_copies': 2412, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 603, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12521, 'num_write_kb': 52500, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1731}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:45:42.138 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:45:42.138 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "39'12322", 'reported_seq': 12440, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:40.169933+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:45:40.169933+0000', 'last_peered': '2024-09-05T16:45:40.169933+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:45:40.169933+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "30'2300", 'ondisk_log_start': "30'2300", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10022, 'log_dups_size': 0, 'ondisk_log_size': 10022, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4183200, 'num_objects': 511, 'num_object_clones': 0, 'num_object_copies': 2044, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2310, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 511, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12320, 'num_write_kb': 51325, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1797}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:45:42.139 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:45:42.139 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "39'12248", 'reported_seq': 12362, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:40.940992+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:45:40.940992+0000', 'last_peered': '2024-09-05T16:45:40.940992+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:45:40.940992+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "29'2200", 'ondisk_log_start': "29'2200", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10048, 'log_dups_size': 0, 'ondisk_log_size': 10048, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4314272, 'num_objects': 527, 'num_object_clones': 0, 'num_object_copies': 2108, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 527, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12246, 'num_write_kb': 51092, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1693}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:45:42.140 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:45:42.895 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:45:42.895 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 28192/34884 objects degraded (80.816%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":746,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":39,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":8721,"data_bytes":71400776,"bytes_used":2805977088,"bytes_avail":383741079552,"bytes_total":386547056640,"degraded_objects":28192,"degraded_total":34884,"degraded_ratio":0.80816420135305589,"recovering_objects_per_sec":6,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":202},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:45:42.919 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:45:42.919 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:45:43.504 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:45:43.504 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:45:43.532 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":403,"stamp":"2024-09-05T16:45:43.079732+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":66207048,"num_objects":8087,"num_object_clones":0,"num_object_copies":32348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8087,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":172063,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1217,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160772,"ondisk_log_size":160772,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2740588,"kb_used_data":133460,"kb_used_omap":14,"kb_used_meta":2607025,"kb_avail":374746772,"statfs":{"total":386547056640,"available":383740694528,"internally_reserved":0,"allocated":136663040,"data_stored":135820716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2669594167},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,4,0,7,15,14,9,7],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21905408,"num_objects":-2674,"num_object_clones":0,"num_object_copies":-10696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-122,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2674,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2674,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":122,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004754"},"pg_stats":[{"pgid":"1.f","version":"39'12425","reported_seq":14412,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:42.285156+0000","last_change":"2024-09-05T16:43:43.436935+0000","last_active":"2024-09-05T16:45:42.285156+0000","last_peered":"2024-09-05T16:45:42.285156+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:45:42.285156+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"33'2400","ondisk_log_start":"33'2400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3923968,"num_objects":479,"num_object_clones":0,"num_object_copies":1916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1298,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":479,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12425,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":964,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-819}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12092","reported_seq":16748,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:45:41.646134+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:45:41.646134+0000","last_peered":"2024-09-05T16:45:41.646134+0000","last_clean":"2024-09-05T16:45:41.646134+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:45:41.646134+0000","last_undegraded":"2024-09-05T16:45:41.646134+0000","last_fullsized":"2024-09-05T16:45:41.646134+0000","mapping_epoch":38,"log_start":"29'2000","ondisk_log_start":"29'2000","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3768320,"num_objects":460,"num_object_clones":0,"num_object_copies":1840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":460,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1405,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12231","reported_seq":12359,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:40.387074+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:45:40.387074+0000","last_peered":"2024-09-05T16:45:40.387074+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:45:40.387074+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"29'2200","ondisk_log_start":"29'2200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4266600,"num_objects":522,"num_object_clones":0,"num_object_copies":2088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":522,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12230,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1710}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12463","reported_seq":17198,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:45:41.522949+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:45:41.522949+0000","last_peered":"2024-09-05T16:45:41.522949+0000","last_clean":"2024-09-05T16:45:41.522949+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:45:41.522949+0000","last_undegraded":"2024-09-05T16:45:41.522949+0000","last_fullsized":"2024-09-05T16:45:41.522949+0000","mapping_epoch":34,"log_start":"33'2400","ondisk_log_start":"33'2400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3858432,"num_objects":471,"num_object_clones":0,"num_object_copies":1884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":471,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6763,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12109","reported_seq":12213,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:42.098106+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:45:42.098106+0000","last_peered":"2024-09-05T16:45:42.098106+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:45:42.098106+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"29'2100","ondisk_log_start":"29'2100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3978400,"num_objects":486,"num_object_clones":0,"num_object_copies":1944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":486,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12107,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1659}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12262","reported_seq":12380,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:42.325137+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:45:42.325137+0000","last_peered":"2024-09-05T16:45:42.325137+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:45:42.325137+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"29'2200","ondisk_log_start":"29'2200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4177920,"num_objects":510,"num_object_clones":0,"num_object_copies":2040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":510,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12262,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1709}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12096","reported_seq":12209,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:40.667158+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:45:40.667158+0000","last_peered":"2024-09-05T16:45:40.667158+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:45:40.667158+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"29'2000","ondisk_log_start":"29'2000","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4633760,"num_objects":566,"num_object_clones":0,"num_object_copies":2264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":566,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12094,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1565}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12246","reported_seq":12338,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:40.854903+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:45:40.854903+0000","last_peered":"2024-09-05T16:45:40.854903+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:45:40.854903+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"30'2200","ondisk_log_start":"30'2200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4363424,"num_objects":533,"num_object_clones":0,"num_object_copies":2132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12244,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1699}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"39'12330","reported_seq":12436,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:42.251591+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:45:42.251591+0000","last_peered":"2024-09-05T16:45:42.251591+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:45:42.251591+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"33'2300","ondisk_log_start":"33'2300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3631424,"num_objects":444,"num_object_clones":0,"num_object_copies":1776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":444,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12327,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1800}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12299","reported_seq":12424,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:40.442235+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:45:40.442235+0000","last_peered":"2024-09-05T16:45:40.442235+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:45:40.442235+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"29'2200","ondisk_log_start":"29'2200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4183200,"num_objects":511,"num_object_clones":0,"num_object_copies":2044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":511,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12297,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1734}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'12599","reported_seq":12731,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:42.080874+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:45:42.080874+0000","last_peered":"2024-09-05T16:45:42.080874+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:45:42.080874+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4288240,"num_objects":524,"num_object_clones":0,"num_object_copies":2096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":524,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12596,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1757}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12237","reported_seq":12341,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:42.192615+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:45:42.192615+0000","last_peered":"2024-09-05T16:45:42.192615+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:45:42.192615+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"29'2200","ondisk_log_start":"29'2200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4274768,"num_objects":522,"num_object_clones":0,"num_object_copies":2088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":522,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12236,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1716}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12304","reported_seq":17045,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:45:41.699893+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:45:41.699893+0000","last_peered":"2024-09-05T16:45:41.699893+0000","last_clean":"2024-09-05T16:45:41.699893+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:45:41.699893+0000","last_undegraded":"2024-09-05T16:45:41.699893+0000","last_fullsized":"2024-09-05T16:45:41.699893+0000","mapping_epoch":36,"log_start":"33'2300","ondisk_log_start":"33'2300","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4128768,"num_objects":504,"num_object_clones":0,"num_object_copies":2016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":504,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3904,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'12609","reported_seq":12723,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:42.106891+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:45:42.106891+0000","last_peered":"2024-09-05T16:45:42.106891+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:45:42.106891+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4232352,"num_objects":517,"num_object_clones":0,"num_object_copies":2068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":517,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12607,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1817}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12322","reported_seq":12440,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:40.169933+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:45:40.169933+0000","last_peered":"2024-09-05T16:45:40.169933+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:45:40.169933+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"30'2300","ondisk_log_start":"30'2300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4183200,"num_objects":511,"num_object_clones":0,"num_object_copies":2044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":511,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12320,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1797}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'12248","reported_seq":12362,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:40.940992+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:45:40.940992+0000","last_peered":"2024-09-05T16:45:40.940992+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:45:40.940992+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"29'2200","ondisk_log_start":"29'2200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4314272,"num_objects":527,"num_object_clones":0,"num_object_copies":2108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":527,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12246,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1693}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":66207048,"num_objects":8087,"num_object_clones":0,"num_object_copies":32348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8087,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":172063,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1217,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":133337088,"data_stored":133337088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160772,"ondisk_log_size":160772,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084428,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722184,"kb_used_data":32440,"kb_used_omap":4,"kb_used_meta":689723,"kb_avail":93649656,"statfs":{"total":96636764160,"available":95897247744,"internally_reserved":0,"allocated":33218560,"data_stored":33007979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":706277075},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,0,3,4,5,3,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986305,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":596840,"kb_used_data":33412,"kb_used_omap":4,"kb_used_meta":563387,"kb_avail":93775000,"statfs":{"total":96636764160,"available":96025600000,"internally_reserved":0,"allocated":34213888,"data_stored":34003307,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":576909010},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,2,4,3,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673111,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":710880,"kb_used_data":34128,"kb_used_omap":1,"kb_used_meta":676734,"kb_avail":93660960,"statfs":{"total":96636764160,"available":95908823040,"internally_reserved":0,"allocated":34947072,"data_stored":34736491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":692976069},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182542,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":710684,"kb_used_data":33480,"kb_used_omap":4,"kb_used_meta":677179,"kb_avail":93661156,"statfs":{"total":96636764160,"available":95909023744,"internally_reserved":0,"allocated":34283520,"data_stored":34072939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":693432013},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2,4,5,2,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":33636352,"data_stored":33636352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":33116160,"data_stored":33116160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":33959936,"data_stored":33959936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32624640,"data_stored":32624640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:45:46.534 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:45:47.146 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:45:47.146 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:45:47.169 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":405,"stamp":"2024-09-05T16:45:47.081292+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":61816136,"num_objects":7551,"num_object_clones":0,"num_object_copies":30204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7551,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":172599,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1217,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160708,"ondisk_log_size":160708,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2724272,"kb_used_data":121328,"kb_used_omap":14,"kb_used_meta":2602865,"kb_avail":374763088,"statfs":{"total":386547056640,"available":383757402112,"internally_reserved":0,"allocated":124239872,"data_stored":123397548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2665334327},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,4,4,14,16,15,9,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21258240,"num_objects":-2595,"num_object_clones":0,"num_object_copies":-10380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-79,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2595,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2595,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004732"},"pg_stats":[{"pgid":"1.f","version":"39'12425","reported_seq":14412,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:42.285156+0000","last_change":"2024-09-05T16:43:43.436935+0000","last_active":"2024-09-05T16:45:42.285156+0000","last_peered":"2024-09-05T16:45:42.285156+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:45:42.285156+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"33'2400","ondisk_log_start":"33'2400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3923968,"num_objects":479,"num_object_clones":0,"num_object_copies":1916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1298,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":479,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12425,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":964,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-819}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12151","reported_seq":16807,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:45:46.758393+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:45:46.758393+0000","last_peered":"2024-09-05T16:45:46.758393+0000","last_clean":"2024-09-05T16:45:46.758393+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:45:46.758393+0000","last_undegraded":"2024-09-05T16:45:46.758393+0000","last_fullsized":"2024-09-05T16:45:46.758393+0000","mapping_epoch":38,"log_start":"29'2100","ondisk_log_start":"29'2100","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3284992,"num_objects":401,"num_object_clones":0,"num_object_copies":1604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1464,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12310","reported_seq":12438,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:45.466478+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:45:45.466478+0000","last_peered":"2024-09-05T16:45:45.466478+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:45:45.466478+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"33'2300","ondisk_log_start":"33'2300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3619432,"num_objects":443,"num_object_clones":0,"num_object_copies":1772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":443,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12309,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1789}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12529","reported_seq":17264,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:45:46.869419+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:45:46.869419+0000","last_peered":"2024-09-05T16:45:46.869419+0000","last_clean":"2024-09-05T16:45:46.869419+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:45:46.869419+0000","last_undegraded":"2024-09-05T16:45:46.869419+0000","last_fullsized":"2024-09-05T16:45:46.869419+0000","mapping_epoch":34,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3317760,"num_objects":405,"num_object_clones":0,"num_object_copies":1620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":405,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6829,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12109","reported_seq":12213,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:42.098106+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:45:42.098106+0000","last_peered":"2024-09-05T16:45:42.098106+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:45:42.098106+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"29'2100","ondisk_log_start":"29'2100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3978400,"num_objects":486,"num_object_clones":0,"num_object_copies":1944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":486,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12107,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1659}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12262","reported_seq":12380,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:42.325137+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:45:42.325137+0000","last_peered":"2024-09-05T16:45:42.325137+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:45:42.325137+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"29'2200","ondisk_log_start":"29'2200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4177920,"num_objects":510,"num_object_clones":0,"num_object_copies":2040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":510,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12262,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1709}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12156","reported_seq":12269,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:45.927147+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:45:45.927147+0000","last_peered":"2024-09-05T16:45:45.927147+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:45:45.927147+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"29'2100","ondisk_log_start":"29'2100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4142240,"num_objects":506,"num_object_clones":0,"num_object_copies":2024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":506,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12154,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1625}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12289","reported_seq":12383,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:45.909664+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:45:45.909664+0000","last_peered":"2024-09-05T16:45:45.909664+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:45:45.909664+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"30'2200","ondisk_log_start":"30'2200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4011168,"num_objects":490,"num_object_clones":0,"num_object_copies":1960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12287,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1742}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"39'12330","reported_seq":12436,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:42.251591+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:45:42.251591+0000","last_peered":"2024-09-05T16:45:42.251591+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:45:42.251591+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"33'2300","ondisk_log_start":"33'2300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3631424,"num_objects":444,"num_object_clones":0,"num_object_copies":1776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":444,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12327,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1800}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12351","reported_seq":12478,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:45.350232+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:45:45.350232+0000","last_peered":"2024-09-05T16:45:45.350232+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:45:45.350232+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"33'2300","ondisk_log_start":"33'2300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3757216,"num_objects":459,"num_object_clones":0,"num_object_copies":1836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":459,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12349,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1786}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'12599","reported_seq":12731,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:42.080874+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:45:42.080874+0000","last_peered":"2024-09-05T16:45:42.080874+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:45:42.080874+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4288240,"num_objects":524,"num_object_clones":0,"num_object_copies":2096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":524,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12596,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1757}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12237","reported_seq":12341,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:42.192615+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:45:42.192615+0000","last_peered":"2024-09-05T16:45:42.192615+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:45:42.192615+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"29'2200","ondisk_log_start":"29'2200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4274768,"num_objects":522,"num_object_clones":0,"num_object_copies":2088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":522,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12236,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1716}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12362","reported_seq":17103,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:45:46.468687+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:45:46.468687+0000","last_peered":"2024-09-05T16:45:46.468687+0000","last_clean":"2024-09-05T16:45:46.468687+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:45:46.468687+0000","last_undegraded":"2024-09-05T16:45:46.468687+0000","last_fullsized":"2024-09-05T16:45:46.468687+0000","mapping_epoch":36,"log_start":"33'2300","ondisk_log_start":"33'2300","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3653632,"num_objects":446,"num_object_clones":0,"num_object_copies":1784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":446,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3962,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'12609","reported_seq":12723,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:42.106891+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:45:42.106891+0000","last_peered":"2024-09-05T16:45:42.106891+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:45:42.106891+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4232352,"num_objects":517,"num_object_clones":0,"num_object_copies":2068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":517,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12607,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1817}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12386","reported_seq":12504,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:45.555369+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:45:45.555369+0000","last_peered":"2024-09-05T16:45:45.555369+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:45:45.555369+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"30'2300","ondisk_log_start":"30'2300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3658912,"num_objects":447,"num_object_clones":0,"num_object_copies":1788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":447,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12384,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1861}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'12303","reported_seq":12417,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:45.838882+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:45:45.838882+0000","last_peered":"2024-09-05T16:45:45.838882+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:45:45.838882+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"33'2300","ondisk_log_start":"33'2300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3863712,"num_objects":472,"num_object_clones":0,"num_object_copies":1888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":472,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12301,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1748}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":61816136,"num_objects":7551,"num_object_clones":0,"num_object_copies":30204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7551,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":172599,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1217,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":120827904,"data_stored":120827904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160708,"ondisk_log_size":160708,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084428,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722184,"kb_used_data":32440,"kb_used_omap":4,"kb_used_meta":689723,"kb_avail":93649656,"statfs":{"total":96636764160,"available":95897247744,"internally_reserved":0,"allocated":33218560,"data_stored":33007979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":706277075},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,0,3,4,5,3,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986306,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":580296,"kb_used_data":29368,"kb_used_omap":4,"kb_used_meta":550907,"kb_avail":93791544,"statfs":{"total":96636764160,"available":96042541056,"internally_reserved":0,"allocated":30072832,"data_stored":29862251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":564129490},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,6,2,0,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673112,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":711040,"kb_used_data":30128,"kb_used_omap":1,"kb_used_meta":680894,"kb_avail":93660800,"statfs":{"total":96636764160,"available":95908659200,"internally_reserved":0,"allocated":30851072,"data_stored":30640491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":697235909},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,1,2,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182543,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":710752,"kb_used_data":29392,"kb_used_omap":4,"kb_used_meta":681339,"kb_avail":93661088,"statfs":{"total":96636764160,"available":95908954112,"internally_reserved":0,"allocated":30097408,"data_stored":29886827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":697691853},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,2,9,8,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":29478912,"data_stored":29478912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28872704,"data_stored":28872704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":29851648,"data_stored":29851648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32624640,"data_stored":32624640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:45:47.171 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:45:47.801 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:45:47.801 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:45:47.833 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":405,"stamp":"2024-09-05T16:45:47.081292+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":61816136,"num_objects":7551,"num_object_clones":0,"num_object_copies":30204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7551,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":172599,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1217,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160708,"ondisk_log_size":160708,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2724272,"kb_used_data":121328,"kb_used_omap":14,"kb_used_meta":2602865,"kb_avail":374763088,"statfs":{"total":386547056640,"available":383757402112,"internally_reserved":0,"allocated":124239872,"data_stored":123397548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2665334327},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,4,4,14,16,15,9,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21258240,"num_objects":-2595,"num_object_clones":0,"num_object_copies":-10380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-79,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2595,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2595,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004732"},"pg_stats":[{"pgid":"1.f","version":"39'12425","reported_seq":14412,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:42.285156+0000","last_change":"2024-09-05T16:43:43.436935+0000","last_active":"2024-09-05T16:45:42.285156+0000","last_peered":"2024-09-05T16:45:42.285156+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:45:42.285156+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"33'2400","ondisk_log_start":"33'2400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3923968,"num_objects":479,"num_object_clones":0,"num_object_copies":1916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1298,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":479,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12425,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":964,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-819}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12151","reported_seq":16807,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:45:46.758393+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:45:46.758393+0000","last_peered":"2024-09-05T16:45:46.758393+0000","last_clean":"2024-09-05T16:45:46.758393+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:45:46.758393+0000","last_undegraded":"2024-09-05T16:45:46.758393+0000","last_fullsized":"2024-09-05T16:45:46.758393+0000","mapping_epoch":38,"log_start":"29'2100","ondisk_log_start":"29'2100","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3284992,"num_objects":401,"num_object_clones":0,"num_object_copies":1604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1464,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12310","reported_seq":12438,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:45.466478+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:45:45.466478+0000","last_peered":"2024-09-05T16:45:45.466478+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:45:45.466478+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"33'2300","ondisk_log_start":"33'2300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3619432,"num_objects":443,"num_object_clones":0,"num_object_copies":1772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":443,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12309,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1789}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12529","reported_seq":17264,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:45:46.869419+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:45:46.869419+0000","last_peered":"2024-09-05T16:45:46.869419+0000","last_clean":"2024-09-05T16:45:46.869419+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:45:46.869419+0000","last_undegraded":"2024-09-05T16:45:46.869419+0000","last_fullsized":"2024-09-05T16:45:46.869419+0000","mapping_epoch":34,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3317760,"num_objects":405,"num_object_clones":0,"num_object_copies":1620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":405,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6829,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12109","reported_seq":12213,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:42.098106+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:45:42.098106+0000","last_peered":"2024-09-05T16:45:42.098106+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:45:42.098106+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"29'2100","ondisk_log_start":"29'2100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3978400,"num_objects":486,"num_object_clones":0,"num_object_copies":1944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":486,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12107,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1659}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12262","reported_seq":12380,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:42.325137+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:45:42.325137+0000","last_peered":"2024-09-05T16:45:42.325137+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:45:42.325137+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"29'2200","ondisk_log_start":"29'2200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4177920,"num_objects":510,"num_object_clones":0,"num_object_copies":2040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":510,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12262,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1709}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12156","reported_seq":12269,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:45.927147+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:45:45.927147+0000","last_peered":"2024-09-05T16:45:45.927147+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:45:45.927147+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"29'2100","ondisk_log_start":"29'2100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4142240,"num_objects":506,"num_object_clones":0,"num_object_copies":2024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":506,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12154,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1625}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12289","reported_seq":12383,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:45.909664+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:45:45.909664+0000","last_peered":"2024-09-05T16:45:45.909664+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:45:45.909664+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"30'2200","ondisk_log_start":"30'2200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4011168,"num_objects":490,"num_object_clones":0,"num_object_copies":1960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12287,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1742}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"39'12330","reported_seq":12436,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:42.251591+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:45:42.251591+0000","last_peered":"2024-09-05T16:45:42.251591+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:45:42.251591+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"33'2300","ondisk_log_start":"33'2300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3631424,"num_objects":444,"num_object_clones":0,"num_object_copies":1776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":444,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12327,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1800}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12351","reported_seq":12478,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:45.350232+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:45:45.350232+0000","last_peered":"2024-09-05T16:45:45.350232+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:45:45.350232+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"33'2300","ondisk_log_start":"33'2300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3757216,"num_objects":459,"num_object_clones":0,"num_object_copies":1836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":459,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12349,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1786}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'12599","reported_seq":12731,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:42.080874+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:45:42.080874+0000","last_peered":"2024-09-05T16:45:42.080874+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:45:42.080874+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4288240,"num_objects":524,"num_object_clones":0,"num_object_copies":2096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":524,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12596,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1757}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12237","reported_seq":12341,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:42.192615+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:45:42.192615+0000","last_peered":"2024-09-05T16:45:42.192615+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:45:42.192615+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"29'2200","ondisk_log_start":"29'2200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4274768,"num_objects":522,"num_object_clones":0,"num_object_copies":2088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":522,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12236,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1716}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12362","reported_seq":17103,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:45:46.468687+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:45:46.468687+0000","last_peered":"2024-09-05T16:45:46.468687+0000","last_clean":"2024-09-05T16:45:46.468687+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:45:46.468687+0000","last_undegraded":"2024-09-05T16:45:46.468687+0000","last_fullsized":"2024-09-05T16:45:46.468687+0000","mapping_epoch":36,"log_start":"33'2300","ondisk_log_start":"33'2300","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3653632,"num_objects":446,"num_object_clones":0,"num_object_copies":1784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":446,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3962,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'12609","reported_seq":12723,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:42.106891+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:45:42.106891+0000","last_peered":"2024-09-05T16:45:42.106891+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:45:42.106891+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4232352,"num_objects":517,"num_object_clones":0,"num_object_copies":2068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":517,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12607,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1817}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12386","reported_seq":12504,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:45.555369+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:45:45.555369+0000","last_peered":"2024-09-05T16:45:45.555369+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:45:45.555369+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"30'2300","ondisk_log_start":"30'2300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3658912,"num_objects":447,"num_object_clones":0,"num_object_copies":1788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":447,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12384,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1861}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'12303","reported_seq":12417,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:45.838882+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:45:45.838882+0000","last_peered":"2024-09-05T16:45:45.838882+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:45:45.838882+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"33'2300","ondisk_log_start":"33'2300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3863712,"num_objects":472,"num_object_clones":0,"num_object_copies":1888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":472,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12301,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1748}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":61816136,"num_objects":7551,"num_object_clones":0,"num_object_copies":30204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7551,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":172599,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1217,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":120827904,"data_stored":120827904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160708,"ondisk_log_size":160708,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084428,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722184,"kb_used_data":32440,"kb_used_omap":4,"kb_used_meta":689723,"kb_avail":93649656,"statfs":{"total":96636764160,"available":95897247744,"internally_reserved":0,"allocated":33218560,"data_stored":33007979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":706277075},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,0,3,4,5,3,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986306,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":580296,"kb_used_data":29368,"kb_used_omap":4,"kb_used_meta":550907,"kb_avail":93791544,"statfs":{"total":96636764160,"available":96042541056,"internally_reserved":0,"allocated":30072832,"data_stored":29862251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":564129490},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,6,2,0,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673112,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":711040,"kb_used_data":30128,"kb_used_omap":1,"kb_used_meta":680894,"kb_avail":93660800,"statfs":{"total":96636764160,"available":95908659200,"internally_reserved":0,"allocated":30851072,"data_stored":30640491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":697235909},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,1,2,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182543,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":710752,"kb_used_data":29392,"kb_used_omap":4,"kb_used_meta":681339,"kb_avail":93661088,"statfs":{"total":96636764160,"available":95908954112,"internally_reserved":0,"allocated":30097408,"data_stored":29886827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":697691853},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,2,9,8,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":29478912,"data_stored":29478912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28872704,"data_stored":28872704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":29851648,"data_stored":29851648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32624640,"data_stored":32624640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:45:47.834 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-05T16:45:47.834 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "39'12425", 'reported_seq': 14412, 'reported_epoch': 39, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:42.285156+0000', 'last_change': '2024-09-05T16:43:43.436935+0000', 'last_active': '2024-09-05T16:45:42.285156+0000', 'last_peered': '2024-09-05T16:45:42.285156+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.828289+0000', 'last_became_peered': '2024-09-05T16:34:56.828289+0000', 'last_unstale': '2024-09-05T16:45:42.285156+0000', 'last_undegraded': '2024-09-05T16:34:56.612160+0000', 'last_fullsized': '2024-09-05T16:34:56.557632+0000', 'mapping_epoch': 32, 'log_start': "33'2400", 'ondisk_log_start': "33'2400", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10025, 'log_dups_size': 0, 'ondisk_log_size': 10025, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3923968, 'num_objects': 479, 'num_object_clones': 0, 'num_object_copies': 1916, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1298, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 479, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12425, 'num_write_kb': 51616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 964, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -819}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:45:47.834 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:45:47.835 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "39'12310", 'reported_seq': 12438, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:45.466478+0000', 'last_change': '2024-09-05T16:34:56.867900+0000', 'last_active': '2024-09-05T16:45:45.466478+0000', 'last_peered': '2024-09-05T16:45:45.466478+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:45:45.466478+0000', 'last_undegraded': '2024-09-05T16:34:56.590972+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 32, 'log_start': "33'2300", 'ondisk_log_start': "33'2300", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10010, 'log_dups_size': 0, 'ondisk_log_size': 10010, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3619432, 'num_objects': 443, 'num_object_clones': 0, 'num_object_copies': 1772, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2233, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 443, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12309, 'num_write_kb': 50999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1789}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:45:47.835 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:45:47.835 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "39'12109", 'reported_seq': 12213, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:42.098106+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:45:42.098106+0000', 'last_peered': '2024-09-05T16:45:42.098106+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:45:42.098106+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "29'2100", 'ondisk_log_start': "29'2100", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10009, 'log_dups_size': 0, 'ondisk_log_size': 10009, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3978400, 'num_objects': 486, 'num_object_clones': 0, 'num_object_copies': 1944, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2147, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 486, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12107, 'num_write_kb': 50373, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1659}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:45:47.835 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:45:47.835 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "39'12262", 'reported_seq': 12380, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:42.325137+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:45:42.325137+0000', 'last_peered': '2024-09-05T16:45:42.325137+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:45:42.325137+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "29'2200", 'ondisk_log_start': "29'2200", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10062, 'log_dups_size': 0, 'ondisk_log_size': 10062, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4177920, 'num_objects': 510, 'num_object_clones': 0, 'num_object_copies': 2040, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 510, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12262, 'num_write_kb': 51088, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1709}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:45:47.835 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:45:47.835 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "39'12156", 'reported_seq': 12269, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:45.927147+0000', 'last_change': '2024-09-05T16:34:56.903436+0000', 'last_active': '2024-09-05T16:45:45.927147+0000', 'last_peered': '2024-09-05T16:45:45.927147+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:45:45.927147+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "29'2100", 'ondisk_log_start': "29'2100", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10056, 'log_dups_size': 0, 'ondisk_log_size': 10056, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4142240, 'num_objects': 506, 'num_object_clones': 0, 'num_object_copies': 2024, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2133, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 506, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12154, 'num_write_kb': 50636, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -1625}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:45:47.835 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-05T16:45:47.835 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "39'12289", 'reported_seq': 12383, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:45.909664+0000', 'last_change': '2024-09-05T16:34:56.883319+0000', 'last_active': '2024-09-05T16:45:45.909664+0000', 'last_peered': '2024-09-05T16:45:45.909664+0000', 'last_clean': '2024-09-05T16:33:53.505305+0000', 'last_became_active': '2024-09-05T16:34:56.873389+0000', 'last_became_peered': '2024-09-05T16:34:56.873389+0000', 'last_unstale': '2024-09-05T16:45:45.909664+0000', 'last_undegraded': '2024-09-05T16:34:56.602567+0000', 'last_fullsized': '2024-09-05T16:34:56.559958+0000', 'mapping_epoch': 32, 'log_start': "30'2200", 'ondisk_log_start': "30'2200", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10089, 'log_dups_size': 0, 'ondisk_log_size': 10089, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4011168, 'num_objects': 490, 'num_object_clones': 0, 'num_object_copies': 1960, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2234, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 490, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12287, 'num_write_kb': 51108, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1742}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:45:47.836 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:45:47.836 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "39'12330", 'reported_seq': 12436, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:42.251591+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:45:42.251591+0000', 'last_peered': '2024-09-05T16:45:42.251591+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:45:42.251591+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "33'2300", 'ondisk_log_start': "33'2300", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3631424, 'num_objects': 444, 'num_object_clones': 0, 'num_object_copies': 1776, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 444, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12327, 'num_write_kb': 51081, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1800}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:45:47.836 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:45:47.836 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "39'12351", 'reported_seq': 12478, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:45.350232+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:45:45.350232+0000', 'last_peered': '2024-09-05T16:45:45.350232+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:45:45.350232+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "33'2300", 'ondisk_log_start': "33'2300", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10051, 'log_dups_size': 0, 'ondisk_log_size': 10051, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3757216, 'num_objects': 459, 'num_object_clones': 0, 'num_object_copies': 1836, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 459, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12349, 'num_write_kb': 51230, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1786}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:45:47.836 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:45:47.836 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "39'12599", 'reported_seq': 12731, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:42.080874+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:45:42.080874+0000', 'last_peered': '2024-09-05T16:45:42.080874+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:45:42.080874+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "33'2500", 'ondisk_log_start': "33'2500", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10099, 'log_dups_size': 0, 'ondisk_log_size': 10099, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4288240, 'num_objects': 524, 'num_object_clones': 0, 'num_object_copies': 2096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2284, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 524, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12596, 'num_write_kb': 52476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1757}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:45:47.836 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:45:47.836 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "39'12237", 'reported_seq': 12341, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:42.192615+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:45:42.192615+0000', 'last_peered': '2024-09-05T16:45:42.192615+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:45:42.192615+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "29'2200", 'ondisk_log_start': "29'2200", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10037, 'log_dups_size': 0, 'ondisk_log_size': 10037, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4274768, 'num_objects': 522, 'num_object_clones': 0, 'num_object_copies': 2088, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 522, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12236, 'num_write_kb': 51031, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1716}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:45:47.836 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:45:47.836 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "39'12609", 'reported_seq': 12723, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:42.106891+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:45:42.106891+0000', 'last_peered': '2024-09-05T16:45:42.106891+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:45:42.106891+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "33'2600", 'ondisk_log_start': "33'2600", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10009, 'log_dups_size': 0, 'ondisk_log_size': 10009, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4232352, 'num_objects': 517, 'num_object_clones': 0, 'num_object_copies': 2068, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 517, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12607, 'num_write_kb': 52500, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1817}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:45:47.837 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:45:47.837 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "39'12386", 'reported_seq': 12504, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:45.555369+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:45:45.555369+0000', 'last_peered': '2024-09-05T16:45:45.555369+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:45:45.555369+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "30'2300", 'ondisk_log_start': "30'2300", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10086, 'log_dups_size': 0, 'ondisk_log_size': 10086, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3658912, 'num_objects': 447, 'num_object_clones': 0, 'num_object_copies': 1788, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2310, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 447, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12384, 'num_write_kb': 51325, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1861}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:45:47.837 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:45:47.837 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "39'12303", 'reported_seq': 12417, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:45.838882+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:45:45.838882+0000', 'last_peered': '2024-09-05T16:45:45.838882+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:45:45.838882+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "33'2300", 'ondisk_log_start': "33'2300", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10003, 'log_dups_size': 0, 'ondisk_log_size': 10003, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3863712, 'num_objects': 472, 'num_object_clones': 0, 'num_object_copies': 1888, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 472, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12301, 'num_write_kb': 51092, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1748}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:45:47.837 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:45:48.592 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:45:48.593 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 28149/30204 objects degraded (93.196%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":751,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":39,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":7551,"data_bytes":61816136,"bytes_used":2789654528,"bytes_avail":383757402112,"bytes_total":386547056640,"degraded_objects":28149,"degraded_total":30204,"degraded_ratio":0.93196265395311884,"recovering_objects_per_sec":6,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":216},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:45:48.614 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:45:48.614 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:45:49.314 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:45:49.314 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:45:49.339 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":406,"stamp":"2024-09-05T16:45:49.081906+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":57908552,"num_objects":7074,"num_object_clones":0,"num_object_copies":28296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28113,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7074,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":173076,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1253,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160885,"ondisk_log_size":160885,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2725788,"kb_used_data":117276,"kb_used_omap":14,"kb_used_meta":2608433,"kb_avail":374761572,"statfs":{"total":386547056640,"available":383755849728,"internally_reserved":0,"allocated":120090624,"data_stored":119248300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2671035959},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,5,4,12,18,18,7,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20455424,"num_objects":-2497,"num_object_clones":0,"num_object_copies":-9988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2497,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2497,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":115,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004496"},"pg_stats":[{"pgid":"1.f","version":"39'12500","reported_seq":14559,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:47.407549+0000","last_change":"2024-09-05T16:43:43.436935+0000","last_active":"2024-09-05T16:45:47.407549+0000","last_peered":"2024-09-05T16:45:47.407549+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:45:47.407549+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"33'2400","ondisk_log_start":"33'2400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3309568,"num_objects":404,"num_object_clones":0,"num_object_copies":1616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":404,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12500,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1000,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-858}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12151","reported_seq":16807,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:45:46.758393+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:45:46.758393+0000","last_peered":"2024-09-05T16:45:46.758393+0000","last_clean":"2024-09-05T16:45:46.758393+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:45:46.758393+0000","last_undegraded":"2024-09-05T16:45:46.758393+0000","last_fullsized":"2024-09-05T16:45:46.758393+0000","mapping_epoch":38,"log_start":"29'2100","ondisk_log_start":"29'2100","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3284992,"num_objects":401,"num_object_clones":0,"num_object_copies":1604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1464,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12310","reported_seq":12438,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:45.466478+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:45:45.466478+0000","last_peered":"2024-09-05T16:45:45.466478+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:45:45.466478+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"33'2300","ondisk_log_start":"33'2300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3619432,"num_objects":443,"num_object_clones":0,"num_object_copies":1772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":443,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12309,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1789}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12529","reported_seq":17264,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:45:46.869419+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:45:46.869419+0000","last_peered":"2024-09-05T16:45:46.869419+0000","last_clean":"2024-09-05T16:45:46.869419+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:45:46.869419+0000","last_undegraded":"2024-09-05T16:45:46.869419+0000","last_fullsized":"2024-09-05T16:45:46.869419+0000","mapping_epoch":34,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3317760,"num_objects":405,"num_object_clones":0,"num_object_copies":1620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":405,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6829,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12172","reported_seq":12276,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:47.164395+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:45:47.164395+0000","last_peered":"2024-09-05T16:45:47.164395+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:45:47.164395+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"29'2100","ondisk_log_start":"29'2100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3462304,"num_objects":423,"num_object_clones":0,"num_object_copies":1692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":423,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12170,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1722}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12332","reported_seq":12451,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:47.401953+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:45:47.401953+0000","last_peered":"2024-09-05T16:45:47.401953+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:45:47.401953+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"33'2300","ondisk_log_start":"33'2300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3604480,"num_objects":440,"num_object_clones":0,"num_object_copies":1760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":440,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12332,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1779}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12156","reported_seq":12269,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:45.927147+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:45:45.927147+0000","last_peered":"2024-09-05T16:45:45.927147+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:45:45.927147+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"29'2100","ondisk_log_start":"29'2100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4142240,"num_objects":506,"num_object_clones":0,"num_object_copies":2024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":506,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12154,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1625}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12289","reported_seq":12383,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:45.909664+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:45:45.909664+0000","last_peered":"2024-09-05T16:45:45.909664+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:45:45.909664+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"30'2200","ondisk_log_start":"30'2200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4011168,"num_objects":490,"num_object_clones":0,"num_object_copies":1960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12287,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1742}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"39'12391","reported_seq":12497,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:47.400767+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:45:47.400767+0000","last_peered":"2024-09-05T16:45:47.400767+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:45:47.400767+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"33'2300","ondisk_log_start":"33'2300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3131712,"num_objects":383,"num_object_clones":0,"num_object_copies":1532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":383,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12388,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1861}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12351","reported_seq":12478,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:45.350232+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:45:45.350232+0000","last_peered":"2024-09-05T16:45:45.350232+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:45:45.350232+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"33'2300","ondisk_log_start":"33'2300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3757216,"num_objects":459,"num_object_clones":0,"num_object_copies":1836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":459,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12349,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1786}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'12657","reported_seq":12789,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:47.330303+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:45:47.330303+0000","last_peered":"2024-09-05T16:45:47.330303+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:45:47.330303+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3813104,"num_objects":466,"num_object_clones":0,"num_object_copies":1864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12654,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1815}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12321","reported_seq":12425,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:47.227464+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:45:47.227464+0000","last_peered":"2024-09-05T16:45:47.227464+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:45:47.227464+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"33'2300","ondisk_log_start":"33'2300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3586640,"num_objects":438,"num_object_clones":0,"num_object_copies":1752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":438,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12320,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1800}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12362","reported_seq":17103,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:45:46.468687+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:45:46.468687+0000","last_peered":"2024-09-05T16:45:46.468687+0000","last_clean":"2024-09-05T16:45:46.468687+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:45:46.468687+0000","last_undegraded":"2024-09-05T16:45:46.468687+0000","last_fullsized":"2024-09-05T16:45:46.468687+0000","mapping_epoch":36,"log_start":"33'2300","ondisk_log_start":"33'2300","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3653632,"num_objects":446,"num_object_clones":0,"num_object_copies":1784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":446,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3962,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'12675","reported_seq":12788,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:47.299073+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:45:47.299073+0000","last_peered":"2024-09-05T16:45:47.299073+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:45:47.299073+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3691680,"num_objects":451,"num_object_clones":0,"num_object_copies":1804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":451,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12673,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1883}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12386","reported_seq":12504,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:45.555369+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:45:45.555369+0000","last_peered":"2024-09-05T16:45:45.555369+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:45:45.555369+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"30'2300","ondisk_log_start":"30'2300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3658912,"num_objects":447,"num_object_clones":0,"num_object_copies":1788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":447,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12384,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1861}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'12303","reported_seq":12417,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:45.838882+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:45:45.838882+0000","last_peered":"2024-09-05T16:45:45.838882+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:45:45.838882+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"33'2300","ondisk_log_start":"33'2300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3863712,"num_objects":472,"num_object_clones":0,"num_object_copies":1888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":472,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12301,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1748}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":57908552,"num_objects":7074,"num_object_clones":0,"num_object_copies":28296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28113,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7074,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":173076,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1253,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":116572160,"data_stored":116572160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160885,"ondisk_log_size":160885,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084429,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":723700,"kb_used_data":28388,"kb_used_omap":4,"kb_used_meta":695291,"kb_avail":93648140,"statfs":{"total":96636764160,"available":95895695360,"internally_reserved":0,"allocated":29069312,"data_stored":28858731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":711978707},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,1,6,8,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986306,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":580296,"kb_used_data":29368,"kb_used_omap":4,"kb_used_meta":550907,"kb_avail":93791544,"statfs":{"total":96636764160,"available":96042541056,"internally_reserved":0,"allocated":30072832,"data_stored":29862251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":564129490},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,6,2,0,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673112,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":711040,"kb_used_data":30128,"kb_used_omap":1,"kb_used_meta":680894,"kb_avail":93660800,"statfs":{"total":96636764160,"available":95908659200,"internally_reserved":0,"allocated":30851072,"data_stored":30640491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":697235909},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,1,2,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182543,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":710752,"kb_used_data":29392,"kb_used_omap":4,"kb_used_meta":681339,"kb_avail":93661088,"statfs":{"total":96636764160,"available":95908954112,"internally_reserved":0,"allocated":30097408,"data_stored":29886827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":697691853},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,2,9,8,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":29478912,"data_stored":29478912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28872704,"data_stored":28872704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":29851648,"data_stored":29851648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28368896,"data_stored":28368896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:45:52.341 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:45:52.986 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:45:52.986 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:45:53.010 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":407,"stamp":"2024-09-05T16:45:51.082680+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":56393032,"num_objects":6889,"num_object_clones":0,"num_object_copies":27556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28113,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6889,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":173261,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1253,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160870,"ondisk_log_size":160870,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2744028,"kb_used_data":113308,"kb_used_omap":14,"kb_used_meta":2630641,"kb_avail":374743332,"statfs":{"total":386547056640,"available":383737171968,"internally_reserved":0,"allocated":116027392,"data_stored":115185068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2693776951},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,5,2,12,22,20,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18227200,"num_objects":-2225,"num_object_clones":0,"num_object_copies":-8900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-79,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2225,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2225,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004400"},"pg_stats":[{"pgid":"1.f","version":"39'12500","reported_seq":14559,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:47.407549+0000","last_change":"2024-09-05T16:43:43.436935+0000","last_active":"2024-09-05T16:45:47.407549+0000","last_peered":"2024-09-05T16:45:47.407549+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:45:47.407549+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"33'2400","ondisk_log_start":"33'2400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3309568,"num_objects":404,"num_object_clones":0,"num_object_copies":1616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":404,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12500,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1000,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-858}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12151","reported_seq":16807,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:45:46.758393+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:45:46.758393+0000","last_peered":"2024-09-05T16:45:46.758393+0000","last_clean":"2024-09-05T16:45:46.758393+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:45:46.758393+0000","last_undegraded":"2024-09-05T16:45:46.758393+0000","last_fullsized":"2024-09-05T16:45:46.758393+0000","mapping_epoch":38,"log_start":"29'2100","ondisk_log_start":"29'2100","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3284992,"num_objects":401,"num_object_clones":0,"num_object_copies":1604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1464,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12376","reported_seq":12504,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:50.669875+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:45:50.669875+0000","last_peered":"2024-09-05T16:45:50.669875+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:45:50.669875+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"33'2300","ondisk_log_start":"33'2300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3078760,"num_objects":377,"num_object_clones":0,"num_object_copies":1508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":377,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12375,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1855}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12529","reported_seq":17264,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:45:46.869419+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:45:46.869419+0000","last_peered":"2024-09-05T16:45:46.869419+0000","last_clean":"2024-09-05T16:45:46.869419+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:45:46.869419+0000","last_undegraded":"2024-09-05T16:45:46.869419+0000","last_fullsized":"2024-09-05T16:45:46.869419+0000","mapping_epoch":34,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3317760,"num_objects":405,"num_object_clones":0,"num_object_copies":1620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":405,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6829,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12172","reported_seq":12276,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:47.164395+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:45:47.164395+0000","last_peered":"2024-09-05T16:45:47.164395+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:45:47.164395+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"29'2100","ondisk_log_start":"29'2100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3462304,"num_objects":423,"num_object_clones":0,"num_object_copies":1692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":423,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12170,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1722}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12332","reported_seq":12451,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:47.401953+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:45:47.401953+0000","last_peered":"2024-09-05T16:45:47.401953+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:45:47.401953+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"33'2300","ondisk_log_start":"33'2300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3604480,"num_objects":440,"num_object_clones":0,"num_object_copies":1760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":440,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12332,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1779}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12156","reported_seq":12269,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:45.927147+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:45:45.927147+0000","last_peered":"2024-09-05T16:45:45.927147+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:45:45.927147+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"29'2100","ondisk_log_start":"29'2100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4142240,"num_objects":506,"num_object_clones":0,"num_object_copies":2024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":506,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12154,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1625}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12289","reported_seq":12383,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:45.909664+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:45:45.909664+0000","last_peered":"2024-09-05T16:45:45.909664+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:45:45.909664+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"30'2200","ondisk_log_start":"30'2200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4011168,"num_objects":490,"num_object_clones":0,"num_object_copies":1960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12287,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1742}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"39'12391","reported_seq":12497,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:47.400767+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:45:47.400767+0000","last_peered":"2024-09-05T16:45:47.400767+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:45:47.400767+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"33'2300","ondisk_log_start":"33'2300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3131712,"num_objects":383,"num_object_clones":0,"num_object_copies":1532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":383,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12388,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1861}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12406","reported_seq":12533,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:50.528649+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:45:50.528649+0000","last_peered":"2024-09-05T16:45:50.528649+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:45:50.528649+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"33'2400","ondisk_log_start":"33'2400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3306656,"num_objects":404,"num_object_clones":0,"num_object_copies":1616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":404,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12404,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1841}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'12657","reported_seq":12789,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:47.330303+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:45:47.330303+0000","last_peered":"2024-09-05T16:45:47.330303+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:45:47.330303+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3813104,"num_objects":466,"num_object_clones":0,"num_object_copies":1864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12654,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1815}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12321","reported_seq":12425,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:47.227464+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:45:47.227464+0000","last_peered":"2024-09-05T16:45:47.227464+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:45:47.227464+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"33'2300","ondisk_log_start":"33'2300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3586640,"num_objects":438,"num_object_clones":0,"num_object_copies":1752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":438,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12320,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1800}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12362","reported_seq":17103,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:45:46.468687+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:45:46.468687+0000","last_peered":"2024-09-05T16:45:46.468687+0000","last_clean":"2024-09-05T16:45:46.468687+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:45:46.468687+0000","last_undegraded":"2024-09-05T16:45:46.468687+0000","last_fullsized":"2024-09-05T16:45:46.468687+0000","mapping_epoch":36,"log_start":"33'2300","ondisk_log_start":"33'2300","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3653632,"num_objects":446,"num_object_clones":0,"num_object_copies":1784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":446,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3962,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'12675","reported_seq":12788,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:47.299073+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:45:47.299073+0000","last_peered":"2024-09-05T16:45:47.299073+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:45:47.299073+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3691680,"num_objects":451,"num_object_clones":0,"num_object_copies":1804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":451,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12673,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1883}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12450","reported_seq":12568,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:50.648659+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:45:50.648659+0000","last_peered":"2024-09-05T16:45:50.648659+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:45:50.648659+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"33'2400","ondisk_log_start":"33'2400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3134624,"num_objects":383,"num_object_clones":0,"num_object_copies":1532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":383,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12448,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1925}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'12303","reported_seq":12417,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:45.838882+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:45:45.838882+0000","last_peered":"2024-09-05T16:45:45.838882+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:45:45.838882+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"33'2300","ondisk_log_start":"33'2300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3863712,"num_objects":472,"num_object_clones":0,"num_object_copies":1888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":472,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12301,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1748}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":56393032,"num_objects":6889,"num_object_clones":0,"num_object_copies":27556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28113,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6889,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":173261,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1253,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":112439296,"data_stored":112439296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160870,"ondisk_log_size":160870,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084429,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":723700,"kb_used_data":28388,"kb_used_omap":4,"kb_used_meta":695291,"kb_avail":93648140,"statfs":{"total":96636764160,"available":95895695360,"internally_reserved":0,"allocated":29069312,"data_stored":28858731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":711978707},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,1,6,8,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986306,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":580296,"kb_used_data":29368,"kb_used_omap":4,"kb_used_meta":550907,"kb_avail":93791544,"statfs":{"total":96636764160,"available":96042541056,"internally_reserved":0,"allocated":30072832,"data_stored":29862251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":564129490},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,6,2,0,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673113,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":729280,"kb_used_data":26160,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":93642560,"statfs":{"total":96636764160,"available":95889981440,"internally_reserved":0,"allocated":26787840,"data_stored":26577259,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":719976901},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,5,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182543,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":710752,"kb_used_data":29392,"kb_used_omap":4,"kb_used_meta":681339,"kb_avail":93661088,"statfs":{"total":96636764160,"available":95908954112,"internally_reserved":0,"allocated":30097408,"data_stored":29886827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":697691853},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,2,9,8,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":29478912,"data_stored":29478912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28872704,"data_stored":28872704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":25718784,"data_stored":25718784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28368896,"data_stored":28368896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:45:53.011 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:45:53.627 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:45:53.628 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:45:53.652 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":408,"stamp":"2024-09-05T16:45:53.083384+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":49741128,"num_objects":6077,"num_object_clones":0,"num_object_copies":24308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28062,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6077,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":174073,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1304,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160782,"ondisk_log_size":160782,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2765992,"kb_used_data":100576,"kb_used_omap":14,"kb_used_meta":2665329,"kb_avail":374721368,"statfs":{"total":386547056640,"available":383714680832,"internally_reserved":0,"allocated":102989824,"data_stored":102147500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2729297463},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,6,7,12,16,5,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21659648,"num_objects":-2644,"num_object_clones":0,"num_object_copies":-10576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2644,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2644,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":130,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004250"},"pg_stats":[{"pgid":"1.f","version":"39'12562","reported_seq":14723,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:52.528673+0000","last_change":"2024-09-05T16:43:43.436935+0000","last_active":"2024-09-05T16:45:52.528673+0000","last_peered":"2024-09-05T16:45:52.528673+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:45:52.528673+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2801664,"num_objects":342,"num_object_clones":0,"num_object_copies":1368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1211,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":342,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12562,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1051,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-869}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12204","reported_seq":16860,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:45:51.951970+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:45:51.951970+0000","last_peered":"2024-09-05T16:45:51.951970+0000","last_clean":"2024-09-05T16:45:51.951970+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:45:51.951970+0000","last_undegraded":"2024-09-05T16:45:51.951970+0000","last_fullsized":"2024-09-05T16:45:51.951970+0000","mapping_epoch":38,"log_start":"29'2200","ondisk_log_start":"29'2200","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2850816,"num_objects":348,"num_object_clones":0,"num_object_copies":1392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":348,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1517,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12376","reported_seq":12504,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:50.669875+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:45:50.669875+0000","last_peered":"2024-09-05T16:45:50.669875+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:45:50.669875+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"33'2300","ondisk_log_start":"33'2300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3078760,"num_objects":377,"num_object_clones":0,"num_object_copies":1508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":377,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12375,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1855}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12591","reported_seq":17325,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:45:51.936325+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:45:51.936325+0000","last_peered":"2024-09-05T16:45:51.936325+0000","last_clean":"2024-09-05T16:45:51.936325+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:45:51.936325+0000","last_undegraded":"2024-09-05T16:45:51.936325+0000","last_fullsized":"2024-09-05T16:45:51.936325+0000","mapping_epoch":34,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2809856,"num_objects":343,"num_object_clones":0,"num_object_copies":1372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":343,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6891,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12236","reported_seq":12340,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:52.312782+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:45:52.312782+0000","last_peered":"2024-09-05T16:45:52.312782+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:45:52.312782+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"33'2200","ondisk_log_start":"33'2200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2938016,"num_objects":359,"num_object_clones":0,"num_object_copies":1436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":359,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12234,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1786}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12398","reported_seq":12518,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:52.496170+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:45:52.496170+0000","last_peered":"2024-09-05T16:45:52.496170+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:45:52.496170+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"33'2300","ondisk_log_start":"33'2300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3063808,"num_objects":374,"num_object_clones":0,"num_object_copies":1496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":374,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12398,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1845}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12227","reported_seq":12340,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:51.123709+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:45:51.123709+0000","last_peered":"2024-09-05T16:45:51.123709+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:45:51.123709+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"33'2200","ondisk_log_start":"33'2200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3560608,"num_objects":435,"num_object_clones":0,"num_object_copies":1740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":435,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12225,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1696}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12365","reported_seq":12458,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:51.056166+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:45:51.056166+0000","last_peered":"2024-09-05T16:45:51.056166+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:45:51.056166+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"33'2300","ondisk_log_start":"33'2300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3388576,"num_objects":414,"num_object_clones":0,"num_object_copies":1656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":414,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12363,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1818}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"39'12445","reported_seq":12550,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:52.364708+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:45:52.364708+0000","last_peered":"2024-09-05T16:45:52.364708+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:45:52.364708+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"33'2400","ondisk_log_start":"33'2400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2689344,"num_objects":329,"num_object_clones":0,"num_object_copies":1316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":329,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12442,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1915}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12406","reported_seq":12533,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:50.528649+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:45:50.528649+0000","last_peered":"2024-09-05T16:45:50.528649+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:45:50.528649+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"33'2400","ondisk_log_start":"33'2400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3306656,"num_objects":404,"num_object_clones":0,"num_object_copies":1616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":404,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12404,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1841}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'12728","reported_seq":12860,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:52.171969+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:45:52.171969+0000","last_peered":"2024-09-05T16:45:52.171969+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:45:52.171969+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3231472,"num_objects":395,"num_object_clones":0,"num_object_copies":1580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":395,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12725,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1886}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12374","reported_seq":12478,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:52.468720+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:45:52.468720+0000","last_peered":"2024-09-05T16:45:52.468720+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:45:52.468720+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"33'2300","ondisk_log_start":"33'2300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3152464,"num_objects":385,"num_object_clones":0,"num_object_copies":1540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":385,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12373,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1853}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12410","reported_seq":17151,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:45:51.913568+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:45:51.913568+0000","last_peered":"2024-09-05T16:45:51.913568+0000","last_clean":"2024-09-05T16:45:51.913568+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:45:51.913568+0000","last_undegraded":"2024-09-05T16:45:51.913568+0000","last_fullsized":"2024-09-05T16:45:51.913568+0000","mapping_epoch":36,"log_start":"33'2400","ondisk_log_start":"33'2400","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3260416,"num_objects":398,"num_object_clones":0,"num_object_copies":1592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":398,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4010,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'12745","reported_seq":12859,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:52.438145+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:45:52.438145+0000","last_peered":"2024-09-05T16:45:52.438145+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:45:52.438145+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3118240,"num_objects":381,"num_object_clones":0,"num_object_copies":1524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":381,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12743,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1953}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12450","reported_seq":12568,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:50.648659+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:45:50.648659+0000","last_peered":"2024-09-05T16:45:50.648659+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:45:50.648659+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"33'2400","ondisk_log_start":"33'2400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3134624,"num_objects":383,"num_object_clones":0,"num_object_copies":1532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":383,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12448,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1925}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'12365","reported_seq":12479,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:50.948538+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:45:50.948538+0000","last_peered":"2024-09-05T16:45:50.948538+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:45:50.948538+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"33'2300","ondisk_log_start":"33'2300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3355808,"num_objects":410,"num_object_clones":0,"num_object_copies":1640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":410,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12363,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1810}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":49741128,"num_objects":6077,"num_object_clones":0,"num_object_copies":24308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28062,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6077,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":174073,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1304,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":100040704,"data_stored":100040704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160782,"ondisk_log_size":160782,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084430,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":723896,"kb_used_data":24424,"kb_used_omap":4,"kb_used_meta":699451,"kb_avail":93647944,"statfs":{"total":96636764160,"available":95895494656,"internally_reserved":0,"allocated":25010176,"data_stored":24799595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":716238547},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,4,0,1,6,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986307,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":583760,"kb_used_data":24504,"kb_used_omap":4,"kb_used_meta":559227,"kb_avail":93788080,"statfs":{"total":96636764160,"available":96038993920,"internally_reserved":0,"allocated":25092096,"data_stored":24881515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":572649170},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,3,3,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673113,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":729280,"kb_used_data":26160,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":93642560,"statfs":{"total":96636764160,"available":95889981440,"internally_reserved":0,"allocated":26787840,"data_stored":26577259,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":719976901},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,5,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182544,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":729056,"kb_used_data":25488,"kb_used_omap":4,"kb_used_meta":703547,"kb_avail":93642784,"statfs":{"total":96636764160,"available":95890210816,"internally_reserved":0,"allocated":26099712,"data_stored":25889131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":720432845},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,3,2,1,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":25387008,"data_stored":25387008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24748032,"data_stored":24748032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":25718784,"data_stored":25718784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":24186880,"data_stored":24186880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:45:53.653 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-05T16:45:53.654 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "39'12562", 'reported_seq': 14723, 'reported_epoch': 39, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:52.528673+0000', 'last_change': '2024-09-05T16:43:43.436935+0000', 'last_active': '2024-09-05T16:45:52.528673+0000', 'last_peered': '2024-09-05T16:45:52.528673+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.828289+0000', 'last_became_peered': '2024-09-05T16:34:56.828289+0000', 'last_unstale': '2024-09-05T16:45:52.528673+0000', 'last_undegraded': '2024-09-05T16:34:56.612160+0000', 'last_fullsized': '2024-09-05T16:34:56.557632+0000', 'mapping_epoch': 32, 'log_start': "33'2500", 'ondisk_log_start': "33'2500", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10062, 'log_dups_size': 0, 'ondisk_log_size': 10062, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2801664, 'num_objects': 342, 'num_object_clones': 0, 'num_object_copies': 1368, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1211, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 342, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12562, 'num_write_kb': 51616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1051, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -869}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:45:53.654 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:45:53.654 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "39'12376", 'reported_seq': 12504, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:50.669875+0000', 'last_change': '2024-09-05T16:34:56.867900+0000', 'last_active': '2024-09-05T16:45:50.669875+0000', 'last_peered': '2024-09-05T16:45:50.669875+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:45:50.669875+0000', 'last_undegraded': '2024-09-05T16:34:56.590972+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 32, 'log_start': "33'2300", 'ondisk_log_start': "33'2300", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10076, 'log_dups_size': 0, 'ondisk_log_size': 10076, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3078760, 'num_objects': 377, 'num_object_clones': 0, 'num_object_copies': 1508, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2233, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 377, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12375, 'num_write_kb': 50999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1855}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:45:53.654 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:45:53.654 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "39'12236", 'reported_seq': 12340, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:52.312782+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:45:52.312782+0000', 'last_peered': '2024-09-05T16:45:52.312782+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:45:52.312782+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "33'2200", 'ondisk_log_start': "33'2200", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10036, 'log_dups_size': 0, 'ondisk_log_size': 10036, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2938016, 'num_objects': 359, 'num_object_clones': 0, 'num_object_copies': 1436, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2147, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 359, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12234, 'num_write_kb': 50373, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1786}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:45:53.654 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:45:53.654 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "39'12398", 'reported_seq': 12518, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:52.496170+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:45:52.496170+0000', 'last_peered': '2024-09-05T16:45:52.496170+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:45:52.496170+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "33'2300", 'ondisk_log_start': "33'2300", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3063808, 'num_objects': 374, 'num_object_clones': 0, 'num_object_copies': 1496, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 374, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12398, 'num_write_kb': 51088, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1845}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:45:53.654 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:45:53.654 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "39'12227", 'reported_seq': 12340, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:51.123709+0000', 'last_change': '2024-09-05T16:34:56.903436+0000', 'last_active': '2024-09-05T16:45:51.123709+0000', 'last_peered': '2024-09-05T16:45:51.123709+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:45:51.123709+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "33'2200", 'ondisk_log_start': "33'2200", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10027, 'log_dups_size': 0, 'ondisk_log_size': 10027, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3560608, 'num_objects': 435, 'num_object_clones': 0, 'num_object_copies': 1740, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2133, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 435, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12225, 'num_write_kb': 50636, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -1696}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:45:53.655 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-05T16:45:53.655 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "39'12365", 'reported_seq': 12458, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:51.056166+0000', 'last_change': '2024-09-05T16:34:56.883319+0000', 'last_active': '2024-09-05T16:45:51.056166+0000', 'last_peered': '2024-09-05T16:45:51.056166+0000', 'last_clean': '2024-09-05T16:33:53.505305+0000', 'last_became_active': '2024-09-05T16:34:56.873389+0000', 'last_became_peered': '2024-09-05T16:34:56.873389+0000', 'last_unstale': '2024-09-05T16:45:51.056166+0000', 'last_undegraded': '2024-09-05T16:34:56.602567+0000', 'last_fullsized': '2024-09-05T16:34:56.559958+0000', 'mapping_epoch': 32, 'log_start': "33'2300", 'ondisk_log_start': "33'2300", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10065, 'log_dups_size': 0, 'ondisk_log_size': 10065, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3388576, 'num_objects': 414, 'num_object_clones': 0, 'num_object_copies': 1656, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2234, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 414, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12363, 'num_write_kb': 51108, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1818}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:45:53.655 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:45:53.656 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "39'12445", 'reported_seq': 12550, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:52.364708+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:45:52.364708+0000', 'last_peered': '2024-09-05T16:45:52.364708+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:45:52.364708+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "33'2400", 'ondisk_log_start': "33'2400", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10045, 'log_dups_size': 0, 'ondisk_log_size': 10045, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2689344, 'num_objects': 329, 'num_object_clones': 0, 'num_object_copies': 1316, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 329, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12442, 'num_write_kb': 51081, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1915}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:45:53.656 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:45:53.656 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "39'12406", 'reported_seq': 12533, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:50.528649+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:45:50.528649+0000', 'last_peered': '2024-09-05T16:45:50.528649+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:45:50.528649+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "33'2400", 'ondisk_log_start': "33'2400", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10006, 'log_dups_size': 0, 'ondisk_log_size': 10006, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3306656, 'num_objects': 404, 'num_object_clones': 0, 'num_object_copies': 1616, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 404, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12404, 'num_write_kb': 51230, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1841}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:45:53.656 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:45:53.656 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "39'12728", 'reported_seq': 12860, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:52.171969+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:45:52.171969+0000', 'last_peered': '2024-09-05T16:45:52.171969+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:45:52.171969+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "33'2700", 'ondisk_log_start': "33'2700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3231472, 'num_objects': 395, 'num_object_clones': 0, 'num_object_copies': 1580, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2284, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 395, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12725, 'num_write_kb': 52476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1886}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:45:53.656 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:45:53.656 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "39'12374", 'reported_seq': 12478, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:52.468720+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:45:52.468720+0000', 'last_peered': '2024-09-05T16:45:52.468720+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:45:52.468720+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "33'2300", 'ondisk_log_start': "33'2300", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10074, 'log_dups_size': 0, 'ondisk_log_size': 10074, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3152464, 'num_objects': 385, 'num_object_clones': 0, 'num_object_copies': 1540, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 385, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12373, 'num_write_kb': 51031, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1853}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:45:53.656 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:45:53.656 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "39'12745", 'reported_seq': 12859, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:52.438145+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:45:52.438145+0000', 'last_peered': '2024-09-05T16:45:52.438145+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:45:52.438145+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "33'2700", 'ondisk_log_start': "33'2700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10045, 'log_dups_size': 0, 'ondisk_log_size': 10045, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3118240, 'num_objects': 381, 'num_object_clones': 0, 'num_object_copies': 1524, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 381, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12743, 'num_write_kb': 52500, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1953}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:45:53.657 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:45:53.657 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "39'12450", 'reported_seq': 12568, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:50.648659+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:45:50.648659+0000', 'last_peered': '2024-09-05T16:45:50.648659+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:45:50.648659+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "33'2400", 'ondisk_log_start': "33'2400", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3134624, 'num_objects': 383, 'num_object_clones': 0, 'num_object_copies': 1532, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2310, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 383, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12448, 'num_write_kb': 51325, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1925}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:45:53.657 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:45:53.657 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "39'12365", 'reported_seq': 12479, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:50.948538+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:45:50.948538+0000', 'last_peered': '2024-09-05T16:45:50.948538+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:45:50.948538+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "33'2300", 'ondisk_log_start': "33'2300", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10065, 'log_dups_size': 0, 'ondisk_log_size': 10065, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3355808, 'num_objects': 410, 'num_object_clones': 0, 'num_object_copies': 1640, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 410, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12363, 'num_write_kb': 51092, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1810}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:45:53.657 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:45:54.405 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:45:54.405 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 28062/24308 objects degraded (115.443%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":757,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":39,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":6077,"data_bytes":49741128,"bytes_used":2832375808,"bytes_avail":383714680832,"bytes_total":386547056640,"degraded_objects":28062,"degraded_total":24308,"degraded_ratio":1.1544347539904558,"recovering_objects_per_sec":10,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":130,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":220},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:45:54.434 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:45:54.434 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:45:55.056 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:45:55.057 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:45:55.078 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":408,"stamp":"2024-09-05T16:45:53.083384+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":49741128,"num_objects":6077,"num_object_clones":0,"num_object_copies":24308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28062,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6077,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":174073,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1304,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160782,"ondisk_log_size":160782,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2765992,"kb_used_data":100576,"kb_used_omap":14,"kb_used_meta":2665329,"kb_avail":374721368,"statfs":{"total":386547056640,"available":383714680832,"internally_reserved":0,"allocated":102989824,"data_stored":102147500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2729297463},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,6,7,12,16,5,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21659648,"num_objects":-2644,"num_object_clones":0,"num_object_copies":-10576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2644,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2644,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":130,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004250"},"pg_stats":[{"pgid":"1.f","version":"39'12562","reported_seq":14723,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:52.528673+0000","last_change":"2024-09-05T16:43:43.436935+0000","last_active":"2024-09-05T16:45:52.528673+0000","last_peered":"2024-09-05T16:45:52.528673+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:45:52.528673+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2801664,"num_objects":342,"num_object_clones":0,"num_object_copies":1368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1211,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":342,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12562,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1051,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-869}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12204","reported_seq":16860,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:45:51.951970+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:45:51.951970+0000","last_peered":"2024-09-05T16:45:51.951970+0000","last_clean":"2024-09-05T16:45:51.951970+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:45:51.951970+0000","last_undegraded":"2024-09-05T16:45:51.951970+0000","last_fullsized":"2024-09-05T16:45:51.951970+0000","mapping_epoch":38,"log_start":"29'2200","ondisk_log_start":"29'2200","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2850816,"num_objects":348,"num_object_clones":0,"num_object_copies":1392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":348,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1517,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12376","reported_seq":12504,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:50.669875+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:45:50.669875+0000","last_peered":"2024-09-05T16:45:50.669875+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:45:50.669875+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"33'2300","ondisk_log_start":"33'2300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3078760,"num_objects":377,"num_object_clones":0,"num_object_copies":1508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":377,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12375,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1855}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12591","reported_seq":17325,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:45:51.936325+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:45:51.936325+0000","last_peered":"2024-09-05T16:45:51.936325+0000","last_clean":"2024-09-05T16:45:51.936325+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:45:51.936325+0000","last_undegraded":"2024-09-05T16:45:51.936325+0000","last_fullsized":"2024-09-05T16:45:51.936325+0000","mapping_epoch":34,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2809856,"num_objects":343,"num_object_clones":0,"num_object_copies":1372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":343,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6891,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12236","reported_seq":12340,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:52.312782+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:45:52.312782+0000","last_peered":"2024-09-05T16:45:52.312782+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:45:52.312782+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"33'2200","ondisk_log_start":"33'2200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2938016,"num_objects":359,"num_object_clones":0,"num_object_copies":1436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":359,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12234,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1786}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12398","reported_seq":12518,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:52.496170+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:45:52.496170+0000","last_peered":"2024-09-05T16:45:52.496170+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:45:52.496170+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"33'2300","ondisk_log_start":"33'2300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3063808,"num_objects":374,"num_object_clones":0,"num_object_copies":1496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":374,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12398,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1845}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12227","reported_seq":12340,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:51.123709+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:45:51.123709+0000","last_peered":"2024-09-05T16:45:51.123709+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:45:51.123709+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"33'2200","ondisk_log_start":"33'2200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3560608,"num_objects":435,"num_object_clones":0,"num_object_copies":1740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":435,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12225,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1696}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12365","reported_seq":12458,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:51.056166+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:45:51.056166+0000","last_peered":"2024-09-05T16:45:51.056166+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:45:51.056166+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"33'2300","ondisk_log_start":"33'2300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3388576,"num_objects":414,"num_object_clones":0,"num_object_copies":1656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":414,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12363,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1818}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"39'12445","reported_seq":12550,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:52.364708+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:45:52.364708+0000","last_peered":"2024-09-05T16:45:52.364708+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:45:52.364708+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"33'2400","ondisk_log_start":"33'2400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2689344,"num_objects":329,"num_object_clones":0,"num_object_copies":1316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":329,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12442,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1915}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12406","reported_seq":12533,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:50.528649+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:45:50.528649+0000","last_peered":"2024-09-05T16:45:50.528649+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:45:50.528649+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"33'2400","ondisk_log_start":"33'2400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3306656,"num_objects":404,"num_object_clones":0,"num_object_copies":1616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":404,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12404,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1841}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'12728","reported_seq":12860,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:52.171969+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:45:52.171969+0000","last_peered":"2024-09-05T16:45:52.171969+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:45:52.171969+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3231472,"num_objects":395,"num_object_clones":0,"num_object_copies":1580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":395,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12725,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1886}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12374","reported_seq":12478,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:52.468720+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:45:52.468720+0000","last_peered":"2024-09-05T16:45:52.468720+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:45:52.468720+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"33'2300","ondisk_log_start":"33'2300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3152464,"num_objects":385,"num_object_clones":0,"num_object_copies":1540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":385,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12373,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1853}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12410","reported_seq":17151,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:45:51.913568+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:45:51.913568+0000","last_peered":"2024-09-05T16:45:51.913568+0000","last_clean":"2024-09-05T16:45:51.913568+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:45:51.913568+0000","last_undegraded":"2024-09-05T16:45:51.913568+0000","last_fullsized":"2024-09-05T16:45:51.913568+0000","mapping_epoch":36,"log_start":"33'2400","ondisk_log_start":"33'2400","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3260416,"num_objects":398,"num_object_clones":0,"num_object_copies":1592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":398,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4010,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'12745","reported_seq":12859,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:52.438145+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:45:52.438145+0000","last_peered":"2024-09-05T16:45:52.438145+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:45:52.438145+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3118240,"num_objects":381,"num_object_clones":0,"num_object_copies":1524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":381,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12743,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1953}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12450","reported_seq":12568,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:50.648659+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:45:50.648659+0000","last_peered":"2024-09-05T16:45:50.648659+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:45:50.648659+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"33'2400","ondisk_log_start":"33'2400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3134624,"num_objects":383,"num_object_clones":0,"num_object_copies":1532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":383,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12448,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1925}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'12365","reported_seq":12479,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:50.948538+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:45:50.948538+0000","last_peered":"2024-09-05T16:45:50.948538+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:45:50.948538+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"33'2300","ondisk_log_start":"33'2300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3355808,"num_objects":410,"num_object_clones":0,"num_object_copies":1640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":410,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12363,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1810}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":49741128,"num_objects":6077,"num_object_clones":0,"num_object_copies":24308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28062,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6077,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":174073,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1304,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":100040704,"data_stored":100040704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160782,"ondisk_log_size":160782,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084430,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":723896,"kb_used_data":24424,"kb_used_omap":4,"kb_used_meta":699451,"kb_avail":93647944,"statfs":{"total":96636764160,"available":95895494656,"internally_reserved":0,"allocated":25010176,"data_stored":24799595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":716238547},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,4,0,1,6,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986307,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":583760,"kb_used_data":24504,"kb_used_omap":4,"kb_used_meta":559227,"kb_avail":93788080,"statfs":{"total":96636764160,"available":96038993920,"internally_reserved":0,"allocated":25092096,"data_stored":24881515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":572649170},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,3,3,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673113,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":729280,"kb_used_data":26160,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":93642560,"statfs":{"total":96636764160,"available":95889981440,"internally_reserved":0,"allocated":26787840,"data_stored":26577259,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":719976901},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,5,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182544,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":729056,"kb_used_data":25488,"kb_used_omap":4,"kb_used_meta":703547,"kb_avail":93642784,"statfs":{"total":96636764160,"available":95890210816,"internally_reserved":0,"allocated":26099712,"data_stored":25889131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":720432845},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,3,2,1,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":25387008,"data_stored":25387008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24748032,"data_stored":24748032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":25718784,"data_stored":25718784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":24186880,"data_stored":24186880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:45:58.080 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:45:58.677 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:45:58.678 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:45:58.709 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":410,"stamp":"2024-09-05T16:45:57.084926+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":44997960,"num_objects":5498,"num_object_clones":0,"num_object_copies":21992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28062,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5498,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":174652,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1304,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160861,"ondisk_log_size":160861,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2763080,"kb_used_data":89304,"kb_used_omap":14,"kb_used_meta":2673649,"kb_avail":374724280,"statfs":{"total":386547056640,"available":383717662720,"internally_reserved":0,"allocated":91447296,"data_stored":90604972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2737817143},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,5,6,7,8,16,8,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21209088,"num_objects":-2589,"num_object_clones":0,"num_object_copies":-10356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-87,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2589,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2589,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":87,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004436"},"pg_stats":[{"pgid":"1.f","version":"39'12562","reported_seq":14723,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:52.528673+0000","last_change":"2024-09-05T16:43:43.436935+0000","last_active":"2024-09-05T16:45:52.528673+0000","last_peered":"2024-09-05T16:45:52.528673+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:45:52.528673+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2801664,"num_objects":342,"num_object_clones":0,"num_object_copies":1368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1211,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":342,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12562,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1051,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-869}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12262","reported_seq":16918,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:45:56.846833+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:45:56.846833+0000","last_peered":"2024-09-05T16:45:56.846833+0000","last_clean":"2024-09-05T16:45:56.846833+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:45:56.846833+0000","last_undegraded":"2024-09-05T16:45:56.846833+0000","last_fullsized":"2024-09-05T16:45:56.846833+0000","mapping_epoch":38,"log_start":"29'2200","ondisk_log_start":"29'2200","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2375680,"num_objects":290,"num_object_clones":0,"num_object_copies":1160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":290,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1575,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12429","reported_seq":12557,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:55.687614+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:45:55.687614+0000","last_peered":"2024-09-05T16:45:55.687614+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:45:55.687614+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"33'2400","ondisk_log_start":"33'2400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2644584,"num_objects":324,"num_object_clones":0,"num_object_copies":1296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":324,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12428,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1908}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12652","reported_seq":17387,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:45:56.870849+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:45:56.870849+0000","last_peered":"2024-09-05T16:45:56.870849+0000","last_clean":"2024-09-05T16:45:56.870849+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:45:56.870849+0000","last_undegraded":"2024-09-05T16:45:56.870849+0000","last_fullsized":"2024-09-05T16:45:56.870849+0000","mapping_epoch":34,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2310144,"num_objects":282,"num_object_clones":0,"num_object_copies":1128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":282,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6952,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12236","reported_seq":12340,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:52.312782+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:45:52.312782+0000","last_peered":"2024-09-05T16:45:52.312782+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:45:52.312782+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"33'2200","ondisk_log_start":"33'2200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2938016,"num_objects":359,"num_object_clones":0,"num_object_copies":1436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":359,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12234,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1786}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12398","reported_seq":12518,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:52.496170+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:45:52.496170+0000","last_peered":"2024-09-05T16:45:52.496170+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:45:52.496170+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"33'2300","ondisk_log_start":"33'2300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3063808,"num_objects":374,"num_object_clones":0,"num_object_copies":1496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":374,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12398,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1845}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12299","reported_seq":12411,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:56.123038+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:45:56.123038+0000","last_peered":"2024-09-05T16:45:56.123038+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:45:56.123038+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"33'2200","ondisk_log_start":"33'2200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2970784,"num_objects":363,"num_object_clones":0,"num_object_copies":1452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":363,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12297,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1768}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12421","reported_seq":12515,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:56.098245+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:45:56.098245+0000","last_peered":"2024-09-05T16:45:56.098245+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:45:56.098245+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"33'2400","ondisk_log_start":"33'2400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2929824,"num_objects":358,"num_object_clones":0,"num_object_copies":1432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":358,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12419,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1874}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"39'12445","reported_seq":12550,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:52.364708+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:45:52.364708+0000","last_peered":"2024-09-05T16:45:52.364708+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:45:52.364708+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"33'2400","ondisk_log_start":"33'2400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2689344,"num_objects":329,"num_object_clones":0,"num_object_copies":1316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":329,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12442,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1915}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12479","reported_seq":12606,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:55.507341+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:45:55.507341+0000","last_peered":"2024-09-05T16:45:55.507341+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:45:55.507341+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"33'2400","ondisk_log_start":"33'2400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2708640,"num_objects":331,"num_object_clones":0,"num_object_copies":1324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":331,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12477,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1914}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'12728","reported_seq":12860,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:52.171969+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:45:52.171969+0000","last_peered":"2024-09-05T16:45:52.171969+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:45:52.171969+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3231472,"num_objects":395,"num_object_clones":0,"num_object_copies":1580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":395,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12725,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1886}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12374","reported_seq":12478,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:52.468720+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:45:52.468720+0000","last_peered":"2024-09-05T16:45:52.468720+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:45:52.468720+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"33'2300","ondisk_log_start":"33'2300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3152464,"num_objects":385,"num_object_clones":0,"num_object_copies":1540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":385,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12373,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1853}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12486","reported_seq":17227,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:45:56.742316+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:45:56.742316+0000","last_peered":"2024-09-05T16:45:56.742316+0000","last_clean":"2024-09-05T16:45:56.742316+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:45:56.742316+0000","last_undegraded":"2024-09-05T16:45:56.742316+0000","last_fullsized":"2024-09-05T16:45:56.742316+0000","mapping_epoch":36,"log_start":"33'2400","ondisk_log_start":"33'2400","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2637824,"num_objects":322,"num_object_clones":0,"num_object_copies":1288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":322,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4086,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'12745","reported_seq":12859,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:52.438145+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:45:52.438145+0000","last_peered":"2024-09-05T16:45:52.438145+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:45:52.438145+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3118240,"num_objects":381,"num_object_clones":0,"num_object_copies":1524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":381,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12743,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1953}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12505","reported_seq":12622,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:55.730942+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:45:55.730942+0000","last_peered":"2024-09-05T16:45:55.730942+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:45:55.730942+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2684064,"num_objects":328,"num_object_clones":0,"num_object_copies":1312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":328,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12503,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1980}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'12440","reported_seq":12554,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:56.138550+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:45:56.138550+0000","last_peered":"2024-09-05T16:45:56.138550+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:45:56.138550+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"33'2400","ondisk_log_start":"33'2400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2741408,"num_objects":335,"num_object_clones":0,"num_object_copies":1340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":335,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12438,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1885}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":44997960,"num_objects":5498,"num_object_clones":0,"num_object_copies":21992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28062,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5498,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":174652,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1304,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":87793664,"data_stored":87793664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160861,"ondisk_log_size":160861,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084430,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":723896,"kb_used_data":24424,"kb_used_omap":4,"kb_used_meta":699451,"kb_avail":93647944,"statfs":{"total":96636764160,"available":95895494656,"internally_reserved":0,"allocated":25010176,"data_stored":24799595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":716238547},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,4,0,1,6,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986308,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":580520,"kb_used_data":21260,"kb_used_omap":4,"kb_used_meta":559227,"kb_avail":93791320,"statfs":{"total":96636764160,"available":96042311680,"internally_reserved":0,"allocated":21770240,"data_stored":21559659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":572649170},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,2,4,6,3,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673114,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":729404,"kb_used_data":22100,"kb_used_omap":1,"kb_used_meta":707262,"kb_avail":93642436,"statfs":{"total":96636764160,"available":95889854464,"internally_reserved":0,"allocated":22630400,"data_stored":22419819,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":724236741},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,2,3,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182545,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":729260,"kb_used_data":21520,"kb_used_omap":4,"kb_used_meta":707707,"kb_avail":93642580,"statfs":{"total":96636764160,"available":95890001920,"internally_reserved":0,"allocated":22036480,"data_stored":21825899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":724692685},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,2,1,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":21299200,"data_stored":21299200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20656128,"data_stored":20656128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":21651456,"data_stored":21651456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":24186880,"data_stored":24186880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:45:58.710 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:45:59.273 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:45:59.516 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:45:59.516 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":411,"stamp":"2024-09-05T16:45:59.085629+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":41631048,"num_objects":5087,"num_object_clones":0,"num_object_copies":20348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5087,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":175063,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1351,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160872,"ondisk_log_size":160872,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2763340,"kb_used_data":85404,"kb_used_omap":14,"kb_used_meta":2677809,"kb_avail":374724020,"statfs":{"total":386547056640,"available":383717396480,"internally_reserved":0,"allocated":87453696,"data_stored":86611372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2742076983},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,3,2,11,14,10,7,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20185088,"num_objects":-2464,"num_object_clones":0,"num_object_copies":-9856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2464,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2464,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":134,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004336"},"pg_stats":[{"pgid":"1.f","version":"39'12629","reported_seq":14884,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:57.677764+0000","last_change":"2024-09-05T16:43:43.436935+0000","last_active":"2024-09-05T16:45:57.677764+0000","last_peered":"2024-09-05T16:45:57.677764+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:45:57.677764+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2252800,"num_objects":275,"num_object_clones":0,"num_object_copies":1100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":275,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12629,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1098,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-889}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12262","reported_seq":16918,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:45:56.846833+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:45:56.846833+0000","last_peered":"2024-09-05T16:45:56.846833+0000","last_clean":"2024-09-05T16:45:56.846833+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:45:56.846833+0000","last_undegraded":"2024-09-05T16:45:56.846833+0000","last_fullsized":"2024-09-05T16:45:56.846833+0000","mapping_epoch":38,"log_start":"29'2200","ondisk_log_start":"29'2200","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2375680,"num_objects":290,"num_object_clones":0,"num_object_copies":1160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":290,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1575,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12429","reported_seq":12557,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:55.687614+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:45:55.687614+0000","last_peered":"2024-09-05T16:45:55.687614+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:45:55.687614+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"33'2400","ondisk_log_start":"33'2400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2644584,"num_objects":324,"num_object_clones":0,"num_object_copies":1296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":324,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12428,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1908}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12652","reported_seq":17387,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:45:56.870849+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:45:56.870849+0000","last_peered":"2024-09-05T16:45:56.870849+0000","last_clean":"2024-09-05T16:45:56.870849+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:45:56.870849+0000","last_undegraded":"2024-09-05T16:45:56.870849+0000","last_fullsized":"2024-09-05T16:45:56.870849+0000","mapping_epoch":34,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2310144,"num_objects":282,"num_object_clones":0,"num_object_copies":1128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":282,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6952,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12291","reported_seq":12395,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:57.523438+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:45:57.523438+0000","last_peered":"2024-09-05T16:45:57.523438+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:45:57.523438+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"33'2200","ondisk_log_start":"33'2200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487456,"num_objects":304,"num_object_clones":0,"num_object_copies":1216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":304,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12289,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1841}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12456","reported_seq":12576,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:57.632837+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:45:57.632837+0000","last_peered":"2024-09-05T16:45:57.632837+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:45:57.632837+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"33'2400","ondisk_log_start":"33'2400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2588672,"num_objects":316,"num_object_clones":0,"num_object_copies":1264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":316,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12456,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1903}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12299","reported_seq":12411,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:56.123038+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:45:56.123038+0000","last_peered":"2024-09-05T16:45:56.123038+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:45:56.123038+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"33'2200","ondisk_log_start":"33'2200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2970784,"num_objects":363,"num_object_clones":0,"num_object_copies":1452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":363,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12297,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1768}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12421","reported_seq":12515,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:56.098245+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:45:56.098245+0000","last_peered":"2024-09-05T16:45:56.098245+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:45:56.098245+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"33'2400","ondisk_log_start":"33'2400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2929824,"num_objects":358,"num_object_clones":0,"num_object_copies":1432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":358,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12419,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1874}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"39'12497","reported_seq":12603,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:57.464241+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:45:57.464241+0000","last_peered":"2024-09-05T16:45:57.464241+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:45:57.464241+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"33'2400","ondisk_log_start":"33'2400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2263360,"num_objects":277,"num_object_clones":0,"num_object_copies":1108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":277,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12494,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1967}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12479","reported_seq":12606,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:55.507341+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:45:55.507341+0000","last_peered":"2024-09-05T16:45:55.507341+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:45:55.507341+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"33'2400","ondisk_log_start":"33'2400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2708640,"num_objects":331,"num_object_clones":0,"num_object_copies":1324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":331,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12477,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1914}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'12793","reported_seq":12924,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:57.650086+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:45:57.650086+0000","last_peered":"2024-09-05T16:45:57.650086+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:45:57.650086+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2698992,"num_objects":330,"num_object_clones":0,"num_object_copies":1320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":330,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12790,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1951}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12431","reported_seq":12535,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:57.406588+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:45:57.406588+0000","last_peered":"2024-09-05T16:45:57.406588+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:45:57.406588+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"33'2400","ondisk_log_start":"33'2400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2685520,"num_objects":328,"num_object_clones":0,"num_object_copies":1312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":328,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12430,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1910}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12486","reported_seq":17227,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:45:56.742316+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:45:56.742316+0000","last_peered":"2024-09-05T16:45:56.742316+0000","last_clean":"2024-09-05T16:45:56.742316+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:45:56.742316+0000","last_undegraded":"2024-09-05T16:45:56.742316+0000","last_fullsized":"2024-09-05T16:45:56.742316+0000","mapping_epoch":36,"log_start":"33'2400","ondisk_log_start":"33'2400","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2637824,"num_objects":322,"num_object_clones":0,"num_object_copies":1288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":322,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4086,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'12802","reported_seq":12916,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:57.523640+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:45:57.523640+0000","last_peered":"2024-09-05T16:45:57.523640+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:45:57.523640+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2651296,"num_objects":324,"num_object_clones":0,"num_object_copies":1296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":324,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12800,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2010}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12505","reported_seq":12622,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:55.730942+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:45:55.730942+0000","last_peered":"2024-09-05T16:45:55.730942+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:45:55.730942+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2684064,"num_objects":328,"num_object_clones":0,"num_object_copies":1312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":328,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12503,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1980}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'12440","reported_seq":12554,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:56.138550+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:45:56.138550+0000","last_peered":"2024-09-05T16:45:56.138550+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:45:56.138550+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"33'2400","ondisk_log_start":"33'2400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2741408,"num_objects":335,"num_object_clones":0,"num_object_copies":1340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":335,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12438,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1885}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":41631048,"num_objects":5087,"num_object_clones":0,"num_object_copies":20348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5087,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":175063,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1351,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":83632128,"data_stored":83632128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160872,"ondisk_log_size":160872,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084431,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":724156,"kb_used_data":20524,"kb_used_omap":4,"kb_used_meta":703611,"kb_avail":93647684,"statfs":{"total":96636764160,"available":95895228416,"internally_reserved":0,"allocated":21016576,"data_stored":20805995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":720498387},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,0,4,7,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986308,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":580520,"kb_used_data":21260,"kb_used_omap":4,"kb_used_meta":559227,"kb_avail":93791320,"statfs":{"total":96636764160,"available":96042311680,"internally_reserved":0,"allocated":21770240,"data_stored":21559659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":572649170},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,2,4,6,3,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673114,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":729404,"kb_used_data":22100,"kb_used_omap":1,"kb_used_meta":707262,"kb_avail":93642436,"statfs":{"total":96636764160,"available":95889854464,"internally_reserved":0,"allocated":22630400,"data_stored":22419819,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":724236741},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,2,3,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182545,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":729260,"kb_used_data":21520,"kb_used_omap":4,"kb_used_meta":707707,"kb_avail":93642580,"statfs":{"total":96636764160,"available":95890001920,"internally_reserved":0,"allocated":22036480,"data_stored":21825899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":724692685},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,2,1,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":21299200,"data_stored":21299200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20656128,"data_stored":20656128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":21651456,"data_stored":21651456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":20025344,"data_stored":20025344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:45:59.517 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-05T16:45:59.517 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "39'12629", 'reported_seq': 14884, 'reported_epoch': 39, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:57.677764+0000', 'last_change': '2024-09-05T16:43:43.436935+0000', 'last_active': '2024-09-05T16:45:57.677764+0000', 'last_peered': '2024-09-05T16:45:57.677764+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.828289+0000', 'last_became_peered': '2024-09-05T16:34:56.828289+0000', 'last_unstale': '2024-09-05T16:45:57.677764+0000', 'last_undegraded': '2024-09-05T16:34:56.612160+0000', 'last_fullsized': '2024-09-05T16:34:56.557632+0000', 'mapping_epoch': 32, 'log_start': "33'2600", 'ondisk_log_start': "33'2600", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10029, 'log_dups_size': 0, 'ondisk_log_size': 10029, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2252800, 'num_objects': 275, 'num_object_clones': 0, 'num_object_copies': 1100, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1164, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 275, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12629, 'num_write_kb': 51616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1098, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -889}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:45:59.517 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:45:59.517 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "39'12429", 'reported_seq': 12557, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:55.687614+0000', 'last_change': '2024-09-05T16:34:56.867900+0000', 'last_active': '2024-09-05T16:45:55.687614+0000', 'last_peered': '2024-09-05T16:45:55.687614+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:45:55.687614+0000', 'last_undegraded': '2024-09-05T16:34:56.590972+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 32, 'log_start': "33'2400", 'ondisk_log_start': "33'2400", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10029, 'log_dups_size': 0, 'ondisk_log_size': 10029, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2644584, 'num_objects': 324, 'num_object_clones': 0, 'num_object_copies': 1296, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2233, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 324, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12428, 'num_write_kb': 50999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1908}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:45:59.518 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:45:59.518 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "39'12291", 'reported_seq': 12395, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:57.523438+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:45:57.523438+0000', 'last_peered': '2024-09-05T16:45:57.523438+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:45:57.523438+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "33'2200", 'ondisk_log_start': "33'2200", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10091, 'log_dups_size': 0, 'ondisk_log_size': 10091, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2487456, 'num_objects': 304, 'num_object_clones': 0, 'num_object_copies': 1216, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2147, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 304, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12289, 'num_write_kb': 50373, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1841}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:45:59.518 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:45:59.518 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "39'12456", 'reported_seq': 12576, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:57.632837+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:45:57.632837+0000', 'last_peered': '2024-09-05T16:45:57.632837+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:45:57.632837+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "33'2400", 'ondisk_log_start': "33'2400", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10056, 'log_dups_size': 0, 'ondisk_log_size': 10056, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2588672, 'num_objects': 316, 'num_object_clones': 0, 'num_object_copies': 1264, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 316, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12456, 'num_write_kb': 51088, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1903}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:45:59.518 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:45:59.518 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "39'12299", 'reported_seq': 12411, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:56.123038+0000', 'last_change': '2024-09-05T16:34:56.903436+0000', 'last_active': '2024-09-05T16:45:56.123038+0000', 'last_peered': '2024-09-05T16:45:56.123038+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:45:56.123038+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "33'2200", 'ondisk_log_start': "33'2200", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10099, 'log_dups_size': 0, 'ondisk_log_size': 10099, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2970784, 'num_objects': 363, 'num_object_clones': 0, 'num_object_copies': 1452, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2133, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 363, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12297, 'num_write_kb': 50636, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -1768}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:45:59.518 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-05T16:45:59.518 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "39'12421", 'reported_seq': 12515, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:56.098245+0000', 'last_change': '2024-09-05T16:34:56.883319+0000', 'last_active': '2024-09-05T16:45:56.098245+0000', 'last_peered': '2024-09-05T16:45:56.098245+0000', 'last_clean': '2024-09-05T16:33:53.505305+0000', 'last_became_active': '2024-09-05T16:34:56.873389+0000', 'last_became_peered': '2024-09-05T16:34:56.873389+0000', 'last_unstale': '2024-09-05T16:45:56.098245+0000', 'last_undegraded': '2024-09-05T16:34:56.602567+0000', 'last_fullsized': '2024-09-05T16:34:56.559958+0000', 'mapping_epoch': 32, 'log_start': "33'2400", 'ondisk_log_start': "33'2400", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10021, 'log_dups_size': 0, 'ondisk_log_size': 10021, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2929824, 'num_objects': 358, 'num_object_clones': 0, 'num_object_copies': 1432, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2234, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 358, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12419, 'num_write_kb': 51108, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1874}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:45:59.518 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:45:59.518 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "39'12497", 'reported_seq': 12603, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:57.464241+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:45:57.464241+0000', 'last_peered': '2024-09-05T16:45:57.464241+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:45:57.464241+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "33'2400", 'ondisk_log_start': "33'2400", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10097, 'log_dups_size': 0, 'ondisk_log_size': 10097, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2263360, 'num_objects': 277, 'num_object_clones': 0, 'num_object_copies': 1108, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 277, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12494, 'num_write_kb': 51081, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1967}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:45:59.518 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:45:59.519 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "39'12479", 'reported_seq': 12606, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:55.507341+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:45:55.507341+0000', 'last_peered': '2024-09-05T16:45:55.507341+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:45:55.507341+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "33'2400", 'ondisk_log_start': "33'2400", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10079, 'log_dups_size': 0, 'ondisk_log_size': 10079, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2708640, 'num_objects': 331, 'num_object_clones': 0, 'num_object_copies': 1324, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 331, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12477, 'num_write_kb': 51230, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1914}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:45:59.519 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:45:59.519 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "39'12793", 'reported_seq': 12924, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:57.650086+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:45:57.650086+0000', 'last_peered': '2024-09-05T16:45:57.650086+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:45:57.650086+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "33'2700", 'ondisk_log_start': "33'2700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10093, 'log_dups_size': 0, 'ondisk_log_size': 10093, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2698992, 'num_objects': 330, 'num_object_clones': 0, 'num_object_copies': 1320, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2284, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 330, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12790, 'num_write_kb': 52476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1951}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:45:59.519 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:45:59.519 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "39'12431", 'reported_seq': 12535, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:57.406588+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:45:57.406588+0000', 'last_peered': '2024-09-05T16:45:57.406588+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:45:57.406588+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "33'2400", 'ondisk_log_start': "33'2400", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10031, 'log_dups_size': 0, 'ondisk_log_size': 10031, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2685520, 'num_objects': 328, 'num_object_clones': 0, 'num_object_copies': 1312, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 328, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12430, 'num_write_kb': 51031, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1910}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:45:59.519 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:45:59.519 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "39'12802", 'reported_seq': 12916, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:57.523640+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:45:57.523640+0000', 'last_peered': '2024-09-05T16:45:57.523640+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:45:57.523640+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "33'2800", 'ondisk_log_start': "33'2800", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10002, 'log_dups_size': 0, 'ondisk_log_size': 10002, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2651296, 'num_objects': 324, 'num_object_clones': 0, 'num_object_copies': 1296, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 324, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12800, 'num_write_kb': 52500, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2010}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:45:59.519 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:45:59.519 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "39'12505", 'reported_seq': 12622, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:55.730942+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:45:55.730942+0000', 'last_peered': '2024-09-05T16:45:55.730942+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:45:55.730942+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "33'2500", 'ondisk_log_start': "33'2500", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10005, 'log_dups_size': 0, 'ondisk_log_size': 10005, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2684064, 'num_objects': 328, 'num_object_clones': 0, 'num_object_copies': 1312, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2310, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 328, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12503, 'num_write_kb': 51325, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1980}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:45:59.519 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:45:59.519 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "39'12440", 'reported_seq': 12554, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:45:56.138550+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:45:56.138550+0000', 'last_peered': '2024-09-05T16:45:56.138550+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:45:56.138550+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "33'2400", 'ondisk_log_start': "33'2400", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2741408, 'num_objects': 335, 'num_object_clones': 0, 'num_object_copies': 1340, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 335, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12438, 'num_write_kb': 51092, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1885}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:45:59.520 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:46:00.242 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:46:00.242 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 28015/20348 objects degraded (137.679%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":763,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":39,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":5087,"data_bytes":41631048,"bytes_used":2829660160,"bytes_avail":383717396480,"bytes_total":386547056640,"degraded_objects":28015,"degraded_total":20348,"degraded_ratio":1.3767937880872814,"recovering_objects_per_sec":11,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":134,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":205},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:46:00.276 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:46:00.276 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:46:01.007 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:46:01.008 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:46:01.031 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":411,"stamp":"2024-09-05T16:45:59.085629+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":41631048,"num_objects":5087,"num_object_clones":0,"num_object_copies":20348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5087,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":175063,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1351,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160872,"ondisk_log_size":160872,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2763340,"kb_used_data":85404,"kb_used_omap":14,"kb_used_meta":2677809,"kb_avail":374724020,"statfs":{"total":386547056640,"available":383717396480,"internally_reserved":0,"allocated":87453696,"data_stored":86611372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2742076983},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,3,2,11,14,10,7,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20185088,"num_objects":-2464,"num_object_clones":0,"num_object_copies":-9856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2464,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2464,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":134,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004336"},"pg_stats":[{"pgid":"1.f","version":"39'12629","reported_seq":14884,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:57.677764+0000","last_change":"2024-09-05T16:43:43.436935+0000","last_active":"2024-09-05T16:45:57.677764+0000","last_peered":"2024-09-05T16:45:57.677764+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:45:57.677764+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2252800,"num_objects":275,"num_object_clones":0,"num_object_copies":1100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":275,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12629,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1098,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-889}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12262","reported_seq":16918,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:45:56.846833+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:45:56.846833+0000","last_peered":"2024-09-05T16:45:56.846833+0000","last_clean":"2024-09-05T16:45:56.846833+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:45:56.846833+0000","last_undegraded":"2024-09-05T16:45:56.846833+0000","last_fullsized":"2024-09-05T16:45:56.846833+0000","mapping_epoch":38,"log_start":"29'2200","ondisk_log_start":"29'2200","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2375680,"num_objects":290,"num_object_clones":0,"num_object_copies":1160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":290,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1575,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12429","reported_seq":12557,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:55.687614+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:45:55.687614+0000","last_peered":"2024-09-05T16:45:55.687614+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:45:55.687614+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"33'2400","ondisk_log_start":"33'2400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2644584,"num_objects":324,"num_object_clones":0,"num_object_copies":1296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":324,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12428,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1908}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12652","reported_seq":17387,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:45:56.870849+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:45:56.870849+0000","last_peered":"2024-09-05T16:45:56.870849+0000","last_clean":"2024-09-05T16:45:56.870849+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:45:56.870849+0000","last_undegraded":"2024-09-05T16:45:56.870849+0000","last_fullsized":"2024-09-05T16:45:56.870849+0000","mapping_epoch":34,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2310144,"num_objects":282,"num_object_clones":0,"num_object_copies":1128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":282,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6952,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12291","reported_seq":12395,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:57.523438+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:45:57.523438+0000","last_peered":"2024-09-05T16:45:57.523438+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:45:57.523438+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"33'2200","ondisk_log_start":"33'2200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487456,"num_objects":304,"num_object_clones":0,"num_object_copies":1216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":304,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12289,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1841}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12456","reported_seq":12576,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:57.632837+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:45:57.632837+0000","last_peered":"2024-09-05T16:45:57.632837+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:45:57.632837+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"33'2400","ondisk_log_start":"33'2400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2588672,"num_objects":316,"num_object_clones":0,"num_object_copies":1264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":316,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12456,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1903}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12299","reported_seq":12411,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:56.123038+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:45:56.123038+0000","last_peered":"2024-09-05T16:45:56.123038+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:45:56.123038+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"33'2200","ondisk_log_start":"33'2200","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2970784,"num_objects":363,"num_object_clones":0,"num_object_copies":1452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":363,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12297,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1768}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12421","reported_seq":12515,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:56.098245+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:45:56.098245+0000","last_peered":"2024-09-05T16:45:56.098245+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:45:56.098245+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"33'2400","ondisk_log_start":"33'2400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2929824,"num_objects":358,"num_object_clones":0,"num_object_copies":1432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":358,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12419,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1874}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"39'12497","reported_seq":12603,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:57.464241+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:45:57.464241+0000","last_peered":"2024-09-05T16:45:57.464241+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:45:57.464241+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"33'2400","ondisk_log_start":"33'2400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2263360,"num_objects":277,"num_object_clones":0,"num_object_copies":1108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":277,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12494,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1967}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12479","reported_seq":12606,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:55.507341+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:45:55.507341+0000","last_peered":"2024-09-05T16:45:55.507341+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:45:55.507341+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"33'2400","ondisk_log_start":"33'2400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2708640,"num_objects":331,"num_object_clones":0,"num_object_copies":1324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":331,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12477,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1914}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'12793","reported_seq":12924,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:57.650086+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:45:57.650086+0000","last_peered":"2024-09-05T16:45:57.650086+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:45:57.650086+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2698992,"num_objects":330,"num_object_clones":0,"num_object_copies":1320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":330,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12790,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1951}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12431","reported_seq":12535,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:57.406588+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:45:57.406588+0000","last_peered":"2024-09-05T16:45:57.406588+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:45:57.406588+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"33'2400","ondisk_log_start":"33'2400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2685520,"num_objects":328,"num_object_clones":0,"num_object_copies":1312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":328,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12430,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1910}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12486","reported_seq":17227,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:45:56.742316+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:45:56.742316+0000","last_peered":"2024-09-05T16:45:56.742316+0000","last_clean":"2024-09-05T16:45:56.742316+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:45:56.742316+0000","last_undegraded":"2024-09-05T16:45:56.742316+0000","last_fullsized":"2024-09-05T16:45:56.742316+0000","mapping_epoch":36,"log_start":"33'2400","ondisk_log_start":"33'2400","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2637824,"num_objects":322,"num_object_clones":0,"num_object_copies":1288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":322,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4086,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'12802","reported_seq":12916,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:57.523640+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:45:57.523640+0000","last_peered":"2024-09-05T16:45:57.523640+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:45:57.523640+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2651296,"num_objects":324,"num_object_clones":0,"num_object_copies":1296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":324,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12800,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2010}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12505","reported_seq":12622,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:55.730942+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:45:55.730942+0000","last_peered":"2024-09-05T16:45:55.730942+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:45:55.730942+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2684064,"num_objects":328,"num_object_clones":0,"num_object_copies":1312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":328,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12503,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1980}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'12440","reported_seq":12554,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:45:56.138550+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:45:56.138550+0000","last_peered":"2024-09-05T16:45:56.138550+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:45:56.138550+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"33'2400","ondisk_log_start":"33'2400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2741408,"num_objects":335,"num_object_clones":0,"num_object_copies":1340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":335,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12438,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1885}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":41631048,"num_objects":5087,"num_object_clones":0,"num_object_copies":20348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5087,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":175063,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1351,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":83632128,"data_stored":83632128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160872,"ondisk_log_size":160872,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084431,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":724156,"kb_used_data":20524,"kb_used_omap":4,"kb_used_meta":703611,"kb_avail":93647684,"statfs":{"total":96636764160,"available":95895228416,"internally_reserved":0,"allocated":21016576,"data_stored":20805995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":720498387},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,0,4,7,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986308,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":580520,"kb_used_data":21260,"kb_used_omap":4,"kb_used_meta":559227,"kb_avail":93791320,"statfs":{"total":96636764160,"available":96042311680,"internally_reserved":0,"allocated":21770240,"data_stored":21559659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":572649170},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,2,4,6,3,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673114,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":729404,"kb_used_data":22100,"kb_used_omap":1,"kb_used_meta":707262,"kb_avail":93642436,"statfs":{"total":96636764160,"available":95889854464,"internally_reserved":0,"allocated":22630400,"data_stored":22419819,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":724236741},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,2,3,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182545,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":729260,"kb_used_data":21520,"kb_used_omap":4,"kb_used_meta":707707,"kb_avail":93642580,"statfs":{"total":96636764160,"available":95890001920,"internally_reserved":0,"allocated":22036480,"data_stored":21825899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":724692685},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,2,1,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":21299200,"data_stored":21299200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20656128,"data_stored":20656128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":21651456,"data_stored":21651456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":20025344,"data_stored":20025344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:46:04.033 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:46:04.630 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:46:04.631 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:46:04.668 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":413,"stamp":"2024-09-05T16:46:03.087099+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":33062216,"num_objects":4041,"num_object_clones":0,"num_object_copies":16164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27971,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4041,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":176109,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1395,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160918,"ondisk_log_size":160918,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2799948,"kb_used_data":67888,"kb_used_omap":14,"kb_used_meta":2731953,"kb_avail":374687412,"statfs":{"total":386547056640,"available":383679909888,"internally_reserved":0,"allocated":69517312,"data_stored":68674988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2797520439},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,6,3,2,13,10,12,12,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-23330816,"num_objects":-2848,"num_object_clones":0,"num_object_copies":-11392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-142,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2848,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2848,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":142,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004419"},"pg_stats":[{"pgid":"1.f","version":"39'12700","reported_seq":15043,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:02.747780+0000","last_change":"2024-09-05T16:43:43.436935+0000","last_active":"2024-09-05T16:46:02.747780+0000","last_peered":"2024-09-05T16:46:02.747780+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:46:02.747780+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1671168,"num_objects":204,"num_object_clones":0,"num_object_copies":816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1120,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":204,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12700,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1142,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-916}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12314","reported_seq":16969,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:46:02.138604+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:46:02.138604+0000","last_peered":"2024-09-05T16:46:02.138604+0000","last_clean":"2024-09-05T16:46:02.138604+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:46:02.138604+0000","last_undegraded":"2024-09-05T16:46:02.138604+0000","last_fullsized":"2024-09-05T16:46:02.138604+0000","mapping_epoch":38,"log_start":"33'2300","ondisk_log_start":"33'2300","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1949696,"num_objects":238,"num_object_clones":0,"num_object_copies":952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":238,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1627,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12491","reported_seq":12619,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:00.683243+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:46:00.683243+0000","last_peered":"2024-09-05T16:46:00.683243+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:46:00.683243+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"33'2400","ondisk_log_start":"33'2400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2136680,"num_objects":262,"num_object_clones":0,"num_object_copies":1048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":262,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12490,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1970}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12706","reported_seq":17441,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:46:02.106730+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:46:02.106730+0000","last_peered":"2024-09-05T16:46:02.106730+0000","last_clean":"2024-09-05T16:46:02.106730+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:46:02.106730+0000","last_undegraded":"2024-09-05T16:46:02.106730+0000","last_fullsized":"2024-09-05T16:46:02.106730+0000","mapping_epoch":34,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1867776,"num_objects":228,"num_object_clones":0,"num_object_copies":912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":228,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7006,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12346","reported_seq":12449,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:02.751010+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:46:02.751010+0000","last_peered":"2024-09-05T16:46:02.751010+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:46:02.751010+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"33'2300","ondisk_log_start":"33'2300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2036896,"num_objects":249,"num_object_clones":0,"num_object_copies":996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":249,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12344,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1896}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12521","reported_seq":12641,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:02.694606+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:46:02.694606+0000","last_peered":"2024-09-05T16:46:02.694606+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:46:02.694606+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2056192,"num_objects":251,"num_object_clones":0,"num_object_copies":1004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":251,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12521,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1968}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12374","reported_seq":12485,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:01.110395+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:46:01.110395+0000","last_peered":"2024-09-05T16:46:01.110395+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:46:01.110395+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"33'2300","ondisk_log_start":"33'2300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2356384,"num_objects":288,"num_object_clones":0,"num_object_copies":1152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":288,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12372,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1843}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12497","reported_seq":12590,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:01.042944+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:46:01.042944+0000","last_peered":"2024-09-05T16:46:01.042944+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:46:01.042944+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"33'2400","ondisk_log_start":"33'2400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2307232,"num_objects":282,"num_object_clones":0,"num_object_copies":1128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":282,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12495,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1950}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"39'12559","reported_seq":12665,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:02.732694+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:46:02.732694+0000","last_peered":"2024-09-05T16:46:02.732694+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:46:02.732694+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1755456,"num_objects":215,"num_object_clones":0,"num_object_copies":860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":215,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12556,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2029}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12567","reported_seq":12694,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:00.663747+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:46:00.663747+0000","last_peered":"2024-09-05T16:46:00.663747+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:46:00.663747+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1987744,"num_objects":243,"num_object_clones":0,"num_object_copies":972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":243,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12565,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2002}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'12860","reported_seq":12992,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:02.722540+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:46:02.722540+0000","last_peered":"2024-09-05T16:46:02.722540+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:46:02.722540+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2150128,"num_objects":263,"num_object_clones":0,"num_object_copies":1052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":263,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12857,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2018}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12497","reported_seq":12601,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:02.615135+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:46:02.615135+0000","last_peered":"2024-09-05T16:46:02.615135+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:46:02.615135+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"33'2400","ondisk_log_start":"33'2400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2144848,"num_objects":262,"num_object_clones":0,"num_object_copies":1048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":262,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12496,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1976}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12543","reported_seq":17284,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:46:01.881930+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:46:01.881930+0000","last_peered":"2024-09-05T16:46:01.881930+0000","last_clean":"2024-09-05T16:46:01.881930+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:46:01.881930+0000","last_undegraded":"2024-09-05T16:46:01.881930+0000","last_fullsized":"2024-09-05T16:46:01.881930+0000","mapping_epoch":36,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2170880,"num_objects":265,"num_object_clones":0,"num_object_copies":1060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":265,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4143,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'12865","reported_seq":12979,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:02.684506+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:46:02.684506+0000","last_peered":"2024-09-05T16:46:02.684506+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:46:02.684506+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2135200,"num_objects":261,"num_object_clones":0,"num_object_copies":1044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":261,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12863,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2073}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12568","reported_seq":12686,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:00.719076+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:46:00.719076+0000","last_peered":"2024-09-05T16:46:00.719076+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:46:00.719076+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2167968,"num_objects":265,"num_object_clones":0,"num_object_copies":1060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":265,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12566,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2043}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'12510","reported_seq":12624,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:01.150273+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:46:01.150273+0000","last_peered":"2024-09-05T16:46:01.150273+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:46:01.150273+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2167968,"num_objects":265,"num_object_clones":0,"num_object_copies":1060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":265,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12508,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1955}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":33062216,"num_objects":4041,"num_object_clones":0,"num_object_copies":16164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27971,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4041,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":176109,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1395,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":67002368,"data_stored":67002368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160918,"ondisk_log_size":160918,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084432,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":741632,"kb_used_data":15792,"kb_used_omap":4,"kb_used_meta":725819,"kb_avail":93630208,"statfs":{"total":96636764160,"available":95877332992,"internally_reserved":0,"allocated":16171008,"data_stored":15960427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":743239379},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,0,1,5,3,1,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986309,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":579896,"kb_used_data":16476,"kb_used_omap":4,"kb_used_meta":563387,"kb_avail":93791944,"statfs":{"total":96636764160,"available":96042950656,"internally_reserved":0,"allocated":16871424,"data_stored":16660843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":576909010},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,1,2,3,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673115,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":730964,"kb_used_data":18100,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":93640876,"statfs":{"total":96636764160,"available":95888257024,"internally_reserved":0,"allocated":18534400,"data_stored":18323819,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":729938373},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,5,1,0,6,1,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182546,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":747456,"kb_used_data":17520,"kb_used_omap":4,"kb_used_meta":729915,"kb_avail":93624384,"statfs":{"total":96636764160,"available":95871369216,"internally_reserved":0,"allocated":17940480,"data_stored":17729899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":747433677},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,1,4,4,4,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":17215488,"data_stored":17215488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16461824,"data_stored":16461824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":17498112,"data_stored":17498112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":15826944,"data_stored":15826944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:46:04.670 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:46:05.305 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:46:05.306 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:46:05.332 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":414,"stamp":"2024-09-05T16:46:05.087962+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":33062216,"num_objects":4041,"num_object_clones":0,"num_object_copies":16164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27971,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4041,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":176109,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1395,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160918,"ondisk_log_size":160918,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2799948,"kb_used_data":67888,"kb_used_omap":14,"kb_used_meta":2731953,"kb_avail":374687412,"statfs":{"total":386547056640,"available":383679909888,"internally_reserved":0,"allocated":69517312,"data_stored":68674988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2797520439},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,6,3,2,13,10,12,12,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16678912,"num_objects":-2036,"num_object_clones":0,"num_object_copies":-8144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-91,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2036,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004577"},"pg_stats":[{"pgid":"1.f","version":"39'12700","reported_seq":15043,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:02.747780+0000","last_change":"2024-09-05T16:43:43.436935+0000","last_active":"2024-09-05T16:46:02.747780+0000","last_peered":"2024-09-05T16:46:02.747780+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:46:02.747780+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1671168,"num_objects":204,"num_object_clones":0,"num_object_copies":816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1120,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":204,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12700,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1142,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-916}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12314","reported_seq":16969,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:46:02.138604+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:46:02.138604+0000","last_peered":"2024-09-05T16:46:02.138604+0000","last_clean":"2024-09-05T16:46:02.138604+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:46:02.138604+0000","last_undegraded":"2024-09-05T16:46:02.138604+0000","last_fullsized":"2024-09-05T16:46:02.138604+0000","mapping_epoch":38,"log_start":"33'2300","ondisk_log_start":"33'2300","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1949696,"num_objects":238,"num_object_clones":0,"num_object_copies":952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":238,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1627,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12491","reported_seq":12619,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:00.683243+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:46:00.683243+0000","last_peered":"2024-09-05T16:46:00.683243+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:46:00.683243+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"33'2400","ondisk_log_start":"33'2400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2136680,"num_objects":262,"num_object_clones":0,"num_object_copies":1048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":262,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12490,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1970}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12706","reported_seq":17441,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:46:02.106730+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:46:02.106730+0000","last_peered":"2024-09-05T16:46:02.106730+0000","last_clean":"2024-09-05T16:46:02.106730+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:46:02.106730+0000","last_undegraded":"2024-09-05T16:46:02.106730+0000","last_fullsized":"2024-09-05T16:46:02.106730+0000","mapping_epoch":34,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1867776,"num_objects":228,"num_object_clones":0,"num_object_copies":912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":228,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7006,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12346","reported_seq":12449,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:02.751010+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:46:02.751010+0000","last_peered":"2024-09-05T16:46:02.751010+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:46:02.751010+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"33'2300","ondisk_log_start":"33'2300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2036896,"num_objects":249,"num_object_clones":0,"num_object_copies":996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":249,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12344,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1896}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12521","reported_seq":12641,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:02.694606+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:46:02.694606+0000","last_peered":"2024-09-05T16:46:02.694606+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:46:02.694606+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2056192,"num_objects":251,"num_object_clones":0,"num_object_copies":1004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":251,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12521,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1968}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12374","reported_seq":12485,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:01.110395+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:46:01.110395+0000","last_peered":"2024-09-05T16:46:01.110395+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:46:01.110395+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"33'2300","ondisk_log_start":"33'2300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2356384,"num_objects":288,"num_object_clones":0,"num_object_copies":1152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":288,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12372,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1843}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12497","reported_seq":12590,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:01.042944+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:46:01.042944+0000","last_peered":"2024-09-05T16:46:01.042944+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:46:01.042944+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"33'2400","ondisk_log_start":"33'2400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2307232,"num_objects":282,"num_object_clones":0,"num_object_copies":1128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":282,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12495,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1950}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"39'12559","reported_seq":12665,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:02.732694+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:46:02.732694+0000","last_peered":"2024-09-05T16:46:02.732694+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:46:02.732694+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1755456,"num_objects":215,"num_object_clones":0,"num_object_copies":860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":215,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12556,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2029}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12567","reported_seq":12694,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:00.663747+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:46:00.663747+0000","last_peered":"2024-09-05T16:46:00.663747+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:46:00.663747+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1987744,"num_objects":243,"num_object_clones":0,"num_object_copies":972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":243,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12565,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2002}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'12860","reported_seq":12992,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:02.722540+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:46:02.722540+0000","last_peered":"2024-09-05T16:46:02.722540+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:46:02.722540+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2150128,"num_objects":263,"num_object_clones":0,"num_object_copies":1052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":263,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12857,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2018}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12497","reported_seq":12601,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:02.615135+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:46:02.615135+0000","last_peered":"2024-09-05T16:46:02.615135+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:46:02.615135+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"33'2400","ondisk_log_start":"33'2400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2144848,"num_objects":262,"num_object_clones":0,"num_object_copies":1048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":262,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12496,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1976}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12543","reported_seq":17284,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:46:01.881930+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:46:01.881930+0000","last_peered":"2024-09-05T16:46:01.881930+0000","last_clean":"2024-09-05T16:46:01.881930+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:46:01.881930+0000","last_undegraded":"2024-09-05T16:46:01.881930+0000","last_fullsized":"2024-09-05T16:46:01.881930+0000","mapping_epoch":36,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2170880,"num_objects":265,"num_object_clones":0,"num_object_copies":1060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":265,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4143,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'12865","reported_seq":12979,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:02.684506+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:46:02.684506+0000","last_peered":"2024-09-05T16:46:02.684506+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:46:02.684506+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2135200,"num_objects":261,"num_object_clones":0,"num_object_copies":1044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":261,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12863,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2073}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12568","reported_seq":12686,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:00.719076+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:46:00.719076+0000","last_peered":"2024-09-05T16:46:00.719076+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:46:00.719076+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2167968,"num_objects":265,"num_object_clones":0,"num_object_copies":1060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":265,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12566,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2043}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'12510","reported_seq":12624,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:01.150273+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:46:01.150273+0000","last_peered":"2024-09-05T16:46:01.150273+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:46:01.150273+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2167968,"num_objects":265,"num_object_clones":0,"num_object_copies":1060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":265,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12508,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1955}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":33062216,"num_objects":4041,"num_object_clones":0,"num_object_copies":16164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27971,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4041,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":176109,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1395,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":67002368,"data_stored":67002368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160918,"ondisk_log_size":160918,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084432,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":741632,"kb_used_data":15792,"kb_used_omap":4,"kb_used_meta":725819,"kb_avail":93630208,"statfs":{"total":96636764160,"available":95877332992,"internally_reserved":0,"allocated":16171008,"data_stored":15960427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":743239379},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,0,1,5,3,1,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986309,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":579896,"kb_used_data":16476,"kb_used_omap":4,"kb_used_meta":563387,"kb_avail":93791944,"statfs":{"total":96636764160,"available":96042950656,"internally_reserved":0,"allocated":16871424,"data_stored":16660843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":576909010},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,1,2,3,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673115,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":730964,"kb_used_data":18100,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":93640876,"statfs":{"total":96636764160,"available":95888257024,"internally_reserved":0,"allocated":18534400,"data_stored":18323819,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":729938373},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,5,1,0,6,1,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182546,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":747456,"kb_used_data":17520,"kb_used_omap":4,"kb_used_meta":729915,"kb_avail":93624384,"statfs":{"total":96636764160,"available":95871369216,"internally_reserved":0,"allocated":17940480,"data_stored":17729899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":747433677},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,1,4,4,4,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":17215488,"data_stored":17215488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16461824,"data_stored":16461824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":17498112,"data_stored":17498112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":15826944,"data_stored":15826944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:46:05.334 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-05T16:46:05.334 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "39'12700", 'reported_seq': 15043, 'reported_epoch': 39, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:02.747780+0000', 'last_change': '2024-09-05T16:43:43.436935+0000', 'last_active': '2024-09-05T16:46:02.747780+0000', 'last_peered': '2024-09-05T16:46:02.747780+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.828289+0000', 'last_became_peered': '2024-09-05T16:34:56.828289+0000', 'last_unstale': '2024-09-05T16:46:02.747780+0000', 'last_undegraded': '2024-09-05T16:34:56.612160+0000', 'last_fullsized': '2024-09-05T16:34:56.557632+0000', 'mapping_epoch': 32, 'log_start': "33'2600", 'ondisk_log_start': "33'2600", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10100, 'log_dups_size': 0, 'ondisk_log_size': 10100, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1671168, 'num_objects': 204, 'num_object_clones': 0, 'num_object_copies': 816, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1120, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 204, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12700, 'num_write_kb': 51616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1142, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -916}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:46:05.334 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:46:05.334 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "39'12491", 'reported_seq': 12619, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:00.683243+0000', 'last_change': '2024-09-05T16:34:56.867900+0000', 'last_active': '2024-09-05T16:46:00.683243+0000', 'last_peered': '2024-09-05T16:46:00.683243+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:46:00.683243+0000', 'last_undegraded': '2024-09-05T16:34:56.590972+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 32, 'log_start': "33'2400", 'ondisk_log_start': "33'2400", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10091, 'log_dups_size': 0, 'ondisk_log_size': 10091, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2136680, 'num_objects': 262, 'num_object_clones': 0, 'num_object_copies': 1048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2233, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 262, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12490, 'num_write_kb': 50999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1970}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:46:05.335 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:46:05.335 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "39'12346", 'reported_seq': 12449, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:02.751010+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:46:02.751010+0000', 'last_peered': '2024-09-05T16:46:02.751010+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:46:02.751010+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "33'2300", 'ondisk_log_start': "33'2300", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10046, 'log_dups_size': 0, 'ondisk_log_size': 10046, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2036896, 'num_objects': 249, 'num_object_clones': 0, 'num_object_copies': 996, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2147, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 249, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12344, 'num_write_kb': 50373, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1896}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:46:05.335 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:46:05.335 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "39'12521", 'reported_seq': 12641, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:02.694606+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:46:02.694606+0000', 'last_peered': '2024-09-05T16:46:02.694606+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:46:02.694606+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "33'2500", 'ondisk_log_start': "33'2500", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10021, 'log_dups_size': 0, 'ondisk_log_size': 10021, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2056192, 'num_objects': 251, 'num_object_clones': 0, 'num_object_copies': 1004, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 251, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12521, 'num_write_kb': 51088, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1968}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:46:05.335 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:46:05.336 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "39'12374", 'reported_seq': 12485, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:01.110395+0000', 'last_change': '2024-09-05T16:34:56.903436+0000', 'last_active': '2024-09-05T16:46:01.110395+0000', 'last_peered': '2024-09-05T16:46:01.110395+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:46:01.110395+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "33'2300", 'ondisk_log_start': "33'2300", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10074, 'log_dups_size': 0, 'ondisk_log_size': 10074, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2356384, 'num_objects': 288, 'num_object_clones': 0, 'num_object_copies': 1152, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2133, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 288, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12372, 'num_write_kb': 50636, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -1843}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:46:05.336 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-05T16:46:05.336 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "39'12497", 'reported_seq': 12590, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:01.042944+0000', 'last_change': '2024-09-05T16:34:56.883319+0000', 'last_active': '2024-09-05T16:46:01.042944+0000', 'last_peered': '2024-09-05T16:46:01.042944+0000', 'last_clean': '2024-09-05T16:33:53.505305+0000', 'last_became_active': '2024-09-05T16:34:56.873389+0000', 'last_became_peered': '2024-09-05T16:34:56.873389+0000', 'last_unstale': '2024-09-05T16:46:01.042944+0000', 'last_undegraded': '2024-09-05T16:34:56.602567+0000', 'last_fullsized': '2024-09-05T16:34:56.559958+0000', 'mapping_epoch': 32, 'log_start': "33'2400", 'ondisk_log_start': "33'2400", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10097, 'log_dups_size': 0, 'ondisk_log_size': 10097, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2307232, 'num_objects': 282, 'num_object_clones': 0, 'num_object_copies': 1128, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2234, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 282, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12495, 'num_write_kb': 51108, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1950}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:46:05.336 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:46:05.336 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "39'12559", 'reported_seq': 12665, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:02.732694+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:46:02.732694+0000', 'last_peered': '2024-09-05T16:46:02.732694+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:46:02.732694+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "33'2500", 'ondisk_log_start': "33'2500", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10059, 'log_dups_size': 0, 'ondisk_log_size': 10059, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1755456, 'num_objects': 215, 'num_object_clones': 0, 'num_object_copies': 860, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 215, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12556, 'num_write_kb': 51081, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2029}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:46:05.337 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:46:05.337 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "39'12567", 'reported_seq': 12694, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:00.663747+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:46:00.663747+0000', 'last_peered': '2024-09-05T16:46:00.663747+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:46:00.663747+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "33'2500", 'ondisk_log_start': "33'2500", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10067, 'log_dups_size': 0, 'ondisk_log_size': 10067, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1987744, 'num_objects': 243, 'num_object_clones': 0, 'num_object_copies': 972, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 243, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12565, 'num_write_kb': 51230, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -2002}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:46:05.337 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:46:05.337 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "39'12860", 'reported_seq': 12992, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:02.722540+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:46:02.722540+0000', 'last_peered': '2024-09-05T16:46:02.722540+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:46:02.722540+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "33'2800", 'ondisk_log_start': "33'2800", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10060, 'log_dups_size': 0, 'ondisk_log_size': 10060, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2150128, 'num_objects': 263, 'num_object_clones': 0, 'num_object_copies': 1052, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2284, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 263, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12857, 'num_write_kb': 52476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2018}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:46:05.338 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:46:05.338 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "39'12497", 'reported_seq': 12601, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:02.615135+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:46:02.615135+0000', 'last_peered': '2024-09-05T16:46:02.615135+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:46:02.615135+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "33'2400", 'ondisk_log_start': "33'2400", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10097, 'log_dups_size': 0, 'ondisk_log_size': 10097, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2144848, 'num_objects': 262, 'num_object_clones': 0, 'num_object_copies': 1048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 262, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12496, 'num_write_kb': 51031, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1976}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:46:05.338 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:46:05.338 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "39'12865", 'reported_seq': 12979, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:02.684506+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:46:02.684506+0000', 'last_peered': '2024-09-05T16:46:02.684506+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:46:02.684506+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "33'2800", 'ondisk_log_start': "33'2800", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10065, 'log_dups_size': 0, 'ondisk_log_size': 10065, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2135200, 'num_objects': 261, 'num_object_clones': 0, 'num_object_copies': 1044, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 261, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12863, 'num_write_kb': 52500, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2073}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:46:05.338 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:46:05.339 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "39'12568", 'reported_seq': 12686, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:00.719076+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:46:00.719076+0000', 'last_peered': '2024-09-05T16:46:00.719076+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:46:00.719076+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "33'2500", 'ondisk_log_start': "33'2500", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10068, 'log_dups_size': 0, 'ondisk_log_size': 10068, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2167968, 'num_objects': 265, 'num_object_clones': 0, 'num_object_copies': 1060, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2310, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 265, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12566, 'num_write_kb': 51325, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -2043}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:46:05.339 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:46:05.339 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "39'12510", 'reported_seq': 12624, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:01.150273+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:46:01.150273+0000', 'last_peered': '2024-09-05T16:46:01.150273+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:46:01.150273+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "33'2500", 'ondisk_log_start': "33'2500", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10010, 'log_dups_size': 0, 'ondisk_log_size': 10010, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2167968, 'num_objects': 265, 'num_object_clones': 0, 'num_object_copies': 1060, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 265, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12508, 'num_write_kb': 51092, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1955}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:46:05.340 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:46:06.069 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:46:06.070 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27971/16164 objects degraded (173.045%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":769,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":39,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":4041,"data_bytes":33062216,"bytes_used":2867146752,"bytes_avail":383679909888,"bytes_total":386547056640,"degraded_objects":27971,"degraded_total":16164,"degraded_ratio":1.7304503835684237,"recovering_objects_per_sec":7,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":169},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:46:06.087 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:46:06.087 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:46:06.734 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:46:06.735 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:46:06.770 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":414,"stamp":"2024-09-05T16:46:05.087962+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":33062216,"num_objects":4041,"num_object_clones":0,"num_object_copies":16164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27971,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4041,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":176109,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1395,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160918,"ondisk_log_size":160918,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2799948,"kb_used_data":67888,"kb_used_omap":14,"kb_used_meta":2731953,"kb_avail":374687412,"statfs":{"total":386547056640,"available":383679909888,"internally_reserved":0,"allocated":69517312,"data_stored":68674988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2797520439},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,6,3,2,13,10,12,12,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16678912,"num_objects":-2036,"num_object_clones":0,"num_object_copies":-8144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-91,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2036,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004577"},"pg_stats":[{"pgid":"1.f","version":"39'12700","reported_seq":15043,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:02.747780+0000","last_change":"2024-09-05T16:43:43.436935+0000","last_active":"2024-09-05T16:46:02.747780+0000","last_peered":"2024-09-05T16:46:02.747780+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:46:02.747780+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1671168,"num_objects":204,"num_object_clones":0,"num_object_copies":816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1120,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":204,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12700,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1142,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-916}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12314","reported_seq":16969,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:46:02.138604+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:46:02.138604+0000","last_peered":"2024-09-05T16:46:02.138604+0000","last_clean":"2024-09-05T16:46:02.138604+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:46:02.138604+0000","last_undegraded":"2024-09-05T16:46:02.138604+0000","last_fullsized":"2024-09-05T16:46:02.138604+0000","mapping_epoch":38,"log_start":"33'2300","ondisk_log_start":"33'2300","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1949696,"num_objects":238,"num_object_clones":0,"num_object_copies":952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":238,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1627,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12491","reported_seq":12619,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:00.683243+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:46:00.683243+0000","last_peered":"2024-09-05T16:46:00.683243+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:46:00.683243+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"33'2400","ondisk_log_start":"33'2400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2136680,"num_objects":262,"num_object_clones":0,"num_object_copies":1048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":262,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12490,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1970}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12706","reported_seq":17441,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:46:02.106730+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:46:02.106730+0000","last_peered":"2024-09-05T16:46:02.106730+0000","last_clean":"2024-09-05T16:46:02.106730+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:46:02.106730+0000","last_undegraded":"2024-09-05T16:46:02.106730+0000","last_fullsized":"2024-09-05T16:46:02.106730+0000","mapping_epoch":34,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1867776,"num_objects":228,"num_object_clones":0,"num_object_copies":912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":228,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7006,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12346","reported_seq":12449,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:02.751010+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:46:02.751010+0000","last_peered":"2024-09-05T16:46:02.751010+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:46:02.751010+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"33'2300","ondisk_log_start":"33'2300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2036896,"num_objects":249,"num_object_clones":0,"num_object_copies":996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":249,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12344,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1896}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12521","reported_seq":12641,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:02.694606+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:46:02.694606+0000","last_peered":"2024-09-05T16:46:02.694606+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:46:02.694606+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2056192,"num_objects":251,"num_object_clones":0,"num_object_copies":1004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":251,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12521,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1968}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12374","reported_seq":12485,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:01.110395+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:46:01.110395+0000","last_peered":"2024-09-05T16:46:01.110395+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:46:01.110395+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"33'2300","ondisk_log_start":"33'2300","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2356384,"num_objects":288,"num_object_clones":0,"num_object_copies":1152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":288,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12372,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1843}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12497","reported_seq":12590,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:01.042944+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:46:01.042944+0000","last_peered":"2024-09-05T16:46:01.042944+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:46:01.042944+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"33'2400","ondisk_log_start":"33'2400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2307232,"num_objects":282,"num_object_clones":0,"num_object_copies":1128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":282,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12495,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1950}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"39'12559","reported_seq":12665,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:02.732694+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:46:02.732694+0000","last_peered":"2024-09-05T16:46:02.732694+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:46:02.732694+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1755456,"num_objects":215,"num_object_clones":0,"num_object_copies":860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":215,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12556,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2029}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12567","reported_seq":12694,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:00.663747+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:46:00.663747+0000","last_peered":"2024-09-05T16:46:00.663747+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:46:00.663747+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1987744,"num_objects":243,"num_object_clones":0,"num_object_copies":972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":243,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12565,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2002}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'12860","reported_seq":12992,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:02.722540+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:46:02.722540+0000","last_peered":"2024-09-05T16:46:02.722540+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:46:02.722540+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2150128,"num_objects":263,"num_object_clones":0,"num_object_copies":1052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":263,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12857,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2018}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12497","reported_seq":12601,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:02.615135+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:46:02.615135+0000","last_peered":"2024-09-05T16:46:02.615135+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:46:02.615135+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"33'2400","ondisk_log_start":"33'2400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2144848,"num_objects":262,"num_object_clones":0,"num_object_copies":1048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":262,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12496,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1976}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12543","reported_seq":17284,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:46:01.881930+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:46:01.881930+0000","last_peered":"2024-09-05T16:46:01.881930+0000","last_clean":"2024-09-05T16:46:01.881930+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:46:01.881930+0000","last_undegraded":"2024-09-05T16:46:01.881930+0000","last_fullsized":"2024-09-05T16:46:01.881930+0000","mapping_epoch":36,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2170880,"num_objects":265,"num_object_clones":0,"num_object_copies":1060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":265,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4143,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'12865","reported_seq":12979,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:02.684506+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:46:02.684506+0000","last_peered":"2024-09-05T16:46:02.684506+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:46:02.684506+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2135200,"num_objects":261,"num_object_clones":0,"num_object_copies":1044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":261,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12863,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2073}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12568","reported_seq":12686,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:00.719076+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:46:00.719076+0000","last_peered":"2024-09-05T16:46:00.719076+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:46:00.719076+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2167968,"num_objects":265,"num_object_clones":0,"num_object_copies":1060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":265,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12566,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2043}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'12510","reported_seq":12624,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:01.150273+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:46:01.150273+0000","last_peered":"2024-09-05T16:46:01.150273+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:46:01.150273+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2167968,"num_objects":265,"num_object_clones":0,"num_object_copies":1060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":265,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12508,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1955}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":33062216,"num_objects":4041,"num_object_clones":0,"num_object_copies":16164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27971,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4041,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":176109,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1395,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":67002368,"data_stored":67002368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160918,"ondisk_log_size":160918,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084432,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":741632,"kb_used_data":15792,"kb_used_omap":4,"kb_used_meta":725819,"kb_avail":93630208,"statfs":{"total":96636764160,"available":95877332992,"internally_reserved":0,"allocated":16171008,"data_stored":15960427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":743239379},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,0,1,5,3,1,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986309,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":579896,"kb_used_data":16476,"kb_used_omap":4,"kb_used_meta":563387,"kb_avail":93791944,"statfs":{"total":96636764160,"available":96042950656,"internally_reserved":0,"allocated":16871424,"data_stored":16660843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":576909010},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,1,2,3,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673115,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":730964,"kb_used_data":18100,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":93640876,"statfs":{"total":96636764160,"available":95888257024,"internally_reserved":0,"allocated":18534400,"data_stored":18323819,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":729938373},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,5,1,0,6,1,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182546,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":747456,"kb_used_data":17520,"kb_used_omap":4,"kb_used_meta":729915,"kb_avail":93624384,"statfs":{"total":96636764160,"available":95871369216,"internally_reserved":0,"allocated":17940480,"data_stored":17729899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":747433677},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,1,4,4,4,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":17215488,"data_stored":17215488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16461824,"data_stored":16461824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":17498112,"data_stored":17498112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":15826944,"data_stored":15826944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:46:09.772 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:46:10.364 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:46:10.365 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:46:10.385 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":416,"stamp":"2024-09-05T16:46:09.089688+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":24837448,"num_objects":3037,"num_object_clones":0,"num_object_copies":12148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27915,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3037,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":177113,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1451,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160722,"ondisk_log_size":160722,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2750712,"kb_used_data":51948,"kb_used_omap":14,"kb_used_meta":2698673,"kb_avail":374736648,"statfs":{"total":386547056640,"available":383730327552,"internally_reserved":0,"allocated":53194752,"data_stored":52352428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2763441719},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,4,2,5,19,8,17,11,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20160512,"num_objects":-2461,"num_object_clones":0,"num_object_copies":-9844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2461,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":147,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004761"},"pg_stats":[{"pgid":"1.f","version":"39'12751","reported_seq":15206,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:07.929177+0000","last_change":"2024-09-05T16:43:43.436935+0000","last_active":"2024-09-05T16:46:07.929177+0000","last_peered":"2024-09-05T16:46:07.929177+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:46:07.929177+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1253376,"num_objects":153,"num_object_clones":0,"num_object_copies":612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1064,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":153,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12751,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1198,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-911}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12367","reported_seq":17023,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:46:07.189314+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:46:07.189314+0000","last_peered":"2024-09-05T16:46:07.189314+0000","last_clean":"2024-09-05T16:46:07.189314+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:46:07.189314+0000","last_undegraded":"2024-09-05T16:46:07.189314+0000","last_fullsized":"2024-09-05T16:46:07.189314+0000","mapping_epoch":38,"log_start":"33'2300","ondisk_log_start":"33'2300","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1515520,"num_objects":185,"num_object_clones":0,"num_object_copies":740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":185,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1680,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12566","reported_seq":12694,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:05.772788+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:46:05.772788+0000","last_peered":"2024-09-05T16:46:05.772788+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:46:05.772788+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1522280,"num_objects":187,"num_object_clones":0,"num_object_copies":748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":187,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12565,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2045}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12763","reported_seq":17498,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:46:07.181678+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:46:07.181678+0000","last_peered":"2024-09-05T16:46:07.181678+0000","last_clean":"2024-09-05T16:46:07.181678+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:46:07.181678+0000","last_undegraded":"2024-09-05T16:46:07.181678+0000","last_fullsized":"2024-09-05T16:46:07.181678+0000","mapping_epoch":34,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1400832,"num_objects":171,"num_object_clones":0,"num_object_copies":684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":171,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7063,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12418","reported_seq":12522,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:07.798018+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:46:07.798018+0000","last_peered":"2024-09-05T16:46:07.798018+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:46:07.798018+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"33'2400","ondisk_log_start":"33'2400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1447072,"num_objects":177,"num_object_clones":0,"num_object_copies":708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12416,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1968}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12581","reported_seq":12700,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:07.731461+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:46:07.731461+0000","last_peered":"2024-09-05T16:46:07.731461+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:46:07.731461+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564672,"num_objects":191,"num_object_clones":0,"num_object_copies":764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":191,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12581,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2028}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12456","reported_seq":12569,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:06.179134+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:46:06.179134+0000","last_peered":"2024-09-05T16:46:06.179134+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:46:06.179134+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"33'2400","ondisk_log_start":"33'2400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1684640,"num_objects":206,"num_object_clones":0,"num_object_copies":824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":206,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12454,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1925}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12571","reported_seq":12660,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:06.099847+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:46:06.099847+0000","last_peered":"2024-09-05T16:46:06.099847+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:46:06.099847+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1701024,"num_objects":208,"num_object_clones":0,"num_object_copies":832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":208,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12569,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2024}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"39'12614","reported_seq":12720,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:07.947644+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:46:07.947644+0000","last_peered":"2024-09-05T16:46:07.947644+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:46:07.947644+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1304896,"num_objects":160,"num_object_clones":0,"num_object_copies":640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":160,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12611,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2084}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12622","reported_seq":12749,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:05.644555+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:46:05.644555+0000","last_peered":"2024-09-05T16:46:05.644555+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:46:05.644555+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1537184,"num_objects":188,"num_object_clones":0,"num_object_copies":752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":188,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12620,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2057}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'12919","reported_seq":13050,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:07.929927+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:46:07.929927+0000","last_peered":"2024-09-05T16:46:07.929927+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:46:07.929927+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1666800,"num_objects":204,"num_object_clones":0,"num_object_copies":816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":204,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12916,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2077}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12542","reported_seq":12646,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:07.773194+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:46:07.773194+0000","last_peered":"2024-09-05T16:46:07.773194+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:46:07.773194+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1776208,"num_objects":217,"num_object_clones":0,"num_object_copies":868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":217,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12541,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2021}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12616","reported_seq":17357,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:46:07.231712+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:46:07.231712+0000","last_peered":"2024-09-05T16:46:07.231712+0000","last_clean":"2024-09-05T16:46:07.231712+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:46:07.231712+0000","last_undegraded":"2024-09-05T16:46:07.231712+0000","last_fullsized":"2024-09-05T16:46:07.231712+0000","mapping_epoch":36,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1572864,"num_objects":192,"num_object_clones":0,"num_object_copies":768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":192,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4216,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'12931","reported_seq":13042,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:07.883489+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:46:07.883489+0000","last_peered":"2024-09-05T16:46:07.883489+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:46:07.883489+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1594528,"num_objects":195,"num_object_clones":0,"num_object_copies":780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":195,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12929,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2139}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12633","reported_seq":12750,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:05.611766+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:46:05.611766+0000","last_peered":"2024-09-05T16:46:05.611766+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:46:05.611766+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1635488,"num_objects":200,"num_object_clones":0,"num_object_copies":800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":200,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12631,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2108}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'12572","reported_seq":12683,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:06.177018+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:46:06.177018+0000","last_peered":"2024-09-05T16:46:06.177018+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:46:06.177018+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1660064,"num_objects":203,"num_object_clones":0,"num_object_copies":812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":203,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12570,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2017}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":24837448,"num_objects":3037,"num_object_clones":0,"num_object_copies":12148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27915,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3037,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":177113,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1451,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":50520064,"data_stored":50520064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160722,"ondisk_log_size":160722,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084433,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":725248,"kb_used_data":11888,"kb_used_omap":4,"kb_used_meta":713339,"kb_avail":93646592,"statfs":{"total":96636764160,"available":95894110208,"internally_reserved":0,"allocated":12173312,"data_stored":11962731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":730459859},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,6,1,5,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986310,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":563452,"kb_used_data":12524,"kb_used_omap":4,"kb_used_meta":550907,"kb_avail":93808388,"statfs":{"total":96636764160,"available":96059789312,"internally_reserved":0,"allocated":12824576,"data_stored":12613995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":564129490},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,0,1,6,1,6,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673116,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731096,"kb_used_data":14080,"kb_used_omap":1,"kb_used_meta":716990,"kb_avail":93640744,"statfs":{"total":96636764160,"available":95888121856,"internally_reserved":0,"allocated":14417920,"data_stored":14207339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":734198213},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,1,3,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182547,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":730916,"kb_used_data":13456,"kb_used_omap":4,"kb_used_meta":717435,"kb_avail":93640924,"statfs":{"total":96636764160,"available":95888306176,"internally_reserved":0,"allocated":13778944,"data_stored":13568363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":734654157},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,6,3,3,2,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13119488,"data_stored":13119488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12349440,"data_stored":12349440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":13393920,"data_stored":13393920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":11657216,"data_stored":11657216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:46:10.386 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:46:11.012 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:46:11.012 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:46:11.036 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":416,"stamp":"2024-09-05T16:46:09.089688+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":24837448,"num_objects":3037,"num_object_clones":0,"num_object_copies":12148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27915,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3037,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":177113,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1451,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160722,"ondisk_log_size":160722,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2750712,"kb_used_data":51948,"kb_used_omap":14,"kb_used_meta":2698673,"kb_avail":374736648,"statfs":{"total":386547056640,"available":383730327552,"internally_reserved":0,"allocated":53194752,"data_stored":52352428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2763441719},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,4,2,5,19,8,17,11,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20160512,"num_objects":-2461,"num_object_clones":0,"num_object_copies":-9844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2461,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":147,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004761"},"pg_stats":[{"pgid":"1.f","version":"39'12751","reported_seq":15206,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:07.929177+0000","last_change":"2024-09-05T16:43:43.436935+0000","last_active":"2024-09-05T16:46:07.929177+0000","last_peered":"2024-09-05T16:46:07.929177+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:46:07.929177+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1253376,"num_objects":153,"num_object_clones":0,"num_object_copies":612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1064,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":153,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12751,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1198,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-911}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12367","reported_seq":17023,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:46:07.189314+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:46:07.189314+0000","last_peered":"2024-09-05T16:46:07.189314+0000","last_clean":"2024-09-05T16:46:07.189314+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:46:07.189314+0000","last_undegraded":"2024-09-05T16:46:07.189314+0000","last_fullsized":"2024-09-05T16:46:07.189314+0000","mapping_epoch":38,"log_start":"33'2300","ondisk_log_start":"33'2300","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1515520,"num_objects":185,"num_object_clones":0,"num_object_copies":740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":185,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1680,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12566","reported_seq":12694,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:05.772788+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:46:05.772788+0000","last_peered":"2024-09-05T16:46:05.772788+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:46:05.772788+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1522280,"num_objects":187,"num_object_clones":0,"num_object_copies":748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":187,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12565,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2045}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12763","reported_seq":17498,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:46:07.181678+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:46:07.181678+0000","last_peered":"2024-09-05T16:46:07.181678+0000","last_clean":"2024-09-05T16:46:07.181678+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:46:07.181678+0000","last_undegraded":"2024-09-05T16:46:07.181678+0000","last_fullsized":"2024-09-05T16:46:07.181678+0000","mapping_epoch":34,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1400832,"num_objects":171,"num_object_clones":0,"num_object_copies":684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":171,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7063,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12418","reported_seq":12522,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:07.798018+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:46:07.798018+0000","last_peered":"2024-09-05T16:46:07.798018+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:46:07.798018+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"33'2400","ondisk_log_start":"33'2400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1447072,"num_objects":177,"num_object_clones":0,"num_object_copies":708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12416,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1968}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12581","reported_seq":12700,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:07.731461+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:46:07.731461+0000","last_peered":"2024-09-05T16:46:07.731461+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:46:07.731461+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564672,"num_objects":191,"num_object_clones":0,"num_object_copies":764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":191,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12581,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2028}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12456","reported_seq":12569,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:06.179134+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:46:06.179134+0000","last_peered":"2024-09-05T16:46:06.179134+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:46:06.179134+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"33'2400","ondisk_log_start":"33'2400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1684640,"num_objects":206,"num_object_clones":0,"num_object_copies":824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":206,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12454,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1925}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12571","reported_seq":12660,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:06.099847+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:46:06.099847+0000","last_peered":"2024-09-05T16:46:06.099847+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:46:06.099847+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1701024,"num_objects":208,"num_object_clones":0,"num_object_copies":832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":208,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12569,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2024}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"39'12614","reported_seq":12720,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:07.947644+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:46:07.947644+0000","last_peered":"2024-09-05T16:46:07.947644+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:46:07.947644+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1304896,"num_objects":160,"num_object_clones":0,"num_object_copies":640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":160,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12611,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2084}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12622","reported_seq":12749,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:05.644555+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:46:05.644555+0000","last_peered":"2024-09-05T16:46:05.644555+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:46:05.644555+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1537184,"num_objects":188,"num_object_clones":0,"num_object_copies":752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":188,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12620,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2057}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'12919","reported_seq":13050,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:07.929927+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:46:07.929927+0000","last_peered":"2024-09-05T16:46:07.929927+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:46:07.929927+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1666800,"num_objects":204,"num_object_clones":0,"num_object_copies":816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":204,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12916,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2077}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12542","reported_seq":12646,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:07.773194+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:46:07.773194+0000","last_peered":"2024-09-05T16:46:07.773194+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:46:07.773194+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1776208,"num_objects":217,"num_object_clones":0,"num_object_copies":868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":217,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12541,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2021}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12616","reported_seq":17357,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:46:07.231712+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:46:07.231712+0000","last_peered":"2024-09-05T16:46:07.231712+0000","last_clean":"2024-09-05T16:46:07.231712+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:46:07.231712+0000","last_undegraded":"2024-09-05T16:46:07.231712+0000","last_fullsized":"2024-09-05T16:46:07.231712+0000","mapping_epoch":36,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1572864,"num_objects":192,"num_object_clones":0,"num_object_copies":768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":192,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4216,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'12931","reported_seq":13042,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:07.883489+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:46:07.883489+0000","last_peered":"2024-09-05T16:46:07.883489+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:46:07.883489+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1594528,"num_objects":195,"num_object_clones":0,"num_object_copies":780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":195,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12929,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2139}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12633","reported_seq":12750,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:05.611766+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:46:05.611766+0000","last_peered":"2024-09-05T16:46:05.611766+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:46:05.611766+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1635488,"num_objects":200,"num_object_clones":0,"num_object_copies":800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":200,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12631,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2108}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'12572","reported_seq":12683,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:06.177018+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:46:06.177018+0000","last_peered":"2024-09-05T16:46:06.177018+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:46:06.177018+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1660064,"num_objects":203,"num_object_clones":0,"num_object_copies":812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":203,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12570,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2017}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":24837448,"num_objects":3037,"num_object_clones":0,"num_object_copies":12148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27915,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3037,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":177113,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1451,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":50520064,"data_stored":50520064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160722,"ondisk_log_size":160722,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084433,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":725248,"kb_used_data":11888,"kb_used_omap":4,"kb_used_meta":713339,"kb_avail":93646592,"statfs":{"total":96636764160,"available":95894110208,"internally_reserved":0,"allocated":12173312,"data_stored":11962731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":730459859},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,6,1,5,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986310,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":563452,"kb_used_data":12524,"kb_used_omap":4,"kb_used_meta":550907,"kb_avail":93808388,"statfs":{"total":96636764160,"available":96059789312,"internally_reserved":0,"allocated":12824576,"data_stored":12613995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":564129490},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,0,1,6,1,6,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673116,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731096,"kb_used_data":14080,"kb_used_omap":1,"kb_used_meta":716990,"kb_avail":93640744,"statfs":{"total":96636764160,"available":95888121856,"internally_reserved":0,"allocated":14417920,"data_stored":14207339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":734198213},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,1,3,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182547,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":730916,"kb_used_data":13456,"kb_used_omap":4,"kb_used_meta":717435,"kb_avail":93640924,"statfs":{"total":96636764160,"available":95888306176,"internally_reserved":0,"allocated":13778944,"data_stored":13568363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":734654157},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,6,3,3,2,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13119488,"data_stored":13119488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12349440,"data_stored":12349440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":13393920,"data_stored":13393920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":11657216,"data_stored":11657216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:46:11.037 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-05T16:46:11.037 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "39'12751", 'reported_seq': 15206, 'reported_epoch': 39, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:07.929177+0000', 'last_change': '2024-09-05T16:43:43.436935+0000', 'last_active': '2024-09-05T16:46:07.929177+0000', 'last_peered': '2024-09-05T16:46:07.929177+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.828289+0000', 'last_became_peered': '2024-09-05T16:34:56.828289+0000', 'last_unstale': '2024-09-05T16:46:07.929177+0000', 'last_undegraded': '2024-09-05T16:34:56.612160+0000', 'last_fullsized': '2024-09-05T16:34:56.557632+0000', 'mapping_epoch': 32, 'log_start': "33'2700", 'ondisk_log_start': "33'2700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10051, 'log_dups_size': 0, 'ondisk_log_size': 10051, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1253376, 'num_objects': 153, 'num_object_clones': 0, 'num_object_copies': 612, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1064, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 153, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12751, 'num_write_kb': 51616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1198, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -911}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:46:11.038 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:46:11.038 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "39'12566", 'reported_seq': 12694, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:05.772788+0000', 'last_change': '2024-09-05T16:34:56.867900+0000', 'last_active': '2024-09-05T16:46:05.772788+0000', 'last_peered': '2024-09-05T16:46:05.772788+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:46:05.772788+0000', 'last_undegraded': '2024-09-05T16:34:56.590972+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 32, 'log_start': "33'2500", 'ondisk_log_start': "33'2500", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10066, 'log_dups_size': 0, 'ondisk_log_size': 10066, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1522280, 'num_objects': 187, 'num_object_clones': 0, 'num_object_copies': 748, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2233, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 187, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12565, 'num_write_kb': 50999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -2045}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:46:11.038 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:46:11.038 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "39'12418", 'reported_seq': 12522, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:07.798018+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:46:07.798018+0000', 'last_peered': '2024-09-05T16:46:07.798018+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:46:07.798018+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "33'2400", 'ondisk_log_start': "33'2400", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10018, 'log_dups_size': 0, 'ondisk_log_size': 10018, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1447072, 'num_objects': 177, 'num_object_clones': 0, 'num_object_copies': 708, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2147, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 177, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12416, 'num_write_kb': 50373, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1968}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:46:11.038 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:46:11.038 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "39'12581", 'reported_seq': 12700, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:07.731461+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:46:07.731461+0000', 'last_peered': '2024-09-05T16:46:07.731461+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:46:07.731461+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "33'2500", 'ondisk_log_start': "33'2500", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10081, 'log_dups_size': 0, 'ondisk_log_size': 10081, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1564672, 'num_objects': 191, 'num_object_clones': 0, 'num_object_copies': 764, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 191, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12581, 'num_write_kb': 51088, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2028}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:46:11.038 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:46:11.038 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "39'12456", 'reported_seq': 12569, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:06.179134+0000', 'last_change': '2024-09-05T16:34:56.903436+0000', 'last_active': '2024-09-05T16:46:06.179134+0000', 'last_peered': '2024-09-05T16:46:06.179134+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:46:06.179134+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "33'2400", 'ondisk_log_start': "33'2400", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10056, 'log_dups_size': 0, 'ondisk_log_size': 10056, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1684640, 'num_objects': 206, 'num_object_clones': 0, 'num_object_copies': 824, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2133, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 206, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12454, 'num_write_kb': 50636, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -1925}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:46:11.038 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-05T16:46:11.038 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "39'12571", 'reported_seq': 12660, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:06.099847+0000', 'last_change': '2024-09-05T16:34:56.883319+0000', 'last_active': '2024-09-05T16:46:06.099847+0000', 'last_peered': '2024-09-05T16:46:06.099847+0000', 'last_clean': '2024-09-05T16:33:53.505305+0000', 'last_became_active': '2024-09-05T16:34:56.873389+0000', 'last_became_peered': '2024-09-05T16:34:56.873389+0000', 'last_unstale': '2024-09-05T16:46:06.099847+0000', 'last_undegraded': '2024-09-05T16:34:56.602567+0000', 'last_fullsized': '2024-09-05T16:34:56.559958+0000', 'mapping_epoch': 32, 'log_start': "33'2500", 'ondisk_log_start': "33'2500", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10071, 'log_dups_size': 0, 'ondisk_log_size': 10071, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1701024, 'num_objects': 208, 'num_object_clones': 0, 'num_object_copies': 832, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2234, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 208, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12569, 'num_write_kb': 51108, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2024}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:46:11.039 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:46:11.039 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "39'12614", 'reported_seq': 12720, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:07.947644+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:46:07.947644+0000', 'last_peered': '2024-09-05T16:46:07.947644+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:46:07.947644+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "33'2600", 'ondisk_log_start': "33'2600", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10014, 'log_dups_size': 0, 'ondisk_log_size': 10014, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1304896, 'num_objects': 160, 'num_object_clones': 0, 'num_object_copies': 640, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 160, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12611, 'num_write_kb': 51081, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2084}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:46:11.039 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:46:11.039 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "39'12622", 'reported_seq': 12749, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:05.644555+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:46:05.644555+0000', 'last_peered': '2024-09-05T16:46:05.644555+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:46:05.644555+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "33'2600", 'ondisk_log_start': "33'2600", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10022, 'log_dups_size': 0, 'ondisk_log_size': 10022, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1537184, 'num_objects': 188, 'num_object_clones': 0, 'num_object_copies': 752, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 188, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12620, 'num_write_kb': 51230, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -2057}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:46:11.039 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:46:11.039 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "39'12919", 'reported_seq': 13050, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:07.929927+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:46:07.929927+0000', 'last_peered': '2024-09-05T16:46:07.929927+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:46:07.929927+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "33'2900", 'ondisk_log_start': "33'2900", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10019, 'log_dups_size': 0, 'ondisk_log_size': 10019, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1666800, 'num_objects': 204, 'num_object_clones': 0, 'num_object_copies': 816, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2284, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 204, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12916, 'num_write_kb': 52476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2077}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:46:11.039 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:46:11.039 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "39'12542", 'reported_seq': 12646, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:07.773194+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:46:07.773194+0000', 'last_peered': '2024-09-05T16:46:07.773194+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:46:07.773194+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "33'2500", 'ondisk_log_start': "33'2500", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10042, 'log_dups_size': 0, 'ondisk_log_size': 10042, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1776208, 'num_objects': 217, 'num_object_clones': 0, 'num_object_copies': 868, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 217, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12541, 'num_write_kb': 51031, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2021}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:46:11.039 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:46:11.039 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "39'12931", 'reported_seq': 13042, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:07.883489+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:46:07.883489+0000', 'last_peered': '2024-09-05T16:46:07.883489+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:46:07.883489+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "33'2900", 'ondisk_log_start': "33'2900", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10031, 'log_dups_size': 0, 'ondisk_log_size': 10031, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1594528, 'num_objects': 195, 'num_object_clones': 0, 'num_object_copies': 780, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 195, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12929, 'num_write_kb': 52500, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2139}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:46:11.040 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:46:11.040 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "39'12633", 'reported_seq': 12750, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:05.611766+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:46:05.611766+0000', 'last_peered': '2024-09-05T16:46:05.611766+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:46:05.611766+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "33'2600", 'ondisk_log_start': "33'2600", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10033, 'log_dups_size': 0, 'ondisk_log_size': 10033, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1635488, 'num_objects': 200, 'num_object_clones': 0, 'num_object_copies': 800, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2310, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 200, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12631, 'num_write_kb': 51325, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -2108}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:46:11.040 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:46:11.040 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "39'12572", 'reported_seq': 12683, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:06.177018+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:46:06.177018+0000', 'last_peered': '2024-09-05T16:46:06.177018+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:46:06.177018+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "33'2500", 'ondisk_log_start': "33'2500", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10072, 'log_dups_size': 0, 'ondisk_log_size': 10072, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1660064, 'num_objects': 203, 'num_object_clones': 0, 'num_object_copies': 812, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 203, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12570, 'num_write_kb': 51092, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2017}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:46:11.040 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:46:11.742 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:46:11.742 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27915/11472 objects degraded (243.332%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":774,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":39,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":2868,"data_bytes":23453000,"bytes_used":2816917504,"bytes_avail":383730139136,"bytes_total":386547056640,"degraded_objects":27915,"degraded_total":11472,"degraded_ratio":2.43331589958159,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":100,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":184},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:46:11.770 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:46:11.770 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:46:12.407 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:46:12.408 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:46:12.430 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":417,"stamp":"2024-09-05T16:46:11.090515+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":23453000,"num_objects":2868,"num_object_clones":0,"num_object_copies":11472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27915,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2868,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":177282,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1451,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160791,"ondisk_log_size":160791,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2750896,"kb_used_data":47972,"kb_used_omap":14,"kb_used_meta":2702833,"kb_avail":374736464,"statfs":{"total":386547056640,"available":383730139136,"internally_reserved":0,"allocated":49123328,"data_stored":48281004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2767701559},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,2,3,22,7,16,9,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18178048,"num_objects":-2219,"num_object_clones":0,"num_object_copies":-8876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-100,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2219,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2219,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":100,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004886"},"pg_stats":[{"pgid":"1.f","version":"39'12751","reported_seq":15206,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:07.929177+0000","last_change":"2024-09-05T16:43:43.436935+0000","last_active":"2024-09-05T16:46:07.929177+0000","last_peered":"2024-09-05T16:46:07.929177+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:46:07.929177+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1253376,"num_objects":153,"num_object_clones":0,"num_object_copies":612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1064,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":153,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12751,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1198,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-911}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12367","reported_seq":17023,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:46:07.189314+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:46:07.189314+0000","last_peered":"2024-09-05T16:46:07.189314+0000","last_clean":"2024-09-05T16:46:07.189314+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:46:07.189314+0000","last_undegraded":"2024-09-05T16:46:07.189314+0000","last_fullsized":"2024-09-05T16:46:07.189314+0000","mapping_epoch":38,"log_start":"33'2300","ondisk_log_start":"33'2300","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1515520,"num_objects":185,"num_object_clones":0,"num_object_copies":740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":185,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1680,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12629","reported_seq":12757,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:10.822338+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:46:10.822338+0000","last_peered":"2024-09-05T16:46:10.822338+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:46:10.822338+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1006184,"num_objects":124,"num_object_clones":0,"num_object_copies":496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":124,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12628,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2108}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12763","reported_seq":17498,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:46:07.181678+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:46:07.181678+0000","last_peered":"2024-09-05T16:46:07.181678+0000","last_clean":"2024-09-05T16:46:07.181678+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:46:07.181678+0000","last_undegraded":"2024-09-05T16:46:07.181678+0000","last_fullsized":"2024-09-05T16:46:07.181678+0000","mapping_epoch":34,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1400832,"num_objects":171,"num_object_clones":0,"num_object_copies":684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":171,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7063,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12418","reported_seq":12522,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:07.798018+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:46:07.798018+0000","last_peered":"2024-09-05T16:46:07.798018+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:46:07.798018+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"33'2400","ondisk_log_start":"33'2400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1447072,"num_objects":177,"num_object_clones":0,"num_object_copies":708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12416,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1968}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12581","reported_seq":12700,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:07.731461+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:46:07.731461+0000","last_peered":"2024-09-05T16:46:07.731461+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:46:07.731461+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564672,"num_objects":191,"num_object_clones":0,"num_object_copies":764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":191,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12581,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2028}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12456","reported_seq":12569,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:06.179134+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:46:06.179134+0000","last_peered":"2024-09-05T16:46:06.179134+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:46:06.179134+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"33'2400","ondisk_log_start":"33'2400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1684640,"num_objects":206,"num_object_clones":0,"num_object_copies":824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":206,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12454,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1925}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12571","reported_seq":12660,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:06.099847+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:46:06.099847+0000","last_peered":"2024-09-05T16:46:06.099847+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:46:06.099847+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1701024,"num_objects":208,"num_object_clones":0,"num_object_copies":832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":208,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12569,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2024}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"39'12614","reported_seq":12720,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:07.947644+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:46:07.947644+0000","last_peered":"2024-09-05T16:46:07.947644+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:46:07.947644+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1304896,"num_objects":160,"num_object_clones":0,"num_object_copies":640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":160,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12611,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2084}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12674","reported_seq":12801,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:10.824141+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:46:10.824141+0000","last_peered":"2024-09-05T16:46:10.824141+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:46:10.824141+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1111200,"num_objects":136,"num_object_clones":0,"num_object_copies":544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12672,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2109}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'12919","reported_seq":13050,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:07.929927+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:46:07.929927+0000","last_peered":"2024-09-05T16:46:07.929927+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:46:07.929927+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1666800,"num_objects":204,"num_object_clones":0,"num_object_copies":816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":204,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12916,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2077}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12542","reported_seq":12646,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:07.773194+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:46:07.773194+0000","last_peered":"2024-09-05T16:46:07.773194+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:46:07.773194+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1776208,"num_objects":217,"num_object_clones":0,"num_object_copies":868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":217,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12541,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2021}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12616","reported_seq":17357,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:46:07.231712+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:46:07.231712+0000","last_peered":"2024-09-05T16:46:07.231712+0000","last_clean":"2024-09-05T16:46:07.231712+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:46:07.231712+0000","last_undegraded":"2024-09-05T16:46:07.231712+0000","last_fullsized":"2024-09-05T16:46:07.231712+0000","mapping_epoch":36,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1572864,"num_objects":192,"num_object_clones":0,"num_object_copies":768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":192,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4216,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'12931","reported_seq":13042,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:07.883489+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:46:07.883489+0000","last_peered":"2024-09-05T16:46:07.883489+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:46:07.883489+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1594528,"num_objects":195,"num_object_clones":0,"num_object_copies":780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":195,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12929,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2139}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12687","reported_seq":12805,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:10.505636+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:46:10.505636+0000","last_peered":"2024-09-05T16:46:10.505636+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:46:10.505636+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1193120,"num_objects":146,"num_object_clones":0,"num_object_copies":584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":146,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12685,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2162}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'12572","reported_seq":12683,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:06.177018+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:46:06.177018+0000","last_peered":"2024-09-05T16:46:06.177018+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:46:06.177018+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1660064,"num_objects":203,"num_object_clones":0,"num_object_copies":812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":203,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12570,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2017}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":23453000,"num_objects":2868,"num_object_clones":0,"num_object_copies":11472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27915,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2868,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":177282,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1451,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":46391296,"data_stored":46391296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160791,"ondisk_log_size":160791,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084433,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":725248,"kb_used_data":11888,"kb_used_omap":4,"kb_used_meta":713339,"kb_avail":93646592,"statfs":{"total":96636764160,"available":95894110208,"internally_reserved":0,"allocated":12173312,"data_stored":11962731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":730459859},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,6,1,5,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986310,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":563452,"kb_used_data":12524,"kb_used_omap":4,"kb_used_meta":550907,"kb_avail":93808388,"statfs":{"total":96636764160,"available":96059789312,"internally_reserved":0,"allocated":12824576,"data_stored":12613995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":564129490},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,0,1,6,1,6,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673117,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731280,"kb_used_data":10104,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93640560,"statfs":{"total":96636764160,"available":95887933440,"internally_reserved":0,"allocated":10346496,"data_stored":10135915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182547,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":730916,"kb_used_data":13456,"kb_used_omap":4,"kb_used_meta":717435,"kb_avail":93640924,"statfs":{"total":96636764160,"available":95888306176,"internally_reserved":0,"allocated":13778944,"data_stored":13568363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":734654157},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,6,3,3,2,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13119488,"data_stored":13119488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12349440,"data_stored":12349440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9265152,"data_stored":9265152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":11657216,"data_stored":11657216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:46:15.432 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:46:16.101 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:46:16.101 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:46:16.120 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":419,"stamp":"2024-09-05T16:46:15.092153+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":16637256,"num_objects":2036,"num_object_clones":0,"num_object_copies":8144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27867,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2036,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":178114,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1499,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160923,"ondisk_log_size":160923,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2751420,"kb_used_data":36000,"kb_used_omap":14,"kb_used_meta":2715313,"kb_avail":374735940,"statfs":{"total":386547056640,"available":383729602560,"internally_reserved":0,"allocated":36864000,"data_stored":36021676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2780481079},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,2,0,2,15,11,13,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16424960,"num_objects":-2005,"num_object_clones":0,"num_object_copies":-8020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2005,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2005,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005053"},"pg_stats":[{"pgid":"1.f","version":"39'12793","reported_seq":15341,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:13.040086+0000","last_change":"2024-09-05T16:43:43.436935+0000","last_active":"2024-09-05T16:46:13.040086+0000","last_peered":"2024-09-05T16:46:13.040086+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:46:13.040086+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":909312,"num_objects":111,"num_object_clones":0,"num_object_copies":444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1016,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":111,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12793,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1246,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-905}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12426","reported_seq":17083,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:46:11.985533+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:46:11.985533+0000","last_peered":"2024-09-05T16:46:11.985533+0000","last_clean":"2024-09-05T16:46:11.985533+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:46:11.985533+0000","last_undegraded":"2024-09-05T16:46:11.985533+0000","last_fullsized":"2024-09-05T16:46:11.985533+0000","mapping_epoch":38,"log_start":"33'2400","ondisk_log_start":"33'2400","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1032192,"num_objects":126,"num_object_clones":0,"num_object_copies":504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":126,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1739,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12629","reported_seq":12757,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:10.822338+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:46:10.822338+0000","last_peered":"2024-09-05T16:46:10.822338+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:46:10.822338+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1006184,"num_objects":124,"num_object_clones":0,"num_object_copies":496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":124,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12628,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2108}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12824","reported_seq":17559,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:46:12.445459+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:46:12.445459+0000","last_peered":"2024-09-05T16:46:12.445459+0000","last_clean":"2024-09-05T16:46:12.445459+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:46:12.445459+0000","last_undegraded":"2024-09-05T16:46:12.445459+0000","last_fullsized":"2024-09-05T16:46:12.445459+0000","mapping_epoch":34,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":901120,"num_objects":110,"num_object_clones":0,"num_object_copies":440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":110,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7124,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12491","reported_seq":12595,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:12.967856+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:46:12.967856+0000","last_peered":"2024-09-05T16:46:12.967856+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:46:12.967856+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"33'2400","ondisk_log_start":"33'2400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":849056,"num_objects":104,"num_object_clones":0,"num_object_copies":416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12489,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2041}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12647","reported_seq":12767,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:12.897439+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:46:12.897439+0000","last_peered":"2024-09-05T16:46:12.897439+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:46:12.897439+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1024000,"num_objects":125,"num_object_clones":0,"num_object_copies":500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":125,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12647,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2094}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12529","reported_seq":12642,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:10.962603+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:46:10.962603+0000","last_peered":"2024-09-05T16:46:10.962603+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:46:10.962603+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1086624,"num_objects":133,"num_object_clones":0,"num_object_copies":532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":133,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12527,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1998}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12639","reported_seq":12733,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:11.128490+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:46:11.128490+0000","last_peered":"2024-09-05T16:46:11.128490+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:46:11.128490+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1143968,"num_objects":140,"num_object_clones":0,"num_object_copies":560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":140,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12637,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2092}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"39'12668","reported_seq":12774,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:12.806329+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:46:12.806329+0000","last_peered":"2024-09-05T16:46:12.806329+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:46:12.806329+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":862528,"num_objects":106,"num_object_clones":0,"num_object_copies":424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":106,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12665,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2138}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12674","reported_seq":12801,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:10.824141+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:46:10.824141+0000","last_peered":"2024-09-05T16:46:10.824141+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:46:10.824141+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1111200,"num_objects":136,"num_object_clones":0,"num_object_copies":544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12672,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2109}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'12988","reported_seq":13120,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:12.753677+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:46:12.753677+0000","last_peered":"2024-09-05T16:46:12.753677+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:46:12.753677+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1101552,"num_objects":135,"num_object_clones":0,"num_object_copies":540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":135,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12985,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2146}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12629","reported_seq":12733,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:12.937479+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:46:12.937479+0000","last_peered":"2024-09-05T16:46:12.937479+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:46:12.937479+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1063504,"num_objects":130,"num_object_clones":0,"num_object_copies":520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":130,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12628,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2108}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12662","reported_seq":17403,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:46:12.224952+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:46:12.224952+0000","last_peered":"2024-09-05T16:46:12.224952+0000","last_clean":"2024-09-05T16:46:12.224952+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:46:12.224952+0000","last_undegraded":"2024-09-05T16:46:12.224952+0000","last_fullsized":"2024-09-05T16:46:12.224952+0000","mapping_epoch":36,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1196032,"num_objects":146,"num_object_clones":0,"num_object_copies":584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":146,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4262,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'12996","reported_seq":13110,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:12.914087+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:46:12.914087+0000","last_peered":"2024-09-05T16:46:12.914087+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:46:12.914087+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1062048,"num_objects":130,"num_object_clones":0,"num_object_copies":520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":130,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12994,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2204}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12687","reported_seq":12805,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:10.505636+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:46:10.505636+0000","last_peered":"2024-09-05T16:46:10.505636+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:46:10.505636+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1193120,"num_objects":146,"num_object_clones":0,"num_object_copies":584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":146,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12685,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2162}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'12641","reported_seq":12755,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:11.104277+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:46:11.104277+0000","last_peered":"2024-09-05T16:46:11.104277+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:46:11.104277+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1094816,"num_objects":134,"num_object_clones":0,"num_object_copies":536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":134,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12639,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2086}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":16637256,"num_objects":2036,"num_object_clones":0,"num_object_copies":8144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27867,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2036,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":178114,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1499,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":34103296,"data_stored":34103296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160923,"ondisk_log_size":160923,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084434,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":725484,"kb_used_data":7964,"kb_used_omap":4,"kb_used_meta":717499,"kb_avail":93646356,"statfs":{"total":96636764160,"available":95893868544,"internally_reserved":0,"allocated":8155136,"data_stored":7944555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":734719699},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,3,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986311,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":563636,"kb_used_data":8548,"kb_used_omap":4,"kb_used_meta":555067,"kb_avail":93808204,"statfs":{"total":96636764160,"available":96059600896,"internally_reserved":0,"allocated":8753152,"data_stored":8542571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":568389330},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,0,1,6,1,6,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673117,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731280,"kb_used_data":10104,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93640560,"statfs":{"total":96636764160,"available":95887933440,"internally_reserved":0,"allocated":10346496,"data_stored":10135915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182548,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731020,"kb_used_data":9384,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93640820,"statfs":{"total":96636764160,"available":95888199680,"internally_reserved":0,"allocated":9609216,"data_stored":9398635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":738913997},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,6,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":9035776,"data_stored":9035776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8200192,"data_stored":8200192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9265152,"data_stored":9265152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":7602176,"data_stored":7602176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:46:16.122 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:46:16.681 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:46:16.682 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:46:16.709 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":419,"stamp":"2024-09-05T16:46:15.092153+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":16637256,"num_objects":2036,"num_object_clones":0,"num_object_copies":8144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27867,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2036,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":178114,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1499,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160923,"ondisk_log_size":160923,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2751420,"kb_used_data":36000,"kb_used_omap":14,"kb_used_meta":2715313,"kb_avail":374735940,"statfs":{"total":386547056640,"available":383729602560,"internally_reserved":0,"allocated":36864000,"data_stored":36021676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2780481079},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,2,0,2,15,11,13,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16424960,"num_objects":-2005,"num_object_clones":0,"num_object_copies":-8020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2005,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2005,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005053"},"pg_stats":[{"pgid":"1.f","version":"39'12793","reported_seq":15341,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:13.040086+0000","last_change":"2024-09-05T16:43:43.436935+0000","last_active":"2024-09-05T16:46:13.040086+0000","last_peered":"2024-09-05T16:46:13.040086+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:46:13.040086+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":909312,"num_objects":111,"num_object_clones":0,"num_object_copies":444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1016,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":111,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12793,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1246,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-905}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12426","reported_seq":17083,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:46:11.985533+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:46:11.985533+0000","last_peered":"2024-09-05T16:46:11.985533+0000","last_clean":"2024-09-05T16:46:11.985533+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:46:11.985533+0000","last_undegraded":"2024-09-05T16:46:11.985533+0000","last_fullsized":"2024-09-05T16:46:11.985533+0000","mapping_epoch":38,"log_start":"33'2400","ondisk_log_start":"33'2400","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1032192,"num_objects":126,"num_object_clones":0,"num_object_copies":504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":126,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1739,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12629","reported_seq":12757,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:10.822338+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:46:10.822338+0000","last_peered":"2024-09-05T16:46:10.822338+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:46:10.822338+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1006184,"num_objects":124,"num_object_clones":0,"num_object_copies":496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":124,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12628,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2108}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12824","reported_seq":17559,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:46:12.445459+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:46:12.445459+0000","last_peered":"2024-09-05T16:46:12.445459+0000","last_clean":"2024-09-05T16:46:12.445459+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:46:12.445459+0000","last_undegraded":"2024-09-05T16:46:12.445459+0000","last_fullsized":"2024-09-05T16:46:12.445459+0000","mapping_epoch":34,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":901120,"num_objects":110,"num_object_clones":0,"num_object_copies":440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":110,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7124,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12491","reported_seq":12595,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:12.967856+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:46:12.967856+0000","last_peered":"2024-09-05T16:46:12.967856+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:46:12.967856+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"33'2400","ondisk_log_start":"33'2400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":849056,"num_objects":104,"num_object_clones":0,"num_object_copies":416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12489,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2041}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12647","reported_seq":12767,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:12.897439+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:46:12.897439+0000","last_peered":"2024-09-05T16:46:12.897439+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:46:12.897439+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1024000,"num_objects":125,"num_object_clones":0,"num_object_copies":500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":125,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12647,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2094}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12529","reported_seq":12642,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:10.962603+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:46:10.962603+0000","last_peered":"2024-09-05T16:46:10.962603+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:46:10.962603+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1086624,"num_objects":133,"num_object_clones":0,"num_object_copies":532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":133,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12527,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1998}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12639","reported_seq":12733,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:11.128490+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:46:11.128490+0000","last_peered":"2024-09-05T16:46:11.128490+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:46:11.128490+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1143968,"num_objects":140,"num_object_clones":0,"num_object_copies":560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":140,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12637,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2092}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"39'12668","reported_seq":12774,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:12.806329+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:46:12.806329+0000","last_peered":"2024-09-05T16:46:12.806329+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:46:12.806329+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":862528,"num_objects":106,"num_object_clones":0,"num_object_copies":424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":106,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12665,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2138}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12674","reported_seq":12801,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:10.824141+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:46:10.824141+0000","last_peered":"2024-09-05T16:46:10.824141+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:46:10.824141+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1111200,"num_objects":136,"num_object_clones":0,"num_object_copies":544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12672,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2109}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'12988","reported_seq":13120,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:12.753677+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:46:12.753677+0000","last_peered":"2024-09-05T16:46:12.753677+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:46:12.753677+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1101552,"num_objects":135,"num_object_clones":0,"num_object_copies":540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":135,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12985,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2146}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12629","reported_seq":12733,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:12.937479+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:46:12.937479+0000","last_peered":"2024-09-05T16:46:12.937479+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:46:12.937479+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1063504,"num_objects":130,"num_object_clones":0,"num_object_copies":520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":130,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12628,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2108}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12662","reported_seq":17403,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:46:12.224952+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:46:12.224952+0000","last_peered":"2024-09-05T16:46:12.224952+0000","last_clean":"2024-09-05T16:46:12.224952+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:46:12.224952+0000","last_undegraded":"2024-09-05T16:46:12.224952+0000","last_fullsized":"2024-09-05T16:46:12.224952+0000","mapping_epoch":36,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1196032,"num_objects":146,"num_object_clones":0,"num_object_copies":584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":146,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4262,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'12996","reported_seq":13110,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:12.914087+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:46:12.914087+0000","last_peered":"2024-09-05T16:46:12.914087+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:46:12.914087+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1062048,"num_objects":130,"num_object_clones":0,"num_object_copies":520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":130,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12994,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2204}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12687","reported_seq":12805,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:10.505636+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:46:10.505636+0000","last_peered":"2024-09-05T16:46:10.505636+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:46:10.505636+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1193120,"num_objects":146,"num_object_clones":0,"num_object_copies":584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":146,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12685,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2162}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'12641","reported_seq":12755,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:11.104277+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:46:11.104277+0000","last_peered":"2024-09-05T16:46:11.104277+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:46:11.104277+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1094816,"num_objects":134,"num_object_clones":0,"num_object_copies":536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":134,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12639,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2086}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":16637256,"num_objects":2036,"num_object_clones":0,"num_object_copies":8144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27867,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2036,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":178114,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1499,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":34103296,"data_stored":34103296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160923,"ondisk_log_size":160923,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084434,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":725484,"kb_used_data":7964,"kb_used_omap":4,"kb_used_meta":717499,"kb_avail":93646356,"statfs":{"total":96636764160,"available":95893868544,"internally_reserved":0,"allocated":8155136,"data_stored":7944555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":734719699},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,3,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986311,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":563636,"kb_used_data":8548,"kb_used_omap":4,"kb_used_meta":555067,"kb_avail":93808204,"statfs":{"total":96636764160,"available":96059600896,"internally_reserved":0,"allocated":8753152,"data_stored":8542571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":568389330},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,0,1,6,1,6,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673117,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731280,"kb_used_data":10104,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93640560,"statfs":{"total":96636764160,"available":95887933440,"internally_reserved":0,"allocated":10346496,"data_stored":10135915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182548,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731020,"kb_used_data":9384,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93640820,"statfs":{"total":96636764160,"available":95888199680,"internally_reserved":0,"allocated":9609216,"data_stored":9398635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":738913997},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,6,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":9035776,"data_stored":9035776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8200192,"data_stored":8200192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9265152,"data_stored":9265152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":7602176,"data_stored":7602176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:46:16.710 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-05T16:46:16.710 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "39'12793", 'reported_seq': 15341, 'reported_epoch': 39, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:13.040086+0000', 'last_change': '2024-09-05T16:43:43.436935+0000', 'last_active': '2024-09-05T16:46:13.040086+0000', 'last_peered': '2024-09-05T16:46:13.040086+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.828289+0000', 'last_became_peered': '2024-09-05T16:34:56.828289+0000', 'last_unstale': '2024-09-05T16:46:13.040086+0000', 'last_undegraded': '2024-09-05T16:34:56.612160+0000', 'last_fullsized': '2024-09-05T16:34:56.557632+0000', 'mapping_epoch': 32, 'log_start': "33'2700", 'ondisk_log_start': "33'2700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10093, 'log_dups_size': 0, 'ondisk_log_size': 10093, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 909312, 'num_objects': 111, 'num_object_clones': 0, 'num_object_copies': 444, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1016, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 111, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12793, 'num_write_kb': 51616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1246, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -905}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:46:16.710 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:46:16.710 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "39'12629", 'reported_seq': 12757, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:10.822338+0000', 'last_change': '2024-09-05T16:34:56.867900+0000', 'last_active': '2024-09-05T16:46:10.822338+0000', 'last_peered': '2024-09-05T16:46:10.822338+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:46:10.822338+0000', 'last_undegraded': '2024-09-05T16:34:56.590972+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 32, 'log_start': "33'2600", 'ondisk_log_start': "33'2600", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10029, 'log_dups_size': 0, 'ondisk_log_size': 10029, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1006184, 'num_objects': 124, 'num_object_clones': 0, 'num_object_copies': 496, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2233, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 124, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12628, 'num_write_kb': 50999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -2108}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:46:16.710 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:46:16.710 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "39'12491", 'reported_seq': 12595, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:12.967856+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:46:12.967856+0000', 'last_peered': '2024-09-05T16:46:12.967856+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:46:12.967856+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "33'2400", 'ondisk_log_start': "33'2400", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10091, 'log_dups_size': 0, 'ondisk_log_size': 10091, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 849056, 'num_objects': 104, 'num_object_clones': 0, 'num_object_copies': 416, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2147, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 104, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12489, 'num_write_kb': 50373, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2041}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:46:16.710 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:46:16.710 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "39'12647", 'reported_seq': 12767, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:12.897439+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:46:12.897439+0000', 'last_peered': '2024-09-05T16:46:12.897439+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:46:12.897439+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "33'2600", 'ondisk_log_start': "33'2600", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10047, 'log_dups_size': 0, 'ondisk_log_size': 10047, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1024000, 'num_objects': 125, 'num_object_clones': 0, 'num_object_copies': 500, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 125, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12647, 'num_write_kb': 51088, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2094}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:46:16.710 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:46:16.711 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "39'12529", 'reported_seq': 12642, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:10.962603+0000', 'last_change': '2024-09-05T16:34:56.903436+0000', 'last_active': '2024-09-05T16:46:10.962603+0000', 'last_peered': '2024-09-05T16:46:10.962603+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:46:10.962603+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "33'2500", 'ondisk_log_start': "33'2500", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10029, 'log_dups_size': 0, 'ondisk_log_size': 10029, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1086624, 'num_objects': 133, 'num_object_clones': 0, 'num_object_copies': 532, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2133, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 133, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12527, 'num_write_kb': 50636, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -1998}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:46:16.711 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-05T16:46:16.711 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "39'12639", 'reported_seq': 12733, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:11.128490+0000', 'last_change': '2024-09-05T16:34:56.883319+0000', 'last_active': '2024-09-05T16:46:11.128490+0000', 'last_peered': '2024-09-05T16:46:11.128490+0000', 'last_clean': '2024-09-05T16:33:53.505305+0000', 'last_became_active': '2024-09-05T16:34:56.873389+0000', 'last_became_peered': '2024-09-05T16:34:56.873389+0000', 'last_unstale': '2024-09-05T16:46:11.128490+0000', 'last_undegraded': '2024-09-05T16:34:56.602567+0000', 'last_fullsized': '2024-09-05T16:34:56.559958+0000', 'mapping_epoch': 32, 'log_start': "33'2600", 'ondisk_log_start': "33'2600", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10039, 'log_dups_size': 0, 'ondisk_log_size': 10039, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1143968, 'num_objects': 140, 'num_object_clones': 0, 'num_object_copies': 560, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2234, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 140, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12637, 'num_write_kb': 51108, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2092}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:46:16.711 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:46:16.711 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "39'12668", 'reported_seq': 12774, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:12.806329+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:46:12.806329+0000', 'last_peered': '2024-09-05T16:46:12.806329+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:46:12.806329+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "33'2600", 'ondisk_log_start': "33'2600", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10068, 'log_dups_size': 0, 'ondisk_log_size': 10068, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 862528, 'num_objects': 106, 'num_object_clones': 0, 'num_object_copies': 424, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 106, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12665, 'num_write_kb': 51081, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2138}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:46:16.711 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:46:16.711 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "39'12674", 'reported_seq': 12801, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:10.824141+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:46:10.824141+0000', 'last_peered': '2024-09-05T16:46:10.824141+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:46:10.824141+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "33'2600", 'ondisk_log_start': "33'2600", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10074, 'log_dups_size': 0, 'ondisk_log_size': 10074, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1111200, 'num_objects': 136, 'num_object_clones': 0, 'num_object_copies': 544, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 136, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12672, 'num_write_kb': 51230, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -2109}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:46:16.711 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:46:16.711 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "39'12988", 'reported_seq': 13120, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:12.753677+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:46:12.753677+0000', 'last_peered': '2024-09-05T16:46:12.753677+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:46:12.753677+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "33'2900", 'ondisk_log_start': "33'2900", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10088, 'log_dups_size': 0, 'ondisk_log_size': 10088, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1101552, 'num_objects': 135, 'num_object_clones': 0, 'num_object_copies': 540, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2284, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 135, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12985, 'num_write_kb': 52476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2146}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:46:16.711 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:46:16.711 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "39'12629", 'reported_seq': 12733, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:12.937479+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:46:12.937479+0000', 'last_peered': '2024-09-05T16:46:12.937479+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:46:12.937479+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "33'2600", 'ondisk_log_start': "33'2600", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10029, 'log_dups_size': 0, 'ondisk_log_size': 10029, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1063504, 'num_objects': 130, 'num_object_clones': 0, 'num_object_copies': 520, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 130, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12628, 'num_write_kb': 51031, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2108}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:46:16.712 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:46:16.712 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "39'12996", 'reported_seq': 13110, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:12.914087+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:46:12.914087+0000', 'last_peered': '2024-09-05T16:46:12.914087+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:46:12.914087+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "33'2900", 'ondisk_log_start': "33'2900", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10096, 'log_dups_size': 0, 'ondisk_log_size': 10096, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1062048, 'num_objects': 130, 'num_object_clones': 0, 'num_object_copies': 520, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 130, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12994, 'num_write_kb': 52500, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2204}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:46:16.712 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:46:16.712 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "39'12687", 'reported_seq': 12805, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:10.505636+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:46:10.505636+0000', 'last_peered': '2024-09-05T16:46:10.505636+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:46:10.505636+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "33'2600", 'ondisk_log_start': "33'2600", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10087, 'log_dups_size': 0, 'ondisk_log_size': 10087, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1193120, 'num_objects': 146, 'num_object_clones': 0, 'num_object_copies': 584, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2310, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 146, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12685, 'num_write_kb': 51325, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -2162}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:46:16.712 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:46:16.712 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "39'12641", 'reported_seq': 12755, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:11.104277+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:46:11.104277+0000', 'last_peered': '2024-09-05T16:46:11.104277+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:46:11.104277+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "33'2600", 'ondisk_log_start': "33'2600", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10041, 'log_dups_size': 0, 'ondisk_log_size': 10041, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1094816, 'num_objects': 134, 'num_object_clones': 0, 'num_object_copies': 536, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 134, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12639, 'num_write_kb': 51092, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2086}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:46:16.712 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:46:17.451 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:46:17.451 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27867/6744 objects degraded (413.212%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":780,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":39,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":1686,"data_bytes":13770056,"bytes_used":2817089536,"bytes_avail":383729967104,"bytes_total":386547056640,"degraded_objects":27867,"degraded_total":6744,"degraded_ratio":4.1321174377224201,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":196},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:46:17.489 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:46:17.490 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:46:18.123 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:46:18.124 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:46:18.147 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":420,"stamp":"2024-09-05T16:46:17.092988+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":13770056,"num_objects":1686,"num_object_clones":0,"num_object_copies":6744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27867,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1686,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":178464,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1499,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161073,"ondisk_log_size":161073,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2751064,"kb_used_data":27352,"kb_used_omap":14,"kb_used_meta":2723633,"kb_avail":374736296,"statfs":{"total":386547056640,"available":383729967104,"internally_reserved":0,"allocated":28008448,"data_stored":27166124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2789000759},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,2,0,3,15,10,14,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19292160,"num_objects":-2355,"num_object_clones":0,"num_object_copies":-9420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2355,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2355,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005026"},"pg_stats":[{"pgid":"1.f","version":"39'12793","reported_seq":15341,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:13.040086+0000","last_change":"2024-09-05T16:43:43.436935+0000","last_active":"2024-09-05T16:46:13.040086+0000","last_peered":"2024-09-05T16:46:13.040086+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:46:13.040086+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":909312,"num_objects":111,"num_object_clones":0,"num_object_copies":444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1016,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":111,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12793,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1246,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-905}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12426","reported_seq":17083,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:46:11.985533+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:46:11.985533+0000","last_peered":"2024-09-05T16:46:11.985533+0000","last_clean":"2024-09-05T16:46:11.985533+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:46:11.985533+0000","last_undegraded":"2024-09-05T16:46:11.985533+0000","last_fullsized":"2024-09-05T16:46:11.985533+0000","mapping_epoch":38,"log_start":"33'2400","ondisk_log_start":"33'2400","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1032192,"num_objects":126,"num_object_clones":0,"num_object_copies":504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":126,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1739,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12685","reported_seq":12811,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:15.976459+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:46:15.976459+0000","last_peered":"2024-09-05T16:46:15.976459+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:46:15.976459+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":547432,"num_objects":68,"num_object_clones":0,"num_object_copies":272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12684,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2164}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12824","reported_seq":17559,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:46:12.445459+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:46:12.445459+0000","last_peered":"2024-09-05T16:46:12.445459+0000","last_clean":"2024-09-05T16:46:12.445459+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:46:12.445459+0000","last_undegraded":"2024-09-05T16:46:12.445459+0000","last_fullsized":"2024-09-05T16:46:12.445459+0000","mapping_epoch":34,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":901120,"num_objects":110,"num_object_clones":0,"num_object_copies":440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":110,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7124,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12491","reported_seq":12595,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:12.967856+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:46:12.967856+0000","last_peered":"2024-09-05T16:46:12.967856+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:46:12.967856+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"33'2400","ondisk_log_start":"33'2400","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":849056,"num_objects":104,"num_object_clones":0,"num_object_copies":416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12489,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2041}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12647","reported_seq":12767,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:12.897439+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:46:12.897439+0000","last_peered":"2024-09-05T16:46:12.897439+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:46:12.897439+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1024000,"num_objects":125,"num_object_clones":0,"num_object_copies":500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":125,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12647,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2094}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12599","reported_seq":12712,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:16.080082+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:46:16.080082+0000","last_peered":"2024-09-05T16:46:16.080082+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:46:16.080082+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":513184,"num_objects":63,"num_object_clones":0,"num_object_copies":252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12597,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2068}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12700","reported_seq":12793,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:16.082423+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:46:16.082423+0000","last_peered":"2024-09-05T16:46:16.082423+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:46:16.082423+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":644256,"num_objects":79,"num_object_clones":0,"num_object_copies":316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":79,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12698,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2153}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"39'12668","reported_seq":12774,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:12.806329+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:46:12.806329+0000","last_peered":"2024-09-05T16:46:12.806329+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:46:12.806329+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":862528,"num_objects":106,"num_object_clones":0,"num_object_copies":424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":106,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12665,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2138}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12734","reported_seq":12861,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:15.655556+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:46:15.655556+0000","last_peered":"2024-09-05T16:46:15.655556+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:46:15.655556+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":619680,"num_objects":76,"num_object_clones":0,"num_object_copies":304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12732,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2169}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'12988","reported_seq":13120,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:12.753677+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:46:12.753677+0000","last_peered":"2024-09-05T16:46:12.753677+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:46:12.753677+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1101552,"num_objects":135,"num_object_clones":0,"num_object_copies":540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":135,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12985,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2146}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12629","reported_seq":12733,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:12.937479+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:46:12.937479+0000","last_peered":"2024-09-05T16:46:12.937479+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:46:12.937479+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1063504,"num_objects":130,"num_object_clones":0,"num_object_copies":520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":130,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12628,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2108}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12662","reported_seq":17403,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:46:12.224952+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:46:12.224952+0000","last_peered":"2024-09-05T16:46:12.224952+0000","last_clean":"2024-09-05T16:46:12.224952+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:46:12.224952+0000","last_undegraded":"2024-09-05T16:46:12.224952+0000","last_fullsized":"2024-09-05T16:46:12.224952+0000","mapping_epoch":36,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1196032,"num_objects":146,"num_object_clones":0,"num_object_copies":584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":146,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4262,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'12996","reported_seq":13110,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:12.914087+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:46:12.914087+0000","last_peered":"2024-09-05T16:46:12.914087+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:46:12.914087+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1062048,"num_objects":130,"num_object_clones":0,"num_object_copies":520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":130,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12994,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2204}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12743","reported_seq":12861,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:15.852443+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:46:15.852443+0000","last_peered":"2024-09-05T16:46:15.852443+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:46:15.852443+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":734368,"num_objects":90,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":90,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12741,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2218}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'12688","reported_seq":12802,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:16.256924+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:46:16.256924+0000","last_peered":"2024-09-05T16:46:16.256924+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:46:16.256924+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":709792,"num_objects":87,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":87,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12686,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2133}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":13770056,"num_objects":1686,"num_object_clones":0,"num_object_copies":6744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27867,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1686,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":178464,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1499,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":25915392,"data_stored":25915392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161073,"ondisk_log_size":161073,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084434,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":725484,"kb_used_data":7964,"kb_used_omap":4,"kb_used_meta":717499,"kb_avail":93646356,"statfs":{"total":96636764160,"available":95893868544,"internally_reserved":0,"allocated":8155136,"data_stored":7944555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":734719699},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,3,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986311,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":563636,"kb_used_data":8548,"kb_used_omap":4,"kb_used_meta":555067,"kb_avail":93808204,"statfs":{"total":96636764160,"available":96059600896,"internally_reserved":0,"allocated":8753152,"data_stored":8542571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":568389330},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,0,1,6,1,6,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673118,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":730720,"kb_used_data":5392,"kb_used_omap":1,"kb_used_meta":725310,"kb_avail":93641120,"statfs":{"total":96636764160,"available":95888506880,"internally_reserved":0,"allocated":5521408,"data_stored":5310827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":742717893},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,5,2,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182549,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731224,"kb_used_data":5448,"kb_used_omap":4,"kb_used_meta":725755,"kb_avail":93640616,"statfs":{"total":96636764160,"available":95887990784,"internally_reserved":0,"allocated":5578752,"data_stored":5368171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":743173837},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,5,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4952064,"data_stored":4952064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8200192,"data_stored":8200192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":5160960,"data_stored":5160960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":7602176,"data_stored":7602176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:46:21.149 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:46:21.780 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:46:21.781 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:46:21.803 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":422,"stamp":"2024-09-05T16:46:21.094789+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6921544,"num_objects":850,"num_object_clones":0,"num_object_copies":3400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27817,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":850,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":179300,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1549,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161009,"ondisk_log_size":161009,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2748292,"kb_used_data":16260,"kb_used_omap":14,"kb_used_meta":2731953,"kb_avail":374739068,"statfs":{"total":386547056640,"available":383732805632,"internally_reserved":0,"allocated":16650240,"data_stored":15807916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2797520439},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,1,5,4,19,11,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17915904,"num_objects":-2187,"num_object_clones":0,"num_object_copies":-8748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-98,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2187,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2187,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005101"},"pg_stats":[{"pgid":"1.f","version":"39'12856","reported_seq":15505,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:18.144381+0000","last_change":"2024-09-05T16:43:43.436935+0000","last_active":"2024-09-05T16:46:18.144381+0000","last_peered":"2024-09-05T16:46:18.144381+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:46:18.144381+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":48,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12856,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1296,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-918}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12487","reported_seq":17144,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:46:17.531333+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:46:17.531333+0000","last_peered":"2024-09-05T16:46:17.531333+0000","last_clean":"2024-09-05T16:46:17.531333+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:46:17.531333+0000","last_undegraded":"2024-09-05T16:46:17.531333+0000","last_fullsized":"2024-09-05T16:46:17.531333+0000","mapping_epoch":38,"log_start":"33'2400","ondisk_log_start":"33'2400","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":532480,"num_objects":65,"num_object_clones":0,"num_object_copies":260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1800,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12736","reported_seq":12864,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:20.838853+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:46:20.838853+0000","last_peered":"2024-09-05T16:46:20.838853+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:46:20.838853+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":129640,"num_objects":17,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12735,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2215}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12881","reported_seq":17616,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:46:17.640398+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:46:17.640398+0000","last_peered":"2024-09-05T16:46:17.640398+0000","last_clean":"2024-09-05T16:46:17.640398+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:46:17.640398+0000","last_undegraded":"2024-09-05T16:46:17.640398+0000","last_fullsized":"2024-09-05T16:46:17.640398+0000","mapping_epoch":34,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":434176,"num_objects":53,"num_object_clones":0,"num_object_copies":212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7181,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12550","reported_seq":12654,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:17.971009+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:46:17.971009+0000","last_peered":"2024-09-05T16:46:17.971009+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:46:17.971009+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":365728,"num_objects":45,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12548,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2100}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12715","reported_seq":12835,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:18.028824+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:46:18.028824+0000","last_peered":"2024-09-05T16:46:18.028824+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:46:18.028824+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":466944,"num_objects":57,"num_object_clones":0,"num_object_copies":228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12715,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2162}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12599","reported_seq":12712,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:16.080082+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:46:16.080082+0000","last_peered":"2024-09-05T16:46:16.080082+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:46:16.080082+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":513184,"num_objects":63,"num_object_clones":0,"num_object_copies":252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12597,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2068}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12700","reported_seq":12793,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:16.082423+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:46:16.082423+0000","last_peered":"2024-09-05T16:46:16.082423+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:46:16.082423+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":644256,"num_objects":79,"num_object_clones":0,"num_object_copies":316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":79,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12698,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2153}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"39'12729","reported_seq":12835,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:17.763168+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:46:17.763168+0000","last_peered":"2024-09-05T16:46:17.763168+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:46:17.763168+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362816,"num_objects":45,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12726,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2199}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12792","reported_seq":12920,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:20.895659+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:46:20.895659+0000","last_peered":"2024-09-05T16:46:20.895659+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:46:20.895659+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":144544,"num_objects":18,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12790,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2227}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13063","reported_seq":13195,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:17.939010+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:46:17.939010+0000","last_peered":"2024-09-05T16:46:17.939010+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:46:17.939010+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"33'3000","ondisk_log_start":"33'3000","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":487152,"num_objects":60,"num_object_clones":0,"num_object_copies":240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13060,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2221}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12696","reported_seq":12800,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:18.046899+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:46:18.046899+0000","last_peered":"2024-09-05T16:46:18.046899+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:46:18.046899+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":514640,"num_objects":63,"num_object_clones":0,"num_object_copies":252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12695,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2175}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12754","reported_seq":17495,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:46:17.273018+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:46:17.273018+0000","last_peered":"2024-09-05T16:46:17.273018+0000","last_clean":"2024-09-05T16:46:17.273018+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:46:17.273018+0000","last_undegraded":"2024-09-05T16:46:17.273018+0000","last_fullsized":"2024-09-05T16:46:17.273018+0000","mapping_epoch":36,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":442368,"num_objects":54,"num_object_clones":0,"num_object_copies":216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4354,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13055","reported_seq":13169,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:17.970907+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:46:17.970907+0000","last_peered":"2024-09-05T16:46:17.970907+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:46:17.970907+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"33'3000","ondisk_log_start":"33'3000","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":578720,"num_objects":71,"num_object_clones":0,"num_object_copies":284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13053,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2263}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12808","reported_seq":12925,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:20.947919+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:46:20.947919+0000","last_peered":"2024-09-05T16:46:20.947919+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:46:20.947919+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":201888,"num_objects":25,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12806,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2283}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'12688","reported_seq":12802,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:16.256924+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:46:16.256924+0000","last_peered":"2024-09-05T16:46:16.256924+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:46:16.256924+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":709792,"num_objects":87,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":87,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12686,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2133}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":6921544,"num_objects":850,"num_object_clones":0,"num_object_copies":3400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27817,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":850,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":179300,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1549,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13570048,"data_stored":13570048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161009,"ondisk_log_size":161009,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084435,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":725712,"kb_used_data":4032,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93646128,"statfs":{"total":96636764160,"available":95893635072,"internally_reserved":0,"allocated":4128768,"data_stored":3918187,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":738979539},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,1,0,1,1,6,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986312,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":563836,"kb_used_data":4588,"kb_used_omap":4,"kb_used_meta":559227,"kb_avail":93808004,"statfs":{"total":96636764160,"available":96059396096,"internally_reserved":0,"allocated":4698112,"data_stored":4487531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":572649170},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,0,5,2,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673119,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":727520,"kb_used_data":2192,"kb_used_omap":1,"kb_used_meta":725310,"kb_avail":93644320,"statfs":{"total":96636764160,"available":95891783680,"internally_reserved":0,"allocated":2244608,"data_stored":2034027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":742717893},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,8,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182549,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731224,"kb_used_data":5448,"kb_used_omap":4,"kb_used_meta":725755,"kb_avail":93640616,"statfs":{"total":96636764160,"available":95887990784,"internally_reserved":0,"allocated":5578752,"data_stored":5368171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":743173837},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,5,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4952064,"data_stored":4952064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4022272,"data_stored":4022272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1069056,"data_stored":1069056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3526656,"data_stored":3526656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:46:21.804 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:46:22.356 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout:Removed 102426 objects 2024-09-05T16:46:22.356 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi087.stdout:Clean up completed and total clean up time :479.801 2024-09-05T16:46:22.373 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:46:22.373 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:46:22.386 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":422,"stamp":"2024-09-05T16:46:21.094789+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6921544,"num_objects":850,"num_object_clones":0,"num_object_copies":3400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27817,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":850,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":179300,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1549,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161009,"ondisk_log_size":161009,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2748292,"kb_used_data":16260,"kb_used_omap":14,"kb_used_meta":2731953,"kb_avail":374739068,"statfs":{"total":386547056640,"available":383732805632,"internally_reserved":0,"allocated":16650240,"data_stored":15807916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2797520439},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,1,5,4,19,11,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17915904,"num_objects":-2187,"num_object_clones":0,"num_object_copies":-8748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-98,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2187,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2187,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005101"},"pg_stats":[{"pgid":"1.f","version":"39'12856","reported_seq":15505,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:18.144381+0000","last_change":"2024-09-05T16:43:43.436935+0000","last_active":"2024-09-05T16:46:18.144381+0000","last_peered":"2024-09-05T16:46:18.144381+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:46:18.144381+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":48,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12856,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1296,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-918}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12487","reported_seq":17144,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:46:17.531333+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:46:17.531333+0000","last_peered":"2024-09-05T16:46:17.531333+0000","last_clean":"2024-09-05T16:46:17.531333+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:46:17.531333+0000","last_undegraded":"2024-09-05T16:46:17.531333+0000","last_fullsized":"2024-09-05T16:46:17.531333+0000","mapping_epoch":38,"log_start":"33'2400","ondisk_log_start":"33'2400","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":532480,"num_objects":65,"num_object_clones":0,"num_object_copies":260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1800,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12736","reported_seq":12864,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:20.838853+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:46:20.838853+0000","last_peered":"2024-09-05T16:46:20.838853+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:46:20.838853+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":129640,"num_objects":17,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12735,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2215}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12881","reported_seq":17616,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:46:17.640398+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:46:17.640398+0000","last_peered":"2024-09-05T16:46:17.640398+0000","last_clean":"2024-09-05T16:46:17.640398+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:46:17.640398+0000","last_undegraded":"2024-09-05T16:46:17.640398+0000","last_fullsized":"2024-09-05T16:46:17.640398+0000","mapping_epoch":34,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":434176,"num_objects":53,"num_object_clones":0,"num_object_copies":212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7181,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12550","reported_seq":12654,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:17.971009+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:46:17.971009+0000","last_peered":"2024-09-05T16:46:17.971009+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:46:17.971009+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":365728,"num_objects":45,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12548,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2100}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12715","reported_seq":12835,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:18.028824+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:46:18.028824+0000","last_peered":"2024-09-05T16:46:18.028824+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:46:18.028824+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":466944,"num_objects":57,"num_object_clones":0,"num_object_copies":228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12715,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2162}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12599","reported_seq":12712,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:16.080082+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:46:16.080082+0000","last_peered":"2024-09-05T16:46:16.080082+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:46:16.080082+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":513184,"num_objects":63,"num_object_clones":0,"num_object_copies":252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12597,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2068}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12700","reported_seq":12793,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:16.082423+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:46:16.082423+0000","last_peered":"2024-09-05T16:46:16.082423+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:46:16.082423+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":644256,"num_objects":79,"num_object_clones":0,"num_object_copies":316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":79,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12698,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2153}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"39'12729","reported_seq":12835,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:17.763168+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:46:17.763168+0000","last_peered":"2024-09-05T16:46:17.763168+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:46:17.763168+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362816,"num_objects":45,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12726,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2199}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12792","reported_seq":12920,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:20.895659+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:46:20.895659+0000","last_peered":"2024-09-05T16:46:20.895659+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:46:20.895659+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":144544,"num_objects":18,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12790,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2227}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13063","reported_seq":13195,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:17.939010+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:46:17.939010+0000","last_peered":"2024-09-05T16:46:17.939010+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:46:17.939010+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"33'3000","ondisk_log_start":"33'3000","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":487152,"num_objects":60,"num_object_clones":0,"num_object_copies":240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13060,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2221}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12696","reported_seq":12800,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:18.046899+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:46:18.046899+0000","last_peered":"2024-09-05T16:46:18.046899+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:46:18.046899+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":514640,"num_objects":63,"num_object_clones":0,"num_object_copies":252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12695,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2175}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12754","reported_seq":17495,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:46:17.273018+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:46:17.273018+0000","last_peered":"2024-09-05T16:46:17.273018+0000","last_clean":"2024-09-05T16:46:17.273018+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:46:17.273018+0000","last_undegraded":"2024-09-05T16:46:17.273018+0000","last_fullsized":"2024-09-05T16:46:17.273018+0000","mapping_epoch":36,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":442368,"num_objects":54,"num_object_clones":0,"num_object_copies":216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4354,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13055","reported_seq":13169,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:17.970907+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:46:17.970907+0000","last_peered":"2024-09-05T16:46:17.970907+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:46:17.970907+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"33'3000","ondisk_log_start":"33'3000","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":578720,"num_objects":71,"num_object_clones":0,"num_object_copies":284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13053,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2263}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12808","reported_seq":12925,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:20.947919+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:46:20.947919+0000","last_peered":"2024-09-05T16:46:20.947919+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:46:20.947919+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":201888,"num_objects":25,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12806,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2283}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'12688","reported_seq":12802,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:16.256924+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:46:16.256924+0000","last_peered":"2024-09-05T16:46:16.256924+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:46:16.256924+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":709792,"num_objects":87,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":87,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12686,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2133}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":6921544,"num_objects":850,"num_object_clones":0,"num_object_copies":3400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27817,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":850,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":179300,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1549,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13570048,"data_stored":13570048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161009,"ondisk_log_size":161009,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084435,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":725712,"kb_used_data":4032,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93646128,"statfs":{"total":96636764160,"available":95893635072,"internally_reserved":0,"allocated":4128768,"data_stored":3918187,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":738979539},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,1,0,1,1,6,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986312,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":563836,"kb_used_data":4588,"kb_used_omap":4,"kb_used_meta":559227,"kb_avail":93808004,"statfs":{"total":96636764160,"available":96059396096,"internally_reserved":0,"allocated":4698112,"data_stored":4487531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":572649170},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,0,5,2,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673119,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":727520,"kb_used_data":2192,"kb_used_omap":1,"kb_used_meta":725310,"kb_avail":93644320,"statfs":{"total":96636764160,"available":95891783680,"internally_reserved":0,"allocated":2244608,"data_stored":2034027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":742717893},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,8,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182549,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731224,"kb_used_data":5448,"kb_used_omap":4,"kb_used_meta":725755,"kb_avail":93640616,"statfs":{"total":96636764160,"available":95887990784,"internally_reserved":0,"allocated":5578752,"data_stored":5368171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":743173837},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,5,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4952064,"data_stored":4952064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4022272,"data_stored":4022272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1069056,"data_stored":1069056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3526656,"data_stored":3526656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:46:22.388 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-05T16:46:22.388 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "39'12856", 'reported_seq': 15505, 'reported_epoch': 39, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:18.144381+0000', 'last_change': '2024-09-05T16:43:43.436935+0000', 'last_active': '2024-09-05T16:46:18.144381+0000', 'last_peered': '2024-09-05T16:46:18.144381+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.828289+0000', 'last_became_peered': '2024-09-05T16:34:56.828289+0000', 'last_unstale': '2024-09-05T16:46:18.144381+0000', 'last_undegraded': '2024-09-05T16:34:56.612160+0000', 'last_fullsized': '2024-09-05T16:34:56.557632+0000', 'mapping_epoch': 32, 'log_start': "33'2800", 'ondisk_log_start': "33'2800", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10056, 'log_dups_size': 0, 'ondisk_log_size': 10056, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 393216, 'num_objects': 48, 'num_object_clones': 0, 'num_object_copies': 192, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 966, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 48, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12856, 'num_write_kb': 51616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1296, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -918}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:46:22.388 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:46:22.388 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "39'12736", 'reported_seq': 12864, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:20.838853+0000', 'last_change': '2024-09-05T16:34:56.867900+0000', 'last_active': '2024-09-05T16:46:20.838853+0000', 'last_peered': '2024-09-05T16:46:20.838853+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:46:20.838853+0000', 'last_undegraded': '2024-09-05T16:34:56.590972+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 32, 'log_start': "33'2700", 'ondisk_log_start': "33'2700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10036, 'log_dups_size': 0, 'ondisk_log_size': 10036, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 129640, 'num_objects': 17, 'num_object_clones': 0, 'num_object_copies': 68, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2233, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 17, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12735, 'num_write_kb': 50999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -2215}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:46:22.388 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:46:22.388 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "39'12550", 'reported_seq': 12654, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:17.971009+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:46:17.971009+0000', 'last_peered': '2024-09-05T16:46:17.971009+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:46:17.971009+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "33'2500", 'ondisk_log_start': "33'2500", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 365728, 'num_objects': 45, 'num_object_clones': 0, 'num_object_copies': 180, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2147, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 45, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12548, 'num_write_kb': 50373, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2100}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:46:22.388 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:46:22.388 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "39'12715", 'reported_seq': 12835, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:18.028824+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:46:18.028824+0000', 'last_peered': '2024-09-05T16:46:18.028824+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:46:18.028824+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "33'2700", 'ondisk_log_start': "33'2700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10015, 'log_dups_size': 0, 'ondisk_log_size': 10015, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 466944, 'num_objects': 57, 'num_object_clones': 0, 'num_object_copies': 228, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 57, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12715, 'num_write_kb': 51088, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2162}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:46:22.389 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:46:22.389 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "39'12599", 'reported_seq': 12712, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:16.080082+0000', 'last_change': '2024-09-05T16:34:56.903436+0000', 'last_active': '2024-09-05T16:46:16.080082+0000', 'last_peered': '2024-09-05T16:46:16.080082+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:46:16.080082+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "33'2500", 'ondisk_log_start': "33'2500", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10099, 'log_dups_size': 0, 'ondisk_log_size': 10099, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 513184, 'num_objects': 63, 'num_object_clones': 0, 'num_object_copies': 252, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2133, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 63, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12597, 'num_write_kb': 50636, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -2068}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:46:22.389 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-05T16:46:22.389 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "39'12700", 'reported_seq': 12793, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:16.082423+0000', 'last_change': '2024-09-05T16:34:56.883319+0000', 'last_active': '2024-09-05T16:46:16.082423+0000', 'last_peered': '2024-09-05T16:46:16.082423+0000', 'last_clean': '2024-09-05T16:33:53.505305+0000', 'last_became_active': '2024-09-05T16:34:56.873389+0000', 'last_became_peered': '2024-09-05T16:34:56.873389+0000', 'last_unstale': '2024-09-05T16:46:16.082423+0000', 'last_undegraded': '2024-09-05T16:34:56.602567+0000', 'last_fullsized': '2024-09-05T16:34:56.559958+0000', 'mapping_epoch': 32, 'log_start': "33'2600", 'ondisk_log_start': "33'2600", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10100, 'log_dups_size': 0, 'ondisk_log_size': 10100, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 644256, 'num_objects': 79, 'num_object_clones': 0, 'num_object_copies': 316, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2234, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 79, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12698, 'num_write_kb': 51108, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2153}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:46:22.389 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:46:22.389 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "39'12729", 'reported_seq': 12835, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:17.763168+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:46:17.763168+0000', 'last_peered': '2024-09-05T16:46:17.763168+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:46:17.763168+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "33'2700", 'ondisk_log_start': "33'2700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10029, 'log_dups_size': 0, 'ondisk_log_size': 10029, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 362816, 'num_objects': 45, 'num_object_clones': 0, 'num_object_copies': 180, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 45, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12726, 'num_write_kb': 51081, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2199}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:46:22.389 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:46:22.389 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "39'12792", 'reported_seq': 12920, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:20.895659+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:46:20.895659+0000', 'last_peered': '2024-09-05T16:46:20.895659+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:46:20.895659+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "33'2700", 'ondisk_log_start': "33'2700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10092, 'log_dups_size': 0, 'ondisk_log_size': 10092, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 144544, 'num_objects': 18, 'num_object_clones': 0, 'num_object_copies': 72, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12790, 'num_write_kb': 51230, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -2227}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:46:22.389 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:46:22.389 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "39'13063", 'reported_seq': 13195, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:17.939010+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:46:17.939010+0000', 'last_peered': '2024-09-05T16:46:17.939010+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:46:17.939010+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "33'3000", 'ondisk_log_start': "33'3000", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10063, 'log_dups_size': 0, 'ondisk_log_size': 10063, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 487152, 'num_objects': 60, 'num_object_clones': 0, 'num_object_copies': 240, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2284, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 60, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13060, 'num_write_kb': 52476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2221}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:46:22.390 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:46:22.390 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "39'12696", 'reported_seq': 12800, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:18.046899+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:46:18.046899+0000', 'last_peered': '2024-09-05T16:46:18.046899+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:46:18.046899+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "33'2600", 'ondisk_log_start': "33'2600", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10096, 'log_dups_size': 0, 'ondisk_log_size': 10096, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 514640, 'num_objects': 63, 'num_object_clones': 0, 'num_object_copies': 252, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 63, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12695, 'num_write_kb': 51031, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2175}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:46:22.390 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:46:22.390 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "39'13055", 'reported_seq': 13169, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:17.970907+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:46:17.970907+0000', 'last_peered': '2024-09-05T16:46:17.970907+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:46:17.970907+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "33'3000", 'ondisk_log_start': "33'3000", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10055, 'log_dups_size': 0, 'ondisk_log_size': 10055, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 578720, 'num_objects': 71, 'num_object_clones': 0, 'num_object_copies': 284, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 71, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13053, 'num_write_kb': 52500, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2263}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:46:22.390 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:46:22.390 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "39'12808", 'reported_seq': 12925, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:20.947919+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:46:20.947919+0000', 'last_peered': '2024-09-05T16:46:20.947919+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:46:20.947919+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "33'2800", 'ondisk_log_start': "33'2800", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10008, 'log_dups_size': 0, 'ondisk_log_size': 10008, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 201888, 'num_objects': 25, 'num_object_clones': 0, 'num_object_copies': 100, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2310, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 25, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12806, 'num_write_kb': 51325, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -2283}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:46:22.390 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:46:22.390 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "39'12688", 'reported_seq': 12802, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:16.256924+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:46:16.256924+0000', 'last_peered': '2024-09-05T16:46:16.256924+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:46:16.256924+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "33'2600", 'ondisk_log_start': "33'2600", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10088, 'log_dups_size': 0, 'ondisk_log_size': 10088, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 709792, 'num_objects': 87, 'num_object_clones': 0, 'num_object_copies': 348, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 87, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12686, 'num_write_kb': 51092, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2133}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:46:22.390 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:46:22.825 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:46:22.825 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27817/3400 objects degraded (818.147%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":786,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":39,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":850,"data_bytes":6921544,"bytes_used":2814251008,"bytes_avail":383732805632,"bytes_total":386547056640,"degraded_objects":27817,"degraded_total":3400,"degraded_ratio":8.1814705882352943,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":182},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:46:22.839 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:46:22.839 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:46:23.160 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:46:23.161 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:46:23.174 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":423,"stamp":"2024-09-05T16:46:23.095464+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3980616,"num_objects":491,"num_object_clones":0,"num_object_copies":1964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27817,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":491,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":179659,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1549,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160768,"ondisk_log_size":160768,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2762412,"kb_used_data":8172,"kb_used_omap":14,"kb_used_meta":2754161,"kb_avail":374724948,"statfs":{"total":386547056640,"available":383718346752,"internally_reserved":0,"allocated":8368128,"data_stored":7525804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2820261431},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,2,6,3,14,12,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19472384,"num_objects":-2377,"num_object_clones":0,"num_object_copies":-9508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-98,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2377,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2377,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004948"},"pg_stats":[{"pgid":"1.f","version":"39'12856","reported_seq":15505,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:18.144381+0000","last_change":"2024-09-05T16:43:43.436935+0000","last_active":"2024-09-05T16:46:18.144381+0000","last_peered":"2024-09-05T16:46:18.144381+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:46:18.144381+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":48,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12856,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1296,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-918}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12552","reported_seq":17209,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:46:22.277035+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:46:22.277035+0000","last_peered":"2024-09-05T16:46:22.277035+0000","last_clean":"2024-09-05T16:46:22.277035+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:46:22.277035+0000","last_undegraded":"2024-09-05T16:46:22.277035+0000","last_fullsized":"2024-09-05T16:46:22.277035+0000","mapping_epoch":38,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1865,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12736","reported_seq":12864,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:20.838853+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:46:20.838853+0000","last_peered":"2024-09-05T16:46:20.838853+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:46:20.838853+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":129640,"num_objects":17,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12735,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2215}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12934","reported_seq":17669,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:46:22.235430+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:46:22.235430+0000","last_peered":"2024-09-05T16:46:22.235430+0000","last_clean":"2024-09-05T16:46:22.235430+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:46:22.235430+0000","last_undegraded":"2024-09-05T16:46:22.235430+0000","last_fullsized":"2024-09-05T16:46:22.235430+0000","mapping_epoch":34,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7234,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12550","reported_seq":12654,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:17.971009+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:46:17.971009+0000","last_peered":"2024-09-05T16:46:17.971009+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:46:17.971009+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":365728,"num_objects":45,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12548,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2100}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12715","reported_seq":12835,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:18.028824+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:46:18.028824+0000","last_peered":"2024-09-05T16:46:18.028824+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:46:18.028824+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":466944,"num_objects":57,"num_object_clones":0,"num_object_copies":228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12715,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2162}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12650","reported_seq":12763,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:21.205618+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:46:21.205618+0000","last_peered":"2024-09-05T16:46:21.205618+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:46:21.205618+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":95392,"num_objects":12,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12648,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2119}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12768","reported_seq":12862,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:21.184215+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:46:21.184215+0000","last_peered":"2024-09-05T16:46:21.184215+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:46:21.184215+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":87200,"num_objects":11,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12766,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2221}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"39'12729","reported_seq":12835,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:17.763168+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:46:17.763168+0000","last_peered":"2024-09-05T16:46:17.763168+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:46:17.763168+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362816,"num_objects":45,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12726,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2199}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12792","reported_seq":12920,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:20.895659+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:46:20.895659+0000","last_peered":"2024-09-05T16:46:20.895659+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:46:20.895659+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":144544,"num_objects":18,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12790,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2227}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13063","reported_seq":13195,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:17.939010+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:46:17.939010+0000","last_peered":"2024-09-05T16:46:17.939010+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:46:17.939010+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"33'3000","ondisk_log_start":"33'3000","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":487152,"num_objects":60,"num_object_clones":0,"num_object_copies":240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13060,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2221}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12696","reported_seq":12800,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:18.046899+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:46:18.046899+0000","last_peered":"2024-09-05T16:46:18.046899+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:46:18.046899+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":514640,"num_objects":63,"num_object_clones":0,"num_object_copies":252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12695,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2175}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12808","reported_seq":17549,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:46:22.264150+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:46:22.264150+0000","last_peered":"2024-09-05T16:46:22.264150+0000","last_clean":"2024-09-05T16:46:22.264150+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:46:22.264150+0000","last_undegraded":"2024-09-05T16:46:22.264150+0000","last_fullsized":"2024-09-05T16:46:22.264150+0000","mapping_epoch":36,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4408,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13055","reported_seq":13169,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:17.970907+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:46:17.970907+0000","last_peered":"2024-09-05T16:46:17.970907+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:46:17.970907+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"33'3000","ondisk_log_start":"33'3000","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":578720,"num_objects":71,"num_object_clones":0,"num_object_copies":284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13053,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2263}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12808","reported_seq":12925,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:20.947919+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:46:20.947919+0000","last_peered":"2024-09-05T16:46:20.947919+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:46:20.947919+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":201888,"num_objects":25,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12806,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2283}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'12756","reported_seq":12870,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:21.125934+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:46:21.125934+0000","last_peered":"2024-09-05T16:46:21.125934+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:46:21.125934+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":152736,"num_objects":19,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12754,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2201}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":3980616,"num_objects":491,"num_object_clones":0,"num_object_copies":1964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27817,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":491,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":179659,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1549,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5500928,"data_stored":5500928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160768,"ondisk_log_size":160768,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084435,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":725712,"kb_used_data":4032,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93646128,"statfs":{"total":96636764160,"available":95893635072,"internally_reserved":0,"allocated":4128768,"data_stored":3918187,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":738979539},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,1,0,1,1,6,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986313,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":581988,"kb_used_data":532,"kb_used_omap":4,"kb_used_meta":581435,"kb_avail":93789852,"statfs":{"total":96636764160,"available":96040808448,"internally_reserved":0,"allocated":544768,"data_stored":334187,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":595390162},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,0,5,2,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673119,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":727520,"kb_used_data":2192,"kb_used_omap":1,"kb_used_meta":725310,"kb_avail":93644320,"statfs":{"total":96636764160,"available":95891783680,"internally_reserved":0,"allocated":2244608,"data_stored":2034027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":742717893},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,8,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182550,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":727192,"kb_used_data":1416,"kb_used_omap":4,"kb_used_meta":725755,"kb_avail":93644648,"statfs":{"total":96636764160,"available":95892119552,"internally_reserved":0,"allocated":1449984,"data_stored":1239403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":743173837},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,0,0,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":811008,"data_stored":811008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":94208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1069056,"data_stored":1069056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3526656,"data_stored":3526656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:46:26.176 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:46:26.484 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:46:26.485 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:46:26.497 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":424,"stamp":"2024-09-05T16:46:25.095866+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":892232,"num_objects":114,"num_object_clones":0,"num_object_copies":456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27646,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":114,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":180036,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1720,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160745,"ondisk_log_size":160745,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2758704,"kb_used_data":4464,"kb_used_omap":14,"kb_used_meta":2754161,"kb_avail":374728656,"statfs":{"total":386547056640,"available":383722143744,"internally_reserved":0,"allocated":4571136,"data_stored":3728812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2820261431},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,2,7,6,14,7,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-15745024,"num_objects":-1922,"num_object_clones":0,"num_object_copies":-7688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1922,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1922,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":221,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004638"},"pg_stats":[{"pgid":"1.f","version":"39'12904","reported_seq":15897,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:23.192696+0000","last_change":"2024-09-05T16:43:43.436935+0000","last_active":"2024-09-05T16:46:23.192696+0000","last_peered":"2024-09-05T16:46:23.192696+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:46:23.192696+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":795,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12904,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1467,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-795}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12552","reported_seq":17209,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:46:22.277035+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:46:22.277035+0000","last_peered":"2024-09-05T16:46:22.277035+0000","last_clean":"2024-09-05T16:46:22.277035+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:46:22.277035+0000","last_undegraded":"2024-09-05T16:46:22.277035+0000","last_fullsized":"2024-09-05T16:46:22.277035+0000","mapping_epoch":38,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1865,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12736","reported_seq":12864,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:20.838853+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:46:20.838853+0000","last_peered":"2024-09-05T16:46:20.838853+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:46:20.838853+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":129640,"num_objects":17,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12735,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2215}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12934","reported_seq":17669,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:46:22.235430+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:46:22.235430+0000","last_peered":"2024-09-05T16:46:22.235430+0000","last_clean":"2024-09-05T16:46:22.235430+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:46:22.235430+0000","last_undegraded":"2024-09-05T16:46:22.235430+0000","last_fullsized":"2024-09-05T16:46:22.235430+0000","mapping_epoch":34,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7234,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12593","reported_seq":12697,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:22.104833+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:46:22.104833+0000","last_peered":"2024-09-05T16:46:22.104833+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:46:22.104833+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12591,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2143}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12772","reported_seq":12892,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:22.230658+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:46:22.230658+0000","last_peered":"2024-09-05T16:46:22.230658+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:46:22.230658+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12772,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2219}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12650","reported_seq":12763,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:21.205618+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:46:21.205618+0000","last_peered":"2024-09-05T16:46:21.205618+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:46:21.205618+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":95392,"num_objects":12,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12648,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2119}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12768","reported_seq":12862,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:21.184215+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:46:21.184215+0000","last_peered":"2024-09-05T16:46:21.184215+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:46:21.184215+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":87200,"num_objects":11,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12766,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2221}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"39'12770","reported_seq":12876,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:22.365101+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:46:22.365101+0000","last_peered":"2024-09-05T16:46:22.365101+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:46:22.365101+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12767,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2240}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12792","reported_seq":12920,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:20.895659+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:46:20.895659+0000","last_peered":"2024-09-05T16:46:20.895659+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:46:20.895659+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":144544,"num_objects":18,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12790,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2227}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13120","reported_seq":13252,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:22.007229+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:46:22.007229+0000","last_peered":"2024-09-05T16:46:22.007229+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:46:22.007229+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13117,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2278}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12758","reported_seq":12862,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:22.317634+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:46:22.317634+0000","last_peered":"2024-09-05T16:46:22.317634+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:46:22.317634+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2237}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12808","reported_seq":17549,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:46:22.264150+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:46:22.264150+0000","last_peered":"2024-09-05T16:46:22.264150+0000","last_clean":"2024-09-05T16:46:22.264150+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:46:22.264150+0000","last_undegraded":"2024-09-05T16:46:22.264150+0000","last_fullsized":"2024-09-05T16:46:22.264150+0000","mapping_epoch":36,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4408,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13124","reported_seq":13238,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:22.285834+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:46:22.285834+0000","last_peered":"2024-09-05T16:46:22.285834+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:46:22.285834+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13122,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2332}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12808","reported_seq":12925,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:20.947919+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:46:20.947919+0000","last_peered":"2024-09-05T16:46:20.947919+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:46:20.947919+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":201888,"num_objects":25,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12806,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2283}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'12756","reported_seq":12870,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:21.125934+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:46:21.125934+0000","last_peered":"2024-09-05T16:46:21.125934+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:46:21.125934+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":152736,"num_objects":19,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12754,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2201}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":892232,"num_objects":114,"num_object_clones":0,"num_object_copies":456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27646,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":114,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":180036,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1720,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1978368,"data_stored":1978368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160745,"ondisk_log_size":160745,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084436,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722004,"kb_used_data":324,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649836,"statfs":{"total":96636764160,"available":95897432064,"internally_reserved":0,"allocated":331776,"data_stored":121195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":738979539},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,1,1,4,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986313,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":581988,"kb_used_data":532,"kb_used_omap":4,"kb_used_meta":581435,"kb_avail":93789852,"statfs":{"total":96636764160,"available":96040808448,"internally_reserved":0,"allocated":544768,"data_stored":334187,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":595390162},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,0,5,2,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673119,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":727520,"kb_used_data":2192,"kb_used_omap":1,"kb_used_meta":725310,"kb_avail":93644320,"statfs":{"total":96636764160,"available":95891783680,"internally_reserved":0,"allocated":2244608,"data_stored":2034027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":742717893},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,8,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182550,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":727192,"kb_used_data":1416,"kb_used_omap":4,"kb_used_meta":725755,"kb_avail":93644648,"statfs":{"total":96636764160,"available":95892119552,"internally_reserved":0,"allocated":1449984,"data_stored":1239403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":743173837},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,0,0,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":811008,"data_stored":811008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":94208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1069056,"data_stored":1069056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:46:26.498 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:46:26.842 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:46:26.843 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:46:26.855 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":425,"stamp":"2024-09-05T16:46:26.744129+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":154928,"num_objects":23,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27646,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":180127,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1720,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2747424,"kb_used_data":1504,"kb_used_omap":14,"kb_used_meta":2745841,"kb_avail":374739936,"statfs":{"total":386547056640,"available":383733694464,"internally_reserved":0,"allocated":1540096,"data_stored":697772,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2811741751},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,1,5,4,6,3,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16482328,"num_objects":-2013,"num_object_clones":0,"num_object_copies":-8052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2013,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2013,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":221,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.651976"},"pg_stats":[{"pgid":"1.f","version":"39'12904","reported_seq":15897,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:23.192696+0000","last_change":"2024-09-05T16:43:43.436935+0000","last_active":"2024-09-05T16:46:23.192696+0000","last_peered":"2024-09-05T16:46:23.192696+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:46:23.192696+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":795,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12904,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1467,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-795}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12552","reported_seq":17209,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:46:22.277035+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:46:22.277035+0000","last_peered":"2024-09-05T16:46:22.277035+0000","last_clean":"2024-09-05T16:46:22.277035+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:46:22.277035+0000","last_undegraded":"2024-09-05T16:46:22.277035+0000","last_fullsized":"2024-09-05T16:46:22.277035+0000","mapping_epoch":38,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1865,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12752","reported_seq":12880,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:22.291716+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:46:22.291716+0000","last_peered":"2024-09-05T16:46:22.291716+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:46:22.291716+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12751,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2231}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12934","reported_seq":17669,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:46:22.235430+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:46:22.235430+0000","last_peered":"2024-09-05T16:46:22.235430+0000","last_clean":"2024-09-05T16:46:22.235430+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:46:22.235430+0000","last_undegraded":"2024-09-05T16:46:22.235430+0000","last_fullsized":"2024-09-05T16:46:22.235430+0000","mapping_epoch":34,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7234,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12593","reported_seq":12697,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:22.104833+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:46:22.104833+0000","last_peered":"2024-09-05T16:46:22.104833+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:46:22.104833+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12591,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2143}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12772","reported_seq":12892,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:22.230658+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:46:22.230658+0000","last_peered":"2024-09-05T16:46:22.230658+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:46:22.230658+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12772,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2219}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12660","reported_seq":12773,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:22.233256+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:46:22.233256+0000","last_peered":"2024-09-05T16:46:22.233256+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:46:22.233256+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12658,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2129}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12777","reported_seq":12871,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:22.171639+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:46:22.171639+0000","last_peered":"2024-09-05T16:46:22.171639+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:46:22.171639+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12775,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2230}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"39'12770","reported_seq":12876,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:22.365101+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:46:22.365101+0000","last_peered":"2024-09-05T16:46:22.365101+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:46:22.365101+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12767,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2240}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12808","reported_seq":12936,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:22.178034+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:46:22.178034+0000","last_peered":"2024-09-05T16:46:22.178034+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:46:22.178034+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12806,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2243}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13120","reported_seq":13252,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:22.007229+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:46:22.007229+0000","last_peered":"2024-09-05T16:46:22.007229+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:46:22.007229+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13117,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2278}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12758","reported_seq":12862,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:22.317634+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:46:22.317634+0000","last_peered":"2024-09-05T16:46:22.317634+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:46:22.317634+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2237}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12808","reported_seq":17549,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-05T16:46:22.264150+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:46:22.264150+0000","last_peered":"2024-09-05T16:46:22.264150+0000","last_clean":"2024-09-05T16:46:22.264150+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:46:22.264150+0000","last_undegraded":"2024-09-05T16:46:22.264150+0000","last_fullsized":"2024-09-05T16:46:22.264150+0000","mapping_epoch":36,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4408,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13124","reported_seq":13238,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:22.285834+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:46:22.285834+0000","last_peered":"2024-09-05T16:46:22.285834+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:46:22.285834+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13122,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2332}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12831","reported_seq":12949,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:22.223782+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:46:22.223782+0000","last_peered":"2024-09-05T16:46:22.223782+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:46:22.223782+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12829,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2306}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'12773","reported_seq":12887,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:22.273515+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:46:22.273515+0000","last_peered":"2024-09-05T16:46:22.273515+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:46:22.273515+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12771,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2218}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":154928,"num_objects":23,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27646,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":180127,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1720,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":106496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084436,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722004,"kb_used_data":324,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649836,"statfs":{"total":96636764160,"available":95897432064,"internally_reserved":0,"allocated":331776,"data_stored":121195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":738979539},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,1,1,4,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986313,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":581988,"kb_used_data":532,"kb_used_omap":4,"kb_used_meta":581435,"kb_avail":93789852,"statfs":{"total":96636764160,"available":96040808448,"internally_reserved":0,"allocated":544768,"data_stored":334187,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":595390162},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,0,5,2,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673120,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713172,"kb_used_data":324,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":93658668,"statfs":{"total":96636764160,"available":95906476032,"internally_reserved":0,"allocated":331776,"data_stored":121195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":729938373},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182551,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":730260,"kb_used_data":324,"kb_used_omap":4,"kb_used_meta":729915,"kb_avail":93641580,"statfs":{"total":96636764160,"available":95888977920,"internally_reserved":0,"allocated":331776,"data_stored":121195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":747433677},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":94208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:46:26.856 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-05T16:46:26.856 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "39'12904", 'reported_seq': 15897, 'reported_epoch': 39, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:23.192696+0000', 'last_change': '2024-09-05T16:43:43.436935+0000', 'last_active': '2024-09-05T16:46:23.192696+0000', 'last_peered': '2024-09-05T16:46:23.192696+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.828289+0000', 'last_became_peered': '2024-09-05T16:34:56.828289+0000', 'last_unstale': '2024-09-05T16:46:23.192696+0000', 'last_undegraded': '2024-09-05T16:34:56.612160+0000', 'last_fullsized': '2024-09-05T16:34:56.557632+0000', 'mapping_epoch': 32, 'log_start': "33'2900", 'ondisk_log_start': "33'2900", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10004, 'log_dups_size': 0, 'ondisk_log_size': 10004, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 795, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12904, 'num_write_kb': 51616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1467, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -795}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:46:26.857 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:46:26.857 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "39'12752", 'reported_seq': 12880, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:22.291716+0000', 'last_change': '2024-09-05T16:34:56.867900+0000', 'last_active': '2024-09-05T16:46:22.291716+0000', 'last_peered': '2024-09-05T16:46:22.291716+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:46:22.291716+0000', 'last_undegraded': '2024-09-05T16:34:56.590972+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 32, 'log_start': "33'2700", 'ondisk_log_start': "33'2700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10052, 'log_dups_size': 0, 'ondisk_log_size': 10052, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6736, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2233, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12751, 'num_write_kb': 50999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -2231}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:46:26.857 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:46:26.857 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "39'12593", 'reported_seq': 12697, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:22.104833+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:46:22.104833+0000', 'last_peered': '2024-09-05T16:46:22.104833+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:46:22.104833+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "33'2500", 'ondisk_log_start': "33'2500", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10093, 'log_dups_size': 0, 'ondisk_log_size': 10093, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2147, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12591, 'num_write_kb': 50373, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2143}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:46:26.857 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:46:26.857 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "39'12772", 'reported_seq': 12892, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:22.230658+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:46:22.230658+0000', 'last_peered': '2024-09-05T16:46:22.230658+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:46:22.230658+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "33'2700", 'ondisk_log_start': "33'2700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10072, 'log_dups_size': 0, 'ondisk_log_size': 10072, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12772, 'num_write_kb': 51088, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2219}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:46:26.857 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:46:26.857 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "39'12660", 'reported_seq': 12773, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:22.233256+0000', 'last_change': '2024-09-05T16:34:56.903436+0000', 'last_active': '2024-09-05T16:46:22.233256+0000', 'last_peered': '2024-09-05T16:46:22.233256+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:46:22.233256+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "33'2600", 'ondisk_log_start': "33'2600", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10060, 'log_dups_size': 0, 'ondisk_log_size': 10060, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2133, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12658, 'num_write_kb': 50636, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -2129}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:46:26.858 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-05T16:46:26.858 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "39'12777", 'reported_seq': 12871, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:22.171639+0000', 'last_change': '2024-09-05T16:34:56.883319+0000', 'last_active': '2024-09-05T16:46:22.171639+0000', 'last_peered': '2024-09-05T16:46:22.171639+0000', 'last_clean': '2024-09-05T16:33:53.505305+0000', 'last_became_active': '2024-09-05T16:34:56.873389+0000', 'last_became_peered': '2024-09-05T16:34:56.873389+0000', 'last_unstale': '2024-09-05T16:46:22.171639+0000', 'last_undegraded': '2024-09-05T16:34:56.602567+0000', 'last_fullsized': '2024-09-05T16:34:56.559958+0000', 'mapping_epoch': 32, 'log_start': "33'2700", 'ondisk_log_start': "33'2700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10077, 'log_dups_size': 0, 'ondisk_log_size': 10077, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2234, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12775, 'num_write_kb': 51108, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2230}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:46:26.858 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:46:26.858 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "39'12770", 'reported_seq': 12876, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:22.365101+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:46:22.365101+0000', 'last_peered': '2024-09-05T16:46:22.365101+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:46:22.365101+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "33'2700", 'ondisk_log_start': "33'2700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10070, 'log_dups_size': 0, 'ondisk_log_size': 10070, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26944, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 16, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12767, 'num_write_kb': 51081, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2240}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:46:26.858 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:46:26.858 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "39'12808", 'reported_seq': 12936, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:22.178034+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:46:22.178034+0000', 'last_peered': '2024-09-05T16:46:22.178034+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:46:22.178034+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "33'2800", 'ondisk_log_start': "33'2800", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10008, 'log_dups_size': 0, 'ondisk_log_size': 10008, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12806, 'num_write_kb': 51230, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -2243}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:46:26.858 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:46:26.858 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "39'13120", 'reported_seq': 13252, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:22.007229+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:46:22.007229+0000', 'last_peered': '2024-09-05T16:46:22.007229+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:46:22.007229+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "33'3100", 'ondisk_log_start': "33'3100", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20208, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2284, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13117, 'num_write_kb': 52476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2278}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:46:26.858 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:46:26.858 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "39'12758", 'reported_seq': 12862, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:22.317634+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:46:22.317634+0000', 'last_peered': '2024-09-05T16:46:22.317634+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:46:22.317634+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "33'2700", 'ondisk_log_start': "33'2700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10058, 'log_dups_size': 0, 'ondisk_log_size': 10058, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6736, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12757, 'num_write_kb': 51031, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2237}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:46:26.859 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:46:26.859 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "39'13124", 'reported_seq': 13238, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:22.285834+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:46:22.285834+0000', 'last_peered': '2024-09-05T16:46:22.285834+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:46:22.285834+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "33'3100", 'ondisk_log_start': "33'3100", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10024, 'log_dups_size': 0, 'ondisk_log_size': 10024, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13122, 'num_write_kb': 52500, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2332}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:46:26.859 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:46:26.859 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "39'12831", 'reported_seq': 12949, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:22.223782+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:46:22.223782+0000', 'last_peered': '2024-09-05T16:46:22.223782+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:46:22.223782+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "33'2800", 'ondisk_log_start': "33'2800", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10031, 'log_dups_size': 0, 'ondisk_log_size': 10031, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2310, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12829, 'num_write_kb': 51325, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -2306}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:46:26.859 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:46:26.859 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "39'12773", 'reported_seq': 12887, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:22.273515+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:46:22.273515+0000', 'last_peered': '2024-09-05T16:46:22.273515+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:46:22.273515+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "33'2700", 'ondisk_log_start': "33'2700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10073, 'log_dups_size': 0, 'ondisk_log_size': 10073, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12771, 'num_write_kb': 51092, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2218}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:46:26.859 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:46:27.244 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:46:27.245 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27646/456 objects degraded (6062.719%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":790,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":40,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":114,"data_bytes":892232,"bytes_used":2824912896,"bytes_avail":383722143744,"bytes_total":386547056640,"degraded_objects":27646,"degraded_total":456,"degraded_ratio":60.627192982456137,"recovering_objects_per_sec":18,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":221,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":160},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:46:27.258 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:46:27.258 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:46:27.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-09-05T16:46:27.571+0000 7f899dd9b640 0 --2- 172.21.15.87:0/1541735224 >> v2:172.21.15.87:3300/0 conn(0x7f89a006f200 0x7f89a012fe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-05T16:46:27.796 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:46:27.797 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:46:27.814 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":427,"stamp":"2024-09-05T16:46:27.748333+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":154928,"num_objects":23,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27646,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":180127,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1720,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2738988,"kb_used_data":1384,"kb_used_omap":14,"kb_used_meta":2737521,"kb_avail":374748372,"statfs":{"total":386547056640,"available":383742332928,"internally_reserved":0,"allocated":1417216,"data_stored":570212,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2803222071},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,1,1,4,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-8192024,"num_objects":-1001,"num_object_clones":0,"num_object_copies":-4004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-171,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1001,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1001,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.654366"},"pg_stats":[{"pgid":"1.f","version":"39'12904","reported_seq":15897,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:23.192696+0000","last_change":"2024-09-05T16:43:43.436935+0000","last_active":"2024-09-05T16:46:23.192696+0000","last_peered":"2024-09-05T16:46:23.192696+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.828289+0000","last_became_peered":"2024-09-05T16:34:56.828289+0000","last_unstale":"2024-09-05T16:46:23.192696+0000","last_undegraded":"2024-09-05T16:34:56.612160+0000","last_fullsized":"2024-09-05T16:34:56.557632+0000","mapping_epoch":32,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":795,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12904,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1467,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-795}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12552","reported_seq":17211,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-05T16:46:26.754074+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:46:26.754074+0000","last_peered":"2024-09-05T16:46:26.754074+0000","last_clean":"2024-09-05T16:46:26.754074+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:46:26.754074+0000","last_undegraded":"2024-09-05T16:46:26.754074+0000","last_fullsized":"2024-09-05T16:46:26.754074+0000","mapping_epoch":38,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1865,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12752","reported_seq":12880,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:22.291716+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:46:22.291716+0000","last_peered":"2024-09-05T16:46:22.291716+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:46:22.291716+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12751,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2231}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12934","reported_seq":17671,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-05T16:46:26.753512+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:46:26.753512+0000","last_peered":"2024-09-05T16:46:26.753512+0000","last_clean":"2024-09-05T16:46:26.753512+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:46:26.753512+0000","last_undegraded":"2024-09-05T16:46:26.753512+0000","last_fullsized":"2024-09-05T16:46:26.753512+0000","mapping_epoch":34,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7234,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12593","reported_seq":12697,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:22.104833+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:46:22.104833+0000","last_peered":"2024-09-05T16:46:22.104833+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:46:22.104833+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12591,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2143}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12772","reported_seq":12892,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:22.230658+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:46:22.230658+0000","last_peered":"2024-09-05T16:46:22.230658+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:46:22.230658+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12772,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2219}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12660","reported_seq":12773,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:22.233256+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:46:22.233256+0000","last_peered":"2024-09-05T16:46:22.233256+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:46:22.233256+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12658,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2129}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12777","reported_seq":12871,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:22.171639+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:46:22.171639+0000","last_peered":"2024-09-05T16:46:22.171639+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:46:22.171639+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12775,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2230}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"39'12770","reported_seq":12876,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:22.365101+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:46:22.365101+0000","last_peered":"2024-09-05T16:46:22.365101+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:46:22.365101+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12767,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2240}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12808","reported_seq":12936,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:22.178034+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:46:22.178034+0000","last_peered":"2024-09-05T16:46:22.178034+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:46:22.178034+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12806,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2243}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13120","reported_seq":13252,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:22.007229+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:46:22.007229+0000","last_peered":"2024-09-05T16:46:22.007229+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:46:22.007229+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13117,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2278}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12758","reported_seq":12862,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:22.317634+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:46:22.317634+0000","last_peered":"2024-09-05T16:46:22.317634+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:46:22.317634+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2237}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12808","reported_seq":17551,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-05T16:46:26.753613+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:46:26.753613+0000","last_peered":"2024-09-05T16:46:26.753613+0000","last_clean":"2024-09-05T16:46:26.753613+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:46:26.753613+0000","last_undegraded":"2024-09-05T16:46:26.753613+0000","last_fullsized":"2024-09-05T16:46:26.753613+0000","mapping_epoch":36,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4408,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13124","reported_seq":13238,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:22.285834+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:46:22.285834+0000","last_peered":"2024-09-05T16:46:22.285834+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:46:22.285834+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13122,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2332}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12831","reported_seq":12949,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:22.223782+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:46:22.223782+0000","last_peered":"2024-09-05T16:46:22.223782+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:46:22.223782+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12829,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2306}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'12773","reported_seq":12887,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:22.273515+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:46:22.273515+0000","last_peered":"2024-09-05T16:46:22.273515+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:46:22.273515+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12771,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2218}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":154928,"num_objects":23,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27646,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":180127,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1720,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":98304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084436,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722004,"kb_used_data":324,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649836,"statfs":{"total":96636764160,"available":95897432064,"internally_reserved":0,"allocated":331776,"data_stored":121195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":738979539},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,1,1,4,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986314,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":573552,"kb_used_data":412,"kb_used_omap":4,"kb_used_meta":573115,"kb_avail":93798288,"statfs":{"total":96636764160,"available":96049446912,"internally_reserved":0,"allocated":421888,"data_stored":206627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":586870482},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673120,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713172,"kb_used_data":324,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":93658668,"statfs":{"total":96636764160,"available":95906476032,"internally_reserved":0,"allocated":331776,"data_stored":121195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":729938373},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182551,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":730260,"kb_used_data":324,"kb_used_omap":4,"kb_used_meta":729915,"kb_avail":93641580,"statfs":{"total":96636764160,"available":95888977920,"internally_reserved":0,"allocated":331776,"data_stored":121195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":747433677},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":86016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:46:30.816 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:46:31.116 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:46:31.116 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:46:31.128 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":429,"stamp":"2024-09-05T16:46:31.096942+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":154928,"num_objects":23,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26851,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":180127,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2515,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2739032,"kb_used_data":1416,"kb_used_omap":14,"kb_used_meta":2737521,"kb_avail":374748328,"statfs":{"total":386547056640,"available":383742287872,"internally_reserved":0,"allocated":1449984,"data_stored":584218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2803222071},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-3825688,"num_objects":-468,"num_object_clones":0,"num_object_copies":-1872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-468,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":468,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":966,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001478"},"pg_stats":[{"pgid":"1.f","version":"39'12904","reported_seq":17500,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-05T16:46:27.826329+0000","last_change":"2024-09-05T16:46:27.826222+0000","last_active":"2024-09-05T16:46:27.826329+0000","last_peered":"2024-09-05T16:46:27.826329+0000","last_clean":"2024-09-05T16:46:27.826329+0000","last_became_active":"2024-09-05T16:46:27.757928+0000","last_became_peered":"2024-09-05T16:46:27.757928+0000","last_unstale":"2024-09-05T16:46:27.826329+0000","last_undegraded":"2024-09-05T16:46:27.826329+0000","last_fullsized":"2024-09-05T16:46:27.826329+0000","mapping_epoch":40,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:32:35.713428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12904,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2262,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12552","reported_seq":17211,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-05T16:46:26.754074+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:46:26.754074+0000","last_peered":"2024-09-05T16:46:26.754074+0000","last_clean":"2024-09-05T16:46:26.754074+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:46:26.754074+0000","last_undegraded":"2024-09-05T16:46:26.754074+0000","last_fullsized":"2024-09-05T16:46:26.754074+0000","mapping_epoch":38,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1865,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12752","reported_seq":12884,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:27.756492+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:46:27.756492+0000","last_peered":"2024-09-05T16:46:27.756492+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:46:27.756492+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12751,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2231}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12934","reported_seq":17671,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-05T16:46:26.753512+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:46:26.753512+0000","last_peered":"2024-09-05T16:46:26.753512+0000","last_clean":"2024-09-05T16:46:26.753512+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:46:26.753512+0000","last_undegraded":"2024-09-05T16:46:26.753512+0000","last_fullsized":"2024-09-05T16:46:26.753512+0000","mapping_epoch":34,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7234,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12593","reported_seq":12702,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:27.753309+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:46:27.753309+0000","last_peered":"2024-09-05T16:46:27.753309+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:46:27.753309+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12591,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2143}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12772","reported_seq":12896,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:27.752538+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:46:27.752538+0000","last_peered":"2024-09-05T16:46:27.752538+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:46:27.752538+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12772,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2219}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12660","reported_seq":12773,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:22.233256+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:46:22.233256+0000","last_peered":"2024-09-05T16:46:22.233256+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:46:22.233256+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12658,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2129}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12777","reported_seq":12871,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:22.171639+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:46:22.171639+0000","last_peered":"2024-09-05T16:46:22.171639+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:46:22.171639+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12775,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2230}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"39'12770","reported_seq":12880,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:27.750634+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:46:27.750634+0000","last_peered":"2024-09-05T16:46:27.750634+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:46:27.750634+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12767,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2240}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12808","reported_seq":12940,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:27.757112+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:46:27.757112+0000","last_peered":"2024-09-05T16:46:27.757112+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:46:27.757112+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12806,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2243}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13120","reported_seq":13256,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:27.750509+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:46:27.750509+0000","last_peered":"2024-09-05T16:46:27.750509+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:46:27.750509+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13117,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2278}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12758","reported_seq":12866,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:27.751040+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:46:27.751040+0000","last_peered":"2024-09-05T16:46:27.751040+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:46:27.751040+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2237}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12808","reported_seq":17551,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-05T16:46:26.753613+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:46:26.753613+0000","last_peered":"2024-09-05T16:46:26.753613+0000","last_clean":"2024-09-05T16:46:26.753613+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:46:26.753613+0000","last_undegraded":"2024-09-05T16:46:26.753613+0000","last_fullsized":"2024-09-05T16:46:26.753613+0000","mapping_epoch":36,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4408,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13124","reported_seq":13242,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:27.753160+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:46:27.753160+0000","last_peered":"2024-09-05T16:46:27.753160+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:46:27.753160+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13122,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2332}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12831","reported_seq":12953,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:27.756269+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:46:27.756269+0000","last_peered":"2024-09-05T16:46:27.756269+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:46:27.756269+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12829,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2306}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'12773","reported_seq":12887,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:22.273515+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:46:22.273515+0000","last_peered":"2024-09-05T16:46:22.273515+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:46:22.273515+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12771,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2218}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":154928,"num_objects":23,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26851,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":180127,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2515,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":98304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084437,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722028,"kb_used_data":340,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649812,"statfs":{"total":96636764160,"available":95897407488,"internally_reserved":0,"allocated":348160,"data_stored":128198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":738979539},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986314,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":573552,"kb_used_data":412,"kb_used_omap":4,"kb_used_meta":573115,"kb_avail":93798288,"statfs":{"total":96636764160,"available":96049446912,"internally_reserved":0,"allocated":421888,"data_stored":206627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":586870482},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673121,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713192,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":93658648,"statfs":{"total":96636764160,"available":95906455552,"internally_reserved":0,"allocated":348160,"data_stored":128198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":729938373},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182551,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":730260,"kb_used_data":324,"kb_used_omap":4,"kb_used_meta":729915,"kb_avail":93641580,"statfs":{"total":96636764160,"available":95888977920,"internally_reserved":0,"allocated":331776,"data_stored":121195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":747433677},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":86016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:46:31.129 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:46:31.429 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:46:31.430 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:46:31.441 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":429,"stamp":"2024-09-05T16:46:31.096942+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":154928,"num_objects":23,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26851,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":180127,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2515,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2739032,"kb_used_data":1416,"kb_used_omap":14,"kb_used_meta":2737521,"kb_avail":374748328,"statfs":{"total":386547056640,"available":383742287872,"internally_reserved":0,"allocated":1449984,"data_stored":584218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2803222071},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-3825688,"num_objects":-468,"num_object_clones":0,"num_object_copies":-1872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-468,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":468,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":966,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001478"},"pg_stats":[{"pgid":"1.f","version":"39'12904","reported_seq":17500,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-05T16:46:27.826329+0000","last_change":"2024-09-05T16:46:27.826222+0000","last_active":"2024-09-05T16:46:27.826329+0000","last_peered":"2024-09-05T16:46:27.826329+0000","last_clean":"2024-09-05T16:46:27.826329+0000","last_became_active":"2024-09-05T16:46:27.757928+0000","last_became_peered":"2024-09-05T16:46:27.757928+0000","last_unstale":"2024-09-05T16:46:27.826329+0000","last_undegraded":"2024-09-05T16:46:27.826329+0000","last_fullsized":"2024-09-05T16:46:27.826329+0000","mapping_epoch":40,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:32:35.713428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12904,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2262,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12552","reported_seq":17211,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-05T16:46:26.754074+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:46:26.754074+0000","last_peered":"2024-09-05T16:46:26.754074+0000","last_clean":"2024-09-05T16:46:26.754074+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:46:26.754074+0000","last_undegraded":"2024-09-05T16:46:26.754074+0000","last_fullsized":"2024-09-05T16:46:26.754074+0000","mapping_epoch":38,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1865,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12752","reported_seq":12884,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:27.756492+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:46:27.756492+0000","last_peered":"2024-09-05T16:46:27.756492+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:46:27.756492+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12751,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2231}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12934","reported_seq":17671,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-05T16:46:26.753512+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:46:26.753512+0000","last_peered":"2024-09-05T16:46:26.753512+0000","last_clean":"2024-09-05T16:46:26.753512+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:46:26.753512+0000","last_undegraded":"2024-09-05T16:46:26.753512+0000","last_fullsized":"2024-09-05T16:46:26.753512+0000","mapping_epoch":34,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7234,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12593","reported_seq":12702,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:27.753309+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:46:27.753309+0000","last_peered":"2024-09-05T16:46:27.753309+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:46:27.753309+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12591,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2143}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12772","reported_seq":12896,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:27.752538+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:46:27.752538+0000","last_peered":"2024-09-05T16:46:27.752538+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:46:27.752538+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12772,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2219}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12660","reported_seq":12773,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:22.233256+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:46:22.233256+0000","last_peered":"2024-09-05T16:46:22.233256+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:46:22.233256+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12658,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2129}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12777","reported_seq":12871,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:22.171639+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:46:22.171639+0000","last_peered":"2024-09-05T16:46:22.171639+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:46:22.171639+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12775,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2230}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"39'12770","reported_seq":12880,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:27.750634+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:46:27.750634+0000","last_peered":"2024-09-05T16:46:27.750634+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:46:27.750634+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12767,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2240}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12808","reported_seq":12940,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:27.757112+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:46:27.757112+0000","last_peered":"2024-09-05T16:46:27.757112+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:46:27.757112+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12806,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2243}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13120","reported_seq":13256,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:27.750509+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:46:27.750509+0000","last_peered":"2024-09-05T16:46:27.750509+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:46:27.750509+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13117,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2278}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12758","reported_seq":12866,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:27.751040+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:46:27.751040+0000","last_peered":"2024-09-05T16:46:27.751040+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:46:27.751040+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2237}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12808","reported_seq":17551,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-05T16:46:26.753613+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:46:26.753613+0000","last_peered":"2024-09-05T16:46:26.753613+0000","last_clean":"2024-09-05T16:46:26.753613+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:46:26.753613+0000","last_undegraded":"2024-09-05T16:46:26.753613+0000","last_fullsized":"2024-09-05T16:46:26.753613+0000","mapping_epoch":36,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4408,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13124","reported_seq":13242,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:27.753160+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:46:27.753160+0000","last_peered":"2024-09-05T16:46:27.753160+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:46:27.753160+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13122,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2332}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12831","reported_seq":12953,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:27.756269+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:46:27.756269+0000","last_peered":"2024-09-05T16:46:27.756269+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:46:27.756269+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12829,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2306}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'12773","reported_seq":12887,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:22.273515+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:46:22.273515+0000","last_peered":"2024-09-05T16:46:22.273515+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:46:22.273515+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12771,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2218}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":154928,"num_objects":23,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26851,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":180127,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2515,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":98304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084437,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722028,"kb_used_data":340,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649812,"statfs":{"total":96636764160,"available":95897407488,"internally_reserved":0,"allocated":348160,"data_stored":128198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":738979539},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986314,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":573552,"kb_used_data":412,"kb_used_omap":4,"kb_used_meta":573115,"kb_avail":93798288,"statfs":{"total":96636764160,"available":96049446912,"internally_reserved":0,"allocated":421888,"data_stored":206627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":586870482},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673121,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713192,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":93658648,"statfs":{"total":96636764160,"available":95906455552,"internally_reserved":0,"allocated":348160,"data_stored":128198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":729938373},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182551,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":730260,"kb_used_data":324,"kb_used_omap":4,"kb_used_meta":729915,"kb_avail":93641580,"statfs":{"total":96636764160,"available":95888977920,"internally_reserved":0,"allocated":331776,"data_stored":121195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":747433677},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":86016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:46:31.442 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:46:31.443 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "39'12752", 'reported_seq': 12884, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:27.756492+0000', 'last_change': '2024-09-05T16:34:56.867900+0000', 'last_active': '2024-09-05T16:46:27.756492+0000', 'last_peered': '2024-09-05T16:46:27.756492+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:46:27.756492+0000', 'last_undegraded': '2024-09-05T16:34:56.590972+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 32, 'log_start': "33'2700", 'ondisk_log_start': "33'2700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10052, 'log_dups_size': 0, 'ondisk_log_size': 10052, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6736, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2233, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12751, 'num_write_kb': 50999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -2231}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:46:31.443 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:46:31.443 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "39'12593", 'reported_seq': 12702, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:27.753309+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:46:27.753309+0000', 'last_peered': '2024-09-05T16:46:27.753309+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:46:27.753309+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "33'2500", 'ondisk_log_start': "33'2500", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10093, 'log_dups_size': 0, 'ondisk_log_size': 10093, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2147, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12591, 'num_write_kb': 50373, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2143}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:46:31.443 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:46:31.443 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "39'12772", 'reported_seq': 12896, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:27.752538+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:46:27.752538+0000', 'last_peered': '2024-09-05T16:46:27.752538+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:46:27.752538+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "33'2700", 'ondisk_log_start': "33'2700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10072, 'log_dups_size': 0, 'ondisk_log_size': 10072, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12772, 'num_write_kb': 51088, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2219}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:46:31.443 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:46:31.443 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "39'12660", 'reported_seq': 12773, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:22.233256+0000', 'last_change': '2024-09-05T16:34:56.903436+0000', 'last_active': '2024-09-05T16:46:22.233256+0000', 'last_peered': '2024-09-05T16:46:22.233256+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:46:22.233256+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "33'2600", 'ondisk_log_start': "33'2600", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10060, 'log_dups_size': 0, 'ondisk_log_size': 10060, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2133, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12658, 'num_write_kb': 50636, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -2129}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:46:31.443 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-05T16:46:31.443 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "39'12777", 'reported_seq': 12871, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:22.171639+0000', 'last_change': '2024-09-05T16:34:56.883319+0000', 'last_active': '2024-09-05T16:46:22.171639+0000', 'last_peered': '2024-09-05T16:46:22.171639+0000', 'last_clean': '2024-09-05T16:33:53.505305+0000', 'last_became_active': '2024-09-05T16:34:56.873389+0000', 'last_became_peered': '2024-09-05T16:34:56.873389+0000', 'last_unstale': '2024-09-05T16:46:22.171639+0000', 'last_undegraded': '2024-09-05T16:34:56.602567+0000', 'last_fullsized': '2024-09-05T16:34:56.559958+0000', 'mapping_epoch': 32, 'log_start': "33'2700", 'ondisk_log_start': "33'2700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10077, 'log_dups_size': 0, 'ondisk_log_size': 10077, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2234, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12775, 'num_write_kb': 51108, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2230}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:46:31.443 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:46:31.443 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "39'12770", 'reported_seq': 12880, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:27.750634+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:46:27.750634+0000', 'last_peered': '2024-09-05T16:46:27.750634+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:46:27.750634+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "33'2700", 'ondisk_log_start': "33'2700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10070, 'log_dups_size': 0, 'ondisk_log_size': 10070, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26944, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 16, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12767, 'num_write_kb': 51081, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2240}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:46:31.444 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:46:31.444 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "39'12808", 'reported_seq': 12940, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:27.757112+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:46:27.757112+0000', 'last_peered': '2024-09-05T16:46:27.757112+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:46:27.757112+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "33'2800", 'ondisk_log_start': "33'2800", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10008, 'log_dups_size': 0, 'ondisk_log_size': 10008, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12806, 'num_write_kb': 51230, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -2243}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:46:31.444 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:46:31.444 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "39'13120", 'reported_seq': 13256, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:27.750509+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:46:27.750509+0000', 'last_peered': '2024-09-05T16:46:27.750509+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:46:27.750509+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "33'3100", 'ondisk_log_start': "33'3100", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20208, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2284, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13117, 'num_write_kb': 52476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2278}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:46:31.444 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:46:31.444 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "39'12758", 'reported_seq': 12866, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:27.751040+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:46:27.751040+0000', 'last_peered': '2024-09-05T16:46:27.751040+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:46:27.751040+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "33'2700", 'ondisk_log_start': "33'2700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10058, 'log_dups_size': 0, 'ondisk_log_size': 10058, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6736, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12757, 'num_write_kb': 51031, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2237}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:46:31.444 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:46:31.444 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "39'13124", 'reported_seq': 13242, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:27.753160+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:46:27.753160+0000', 'last_peered': '2024-09-05T16:46:27.753160+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:46:27.753160+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "33'3100", 'ondisk_log_start': "33'3100", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10024, 'log_dups_size': 0, 'ondisk_log_size': 10024, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13122, 'num_write_kb': 52500, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2332}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:46:31.444 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:46:31.444 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "39'12831", 'reported_seq': 12953, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:27.756269+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:46:27.756269+0000', 'last_peered': '2024-09-05T16:46:27.756269+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:46:27.756269+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "33'2800", 'ondisk_log_start': "33'2800", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10031, 'log_dups_size': 0, 'ondisk_log_size': 10031, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2310, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12829, 'num_write_kb': 51325, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -2306}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:46:31.444 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:46:31.445 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "39'12773", 'reported_seq': 12887, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:22.273515+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:46:22.273515+0000', 'last_peered': '2024-09-05T16:46:22.273515+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:46:22.273515+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "33'2700", 'ondisk_log_start': "33'2700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10073, 'log_dups_size': 0, 'ondisk_log_size': 10073, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12771, 'num_write_kb': 51092, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2218}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:46:31.445 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:46:31.831 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:46:31.831 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 26851/92 objects degraded (29185.870%), 12 pgs degraded, 12 pgs undersized","count":24},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":795,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":41,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":4}],"num_pgs":16,"num_pools":1,"num_objects":23,"data_bytes":154928,"bytes_used":2804768768,"bytes_avail":383742287872,"bytes_total":386547056640,"degraded_objects":26851,"degraded_total":92,"degraded_ratio":291.85869565217394,"recovering_objects_per_sec":120,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":966,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":58},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:46:31.844 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:46:31.844 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:46:32.160 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:46:32.161 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:46:32.174 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":429,"stamp":"2024-09-05T16:46:31.096942+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":154928,"num_objects":23,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26851,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":180127,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2515,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2739032,"kb_used_data":1416,"kb_used_omap":14,"kb_used_meta":2737521,"kb_avail":374748328,"statfs":{"total":386547056640,"available":383742287872,"internally_reserved":0,"allocated":1449984,"data_stored":584218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2803222071},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-3825688,"num_objects":-468,"num_object_clones":0,"num_object_copies":-1872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-468,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":468,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":966,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001478"},"pg_stats":[{"pgid":"1.f","version":"39'12904","reported_seq":17500,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-05T16:46:27.826329+0000","last_change":"2024-09-05T16:46:27.826222+0000","last_active":"2024-09-05T16:46:27.826329+0000","last_peered":"2024-09-05T16:46:27.826329+0000","last_clean":"2024-09-05T16:46:27.826329+0000","last_became_active":"2024-09-05T16:46:27.757928+0000","last_became_peered":"2024-09-05T16:46:27.757928+0000","last_unstale":"2024-09-05T16:46:27.826329+0000","last_undegraded":"2024-09-05T16:46:27.826329+0000","last_fullsized":"2024-09-05T16:46:27.826329+0000","mapping_epoch":40,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:32:35.713428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12904,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2262,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12552","reported_seq":17211,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-05T16:46:26.754074+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:46:26.754074+0000","last_peered":"2024-09-05T16:46:26.754074+0000","last_clean":"2024-09-05T16:46:26.754074+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:46:26.754074+0000","last_undegraded":"2024-09-05T16:46:26.754074+0000","last_fullsized":"2024-09-05T16:46:26.754074+0000","mapping_epoch":38,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1865,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12752","reported_seq":12884,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:27.756492+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:46:27.756492+0000","last_peered":"2024-09-05T16:46:27.756492+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:46:27.756492+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12751,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2231}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12934","reported_seq":17671,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-05T16:46:26.753512+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:46:26.753512+0000","last_peered":"2024-09-05T16:46:26.753512+0000","last_clean":"2024-09-05T16:46:26.753512+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:46:26.753512+0000","last_undegraded":"2024-09-05T16:46:26.753512+0000","last_fullsized":"2024-09-05T16:46:26.753512+0000","mapping_epoch":34,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7234,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12593","reported_seq":12702,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:27.753309+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:46:27.753309+0000","last_peered":"2024-09-05T16:46:27.753309+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:46:27.753309+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12591,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2143}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12772","reported_seq":12896,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:27.752538+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:46:27.752538+0000","last_peered":"2024-09-05T16:46:27.752538+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:46:27.752538+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12772,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2219}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12660","reported_seq":12773,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:22.233256+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:46:22.233256+0000","last_peered":"2024-09-05T16:46:22.233256+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:46:22.233256+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12658,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2129}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12777","reported_seq":12871,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:22.171639+0000","last_change":"2024-09-05T16:34:56.883319+0000","last_active":"2024-09-05T16:46:22.171639+0000","last_peered":"2024-09-05T16:46:22.171639+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:46:22.171639+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12775,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2230}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"39'12770","reported_seq":12880,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:27.750634+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:46:27.750634+0000","last_peered":"2024-09-05T16:46:27.750634+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:46:27.750634+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12767,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2240}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12808","reported_seq":12940,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:27.757112+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:46:27.757112+0000","last_peered":"2024-09-05T16:46:27.757112+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:46:27.757112+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12806,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2243}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13120","reported_seq":13256,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:27.750509+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:46:27.750509+0000","last_peered":"2024-09-05T16:46:27.750509+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:46:27.750509+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13117,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2278}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12758","reported_seq":12866,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:27.751040+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:46:27.751040+0000","last_peered":"2024-09-05T16:46:27.751040+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:46:27.751040+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2237}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12808","reported_seq":17551,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-05T16:46:26.753613+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:46:26.753613+0000","last_peered":"2024-09-05T16:46:26.753613+0000","last_clean":"2024-09-05T16:46:26.753613+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:46:26.753613+0000","last_undegraded":"2024-09-05T16:46:26.753613+0000","last_fullsized":"2024-09-05T16:46:26.753613+0000","mapping_epoch":36,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4408,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13124","reported_seq":13242,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:27.753160+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:46:27.753160+0000","last_peered":"2024-09-05T16:46:27.753160+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:46:27.753160+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13122,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2332}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12831","reported_seq":12953,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:27.756269+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:46:27.756269+0000","last_peered":"2024-09-05T16:46:27.756269+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:46:27.756269+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12829,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2306}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'12773","reported_seq":12887,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:22.273515+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:46:22.273515+0000","last_peered":"2024-09-05T16:46:22.273515+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:46:22.273515+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12771,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2218}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":154928,"num_objects":23,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26851,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":180127,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2515,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":98304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084437,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722028,"kb_used_data":340,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649812,"statfs":{"total":96636764160,"available":95897407488,"internally_reserved":0,"allocated":348160,"data_stored":128198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":738979539},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986314,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":573552,"kb_used_data":412,"kb_used_omap":4,"kb_used_meta":573115,"kb_avail":93798288,"statfs":{"total":96636764160,"available":96049446912,"internally_reserved":0,"allocated":421888,"data_stored":206627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":586870482},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673121,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713192,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":93658648,"statfs":{"total":96636764160,"available":95906455552,"internally_reserved":0,"allocated":348160,"data_stored":128198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":729938373},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182551,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":730260,"kb_used_data":324,"kb_used_omap":4,"kb_used_meta":729915,"kb_avail":93641580,"statfs":{"total":96636764160,"available":95888977920,"internally_reserved":0,"allocated":331776,"data_stored":121195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":747433677},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":86016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:46:35.176 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:46:35.509 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:46:35.510 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:46:35.523 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":431,"stamp":"2024-09-05T16:46:35.097790+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":154928,"num_objects":23,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26132,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":180127,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3234,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2765428,"kb_used_data":1440,"kb_used_omap":14,"kb_used_meta":2763889,"kb_avail":374721932,"statfs":{"total":386547056640,"available":383715258368,"internally_reserved":0,"allocated":1474560,"data_stored":594712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2830222903},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1514,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1514,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.353661"},"pg_stats":[{"pgid":"1.f","version":"39'12904","reported_seq":17500,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-05T16:46:27.826329+0000","last_change":"2024-09-05T16:46:27.826222+0000","last_active":"2024-09-05T16:46:27.826329+0000","last_peered":"2024-09-05T16:46:27.826329+0000","last_clean":"2024-09-05T16:46:27.826329+0000","last_became_active":"2024-09-05T16:46:27.757928+0000","last_became_peered":"2024-09-05T16:46:27.757928+0000","last_unstale":"2024-09-05T16:46:27.826329+0000","last_undegraded":"2024-09-05T16:46:27.826329+0000","last_fullsized":"2024-09-05T16:46:27.826329+0000","mapping_epoch":40,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:32:35.713428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12904,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2262,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12552","reported_seq":17213,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-05T16:46:27.756436+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:46:27.756436+0000","last_peered":"2024-09-05T16:46:27.756436+0000","last_clean":"2024-09-05T16:46:27.756436+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:46:27.756436+0000","last_undegraded":"2024-09-05T16:46:27.756436+0000","last_fullsized":"2024-09-05T16:46:27.756436+0000","mapping_epoch":38,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1865,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12752","reported_seq":12884,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:27.756492+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:46:27.756492+0000","last_peered":"2024-09-05T16:46:27.756492+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:46:27.756492+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12751,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2231}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12934","reported_seq":17673,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-05T16:46:27.755577+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:46:27.755577+0000","last_peered":"2024-09-05T16:46:27.755577+0000","last_clean":"2024-09-05T16:46:27.755577+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:46:27.755577+0000","last_undegraded":"2024-09-05T16:46:27.755577+0000","last_fullsized":"2024-09-05T16:46:27.755577+0000","mapping_epoch":34,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7234,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12593","reported_seq":12702,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:27.753309+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:46:27.753309+0000","last_peered":"2024-09-05T16:46:27.753309+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:46:27.753309+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12591,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2143}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12772","reported_seq":12896,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:27.752538+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:46:27.752538+0000","last_peered":"2024-09-05T16:46:27.752538+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:46:27.752538+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12772,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2219}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12660","reported_seq":12777,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:27.749807+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:46:27.749807+0000","last_peered":"2024-09-05T16:46:27.749807+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:46:27.749807+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12658,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2129}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12777","reported_seq":14314,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:31.343310+0000","last_change":"2024-09-05T16:46:26.428781+0000","last_active":"2024-09-05T16:46:31.343310+0000","last_peered":"2024-09-05T16:46:31.343310+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:46:31.343310+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1515,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12775,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":728,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1513}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"39'12770","reported_seq":12880,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:27.750634+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:46:27.750634+0000","last_peered":"2024-09-05T16:46:27.750634+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:46:27.750634+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12767,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2240}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12808","reported_seq":12940,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:27.757112+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:46:27.757112+0000","last_peered":"2024-09-05T16:46:27.757112+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:46:27.757112+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12806,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2243}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13120","reported_seq":13256,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:27.750509+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:46:27.750509+0000","last_peered":"2024-09-05T16:46:27.750509+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:46:27.750509+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13117,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2278}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12758","reported_seq":12866,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:27.751040+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:46:27.751040+0000","last_peered":"2024-09-05T16:46:27.751040+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:46:27.751040+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2237}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12808","reported_seq":17553,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-05T16:46:27.755657+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:46:27.755657+0000","last_peered":"2024-09-05T16:46:27.755657+0000","last_clean":"2024-09-05T16:46:27.755657+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:46:27.755657+0000","last_undegraded":"2024-09-05T16:46:27.755657+0000","last_fullsized":"2024-09-05T16:46:27.755657+0000","mapping_epoch":36,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4408,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13124","reported_seq":13242,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:27.753160+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:46:27.753160+0000","last_peered":"2024-09-05T16:46:27.753160+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:46:27.753160+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13122,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2332}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12831","reported_seq":12953,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:27.756269+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:46:27.756269+0000","last_peered":"2024-09-05T16:46:27.756269+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:46:27.756269+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12829,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2306}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'12773","reported_seq":12891,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:27.750088+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:46:27.750088+0000","last_peered":"2024-09-05T16:46:27.750088+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:46:27.750088+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12771,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2218}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":154928,"num_objects":23,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26132,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":180127,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3234,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":98304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084438,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722028,"kb_used_data":340,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649812,"statfs":{"total":96636764160,"available":95897407488,"internally_reserved":0,"allocated":348160,"data_stored":128198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":738979539},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986315,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":599924,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":599483,"kb_avail":93771916,"statfs":{"total":96636764160,"available":96022441984,"internally_reserved":0,"allocated":430080,"data_stored":210118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":613871314},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673121,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713192,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":93658648,"statfs":{"total":96636764160,"available":95906455552,"internally_reserved":0,"allocated":348160,"data_stored":128198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":729938373},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182552,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":730284,"kb_used_data":340,"kb_used_omap":4,"kb_used_meta":729915,"kb_avail":93641556,"statfs":{"total":96636764160,"available":95888953344,"internally_reserved":0,"allocated":348160,"data_stored":128198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":747433677},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":86016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:46:35.524 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:46:35.843 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:46:35.843 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:46:35.856 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":431,"stamp":"2024-09-05T16:46:35.097790+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":154928,"num_objects":23,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26132,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":180127,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3234,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2765428,"kb_used_data":1440,"kb_used_omap":14,"kb_used_meta":2763889,"kb_avail":374721932,"statfs":{"total":386547056640,"available":383715258368,"internally_reserved":0,"allocated":1474560,"data_stored":594712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2830222903},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1514,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1514,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.353661"},"pg_stats":[{"pgid":"1.f","version":"39'12904","reported_seq":17500,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-05T16:46:27.826329+0000","last_change":"2024-09-05T16:46:27.826222+0000","last_active":"2024-09-05T16:46:27.826329+0000","last_peered":"2024-09-05T16:46:27.826329+0000","last_clean":"2024-09-05T16:46:27.826329+0000","last_became_active":"2024-09-05T16:46:27.757928+0000","last_became_peered":"2024-09-05T16:46:27.757928+0000","last_unstale":"2024-09-05T16:46:27.826329+0000","last_undegraded":"2024-09-05T16:46:27.826329+0000","last_fullsized":"2024-09-05T16:46:27.826329+0000","mapping_epoch":40,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:32:35.713428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12904,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2262,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12552","reported_seq":17213,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-05T16:46:27.756436+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:46:27.756436+0000","last_peered":"2024-09-05T16:46:27.756436+0000","last_clean":"2024-09-05T16:46:27.756436+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:46:27.756436+0000","last_undegraded":"2024-09-05T16:46:27.756436+0000","last_fullsized":"2024-09-05T16:46:27.756436+0000","mapping_epoch":38,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1865,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12752","reported_seq":12884,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:27.756492+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:46:27.756492+0000","last_peered":"2024-09-05T16:46:27.756492+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:46:27.756492+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12751,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2231}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12934","reported_seq":17673,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-05T16:46:27.755577+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:46:27.755577+0000","last_peered":"2024-09-05T16:46:27.755577+0000","last_clean":"2024-09-05T16:46:27.755577+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:46:27.755577+0000","last_undegraded":"2024-09-05T16:46:27.755577+0000","last_fullsized":"2024-09-05T16:46:27.755577+0000","mapping_epoch":34,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7234,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12593","reported_seq":12702,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:27.753309+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:46:27.753309+0000","last_peered":"2024-09-05T16:46:27.753309+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:46:27.753309+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12591,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2143}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12772","reported_seq":12896,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:27.752538+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:46:27.752538+0000","last_peered":"2024-09-05T16:46:27.752538+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:46:27.752538+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12772,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2219}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12660","reported_seq":12777,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:27.749807+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:46:27.749807+0000","last_peered":"2024-09-05T16:46:27.749807+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:46:27.749807+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12658,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2129}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12777","reported_seq":14314,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:31.343310+0000","last_change":"2024-09-05T16:46:26.428781+0000","last_active":"2024-09-05T16:46:31.343310+0000","last_peered":"2024-09-05T16:46:31.343310+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:46:31.343310+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1515,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12775,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":728,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1513}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"39'12770","reported_seq":12880,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:27.750634+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:46:27.750634+0000","last_peered":"2024-09-05T16:46:27.750634+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:46:27.750634+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12767,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2240}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12808","reported_seq":12940,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:27.757112+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:46:27.757112+0000","last_peered":"2024-09-05T16:46:27.757112+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:46:27.757112+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12806,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2243}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13120","reported_seq":13256,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:27.750509+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:46:27.750509+0000","last_peered":"2024-09-05T16:46:27.750509+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:46:27.750509+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13117,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2278}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12758","reported_seq":12866,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:27.751040+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:46:27.751040+0000","last_peered":"2024-09-05T16:46:27.751040+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:46:27.751040+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2237}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12808","reported_seq":17553,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-05T16:46:27.755657+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:46:27.755657+0000","last_peered":"2024-09-05T16:46:27.755657+0000","last_clean":"2024-09-05T16:46:27.755657+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:46:27.755657+0000","last_undegraded":"2024-09-05T16:46:27.755657+0000","last_fullsized":"2024-09-05T16:46:27.755657+0000","mapping_epoch":36,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4408,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13124","reported_seq":13242,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:27.753160+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:46:27.753160+0000","last_peered":"2024-09-05T16:46:27.753160+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:46:27.753160+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13122,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2332}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12831","reported_seq":12953,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:27.756269+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:46:27.756269+0000","last_peered":"2024-09-05T16:46:27.756269+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:46:27.756269+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12829,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2306}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'12773","reported_seq":12891,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:27.750088+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:46:27.750088+0000","last_peered":"2024-09-05T16:46:27.750088+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:46:27.750088+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12771,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2218}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":154928,"num_objects":23,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26132,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":180127,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3234,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":98304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084438,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722028,"kb_used_data":340,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649812,"statfs":{"total":96636764160,"available":95897407488,"internally_reserved":0,"allocated":348160,"data_stored":128198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":738979539},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986315,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":599924,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":599483,"kb_avail":93771916,"statfs":{"total":96636764160,"available":96022441984,"internally_reserved":0,"allocated":430080,"data_stored":210118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":613871314},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673121,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713192,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":93658648,"statfs":{"total":96636764160,"available":95906455552,"internally_reserved":0,"allocated":348160,"data_stored":128198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":729938373},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182552,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":730284,"kb_used_data":340,"kb_used_omap":4,"kb_used_meta":729915,"kb_avail":93641556,"statfs":{"total":96636764160,"available":95888953344,"internally_reserved":0,"allocated":348160,"data_stored":128198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":747433677},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":86016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:46:35.857 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:46:35.857 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "39'12752", 'reported_seq': 12884, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:27.756492+0000', 'last_change': '2024-09-05T16:34:56.867900+0000', 'last_active': '2024-09-05T16:46:27.756492+0000', 'last_peered': '2024-09-05T16:46:27.756492+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:46:27.756492+0000', 'last_undegraded': '2024-09-05T16:34:56.590972+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 32, 'log_start': "33'2700", 'ondisk_log_start': "33'2700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10052, 'log_dups_size': 0, 'ondisk_log_size': 10052, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6736, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2233, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12751, 'num_write_kb': 50999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -2231}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:46:35.857 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:46:35.857 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "39'12593", 'reported_seq': 12702, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:27.753309+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:46:27.753309+0000', 'last_peered': '2024-09-05T16:46:27.753309+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:46:27.753309+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "33'2500", 'ondisk_log_start': "33'2500", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10093, 'log_dups_size': 0, 'ondisk_log_size': 10093, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2147, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12591, 'num_write_kb': 50373, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2143}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:46:35.857 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:46:35.857 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "39'12772", 'reported_seq': 12896, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:27.752538+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:46:27.752538+0000', 'last_peered': '2024-09-05T16:46:27.752538+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:46:27.752538+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "33'2700", 'ondisk_log_start': "33'2700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10072, 'log_dups_size': 0, 'ondisk_log_size': 10072, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12772, 'num_write_kb': 51088, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2219}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:46:35.857 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:46:35.857 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "39'12660", 'reported_seq': 12777, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:27.749807+0000', 'last_change': '2024-09-05T16:34:56.903436+0000', 'last_active': '2024-09-05T16:46:27.749807+0000', 'last_peered': '2024-09-05T16:46:27.749807+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:46:27.749807+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "33'2600", 'ondisk_log_start': "33'2600", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10060, 'log_dups_size': 0, 'ondisk_log_size': 10060, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2133, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12658, 'num_write_kb': 50636, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -2129}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:46:35.858 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-05T16:46:35.858 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "39'12777", 'reported_seq': 14314, 'reported_epoch': 41, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:31.343310+0000', 'last_change': '2024-09-05T16:46:26.428781+0000', 'last_active': '2024-09-05T16:46:31.343310+0000', 'last_peered': '2024-09-05T16:46:31.343310+0000', 'last_clean': '2024-09-05T16:33:53.505305+0000', 'last_became_active': '2024-09-05T16:34:56.873389+0000', 'last_became_peered': '2024-09-05T16:34:56.873389+0000', 'last_unstale': '2024-09-05T16:46:31.343310+0000', 'last_undegraded': '2024-09-05T16:34:56.602567+0000', 'last_fullsized': '2024-09-05T16:34:56.559958+0000', 'mapping_epoch': 32, 'log_start': "33'2700", 'ondisk_log_start': "33'2700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10077, 'log_dups_size': 0, 'ondisk_log_size': 10077, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1515, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12775, 'num_write_kb': 51108, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 728, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1513}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:46:35.858 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:46:35.858 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "39'12770", 'reported_seq': 12880, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:27.750634+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:46:27.750634+0000', 'last_peered': '2024-09-05T16:46:27.750634+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:46:27.750634+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "33'2700", 'ondisk_log_start': "33'2700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10070, 'log_dups_size': 0, 'ondisk_log_size': 10070, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26944, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 16, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12767, 'num_write_kb': 51081, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2240}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:46:35.858 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:46:35.858 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "39'12808", 'reported_seq': 12940, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:27.757112+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:46:27.757112+0000', 'last_peered': '2024-09-05T16:46:27.757112+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:46:27.757112+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "33'2800", 'ondisk_log_start': "33'2800", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10008, 'log_dups_size': 0, 'ondisk_log_size': 10008, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12806, 'num_write_kb': 51230, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -2243}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:46:35.858 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:46:35.858 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "39'13120", 'reported_seq': 13256, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:27.750509+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:46:27.750509+0000', 'last_peered': '2024-09-05T16:46:27.750509+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:46:27.750509+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "33'3100", 'ondisk_log_start': "33'3100", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20208, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2284, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13117, 'num_write_kb': 52476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2278}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:46:35.858 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:46:35.858 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "39'12758", 'reported_seq': 12866, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:27.751040+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:46:27.751040+0000', 'last_peered': '2024-09-05T16:46:27.751040+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:46:27.751040+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "33'2700", 'ondisk_log_start': "33'2700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10058, 'log_dups_size': 0, 'ondisk_log_size': 10058, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6736, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12757, 'num_write_kb': 51031, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2237}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:46:35.858 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:46:35.859 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "39'13124", 'reported_seq': 13242, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:27.753160+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:46:27.753160+0000', 'last_peered': '2024-09-05T16:46:27.753160+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:46:27.753160+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "33'3100", 'ondisk_log_start': "33'3100", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10024, 'log_dups_size': 0, 'ondisk_log_size': 10024, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13122, 'num_write_kb': 52500, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2332}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:46:35.859 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:46:35.859 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "39'12831", 'reported_seq': 12953, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:27.756269+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:46:27.756269+0000', 'last_peered': '2024-09-05T16:46:27.756269+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:46:27.756269+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "33'2800", 'ondisk_log_start': "33'2800", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10031, 'log_dups_size': 0, 'ondisk_log_size': 10031, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2310, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12829, 'num_write_kb': 51325, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -2306}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:46:35.859 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:46:35.859 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "39'12773", 'reported_seq': 12891, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:27.750088+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:46:27.750088+0000', 'last_peered': '2024-09-05T16:46:27.750088+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:46:27.750088+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "33'2700", 'ondisk_log_start': "33'2700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10073, 'log_dups_size': 0, 'ondisk_log_size': 10073, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12771, 'num_write_kb': 51092, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2218}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:46:35.859 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:46:36.250 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:46:36.250 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 26132/92 objects degraded (28404.348%), 12 pgs degraded, 12 pgs undersized","count":24},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":799,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":41,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+clean","count":4},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":23,"data_bytes":154928,"bytes_used":2831798272,"bytes_avail":383715258368,"bytes_total":386547056640,"degraded_objects":26132,"degraded_total":92,"degraded_ratio":284.04347826086956,"recovering_objects_per_sec":181,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1514,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:46:36.263 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:46:36.264 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:46:36.560 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:46:36.560 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:46:36.572 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":431,"stamp":"2024-09-05T16:46:35.097790+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":154928,"num_objects":23,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26132,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":180127,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3234,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2765428,"kb_used_data":1440,"kb_used_omap":14,"kb_used_meta":2763889,"kb_avail":374721932,"statfs":{"total":386547056640,"available":383715258368,"internally_reserved":0,"allocated":1474560,"data_stored":594712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2830222903},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1514,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1514,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.353661"},"pg_stats":[{"pgid":"1.f","version":"39'12904","reported_seq":17500,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-05T16:46:27.826329+0000","last_change":"2024-09-05T16:46:27.826222+0000","last_active":"2024-09-05T16:46:27.826329+0000","last_peered":"2024-09-05T16:46:27.826329+0000","last_clean":"2024-09-05T16:46:27.826329+0000","last_became_active":"2024-09-05T16:46:27.757928+0000","last_became_peered":"2024-09-05T16:46:27.757928+0000","last_unstale":"2024-09-05T16:46:27.826329+0000","last_undegraded":"2024-09-05T16:46:27.826329+0000","last_fullsized":"2024-09-05T16:46:27.826329+0000","mapping_epoch":40,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:32:35.713428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12904,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2262,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12552","reported_seq":17213,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-05T16:46:27.756436+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:46:27.756436+0000","last_peered":"2024-09-05T16:46:27.756436+0000","last_clean":"2024-09-05T16:46:27.756436+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:46:27.756436+0000","last_undegraded":"2024-09-05T16:46:27.756436+0000","last_fullsized":"2024-09-05T16:46:27.756436+0000","mapping_epoch":38,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1865,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12752","reported_seq":12884,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:27.756492+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:46:27.756492+0000","last_peered":"2024-09-05T16:46:27.756492+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:46:27.756492+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12751,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2231}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12934","reported_seq":17673,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-05T16:46:27.755577+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:46:27.755577+0000","last_peered":"2024-09-05T16:46:27.755577+0000","last_clean":"2024-09-05T16:46:27.755577+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:46:27.755577+0000","last_undegraded":"2024-09-05T16:46:27.755577+0000","last_fullsized":"2024-09-05T16:46:27.755577+0000","mapping_epoch":34,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7234,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12593","reported_seq":12702,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:27.753309+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:46:27.753309+0000","last_peered":"2024-09-05T16:46:27.753309+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:46:27.753309+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12591,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2143}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12772","reported_seq":12896,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:27.752538+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:46:27.752538+0000","last_peered":"2024-09-05T16:46:27.752538+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:46:27.752538+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12772,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2219}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12660","reported_seq":12777,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:27.749807+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:46:27.749807+0000","last_peered":"2024-09-05T16:46:27.749807+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:46:27.749807+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12658,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2129}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12777","reported_seq":14314,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:31.343310+0000","last_change":"2024-09-05T16:46:26.428781+0000","last_active":"2024-09-05T16:46:31.343310+0000","last_peered":"2024-09-05T16:46:31.343310+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:46:31.343310+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1515,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12775,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":728,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1513}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"39'12770","reported_seq":12880,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:27.750634+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:46:27.750634+0000","last_peered":"2024-09-05T16:46:27.750634+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:46:27.750634+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12767,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2240}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12808","reported_seq":12940,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:27.757112+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:46:27.757112+0000","last_peered":"2024-09-05T16:46:27.757112+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:46:27.757112+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12806,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2243}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13120","reported_seq":13256,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:27.750509+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:46:27.750509+0000","last_peered":"2024-09-05T16:46:27.750509+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:46:27.750509+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13117,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2278}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12758","reported_seq":12866,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:27.751040+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:46:27.751040+0000","last_peered":"2024-09-05T16:46:27.751040+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:46:27.751040+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2237}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12808","reported_seq":17553,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-05T16:46:27.755657+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:46:27.755657+0000","last_peered":"2024-09-05T16:46:27.755657+0000","last_clean":"2024-09-05T16:46:27.755657+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:46:27.755657+0000","last_undegraded":"2024-09-05T16:46:27.755657+0000","last_fullsized":"2024-09-05T16:46:27.755657+0000","mapping_epoch":36,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4408,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13124","reported_seq":13242,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:27.753160+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:46:27.753160+0000","last_peered":"2024-09-05T16:46:27.753160+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:46:27.753160+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13122,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2332}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12831","reported_seq":12953,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:27.756269+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:46:27.756269+0000","last_peered":"2024-09-05T16:46:27.756269+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:46:27.756269+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12829,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2306}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'12773","reported_seq":12891,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:27.750088+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:46:27.750088+0000","last_peered":"2024-09-05T16:46:27.750088+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:46:27.750088+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12771,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2218}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":154928,"num_objects":23,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26132,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":180127,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3234,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":98304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084438,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722028,"kb_used_data":340,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649812,"statfs":{"total":96636764160,"available":95897407488,"internally_reserved":0,"allocated":348160,"data_stored":128198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":738979539},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986315,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":599924,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":599483,"kb_avail":93771916,"statfs":{"total":96636764160,"available":96022441984,"internally_reserved":0,"allocated":430080,"data_stored":210118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":613871314},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673121,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713192,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":93658648,"statfs":{"total":96636764160,"available":95906455552,"internally_reserved":0,"allocated":348160,"data_stored":128198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":729938373},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182552,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":730284,"kb_used_data":340,"kb_used_omap":4,"kb_used_meta":729915,"kb_avail":93641556,"statfs":{"total":96636764160,"available":95888953344,"internally_reserved":0,"allocated":348160,"data_stored":128198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":747433677},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":86016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:46:39.574 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:46:39.925 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:46:39.925 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:46:39.938 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":434,"stamp":"2024-09-05T16:46:39.172423+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":154928,"num_objects":23,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25227,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":180127,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4139,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2757108,"kb_used_data":1440,"kb_used_omap":14,"kb_used_meta":2755569,"kb_avail":374730252,"statfs":{"total":386547056640,"available":383723778048,"internally_reserved":0,"allocated":1474560,"data_stored":594712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2821703223},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1624,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1624,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.075841"},"pg_stats":[{"pgid":"1.f","version":"39'12904","reported_seq":17500,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-05T16:46:27.826329+0000","last_change":"2024-09-05T16:46:27.826222+0000","last_active":"2024-09-05T16:46:27.826329+0000","last_peered":"2024-09-05T16:46:27.826329+0000","last_clean":"2024-09-05T16:46:27.826329+0000","last_became_active":"2024-09-05T16:46:27.757928+0000","last_became_peered":"2024-09-05T16:46:27.757928+0000","last_unstale":"2024-09-05T16:46:27.826329+0000","last_undegraded":"2024-09-05T16:46:27.826329+0000","last_fullsized":"2024-09-05T16:46:27.826329+0000","mapping_epoch":40,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:32:35.713428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12904,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2262,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12552","reported_seq":17213,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-05T16:46:27.756436+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:46:27.756436+0000","last_peered":"2024-09-05T16:46:27.756436+0000","last_clean":"2024-09-05T16:46:27.756436+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:46:27.756436+0000","last_undegraded":"2024-09-05T16:46:27.756436+0000","last_fullsized":"2024-09-05T16:46:27.756436+0000","mapping_epoch":38,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1865,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12752","reported_seq":12884,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:27.756492+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:46:27.756492+0000","last_peered":"2024-09-05T16:46:27.756492+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:46:27.756492+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12751,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2231}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12934","reported_seq":17673,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-05T16:46:27.755577+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:46:27.755577+0000","last_peered":"2024-09-05T16:46:27.755577+0000","last_clean":"2024-09-05T16:46:27.755577+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:46:27.755577+0000","last_undegraded":"2024-09-05T16:46:27.755577+0000","last_fullsized":"2024-09-05T16:46:27.755577+0000","mapping_epoch":34,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7234,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12593","reported_seq":12702,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:27.753309+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:46:27.753309+0000","last_peered":"2024-09-05T16:46:27.753309+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:46:27.753309+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12591,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2143}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12772","reported_seq":12896,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:27.752538+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:46:27.752538+0000","last_peered":"2024-09-05T16:46:27.752538+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:46:27.752538+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12772,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2219}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12660","reported_seq":12777,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:27.749807+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:46:27.749807+0000","last_peered":"2024-09-05T16:46:27.749807+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:46:27.749807+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12658,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2129}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12777","reported_seq":16124,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:36.346936+0000","last_change":"2024-09-05T16:46:26.428781+0000","last_active":"2024-09-05T16:46:36.346936+0000","last_peered":"2024-09-05T16:46:36.346936+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:46:36.346936+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":610,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12775,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1633,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-608}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"39'12770","reported_seq":12880,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:27.750634+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:46:27.750634+0000","last_peered":"2024-09-05T16:46:27.750634+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:46:27.750634+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12767,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2240}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12808","reported_seq":12940,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:27.757112+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:46:27.757112+0000","last_peered":"2024-09-05T16:46:27.757112+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:46:27.757112+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12806,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2243}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13120","reported_seq":13256,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:27.750509+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:46:27.750509+0000","last_peered":"2024-09-05T16:46:27.750509+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:46:27.750509+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13117,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2278}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12758","reported_seq":12866,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:27.751040+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:46:27.751040+0000","last_peered":"2024-09-05T16:46:27.751040+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:46:27.751040+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2237}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12808","reported_seq":17553,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-05T16:46:27.755657+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:46:27.755657+0000","last_peered":"2024-09-05T16:46:27.755657+0000","last_clean":"2024-09-05T16:46:27.755657+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:46:27.755657+0000","last_undegraded":"2024-09-05T16:46:27.755657+0000","last_fullsized":"2024-09-05T16:46:27.755657+0000","mapping_epoch":36,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4408,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13124","reported_seq":13242,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:27.753160+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:46:27.753160+0000","last_peered":"2024-09-05T16:46:27.753160+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:46:27.753160+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13122,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2332}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12831","reported_seq":12953,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:27.756269+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:46:27.756269+0000","last_peered":"2024-09-05T16:46:27.756269+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:46:27.756269+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12829,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2306}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'12773","reported_seq":12891,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:27.750088+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:46:27.750088+0000","last_peered":"2024-09-05T16:46:27.750088+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:46:27.750088+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12771,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2218}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":154928,"num_objects":23,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25227,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":180127,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4139,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":98304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084439,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722028,"kb_used_data":340,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649812,"statfs":{"total":96636764160,"available":95897407488,"internally_reserved":0,"allocated":348160,"data_stored":128198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":738979539},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986316,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":591604,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":591163,"kb_avail":93780236,"statfs":{"total":96636764160,"available":96030961664,"internally_reserved":0,"allocated":430080,"data_stored":210118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":605351634},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673122,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713192,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":93658648,"statfs":{"total":96636764160,"available":95906455552,"internally_reserved":0,"allocated":348160,"data_stored":128198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":729938373},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182553,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":730284,"kb_used_data":340,"kb_used_omap":4,"kb_used_meta":729915,"kb_avail":93641556,"statfs":{"total":96636764160,"available":95888953344,"internally_reserved":0,"allocated":348160,"data_stored":128198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":747433677},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":86016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:46:39.939 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:46:40.295 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:46:40.295 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:46:40.309 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":435,"stamp":"2024-09-05T16:46:40.177038+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":154928,"num_objects":23,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25227,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":180127,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4139,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2757108,"kb_used_data":1440,"kb_used_omap":14,"kb_used_meta":2755569,"kb_avail":374730252,"statfs":{"total":386547056640,"available":383723778048,"internally_reserved":0,"allocated":1474560,"data_stored":594712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2821703223},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1624,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1624,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.080096"},"pg_stats":[{"pgid":"1.f","version":"39'12904","reported_seq":17500,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-05T16:46:27.826329+0000","last_change":"2024-09-05T16:46:27.826222+0000","last_active":"2024-09-05T16:46:27.826329+0000","last_peered":"2024-09-05T16:46:27.826329+0000","last_clean":"2024-09-05T16:46:27.826329+0000","last_became_active":"2024-09-05T16:46:27.757928+0000","last_became_peered":"2024-09-05T16:46:27.757928+0000","last_unstale":"2024-09-05T16:46:27.826329+0000","last_undegraded":"2024-09-05T16:46:27.826329+0000","last_fullsized":"2024-09-05T16:46:27.826329+0000","mapping_epoch":40,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:32:35.713428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12904,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2262,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12552","reported_seq":17213,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-05T16:46:27.756436+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:46:27.756436+0000","last_peered":"2024-09-05T16:46:27.756436+0000","last_clean":"2024-09-05T16:46:27.756436+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:46:27.756436+0000","last_undegraded":"2024-09-05T16:46:27.756436+0000","last_fullsized":"2024-09-05T16:46:27.756436+0000","mapping_epoch":38,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1865,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12752","reported_seq":12884,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:27.756492+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:46:27.756492+0000","last_peered":"2024-09-05T16:46:27.756492+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:46:27.756492+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12751,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2231}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12934","reported_seq":17673,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-05T16:46:27.755577+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:46:27.755577+0000","last_peered":"2024-09-05T16:46:27.755577+0000","last_clean":"2024-09-05T16:46:27.755577+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:46:27.755577+0000","last_undegraded":"2024-09-05T16:46:27.755577+0000","last_fullsized":"2024-09-05T16:46:27.755577+0000","mapping_epoch":34,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7234,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12593","reported_seq":12702,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:27.753309+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:46:27.753309+0000","last_peered":"2024-09-05T16:46:27.753309+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:46:27.753309+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12591,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2143}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12772","reported_seq":12896,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:27.752538+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:46:27.752538+0000","last_peered":"2024-09-05T16:46:27.752538+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:46:27.752538+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12772,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2219}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12660","reported_seq":12777,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:27.749807+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:46:27.749807+0000","last_peered":"2024-09-05T16:46:27.749807+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:46:27.749807+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12658,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2129}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12777","reported_seq":16124,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:36.346936+0000","last_change":"2024-09-05T16:46:26.428781+0000","last_active":"2024-09-05T16:46:36.346936+0000","last_peered":"2024-09-05T16:46:36.346936+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:46:36.346936+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":610,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12775,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1633,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-608}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"39'12770","reported_seq":12880,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:27.750634+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:46:27.750634+0000","last_peered":"2024-09-05T16:46:27.750634+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:46:27.750634+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12767,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2240}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12808","reported_seq":12940,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:27.757112+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:46:27.757112+0000","last_peered":"2024-09-05T16:46:27.757112+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:46:27.757112+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12806,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2243}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13120","reported_seq":13256,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:27.750509+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:46:27.750509+0000","last_peered":"2024-09-05T16:46:27.750509+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:46:27.750509+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13117,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2278}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12758","reported_seq":12866,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:27.751040+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:46:27.751040+0000","last_peered":"2024-09-05T16:46:27.751040+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:46:27.751040+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2237}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12808","reported_seq":17553,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-05T16:46:27.755657+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:46:27.755657+0000","last_peered":"2024-09-05T16:46:27.755657+0000","last_clean":"2024-09-05T16:46:27.755657+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:46:27.755657+0000","last_undegraded":"2024-09-05T16:46:27.755657+0000","last_fullsized":"2024-09-05T16:46:27.755657+0000","mapping_epoch":36,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4408,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13124","reported_seq":13242,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:27.753160+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:46:27.753160+0000","last_peered":"2024-09-05T16:46:27.753160+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:46:27.753160+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13122,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2332}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12831","reported_seq":12953,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:27.756269+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:46:27.756269+0000","last_peered":"2024-09-05T16:46:27.756269+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:46:27.756269+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12829,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2306}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'12773","reported_seq":12891,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:27.750088+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:46:27.750088+0000","last_peered":"2024-09-05T16:46:27.750088+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:46:27.750088+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12771,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2218}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":154928,"num_objects":23,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25227,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":180127,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4139,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":98304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084439,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722028,"kb_used_data":340,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649812,"statfs":{"total":96636764160,"available":95897407488,"internally_reserved":0,"allocated":348160,"data_stored":128198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":738979539},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986316,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":591604,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":591163,"kb_avail":93780236,"statfs":{"total":96636764160,"available":96030961664,"internally_reserved":0,"allocated":430080,"data_stored":210118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":605351634},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673122,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713192,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":93658648,"statfs":{"total":96636764160,"available":95906455552,"internally_reserved":0,"allocated":348160,"data_stored":128198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":729938373},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182553,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":730284,"kb_used_data":340,"kb_used_omap":4,"kb_used_meta":729915,"kb_avail":93641556,"statfs":{"total":96636764160,"available":95888953344,"internally_reserved":0,"allocated":348160,"data_stored":128198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":747433677},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":86016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:46:40.310 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:46:40.310 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "39'12752", 'reported_seq': 12884, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:27.756492+0000', 'last_change': '2024-09-05T16:34:56.867900+0000', 'last_active': '2024-09-05T16:46:27.756492+0000', 'last_peered': '2024-09-05T16:46:27.756492+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:46:27.756492+0000', 'last_undegraded': '2024-09-05T16:34:56.590972+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 32, 'log_start': "33'2700", 'ondisk_log_start': "33'2700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10052, 'log_dups_size': 0, 'ondisk_log_size': 10052, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6736, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2233, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12751, 'num_write_kb': 50999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -2231}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:46:40.310 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:46:40.310 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "39'12593", 'reported_seq': 12702, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:27.753309+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:46:27.753309+0000', 'last_peered': '2024-09-05T16:46:27.753309+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:46:27.753309+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "33'2500", 'ondisk_log_start': "33'2500", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10093, 'log_dups_size': 0, 'ondisk_log_size': 10093, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2147, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12591, 'num_write_kb': 50373, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2143}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:46:40.311 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:46:40.311 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "39'12772", 'reported_seq': 12896, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:27.752538+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:46:27.752538+0000', 'last_peered': '2024-09-05T16:46:27.752538+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:46:27.752538+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "33'2700", 'ondisk_log_start': "33'2700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10072, 'log_dups_size': 0, 'ondisk_log_size': 10072, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12772, 'num_write_kb': 51088, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2219}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:46:40.311 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:46:40.311 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "39'12660", 'reported_seq': 12777, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:27.749807+0000', 'last_change': '2024-09-05T16:34:56.903436+0000', 'last_active': '2024-09-05T16:46:27.749807+0000', 'last_peered': '2024-09-05T16:46:27.749807+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:46:27.749807+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "33'2600", 'ondisk_log_start': "33'2600", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10060, 'log_dups_size': 0, 'ondisk_log_size': 10060, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2133, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12658, 'num_write_kb': 50636, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -2129}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:46:40.311 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-05T16:46:40.311 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "39'12777", 'reported_seq': 16124, 'reported_epoch': 41, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:36.346936+0000', 'last_change': '2024-09-05T16:46:26.428781+0000', 'last_active': '2024-09-05T16:46:36.346936+0000', 'last_peered': '2024-09-05T16:46:36.346936+0000', 'last_clean': '2024-09-05T16:33:53.505305+0000', 'last_became_active': '2024-09-05T16:34:56.873389+0000', 'last_became_peered': '2024-09-05T16:34:56.873389+0000', 'last_unstale': '2024-09-05T16:46:36.346936+0000', 'last_undegraded': '2024-09-05T16:34:56.602567+0000', 'last_fullsized': '2024-09-05T16:34:56.559958+0000', 'mapping_epoch': 32, 'log_start': "33'2700", 'ondisk_log_start': "33'2700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10077, 'log_dups_size': 0, 'ondisk_log_size': 10077, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 610, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12775, 'num_write_kb': 51108, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1633, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -608}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:46:40.311 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:46:40.311 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "39'12770", 'reported_seq': 12880, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:27.750634+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:46:27.750634+0000', 'last_peered': '2024-09-05T16:46:27.750634+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:46:27.750634+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "33'2700", 'ondisk_log_start': "33'2700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10070, 'log_dups_size': 0, 'ondisk_log_size': 10070, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26944, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 16, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12767, 'num_write_kb': 51081, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2240}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:46:40.311 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:46:40.311 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "39'12808", 'reported_seq': 12940, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:27.757112+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:46:27.757112+0000', 'last_peered': '2024-09-05T16:46:27.757112+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:46:27.757112+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "33'2800", 'ondisk_log_start': "33'2800", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10008, 'log_dups_size': 0, 'ondisk_log_size': 10008, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12806, 'num_write_kb': 51230, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -2243}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:46:40.311 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:46:40.311 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "39'13120", 'reported_seq': 13256, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:27.750509+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:46:27.750509+0000', 'last_peered': '2024-09-05T16:46:27.750509+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:46:27.750509+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "33'3100", 'ondisk_log_start': "33'3100", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20208, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2284, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13117, 'num_write_kb': 52476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2278}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:46:40.312 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:46:40.312 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "39'12758", 'reported_seq': 12866, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:27.751040+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:46:27.751040+0000', 'last_peered': '2024-09-05T16:46:27.751040+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:46:27.751040+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "33'2700", 'ondisk_log_start': "33'2700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10058, 'log_dups_size': 0, 'ondisk_log_size': 10058, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6736, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12757, 'num_write_kb': 51031, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2237}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:46:40.312 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:46:40.312 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "39'13124", 'reported_seq': 13242, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:27.753160+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:46:27.753160+0000', 'last_peered': '2024-09-05T16:46:27.753160+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:46:27.753160+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "33'3100", 'ondisk_log_start': "33'3100", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10024, 'log_dups_size': 0, 'ondisk_log_size': 10024, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13122, 'num_write_kb': 52500, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2332}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:46:40.312 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:46:40.312 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "39'12831", 'reported_seq': 12953, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:27.756269+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:46:27.756269+0000', 'last_peered': '2024-09-05T16:46:27.756269+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:46:27.756269+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "33'2800", 'ondisk_log_start': "33'2800", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10031, 'log_dups_size': 0, 'ondisk_log_size': 10031, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2310, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12829, 'num_write_kb': 51325, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -2306}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:46:40.312 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:46:40.312 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "39'12773", 'reported_seq': 12891, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:27.750088+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:46:27.750088+0000', 'last_peered': '2024-09-05T16:46:27.750088+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:46:27.750088+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "33'2700", 'ondisk_log_start': "33'2700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10073, 'log_dups_size': 0, 'ondisk_log_size': 10073, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12771, 'num_write_kb': 51092, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2218}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:46:40.312 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:46:40.710 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:46:40.710 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 25227/92 objects degraded (27420.652%), 12 pgs degraded, 12 pgs undersized","count":24},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":803,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":43,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+clean","count":4},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":23,"data_bytes":154928,"bytes_used":2823278592,"bytes_avail":383723778048,"bytes_total":386547056640,"degraded_objects":25227,"degraded_total":92,"degraded_ratio":274.20652173913044,"recovering_objects_per_sec":213,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":2419,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:46:40.723 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:46:40.723 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:46:41.026 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:46:41.027 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:46:41.039 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":435,"stamp":"2024-09-05T16:46:40.177038+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":154928,"num_objects":23,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25227,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":180127,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4139,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2757108,"kb_used_data":1440,"kb_used_omap":14,"kb_used_meta":2755569,"kb_avail":374730252,"statfs":{"total":386547056640,"available":383723778048,"internally_reserved":0,"allocated":1474560,"data_stored":594712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2821703223},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1624,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1624,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.080096"},"pg_stats":[{"pgid":"1.f","version":"39'12904","reported_seq":17500,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-05T16:46:27.826329+0000","last_change":"2024-09-05T16:46:27.826222+0000","last_active":"2024-09-05T16:46:27.826329+0000","last_peered":"2024-09-05T16:46:27.826329+0000","last_clean":"2024-09-05T16:46:27.826329+0000","last_became_active":"2024-09-05T16:46:27.757928+0000","last_became_peered":"2024-09-05T16:46:27.757928+0000","last_unstale":"2024-09-05T16:46:27.826329+0000","last_undegraded":"2024-09-05T16:46:27.826329+0000","last_fullsized":"2024-09-05T16:46:27.826329+0000","mapping_epoch":40,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:32:35.713428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12904,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2262,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12552","reported_seq":17213,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-05T16:46:27.756436+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:46:27.756436+0000","last_peered":"2024-09-05T16:46:27.756436+0000","last_clean":"2024-09-05T16:46:27.756436+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:46:27.756436+0000","last_undegraded":"2024-09-05T16:46:27.756436+0000","last_fullsized":"2024-09-05T16:46:27.756436+0000","mapping_epoch":38,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1865,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12752","reported_seq":12884,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:27.756492+0000","last_change":"2024-09-05T16:34:56.867900+0000","last_active":"2024-09-05T16:46:27.756492+0000","last_peered":"2024-09-05T16:46:27.756492+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:46:27.756492+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12751,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2231}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12934","reported_seq":17673,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-05T16:46:27.755577+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:46:27.755577+0000","last_peered":"2024-09-05T16:46:27.755577+0000","last_clean":"2024-09-05T16:46:27.755577+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:46:27.755577+0000","last_undegraded":"2024-09-05T16:46:27.755577+0000","last_fullsized":"2024-09-05T16:46:27.755577+0000","mapping_epoch":34,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7234,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12593","reported_seq":12702,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:27.753309+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:46:27.753309+0000","last_peered":"2024-09-05T16:46:27.753309+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:46:27.753309+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12591,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2143}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12772","reported_seq":12896,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:27.752538+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:46:27.752538+0000","last_peered":"2024-09-05T16:46:27.752538+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:46:27.752538+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12772,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2219}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12660","reported_seq":12777,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:27.749807+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:46:27.749807+0000","last_peered":"2024-09-05T16:46:27.749807+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:46:27.749807+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12658,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2129}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12777","reported_seq":16124,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:36.346936+0000","last_change":"2024-09-05T16:46:26.428781+0000","last_active":"2024-09-05T16:46:36.346936+0000","last_peered":"2024-09-05T16:46:36.346936+0000","last_clean":"2024-09-05T16:33:53.505305+0000","last_became_active":"2024-09-05T16:34:56.873389+0000","last_became_peered":"2024-09-05T16:34:56.873389+0000","last_unstale":"2024-09-05T16:46:36.346936+0000","last_undegraded":"2024-09-05T16:34:56.602567+0000","last_fullsized":"2024-09-05T16:34:56.559958+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":610,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12775,"num_write_kb":51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1633,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-608}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"39'12770","reported_seq":12880,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:27.750634+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:46:27.750634+0000","last_peered":"2024-09-05T16:46:27.750634+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:46:27.750634+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12767,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2240}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12808","reported_seq":12940,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:27.757112+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:46:27.757112+0000","last_peered":"2024-09-05T16:46:27.757112+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:46:27.757112+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12806,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2243}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13120","reported_seq":13256,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:27.750509+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:46:27.750509+0000","last_peered":"2024-09-05T16:46:27.750509+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:46:27.750509+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13117,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2278}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12758","reported_seq":12866,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:27.751040+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:46:27.751040+0000","last_peered":"2024-09-05T16:46:27.751040+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:46:27.751040+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2237}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12808","reported_seq":17553,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-05T16:46:27.755657+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:46:27.755657+0000","last_peered":"2024-09-05T16:46:27.755657+0000","last_clean":"2024-09-05T16:46:27.755657+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:46:27.755657+0000","last_undegraded":"2024-09-05T16:46:27.755657+0000","last_fullsized":"2024-09-05T16:46:27.755657+0000","mapping_epoch":36,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4408,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13124","reported_seq":13242,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:27.753160+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:46:27.753160+0000","last_peered":"2024-09-05T16:46:27.753160+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:46:27.753160+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13122,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2332}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12831","reported_seq":12953,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:27.756269+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:46:27.756269+0000","last_peered":"2024-09-05T16:46:27.756269+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:46:27.756269+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12829,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2306}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'12773","reported_seq":12891,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:27.750088+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:46:27.750088+0000","last_peered":"2024-09-05T16:46:27.750088+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:46:27.750088+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12771,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2218}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":154928,"num_objects":23,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25227,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":180127,"num_write_kb":672715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4139,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":98304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084439,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722028,"kb_used_data":340,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649812,"statfs":{"total":96636764160,"available":95897407488,"internally_reserved":0,"allocated":348160,"data_stored":128198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":738979539},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986316,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":591604,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":591163,"kb_avail":93780236,"statfs":{"total":96636764160,"available":96030961664,"internally_reserved":0,"allocated":430080,"data_stored":210118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":605351634},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673122,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713192,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":93658648,"statfs":{"total":96636764160,"available":95906455552,"internally_reserved":0,"allocated":348160,"data_stored":128198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":729938373},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182553,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":730284,"kb_used_data":340,"kb_used_omap":4,"kb_used_meta":729915,"kb_avail":93641556,"statfs":{"total":96636764160,"available":95888953344,"internally_reserved":0,"allocated":348160,"data_stored":128198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":747433677},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":86016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:46:44.042 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:46:44.377 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:46:44.378 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:46:44.389 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":437,"stamp":"2024-09-05T16:46:43.099391+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":141456,"num_objects":21,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":167352,"num_write_kb":621607,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2908,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2765476,"kb_used_data":1484,"kb_used_omap":14,"kb_used_meta":2763889,"kb_avail":374721884,"statfs":{"total":386547056640,"available":383715209216,"internally_reserved":0,"allocated":1519616,"data_stored":611403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2830222903},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13472,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1917,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12775,"num_write_kb":-51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001600"},"pg_stats":[{"pgid":"1.f","version":"39'12904","reported_seq":17500,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-05T16:46:27.826329+0000","last_change":"2024-09-05T16:46:27.826222+0000","last_active":"2024-09-05T16:46:27.826329+0000","last_peered":"2024-09-05T16:46:27.826329+0000","last_clean":"2024-09-05T16:46:27.826329+0000","last_became_active":"2024-09-05T16:46:27.757928+0000","last_became_peered":"2024-09-05T16:46:27.757928+0000","last_unstale":"2024-09-05T16:46:27.826329+0000","last_undegraded":"2024-09-05T16:46:27.826329+0000","last_fullsized":"2024-09-05T16:46:27.826329+0000","mapping_epoch":40,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:32:35.713428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12904,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2262,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12552","reported_seq":17217,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-05T16:46:40.178538+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:46:40.178538+0000","last_peered":"2024-09-05T16:46:40.178538+0000","last_clean":"2024-09-05T16:46:40.178538+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:46:40.178538+0000","last_undegraded":"2024-09-05T16:46:40.178538+0000","last_fullsized":"2024-09-05T16:46:40.178538+0000","mapping_epoch":38,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1865,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12752","reported_seq":13693,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:41.014188+0000","last_change":"2024-09-05T16:46:38.780739+0000","last_active":"2024-09-05T16:46:41.014188+0000","last_peered":"2024-09-05T16:46:41.014188+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:46:41.014188+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1831,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12751,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":421,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1830}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12934","reported_seq":17677,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-05T16:46:40.177938+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:46:40.177938+0000","last_peered":"2024-09-05T16:46:40.177938+0000","last_clean":"2024-09-05T16:46:40.177938+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:46:40.177938+0000","last_undegraded":"2024-09-05T16:46:40.177938+0000","last_fullsized":"2024-09-05T16:46:40.177938+0000","mapping_epoch":34,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7234,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12593","reported_seq":12702,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:27.753309+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:46:27.753309+0000","last_peered":"2024-09-05T16:46:27.753309+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:46:27.753309+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12591,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2143}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12772","reported_seq":12896,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:27.752538+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:46:27.752538+0000","last_peered":"2024-09-05T16:46:27.752538+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:46:27.752538+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12772,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2219}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12660","reported_seq":12782,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:40.177421+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:46:40.177421+0000","last_peered":"2024-09-05T16:46:40.177421+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:46:40.177421+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12658,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2129}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12777","reported_seq":12894,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-05T16:46:40.458401+0000","last_change":"2024-09-05T16:46:40.458315+0000","last_active":"2024-09-05T16:46:40.458401+0000","last_peered":"2024-09-05T16:46:40.458401+0000","last_clean":"2024-09-05T16:46:40.458401+0000","last_became_active":"2024-09-05T16:46:40.184288+0000","last_became_peered":"2024-09-05T16:46:40.184288+0000","last_unstale":"2024-09-05T16:46:40.458401+0000","last_undegraded":"2024-09-05T16:46:40.458401+0000","last_fullsized":"2024-09-05T16:46:40.458401+0000","mapping_epoch":42,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12777","last_scrub_stamp":"2024-09-05T16:46:40.458226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:40.458226+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:38:58.734210+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'12770","reported_seq":12880,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:27.750634+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:46:27.750634+0000","last_peered":"2024-09-05T16:46:27.750634+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:46:27.750634+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12767,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2240}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12808","reported_seq":12944,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:40.186355+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:46:40.186355+0000","last_peered":"2024-09-05T16:46:40.186355+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:46:40.186355+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12806,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2243}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13120","reported_seq":13256,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:27.750509+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:46:27.750509+0000","last_peered":"2024-09-05T16:46:27.750509+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:46:27.750509+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13117,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2278}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12758","reported_seq":12866,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:27.751040+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:46:27.751040+0000","last_peered":"2024-09-05T16:46:27.751040+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:46:27.751040+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2237}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12808","reported_seq":17557,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-05T16:46:40.178047+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:46:40.178047+0000","last_peered":"2024-09-05T16:46:40.178047+0000","last_clean":"2024-09-05T16:46:40.178047+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:46:40.178047+0000","last_undegraded":"2024-09-05T16:46:40.178047+0000","last_fullsized":"2024-09-05T16:46:40.178047+0000","mapping_epoch":36,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4408,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13124","reported_seq":13242,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:27.753160+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:46:27.753160+0000","last_peered":"2024-09-05T16:46:27.753160+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:46:27.753160+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13122,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2332}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12831","reported_seq":12957,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:40.182008+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:46:40.182008+0000","last_peered":"2024-09-05T16:46:40.182008+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:46:40.182008+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12829,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2306}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'12773","reported_seq":12895,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:40.176436+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:46:40.176436+0000","last_peered":"2024-09-05T16:46:40.176436+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:46:40.176436+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12771,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2218}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":141456,"num_objects":21,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":167352,"num_write_kb":621607,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2908,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":94208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084439,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722028,"kb_used_data":340,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649812,"statfs":{"total":96636764160,"available":95897407488,"internally_reserved":0,"allocated":348160,"data_stored":128198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":738979539},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986317,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":599936,"kb_used_data":432,"kb_used_omap":4,"kb_used_meta":599483,"kb_avail":93771904,"statfs":{"total":96636764160,"available":96022429696,"internally_reserved":0,"allocated":442368,"data_stored":212951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":613871314},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673123,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713212,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":93658628,"statfs":{"total":96636764160,"available":95906435072,"internally_reserved":0,"allocated":364544,"data_stored":135127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":729938373},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182554,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":730300,"kb_used_data":356,"kb_used_omap":4,"kb_used_meta":729915,"kb_avail":93641540,"statfs":{"total":96636764160,"available":95888936960,"internally_reserved":0,"allocated":364544,"data_stored":135127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":747433677},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":81920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:46:44.391 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:46:44.712 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:46:44.713 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:46:44.725 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":437,"stamp":"2024-09-05T16:46:43.099391+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":141456,"num_objects":21,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":167352,"num_write_kb":621607,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2908,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2765476,"kb_used_data":1484,"kb_used_omap":14,"kb_used_meta":2763889,"kb_avail":374721884,"statfs":{"total":386547056640,"available":383715209216,"internally_reserved":0,"allocated":1519616,"data_stored":611403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2830222903},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13472,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1917,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12775,"num_write_kb":-51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001600"},"pg_stats":[{"pgid":"1.f","version":"39'12904","reported_seq":17500,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-05T16:46:27.826329+0000","last_change":"2024-09-05T16:46:27.826222+0000","last_active":"2024-09-05T16:46:27.826329+0000","last_peered":"2024-09-05T16:46:27.826329+0000","last_clean":"2024-09-05T16:46:27.826329+0000","last_became_active":"2024-09-05T16:46:27.757928+0000","last_became_peered":"2024-09-05T16:46:27.757928+0000","last_unstale":"2024-09-05T16:46:27.826329+0000","last_undegraded":"2024-09-05T16:46:27.826329+0000","last_fullsized":"2024-09-05T16:46:27.826329+0000","mapping_epoch":40,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:32:35.713428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12904,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2262,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12552","reported_seq":17217,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-05T16:46:40.178538+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:46:40.178538+0000","last_peered":"2024-09-05T16:46:40.178538+0000","last_clean":"2024-09-05T16:46:40.178538+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:46:40.178538+0000","last_undegraded":"2024-09-05T16:46:40.178538+0000","last_fullsized":"2024-09-05T16:46:40.178538+0000","mapping_epoch":38,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1865,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12752","reported_seq":13693,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:41.014188+0000","last_change":"2024-09-05T16:46:38.780739+0000","last_active":"2024-09-05T16:46:41.014188+0000","last_peered":"2024-09-05T16:46:41.014188+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:46:41.014188+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1831,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12751,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":421,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1830}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12934","reported_seq":17677,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-05T16:46:40.177938+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:46:40.177938+0000","last_peered":"2024-09-05T16:46:40.177938+0000","last_clean":"2024-09-05T16:46:40.177938+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:46:40.177938+0000","last_undegraded":"2024-09-05T16:46:40.177938+0000","last_fullsized":"2024-09-05T16:46:40.177938+0000","mapping_epoch":34,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7234,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12593","reported_seq":12702,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:27.753309+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:46:27.753309+0000","last_peered":"2024-09-05T16:46:27.753309+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:46:27.753309+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12591,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2143}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12772","reported_seq":12896,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:27.752538+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:46:27.752538+0000","last_peered":"2024-09-05T16:46:27.752538+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:46:27.752538+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12772,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2219}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12660","reported_seq":12782,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:40.177421+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:46:40.177421+0000","last_peered":"2024-09-05T16:46:40.177421+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:46:40.177421+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12658,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2129}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12777","reported_seq":12894,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-05T16:46:40.458401+0000","last_change":"2024-09-05T16:46:40.458315+0000","last_active":"2024-09-05T16:46:40.458401+0000","last_peered":"2024-09-05T16:46:40.458401+0000","last_clean":"2024-09-05T16:46:40.458401+0000","last_became_active":"2024-09-05T16:46:40.184288+0000","last_became_peered":"2024-09-05T16:46:40.184288+0000","last_unstale":"2024-09-05T16:46:40.458401+0000","last_undegraded":"2024-09-05T16:46:40.458401+0000","last_fullsized":"2024-09-05T16:46:40.458401+0000","mapping_epoch":42,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12777","last_scrub_stamp":"2024-09-05T16:46:40.458226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:40.458226+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:38:58.734210+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'12770","reported_seq":12880,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:27.750634+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:46:27.750634+0000","last_peered":"2024-09-05T16:46:27.750634+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:46:27.750634+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12767,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2240}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12808","reported_seq":12944,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:40.186355+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:46:40.186355+0000","last_peered":"2024-09-05T16:46:40.186355+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:46:40.186355+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12806,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2243}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13120","reported_seq":13256,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:27.750509+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:46:27.750509+0000","last_peered":"2024-09-05T16:46:27.750509+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:46:27.750509+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13117,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2278}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12758","reported_seq":12866,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:27.751040+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:46:27.751040+0000","last_peered":"2024-09-05T16:46:27.751040+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:46:27.751040+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2237}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12808","reported_seq":17557,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-05T16:46:40.178047+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:46:40.178047+0000","last_peered":"2024-09-05T16:46:40.178047+0000","last_clean":"2024-09-05T16:46:40.178047+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:46:40.178047+0000","last_undegraded":"2024-09-05T16:46:40.178047+0000","last_fullsized":"2024-09-05T16:46:40.178047+0000","mapping_epoch":36,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4408,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13124","reported_seq":13242,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:27.753160+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:46:27.753160+0000","last_peered":"2024-09-05T16:46:27.753160+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:46:27.753160+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13122,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2332}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12831","reported_seq":12957,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:40.182008+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:46:40.182008+0000","last_peered":"2024-09-05T16:46:40.182008+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:46:40.182008+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12829,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2306}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'12773","reported_seq":12895,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:40.176436+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:46:40.176436+0000","last_peered":"2024-09-05T16:46:40.176436+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:46:40.176436+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12771,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2218}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":141456,"num_objects":21,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":167352,"num_write_kb":621607,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2908,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":94208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084439,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722028,"kb_used_data":340,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649812,"statfs":{"total":96636764160,"available":95897407488,"internally_reserved":0,"allocated":348160,"data_stored":128198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":738979539},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986317,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":599936,"kb_used_data":432,"kb_used_omap":4,"kb_used_meta":599483,"kb_avail":93771904,"statfs":{"total":96636764160,"available":96022429696,"internally_reserved":0,"allocated":442368,"data_stored":212951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":613871314},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673123,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713212,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":93658628,"statfs":{"total":96636764160,"available":95906435072,"internally_reserved":0,"allocated":364544,"data_stored":135127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":729938373},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182554,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":730300,"kb_used_data":356,"kb_used_omap":4,"kb_used_meta":729915,"kb_avail":93641540,"statfs":{"total":96636764160,"available":95888936960,"internally_reserved":0,"allocated":364544,"data_stored":135127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":747433677},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":81920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:46:44.726 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:46:44.726 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "39'12752", 'reported_seq': 13693, 'reported_epoch': 43, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:41.014188+0000', 'last_change': '2024-09-05T16:46:38.780739+0000', 'last_active': '2024-09-05T16:46:41.014188+0000', 'last_peered': '2024-09-05T16:46:41.014188+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:46:41.014188+0000', 'last_undegraded': '2024-09-05T16:34:56.590972+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 32, 'log_start': "33'2700", 'ondisk_log_start': "33'2700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10052, 'log_dups_size': 0, 'ondisk_log_size': 10052, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6736, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1831, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12751, 'num_write_kb': 50999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 421, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1830}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:46:44.727 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:46:44.727 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "39'12593", 'reported_seq': 12702, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:27.753309+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:46:27.753309+0000', 'last_peered': '2024-09-05T16:46:27.753309+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:46:27.753309+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "33'2500", 'ondisk_log_start': "33'2500", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10093, 'log_dups_size': 0, 'ondisk_log_size': 10093, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2147, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12591, 'num_write_kb': 50373, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2143}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:46:44.727 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:46:44.727 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "39'12772", 'reported_seq': 12896, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:27.752538+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:46:27.752538+0000', 'last_peered': '2024-09-05T16:46:27.752538+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:46:27.752538+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "33'2700", 'ondisk_log_start': "33'2700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10072, 'log_dups_size': 0, 'ondisk_log_size': 10072, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12772, 'num_write_kb': 51088, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2219}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:46:44.727 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:46:44.727 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "39'12660", 'reported_seq': 12782, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:40.177421+0000', 'last_change': '2024-09-05T16:34:56.903436+0000', 'last_active': '2024-09-05T16:46:40.177421+0000', 'last_peered': '2024-09-05T16:46:40.177421+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:46:40.177421+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "33'2600", 'ondisk_log_start': "33'2600", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10060, 'log_dups_size': 0, 'ondisk_log_size': 10060, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2133, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12658, 'num_write_kb': 50636, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -2129}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:46:44.727 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:46:44.727 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "39'12770", 'reported_seq': 12880, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:27.750634+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:46:27.750634+0000', 'last_peered': '2024-09-05T16:46:27.750634+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:46:27.750634+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "33'2700", 'ondisk_log_start': "33'2700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10070, 'log_dups_size': 0, 'ondisk_log_size': 10070, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26944, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 16, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12767, 'num_write_kb': 51081, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2240}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:46:44.727 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:46:44.727 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "39'12808", 'reported_seq': 12944, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:40.186355+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:46:40.186355+0000', 'last_peered': '2024-09-05T16:46:40.186355+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:46:40.186355+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "33'2800", 'ondisk_log_start': "33'2800", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10008, 'log_dups_size': 0, 'ondisk_log_size': 10008, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12806, 'num_write_kb': 51230, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -2243}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:46:44.728 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:46:44.728 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "39'13120", 'reported_seq': 13256, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:27.750509+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:46:27.750509+0000', 'last_peered': '2024-09-05T16:46:27.750509+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:46:27.750509+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "33'3100", 'ondisk_log_start': "33'3100", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20208, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2284, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13117, 'num_write_kb': 52476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2278}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:46:44.728 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:46:44.728 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "39'12758", 'reported_seq': 12866, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:27.751040+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:46:27.751040+0000', 'last_peered': '2024-09-05T16:46:27.751040+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:46:27.751040+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "33'2700", 'ondisk_log_start': "33'2700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10058, 'log_dups_size': 0, 'ondisk_log_size': 10058, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6736, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12757, 'num_write_kb': 51031, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2237}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:46:44.728 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:46:44.728 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "39'13124", 'reported_seq': 13242, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:27.753160+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:46:27.753160+0000', 'last_peered': '2024-09-05T16:46:27.753160+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:46:27.753160+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "33'3100", 'ondisk_log_start': "33'3100", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10024, 'log_dups_size': 0, 'ondisk_log_size': 10024, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13122, 'num_write_kb': 52500, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2332}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:46:44.728 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:46:44.728 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "39'12831", 'reported_seq': 12957, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:40.182008+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:46:40.182008+0000', 'last_peered': '2024-09-05T16:46:40.182008+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:46:40.182008+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "33'2800", 'ondisk_log_start': "33'2800", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10031, 'log_dups_size': 0, 'ondisk_log_size': 10031, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2310, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12829, 'num_write_kb': 51325, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -2306}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:46:44.728 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:46:44.728 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "39'12773", 'reported_seq': 12895, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:40.176436+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:46:40.176436+0000', 'last_peered': '2024-09-05T16:46:40.176436+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:46:40.176436+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "33'2700", 'ondisk_log_start': "33'2700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10073, 'log_dups_size': 0, 'ondisk_log_size': 10073, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12771, 'num_write_kb': 51092, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2218}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:46:44.729 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:46:45.157 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:46:45.158 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 24215/84 objects degraded (28827.381%), 11 pgs degraded, 11 pgs undersized","count":22},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":808,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":43,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+clean","count":5},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":21,"data_bytes":141456,"bytes_used":2831863808,"bytes_avail":383715192832,"bytes_total":386547056640,"degraded_objects":24215,"degraded_total":84,"degraded_ratio":288.27380952380952},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:46:45.171 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-09-05T16:46:45.171 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:46:45.469 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:46:45.470 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:46:45.482 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":438,"stamp":"2024-09-05T16:46:45.099740+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":141456,"num_objects":21,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":167352,"num_write_kb":621607,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2908,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2765492,"kb_used_data":1500,"kb_used_omap":14,"kb_used_meta":2763889,"kb_avail":374721868,"statfs":{"total":386547056640,"available":383715192832,"internally_reserved":0,"allocated":1536000,"data_stored":618332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2830222903},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13472,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1012,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12775,"num_write_kb":-51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1231,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001582"},"pg_stats":[{"pgid":"1.f","version":"39'12904","reported_seq":17504,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-05T16:46:40.183431+0000","last_change":"2024-09-05T16:46:27.826222+0000","last_active":"2024-09-05T16:46:40.183431+0000","last_peered":"2024-09-05T16:46:40.183431+0000","last_clean":"2024-09-05T16:46:40.183431+0000","last_became_active":"2024-09-05T16:46:27.757928+0000","last_became_peered":"2024-09-05T16:46:27.757928+0000","last_unstale":"2024-09-05T16:46:40.183431+0000","last_undegraded":"2024-09-05T16:46:40.183431+0000","last_fullsized":"2024-09-05T16:46:40.183431+0000","mapping_epoch":40,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:32:35.713428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12904,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2262,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12552","reported_seq":17217,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-05T16:46:40.178538+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:46:40.178538+0000","last_peered":"2024-09-05T16:46:40.178538+0000","last_clean":"2024-09-05T16:46:40.178538+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:46:40.178538+0000","last_undegraded":"2024-09-05T16:46:40.178538+0000","last_fullsized":"2024-09-05T16:46:40.178538+0000","mapping_epoch":38,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1865,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12752","reported_seq":13693,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:41.014188+0000","last_change":"2024-09-05T16:46:38.780739+0000","last_active":"2024-09-05T16:46:41.014188+0000","last_peered":"2024-09-05T16:46:41.014188+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:46:41.014188+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1831,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12751,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":421,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1830}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12934","reported_seq":17677,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-05T16:46:40.177938+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:46:40.177938+0000","last_peered":"2024-09-05T16:46:40.177938+0000","last_clean":"2024-09-05T16:46:40.177938+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:46:40.177938+0000","last_undegraded":"2024-09-05T16:46:40.177938+0000","last_fullsized":"2024-09-05T16:46:40.177938+0000","mapping_epoch":34,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7234,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12593","reported_seq":12706,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:40.180702+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:46:40.180702+0000","last_peered":"2024-09-05T16:46:40.180702+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:46:40.180702+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12591,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2143}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12772","reported_seq":12900,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:40.183200+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:46:40.183200+0000","last_peered":"2024-09-05T16:46:40.183200+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:46:40.183200+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12772,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2219}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12660","reported_seq":12782,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:40.177421+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:46:40.177421+0000","last_peered":"2024-09-05T16:46:40.177421+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:46:40.177421+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12658,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2129}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12777","reported_seq":12894,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-05T16:46:40.458401+0000","last_change":"2024-09-05T16:46:40.458315+0000","last_active":"2024-09-05T16:46:40.458401+0000","last_peered":"2024-09-05T16:46:40.458401+0000","last_clean":"2024-09-05T16:46:40.458401+0000","last_became_active":"2024-09-05T16:46:40.184288+0000","last_became_peered":"2024-09-05T16:46:40.184288+0000","last_unstale":"2024-09-05T16:46:40.458401+0000","last_undegraded":"2024-09-05T16:46:40.458401+0000","last_fullsized":"2024-09-05T16:46:40.458401+0000","mapping_epoch":42,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12777","last_scrub_stamp":"2024-09-05T16:46:40.458226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:40.458226+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:38:58.734210+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'12770","reported_seq":12884,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:40.182678+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:46:40.182678+0000","last_peered":"2024-09-05T16:46:40.182678+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:46:40.182678+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12767,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2240}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12808","reported_seq":12944,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:40.186355+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:46:40.186355+0000","last_peered":"2024-09-05T16:46:40.186355+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:46:40.186355+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12806,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2243}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13120","reported_seq":13260,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:40.180757+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:46:40.180757+0000","last_peered":"2024-09-05T16:46:40.180757+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:46:40.180757+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13117,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2278}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12758","reported_seq":12870,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:40.183507+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:46:40.183507+0000","last_peered":"2024-09-05T16:46:40.183507+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:46:40.183507+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2237}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12808","reported_seq":17557,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-05T16:46:40.178047+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:46:40.178047+0000","last_peered":"2024-09-05T16:46:40.178047+0000","last_clean":"2024-09-05T16:46:40.178047+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:46:40.178047+0000","last_undegraded":"2024-09-05T16:46:40.178047+0000","last_fullsized":"2024-09-05T16:46:40.178047+0000","mapping_epoch":36,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4408,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13124","reported_seq":13246,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:40.180923+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:46:40.180923+0000","last_peered":"2024-09-05T16:46:40.180923+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:46:40.180923+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13122,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2332}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12831","reported_seq":12957,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:40.182008+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:46:40.182008+0000","last_peered":"2024-09-05T16:46:40.182008+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:46:40.182008+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12829,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2306}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'12773","reported_seq":12895,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:40.176436+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:46:40.176436+0000","last_peered":"2024-09-05T16:46:40.176436+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:46:40.176436+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12771,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2218}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":141456,"num_objects":21,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":167352,"num_write_kb":621607,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2908,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":94208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084440,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722044,"kb_used_data":356,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649796,"statfs":{"total":96636764160,"available":95897391104,"internally_reserved":0,"allocated":364544,"data_stored":135127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":738979539},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986317,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":599936,"kb_used_data":432,"kb_used_omap":4,"kb_used_meta":599483,"kb_avail":93771904,"statfs":{"total":96636764160,"available":96022429696,"internally_reserved":0,"allocated":442368,"data_stored":212951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":613871314},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673123,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713212,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":93658628,"statfs":{"total":96636764160,"available":95906435072,"internally_reserved":0,"allocated":364544,"data_stored":135127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":729938373},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182554,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":730300,"kb_used_data":356,"kb_used_omap":4,"kb_used_meta":729915,"kb_avail":93641540,"statfs":{"total":96636764160,"available":95888936960,"internally_reserved":0,"allocated":364544,"data_stored":135127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":747433677},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":81920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:46:48.484 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:46:48.798 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:46:48.799 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:46:48.811 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":439,"stamp":"2024-09-05T16:46:47.100222+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":141456,"num_objects":21,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23370,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":167352,"num_write_kb":621607,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3753,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2765492,"kb_used_data":1500,"kb_used_omap":14,"kb_used_meta":2763889,"kb_avail":374721868,"statfs":{"total":386547056640,"available":383715192832,"internally_reserved":0,"allocated":1536000,"data_stored":618332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2830222903},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13472,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12775,"num_write_kb":-51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-386,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001663"},"pg_stats":[{"pgid":"1.f","version":"39'12904","reported_seq":17504,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-05T16:46:40.183431+0000","last_change":"2024-09-05T16:46:27.826222+0000","last_active":"2024-09-05T16:46:40.183431+0000","last_peered":"2024-09-05T16:46:40.183431+0000","last_clean":"2024-09-05T16:46:40.183431+0000","last_became_active":"2024-09-05T16:46:27.757928+0000","last_became_peered":"2024-09-05T16:46:27.757928+0000","last_unstale":"2024-09-05T16:46:40.183431+0000","last_undegraded":"2024-09-05T16:46:40.183431+0000","last_fullsized":"2024-09-05T16:46:40.183431+0000","mapping_epoch":40,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:32:35.713428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12904,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2262,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12552","reported_seq":17217,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-05T16:46:40.178538+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:46:40.178538+0000","last_peered":"2024-09-05T16:46:40.178538+0000","last_clean":"2024-09-05T16:46:40.178538+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:46:40.178538+0000","last_undegraded":"2024-09-05T16:46:40.178538+0000","last_fullsized":"2024-09-05T16:46:40.178538+0000","mapping_epoch":38,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1865,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12752","reported_seq":15383,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:46.014072+0000","last_change":"2024-09-05T16:46:38.780739+0000","last_active":"2024-09-05T16:46:46.014072+0000","last_peered":"2024-09-05T16:46:46.014072+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:46:46.014072+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":986,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12751,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1266,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-985}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12934","reported_seq":17677,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-05T16:46:40.177938+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:46:40.177938+0000","last_peered":"2024-09-05T16:46:40.177938+0000","last_clean":"2024-09-05T16:46:40.177938+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:46:40.177938+0000","last_undegraded":"2024-09-05T16:46:40.177938+0000","last_fullsized":"2024-09-05T16:46:40.177938+0000","mapping_epoch":34,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7234,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12593","reported_seq":12706,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:40.180702+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:46:40.180702+0000","last_peered":"2024-09-05T16:46:40.180702+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:46:40.180702+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12591,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2143}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12772","reported_seq":12900,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:40.183200+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:46:40.183200+0000","last_peered":"2024-09-05T16:46:40.183200+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:46:40.183200+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12772,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2219}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12660","reported_seq":12782,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:40.177421+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:46:40.177421+0000","last_peered":"2024-09-05T16:46:40.177421+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:46:40.177421+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12658,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2129}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12777","reported_seq":12894,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-05T16:46:40.458401+0000","last_change":"2024-09-05T16:46:40.458315+0000","last_active":"2024-09-05T16:46:40.458401+0000","last_peered":"2024-09-05T16:46:40.458401+0000","last_clean":"2024-09-05T16:46:40.458401+0000","last_became_active":"2024-09-05T16:46:40.184288+0000","last_became_peered":"2024-09-05T16:46:40.184288+0000","last_unstale":"2024-09-05T16:46:40.458401+0000","last_undegraded":"2024-09-05T16:46:40.458401+0000","last_fullsized":"2024-09-05T16:46:40.458401+0000","mapping_epoch":42,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12777","last_scrub_stamp":"2024-09-05T16:46:40.458226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:40.458226+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:38:58.734210+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'12770","reported_seq":12884,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:40.182678+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:46:40.182678+0000","last_peered":"2024-09-05T16:46:40.182678+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:46:40.182678+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12767,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2240}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12808","reported_seq":12944,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:40.186355+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:46:40.186355+0000","last_peered":"2024-09-05T16:46:40.186355+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:46:40.186355+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12806,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2243}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13120","reported_seq":13260,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:40.180757+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:46:40.180757+0000","last_peered":"2024-09-05T16:46:40.180757+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:46:40.180757+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13117,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2278}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12758","reported_seq":12870,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:40.183507+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:46:40.183507+0000","last_peered":"2024-09-05T16:46:40.183507+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:46:40.183507+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2237}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12808","reported_seq":17557,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-05T16:46:40.178047+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:46:40.178047+0000","last_peered":"2024-09-05T16:46:40.178047+0000","last_clean":"2024-09-05T16:46:40.178047+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:46:40.178047+0000","last_undegraded":"2024-09-05T16:46:40.178047+0000","last_fullsized":"2024-09-05T16:46:40.178047+0000","mapping_epoch":36,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4408,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13124","reported_seq":13246,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:40.180923+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:46:40.180923+0000","last_peered":"2024-09-05T16:46:40.180923+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:46:40.180923+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13122,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2332}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12831","reported_seq":12957,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:40.182008+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:46:40.182008+0000","last_peered":"2024-09-05T16:46:40.182008+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:46:40.182008+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12829,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2306}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'12773","reported_seq":12895,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:40.176436+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:46:40.176436+0000","last_peered":"2024-09-05T16:46:40.176436+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:46:40.176436+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12771,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2218}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":141456,"num_objects":21,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23370,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":167352,"num_write_kb":621607,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3753,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":94208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084440,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722044,"kb_used_data":356,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649796,"statfs":{"total":96636764160,"available":95897391104,"internally_reserved":0,"allocated":364544,"data_stored":135127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":738979539},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986317,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":599936,"kb_used_data":432,"kb_used_omap":4,"kb_used_meta":599483,"kb_avail":93771904,"statfs":{"total":96636764160,"available":96022429696,"internally_reserved":0,"allocated":442368,"data_stored":212951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":613871314},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673124,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713212,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":93658628,"statfs":{"total":96636764160,"available":95906435072,"internally_reserved":0,"allocated":364544,"data_stored":135127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":729938373},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182555,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":730300,"kb_used_data":356,"kb_used_omap":4,"kb_used_meta":729915,"kb_avail":93641540,"statfs":{"total":96636764160,"available":95888936960,"internally_reserved":0,"allocated":364544,"data_stored":135127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":747433677},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":81920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:46:48.812 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:46:49.122 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:46:49.122 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:46:49.134 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":440,"stamp":"2024-09-05T16:46:49.100594+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":141456,"num_objects":21,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23370,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":167352,"num_write_kb":621607,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3753,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2753012,"kb_used_data":1500,"kb_used_omap":14,"kb_used_meta":2751409,"kb_avail":374734348,"statfs":{"total":386547056640,"available":383727972352,"internally_reserved":0,"allocated":1536000,"data_stored":618332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2817443383},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13472,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12775,"num_write_kb":-51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-386,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.928170"},"pg_stats":[{"pgid":"1.f","version":"39'12904","reported_seq":17504,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-05T16:46:40.183431+0000","last_change":"2024-09-05T16:46:27.826222+0000","last_active":"2024-09-05T16:46:40.183431+0000","last_peered":"2024-09-05T16:46:40.183431+0000","last_clean":"2024-09-05T16:46:40.183431+0000","last_became_active":"2024-09-05T16:46:27.757928+0000","last_became_peered":"2024-09-05T16:46:27.757928+0000","last_unstale":"2024-09-05T16:46:40.183431+0000","last_undegraded":"2024-09-05T16:46:40.183431+0000","last_fullsized":"2024-09-05T16:46:40.183431+0000","mapping_epoch":40,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:32:35.713428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12904,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2262,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12552","reported_seq":17217,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-05T16:46:40.178538+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:46:40.178538+0000","last_peered":"2024-09-05T16:46:40.178538+0000","last_clean":"2024-09-05T16:46:40.178538+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:46:40.178538+0000","last_undegraded":"2024-09-05T16:46:40.178538+0000","last_fullsized":"2024-09-05T16:46:40.178538+0000","mapping_epoch":38,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1865,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12752","reported_seq":15383,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:46.014072+0000","last_change":"2024-09-05T16:46:38.780739+0000","last_active":"2024-09-05T16:46:46.014072+0000","last_peered":"2024-09-05T16:46:46.014072+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:46:46.014072+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":986,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12751,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1266,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-985}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12934","reported_seq":17677,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-05T16:46:40.177938+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:46:40.177938+0000","last_peered":"2024-09-05T16:46:40.177938+0000","last_clean":"2024-09-05T16:46:40.177938+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:46:40.177938+0000","last_undegraded":"2024-09-05T16:46:40.177938+0000","last_fullsized":"2024-09-05T16:46:40.177938+0000","mapping_epoch":34,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7234,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12593","reported_seq":12706,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:40.180702+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:46:40.180702+0000","last_peered":"2024-09-05T16:46:40.180702+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:46:40.180702+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12591,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2143}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12772","reported_seq":12900,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:40.183200+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:46:40.183200+0000","last_peered":"2024-09-05T16:46:40.183200+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:46:40.183200+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12772,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2219}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12660","reported_seq":12782,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:40.177421+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:46:40.177421+0000","last_peered":"2024-09-05T16:46:40.177421+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:46:40.177421+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12658,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2129}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12777","reported_seq":12894,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-05T16:46:40.458401+0000","last_change":"2024-09-05T16:46:40.458315+0000","last_active":"2024-09-05T16:46:40.458401+0000","last_peered":"2024-09-05T16:46:40.458401+0000","last_clean":"2024-09-05T16:46:40.458401+0000","last_became_active":"2024-09-05T16:46:40.184288+0000","last_became_peered":"2024-09-05T16:46:40.184288+0000","last_unstale":"2024-09-05T16:46:40.458401+0000","last_undegraded":"2024-09-05T16:46:40.458401+0000","last_fullsized":"2024-09-05T16:46:40.458401+0000","mapping_epoch":42,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12777","last_scrub_stamp":"2024-09-05T16:46:40.458226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:40.458226+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:38:58.734210+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'12770","reported_seq":12884,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:40.182678+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:46:40.182678+0000","last_peered":"2024-09-05T16:46:40.182678+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:46:40.182678+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12767,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2240}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12808","reported_seq":12944,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:40.186355+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:46:40.186355+0000","last_peered":"2024-09-05T16:46:40.186355+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:46:40.186355+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12806,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2243}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13120","reported_seq":13260,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:40.180757+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:46:40.180757+0000","last_peered":"2024-09-05T16:46:40.180757+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:46:40.180757+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13117,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2278}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12758","reported_seq":12870,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:40.183507+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:46:40.183507+0000","last_peered":"2024-09-05T16:46:40.183507+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:46:40.183507+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2237}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12808","reported_seq":17557,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-05T16:46:40.178047+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:46:40.178047+0000","last_peered":"2024-09-05T16:46:40.178047+0000","last_clean":"2024-09-05T16:46:40.178047+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:46:40.178047+0000","last_undegraded":"2024-09-05T16:46:40.178047+0000","last_fullsized":"2024-09-05T16:46:40.178047+0000","mapping_epoch":36,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4408,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13124","reported_seq":13246,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:40.180923+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:46:40.180923+0000","last_peered":"2024-09-05T16:46:40.180923+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:46:40.180923+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13122,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2332}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12831","reported_seq":12957,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:40.182008+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:46:40.182008+0000","last_peered":"2024-09-05T16:46:40.182008+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:46:40.182008+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12829,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2306}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'12773","reported_seq":12895,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:40.176436+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:46:40.176436+0000","last_peered":"2024-09-05T16:46:40.176436+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:46:40.176436+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12771,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2218}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":141456,"num_objects":21,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23370,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":167352,"num_write_kb":621607,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3753,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":94208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084441,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722044,"kb_used_data":356,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649796,"statfs":{"total":96636764160,"available":95897391104,"internally_reserved":0,"allocated":364544,"data_stored":135127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":738979539},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986318,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":587456,"kb_used_data":432,"kb_used_omap":4,"kb_used_meta":587003,"kb_avail":93784384,"statfs":{"total":96636764160,"available":96035209216,"internally_reserved":0,"allocated":442368,"data_stored":212951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":601091794},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673124,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713212,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":93658628,"statfs":{"total":96636764160,"available":95906435072,"internally_reserved":0,"allocated":364544,"data_stored":135127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":729938373},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182555,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":730300,"kb_used_data":356,"kb_used_omap":4,"kb_used_meta":729915,"kb_avail":93641540,"statfs":{"total":96636764160,"available":95888936960,"internally_reserved":0,"allocated":364544,"data_stored":135127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":747433677},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":81920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:46:49.135 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:46:49.135 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "39'12752", 'reported_seq': 15383, 'reported_epoch': 43, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:46.014072+0000', 'last_change': '2024-09-05T16:46:38.780739+0000', 'last_active': '2024-09-05T16:46:46.014072+0000', 'last_peered': '2024-09-05T16:46:46.014072+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:46:46.014072+0000', 'last_undegraded': '2024-09-05T16:34:56.590972+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 32, 'log_start': "33'2700", 'ondisk_log_start': "33'2700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10052, 'log_dups_size': 0, 'ondisk_log_size': 10052, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6736, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 986, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12751, 'num_write_kb': 50999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1266, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -985}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:46:49.135 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:46:49.135 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "39'12593", 'reported_seq': 12706, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:40.180702+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:46:40.180702+0000', 'last_peered': '2024-09-05T16:46:40.180702+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:46:40.180702+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "33'2500", 'ondisk_log_start': "33'2500", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10093, 'log_dups_size': 0, 'ondisk_log_size': 10093, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2147, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12591, 'num_write_kb': 50373, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2143}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:46:49.136 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:46:49.136 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "39'12772", 'reported_seq': 12900, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:40.183200+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:46:40.183200+0000', 'last_peered': '2024-09-05T16:46:40.183200+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:46:40.183200+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "33'2700", 'ondisk_log_start': "33'2700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10072, 'log_dups_size': 0, 'ondisk_log_size': 10072, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12772, 'num_write_kb': 51088, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2219}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:46:49.136 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:46:49.136 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "39'12660", 'reported_seq': 12782, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:40.177421+0000', 'last_change': '2024-09-05T16:34:56.903436+0000', 'last_active': '2024-09-05T16:46:40.177421+0000', 'last_peered': '2024-09-05T16:46:40.177421+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:46:40.177421+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "33'2600", 'ondisk_log_start': "33'2600", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10060, 'log_dups_size': 0, 'ondisk_log_size': 10060, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2133, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12658, 'num_write_kb': 50636, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -2129}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:46:49.136 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:46:49.136 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "39'12770", 'reported_seq': 12884, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:40.182678+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:46:40.182678+0000', 'last_peered': '2024-09-05T16:46:40.182678+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:46:40.182678+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "33'2700", 'ondisk_log_start': "33'2700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10070, 'log_dups_size': 0, 'ondisk_log_size': 10070, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26944, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 16, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12767, 'num_write_kb': 51081, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2240}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:46:49.136 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:46:49.136 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "39'12808", 'reported_seq': 12944, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:40.186355+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:46:40.186355+0000', 'last_peered': '2024-09-05T16:46:40.186355+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:46:40.186355+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "33'2800", 'ondisk_log_start': "33'2800", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10008, 'log_dups_size': 0, 'ondisk_log_size': 10008, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12806, 'num_write_kb': 51230, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -2243}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:46:49.136 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:46:49.136 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "39'13120", 'reported_seq': 13260, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:40.180757+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:46:40.180757+0000', 'last_peered': '2024-09-05T16:46:40.180757+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:46:40.180757+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "33'3100", 'ondisk_log_start': "33'3100", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20208, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2284, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13117, 'num_write_kb': 52476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2278}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:46:49.136 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:46:49.137 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "39'12758", 'reported_seq': 12870, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:40.183507+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:46:40.183507+0000', 'last_peered': '2024-09-05T16:46:40.183507+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:46:40.183507+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "33'2700", 'ondisk_log_start': "33'2700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10058, 'log_dups_size': 0, 'ondisk_log_size': 10058, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6736, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12757, 'num_write_kb': 51031, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2237}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:46:49.137 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:46:49.137 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "39'13124", 'reported_seq': 13246, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:40.180923+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:46:40.180923+0000', 'last_peered': '2024-09-05T16:46:40.180923+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:46:40.180923+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "33'3100", 'ondisk_log_start': "33'3100", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10024, 'log_dups_size': 0, 'ondisk_log_size': 10024, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13122, 'num_write_kb': 52500, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2332}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:46:49.137 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:46:49.137 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "39'12831", 'reported_seq': 12957, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:40.182008+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:46:40.182008+0000', 'last_peered': '2024-09-05T16:46:40.182008+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:46:40.182008+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "33'2800", 'ondisk_log_start': "33'2800", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10031, 'log_dups_size': 0, 'ondisk_log_size': 10031, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2310, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12829, 'num_write_kb': 51325, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -2306}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:46:49.137 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:46:49.137 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "39'12773", 'reported_seq': 12895, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:40.176436+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:46:40.176436+0000', 'last_peered': '2024-09-05T16:46:40.176436+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:46:40.176436+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "33'2700", 'ondisk_log_start': "33'2700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10073, 'log_dups_size': 0, 'ondisk_log_size': 10073, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12771, 'num_write_kb': 51092, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2218}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:46:49.137 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:46:49.523 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:46:49.523 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 23370/84 objects degraded (27821.429%), 11 pgs degraded, 11 pgs undersized","count":22},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":812,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":43,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+clean","count":5},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":21,"data_bytes":141456,"bytes_used":2819084288,"bytes_avail":383727972352,"bytes_total":386547056640,"degraded_objects":23370,"degraded_total":84,"degraded_ratio":278.21428571428572},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:46:49.538 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-09-05T16:46:49.538 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:46:49.836 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:46:49.837 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:46:49.849 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":440,"stamp":"2024-09-05T16:46:49.100594+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":141456,"num_objects":21,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23370,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":167352,"num_write_kb":621607,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3753,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2753012,"kb_used_data":1500,"kb_used_omap":14,"kb_used_meta":2751409,"kb_avail":374734348,"statfs":{"total":386547056640,"available":383727972352,"internally_reserved":0,"allocated":1536000,"data_stored":618332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2817443383},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13472,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12775,"num_write_kb":-51108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-386,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.928170"},"pg_stats":[{"pgid":"1.f","version":"39'12904","reported_seq":17504,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-05T16:46:40.183431+0000","last_change":"2024-09-05T16:46:27.826222+0000","last_active":"2024-09-05T16:46:40.183431+0000","last_peered":"2024-09-05T16:46:40.183431+0000","last_clean":"2024-09-05T16:46:40.183431+0000","last_became_active":"2024-09-05T16:46:27.757928+0000","last_became_peered":"2024-09-05T16:46:27.757928+0000","last_unstale":"2024-09-05T16:46:40.183431+0000","last_undegraded":"2024-09-05T16:46:40.183431+0000","last_fullsized":"2024-09-05T16:46:40.183431+0000","mapping_epoch":40,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:32:35.713428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12904,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2262,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12552","reported_seq":17217,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-05T16:46:40.178538+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:46:40.178538+0000","last_peered":"2024-09-05T16:46:40.178538+0000","last_clean":"2024-09-05T16:46:40.178538+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:46:40.178538+0000","last_undegraded":"2024-09-05T16:46:40.178538+0000","last_fullsized":"2024-09-05T16:46:40.178538+0000","mapping_epoch":38,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1865,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12752","reported_seq":15383,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:46.014072+0000","last_change":"2024-09-05T16:46:38.780739+0000","last_active":"2024-09-05T16:46:46.014072+0000","last_peered":"2024-09-05T16:46:46.014072+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:46:46.014072+0000","last_undegraded":"2024-09-05T16:34:56.590972+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":986,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12751,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1266,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-985}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12934","reported_seq":17677,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-05T16:46:40.177938+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:46:40.177938+0000","last_peered":"2024-09-05T16:46:40.177938+0000","last_clean":"2024-09-05T16:46:40.177938+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:46:40.177938+0000","last_undegraded":"2024-09-05T16:46:40.177938+0000","last_fullsized":"2024-09-05T16:46:40.177938+0000","mapping_epoch":34,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7234,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12593","reported_seq":12706,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:40.180702+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:46:40.180702+0000","last_peered":"2024-09-05T16:46:40.180702+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:46:40.180702+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12591,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2143}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12772","reported_seq":12900,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:40.183200+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:46:40.183200+0000","last_peered":"2024-09-05T16:46:40.183200+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:46:40.183200+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12772,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2219}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12660","reported_seq":12782,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:40.177421+0000","last_change":"2024-09-05T16:34:56.903436+0000","last_active":"2024-09-05T16:46:40.177421+0000","last_peered":"2024-09-05T16:46:40.177421+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:46:40.177421+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12658,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2129}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12777","reported_seq":12894,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-05T16:46:40.458401+0000","last_change":"2024-09-05T16:46:40.458315+0000","last_active":"2024-09-05T16:46:40.458401+0000","last_peered":"2024-09-05T16:46:40.458401+0000","last_clean":"2024-09-05T16:46:40.458401+0000","last_became_active":"2024-09-05T16:46:40.184288+0000","last_became_peered":"2024-09-05T16:46:40.184288+0000","last_unstale":"2024-09-05T16:46:40.458401+0000","last_undegraded":"2024-09-05T16:46:40.458401+0000","last_fullsized":"2024-09-05T16:46:40.458401+0000","mapping_epoch":42,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12777","last_scrub_stamp":"2024-09-05T16:46:40.458226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:40.458226+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:38:58.734210+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'12770","reported_seq":12884,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:40.182678+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:46:40.182678+0000","last_peered":"2024-09-05T16:46:40.182678+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:46:40.182678+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12767,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2240}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12808","reported_seq":12944,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:40.186355+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:46:40.186355+0000","last_peered":"2024-09-05T16:46:40.186355+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:46:40.186355+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12806,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2243}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13120","reported_seq":13260,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:40.180757+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:46:40.180757+0000","last_peered":"2024-09-05T16:46:40.180757+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:46:40.180757+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13117,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2278}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12758","reported_seq":12870,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:40.183507+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:46:40.183507+0000","last_peered":"2024-09-05T16:46:40.183507+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:46:40.183507+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2237}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12808","reported_seq":17557,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-05T16:46:40.178047+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:46:40.178047+0000","last_peered":"2024-09-05T16:46:40.178047+0000","last_clean":"2024-09-05T16:46:40.178047+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:46:40.178047+0000","last_undegraded":"2024-09-05T16:46:40.178047+0000","last_fullsized":"2024-09-05T16:46:40.178047+0000","mapping_epoch":36,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4408,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13124","reported_seq":13246,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:40.180923+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:46:40.180923+0000","last_peered":"2024-09-05T16:46:40.180923+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:46:40.180923+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13122,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2332}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12831","reported_seq":12957,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:40.182008+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:46:40.182008+0000","last_peered":"2024-09-05T16:46:40.182008+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:46:40.182008+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12829,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2306}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'12773","reported_seq":12895,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:40.176436+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:46:40.176436+0000","last_peered":"2024-09-05T16:46:40.176436+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:46:40.176436+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12771,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2218}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":141456,"num_objects":21,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23370,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":167352,"num_write_kb":621607,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3753,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":94208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084441,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722044,"kb_used_data":356,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649796,"statfs":{"total":96636764160,"available":95897391104,"internally_reserved":0,"allocated":364544,"data_stored":135127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":738979539},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986318,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":587456,"kb_used_data":432,"kb_used_omap":4,"kb_used_meta":587003,"kb_avail":93784384,"statfs":{"total":96636764160,"available":96035209216,"internally_reserved":0,"allocated":442368,"data_stored":212951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":601091794},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673124,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713212,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":93658628,"statfs":{"total":96636764160,"available":95906435072,"internally_reserved":0,"allocated":364544,"data_stored":135127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":729938373},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182555,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":730300,"kb_used_data":356,"kb_used_omap":4,"kb_used_meta":729915,"kb_avail":93641540,"statfs":{"total":96636764160,"available":95888936960,"internally_reserved":0,"allocated":364544,"data_stored":135127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":747433677},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":81920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:46:52.852 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:46:53.192 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:46:53.193 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:46:53.207 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":444,"stamp":"2024-09-05T16:46:53.101357+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":141456,"num_objects":21,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":167352,"num_write_kb":621607,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4872,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2761356,"kb_used_data":1524,"kb_used_omap":14,"kb_used_meta":2759729,"kb_avail":374726004,"statfs":{"total":386547056640,"available":383719428096,"internally_reserved":0,"allocated":1560576,"data_stored":623871,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2825963063},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1964,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001617"},"pg_stats":[{"pgid":"1.f","version":"39'12904","reported_seq":17504,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-05T16:46:40.183431+0000","last_change":"2024-09-05T16:46:27.826222+0000","last_active":"2024-09-05T16:46:40.183431+0000","last_peered":"2024-09-05T16:46:40.183431+0000","last_clean":"2024-09-05T16:46:40.183431+0000","last_became_active":"2024-09-05T16:46:27.757928+0000","last_became_peered":"2024-09-05T16:46:27.757928+0000","last_unstale":"2024-09-05T16:46:40.183431+0000","last_undegraded":"2024-09-05T16:46:40.183431+0000","last_fullsized":"2024-09-05T16:46:40.183431+0000","mapping_epoch":40,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:32:35.713428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12904,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2262,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12552","reported_seq":17221,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-05T16:46:51.660993+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:46:51.660993+0000","last_peered":"2024-09-05T16:46:51.660993+0000","last_clean":"2024-09-05T16:46:51.660993+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:46:51.660993+0000","last_undegraded":"2024-09-05T16:46:51.660993+0000","last_fullsized":"2024-09-05T16:46:51.660993+0000","mapping_epoch":38,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1865,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12752","reported_seq":17357,"reported_epoch":43,"state":"peering","last_fresh":"2024-09-05T16:46:50.436402+0000","last_change":"2024-09-05T16:46:50.662459+0000","last_active":"2024-09-05T16:46:50.662459+0000","last_peered":"2024-09-05T16:46:50.436402+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:46:50.436402+0000","last_undegraded":"2024-09-05T16:46:50.436402+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":44,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12751,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2252,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,3],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12934","reported_seq":17681,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-05T16:46:51.660302+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:46:51.660302+0000","last_peered":"2024-09-05T16:46:51.660302+0000","last_clean":"2024-09-05T16:46:51.660302+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:46:51.660302+0000","last_undegraded":"2024-09-05T16:46:51.660302+0000","last_fullsized":"2024-09-05T16:46:51.660302+0000","mapping_epoch":34,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7234,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12593","reported_seq":12706,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:40.180702+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:46:40.180702+0000","last_peered":"2024-09-05T16:46:40.180702+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:46:40.180702+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12591,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2143}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12772","reported_seq":12900,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:40.183200+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:46:40.183200+0000","last_peered":"2024-09-05T16:46:40.183200+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:46:40.183200+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12772,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2219}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12660","reported_seq":13051,"reported_epoch":44,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:51.347437+0000","last_change":"2024-09-05T16:46:50.437247+0000","last_active":"2024-09-05T16:46:51.347437+0000","last_peered":"2024-09-05T16:46:51.347437+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:46:51.347437+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2000,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12658,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":151,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1998}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12777","reported_seq":12898,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-05T16:46:51.660614+0000","last_change":"2024-09-05T16:46:40.458315+0000","last_active":"2024-09-05T16:46:51.660614+0000","last_peered":"2024-09-05T16:46:51.660614+0000","last_clean":"2024-09-05T16:46:51.660614+0000","last_became_active":"2024-09-05T16:46:40.184288+0000","last_became_peered":"2024-09-05T16:46:40.184288+0000","last_unstale":"2024-09-05T16:46:51.660614+0000","last_undegraded":"2024-09-05T16:46:51.660614+0000","last_fullsized":"2024-09-05T16:46:51.660614+0000","mapping_epoch":42,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12777","last_scrub_stamp":"2024-09-05T16:46:40.458226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:40.458226+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:38:58.734210+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'12770","reported_seq":12884,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:40.182678+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:46:40.182678+0000","last_peered":"2024-09-05T16:46:40.182678+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:46:40.182678+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12767,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2240}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12808","reported_seq":12946,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:50.663154+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:46:50.663154+0000","last_peered":"2024-09-05T16:46:50.663154+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:46:50.663154+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12806,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2243}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13120","reported_seq":13260,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:40.180757+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:46:40.180757+0000","last_peered":"2024-09-05T16:46:40.180757+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:46:40.180757+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13117,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2278}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12758","reported_seq":12870,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:40.183507+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:46:40.183507+0000","last_peered":"2024-09-05T16:46:40.183507+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:46:40.183507+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2237}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12808","reported_seq":17561,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-05T16:46:51.660402+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:46:51.660402+0000","last_peered":"2024-09-05T16:46:51.660402+0000","last_clean":"2024-09-05T16:46:51.660402+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:46:51.660402+0000","last_undegraded":"2024-09-05T16:46:51.660402+0000","last_fullsized":"2024-09-05T16:46:51.660402+0000","mapping_epoch":36,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4408,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13124","reported_seq":13246,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:40.180923+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:46:40.180923+0000","last_peered":"2024-09-05T16:46:40.180923+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:46:40.180923+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13122,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2332}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12831","reported_seq":12959,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:50.658445+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:46:50.658445+0000","last_peered":"2024-09-05T16:46:50.658445+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:46:50.658445+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12829,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2306}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'12773","reported_seq":12897,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:50.655396+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:46:50.655396+0000","last_peered":"2024-09-05T16:46:50.655396+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:46:50.655396+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12771,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2218}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":141456,"num_objects":21,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":167352,"num_write_kb":621607,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4872,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":86016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084441,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722044,"kb_used_data":356,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649796,"statfs":{"total":96636764160,"available":95897391104,"internally_reserved":0,"allocated":364544,"data_stored":135127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":738979539},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986319,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":595792,"kb_used_data":440,"kb_used_omap":4,"kb_used_meta":595323,"kb_avail":93776048,"statfs":{"total":96636764160,"available":96026673152,"internally_reserved":0,"allocated":450560,"data_stored":211614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":609611474},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673125,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713216,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":93658624,"statfs":{"total":96636764160,"available":95906430976,"internally_reserved":0,"allocated":372736,"data_stored":138565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":729938373},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182556,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":730304,"kb_used_data":364,"kb_used_omap":4,"kb_used_meta":729915,"kb_avail":93641536,"statfs":{"total":96636764160,"available":95888932864,"internally_reserved":0,"allocated":372736,"data_stored":138565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":747433677},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":73728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:46:53.209 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:46:53.552 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:46:53.553 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:46:53.566 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":444,"stamp":"2024-09-05T16:46:53.101357+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":141456,"num_objects":21,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":167352,"num_write_kb":621607,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4872,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2761356,"kb_used_data":1524,"kb_used_omap":14,"kb_used_meta":2759729,"kb_avail":374726004,"statfs":{"total":386547056640,"available":383719428096,"internally_reserved":0,"allocated":1560576,"data_stored":623871,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2825963063},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1964,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001617"},"pg_stats":[{"pgid":"1.f","version":"39'12904","reported_seq":17504,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-05T16:46:40.183431+0000","last_change":"2024-09-05T16:46:27.826222+0000","last_active":"2024-09-05T16:46:40.183431+0000","last_peered":"2024-09-05T16:46:40.183431+0000","last_clean":"2024-09-05T16:46:40.183431+0000","last_became_active":"2024-09-05T16:46:27.757928+0000","last_became_peered":"2024-09-05T16:46:27.757928+0000","last_unstale":"2024-09-05T16:46:40.183431+0000","last_undegraded":"2024-09-05T16:46:40.183431+0000","last_fullsized":"2024-09-05T16:46:40.183431+0000","mapping_epoch":40,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:32:35.713428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12904,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2262,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12552","reported_seq":17221,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-05T16:46:51.660993+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:46:51.660993+0000","last_peered":"2024-09-05T16:46:51.660993+0000","last_clean":"2024-09-05T16:46:51.660993+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:46:51.660993+0000","last_undegraded":"2024-09-05T16:46:51.660993+0000","last_fullsized":"2024-09-05T16:46:51.660993+0000","mapping_epoch":38,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1865,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12752","reported_seq":17357,"reported_epoch":43,"state":"peering","last_fresh":"2024-09-05T16:46:50.436402+0000","last_change":"2024-09-05T16:46:50.662459+0000","last_active":"2024-09-05T16:46:50.662459+0000","last_peered":"2024-09-05T16:46:50.436402+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:46:50.436402+0000","last_undegraded":"2024-09-05T16:46:50.436402+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":44,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12751,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2252,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,3],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12934","reported_seq":17681,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-05T16:46:51.660302+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:46:51.660302+0000","last_peered":"2024-09-05T16:46:51.660302+0000","last_clean":"2024-09-05T16:46:51.660302+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:46:51.660302+0000","last_undegraded":"2024-09-05T16:46:51.660302+0000","last_fullsized":"2024-09-05T16:46:51.660302+0000","mapping_epoch":34,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7234,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12593","reported_seq":12706,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:40.180702+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:46:40.180702+0000","last_peered":"2024-09-05T16:46:40.180702+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:46:40.180702+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12591,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2143}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12772","reported_seq":12900,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:40.183200+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:46:40.183200+0000","last_peered":"2024-09-05T16:46:40.183200+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:46:40.183200+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12772,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2219}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12660","reported_seq":13051,"reported_epoch":44,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:51.347437+0000","last_change":"2024-09-05T16:46:50.437247+0000","last_active":"2024-09-05T16:46:51.347437+0000","last_peered":"2024-09-05T16:46:51.347437+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:46:51.347437+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2000,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12658,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":151,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1998}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12777","reported_seq":12898,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-05T16:46:51.660614+0000","last_change":"2024-09-05T16:46:40.458315+0000","last_active":"2024-09-05T16:46:51.660614+0000","last_peered":"2024-09-05T16:46:51.660614+0000","last_clean":"2024-09-05T16:46:51.660614+0000","last_became_active":"2024-09-05T16:46:40.184288+0000","last_became_peered":"2024-09-05T16:46:40.184288+0000","last_unstale":"2024-09-05T16:46:51.660614+0000","last_undegraded":"2024-09-05T16:46:51.660614+0000","last_fullsized":"2024-09-05T16:46:51.660614+0000","mapping_epoch":42,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12777","last_scrub_stamp":"2024-09-05T16:46:40.458226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:40.458226+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:38:58.734210+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'12770","reported_seq":12884,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:40.182678+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:46:40.182678+0000","last_peered":"2024-09-05T16:46:40.182678+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:46:40.182678+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12767,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2240}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12808","reported_seq":12946,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:50.663154+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:46:50.663154+0000","last_peered":"2024-09-05T16:46:50.663154+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:46:50.663154+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12806,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2243}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13120","reported_seq":13260,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:40.180757+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:46:40.180757+0000","last_peered":"2024-09-05T16:46:40.180757+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:46:40.180757+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13117,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2278}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12758","reported_seq":12870,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:40.183507+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:46:40.183507+0000","last_peered":"2024-09-05T16:46:40.183507+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:46:40.183507+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2237}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12808","reported_seq":17561,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-05T16:46:51.660402+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:46:51.660402+0000","last_peered":"2024-09-05T16:46:51.660402+0000","last_clean":"2024-09-05T16:46:51.660402+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:46:51.660402+0000","last_undegraded":"2024-09-05T16:46:51.660402+0000","last_fullsized":"2024-09-05T16:46:51.660402+0000","mapping_epoch":36,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4408,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13124","reported_seq":13246,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:40.180923+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:46:40.180923+0000","last_peered":"2024-09-05T16:46:40.180923+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:46:40.180923+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13122,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2332}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12831","reported_seq":12959,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:50.658445+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:46:50.658445+0000","last_peered":"2024-09-05T16:46:50.658445+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:46:50.658445+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12829,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2306}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'12773","reported_seq":12897,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:50.655396+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:46:50.655396+0000","last_peered":"2024-09-05T16:46:50.655396+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:46:50.655396+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12771,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2218}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":141456,"num_objects":21,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":167352,"num_write_kb":621607,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4872,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":86016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084441,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722044,"kb_used_data":356,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649796,"statfs":{"total":96636764160,"available":95897391104,"internally_reserved":0,"allocated":364544,"data_stored":135127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":738979539},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986319,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":595792,"kb_used_data":440,"kb_used_omap":4,"kb_used_meta":595323,"kb_avail":93776048,"statfs":{"total":96636764160,"available":96026673152,"internally_reserved":0,"allocated":450560,"data_stored":211614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":609611474},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673125,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713216,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":93658624,"statfs":{"total":96636764160,"available":95906430976,"internally_reserved":0,"allocated":372736,"data_stored":138565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":729938373},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182556,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":730304,"kb_used_data":364,"kb_used_omap":4,"kb_used_meta":729915,"kb_avail":93641536,"statfs":{"total":96636764160,"available":95888932864,"internally_reserved":0,"allocated":372736,"data_stored":138565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":747433677},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":73728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:46:53.567 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-05T16:46:53.567 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "39'12752", 'reported_seq': 17357, 'reported_epoch': 43, 'state': 'peering', 'last_fresh': '2024-09-05T16:46:50.436402+0000', 'last_change': '2024-09-05T16:46:50.662459+0000', 'last_active': '2024-09-05T16:46:50.662459+0000', 'last_peered': '2024-09-05T16:46:50.436402+0000', 'last_clean': '2024-09-05T16:33:53.454690+0000', 'last_became_active': '2024-09-05T16:34:56.859218+0000', 'last_became_peered': '2024-09-05T16:34:56.859218+0000', 'last_unstale': '2024-09-05T16:46:50.436402+0000', 'last_undegraded': '2024-09-05T16:46:50.436402+0000', 'last_fullsized': '2024-09-05T16:34:56.560384+0000', 'mapping_epoch': 44, 'log_start': "33'2700", 'ondisk_log_start': "33'2700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10052, 'log_dups_size': 0, 'ondisk_log_size': 10052, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6736, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12751, 'num_write_kb': 50999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2252, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 1, 3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [0, 1, 3], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:46:53.567 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:46:53.567 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "39'12593", 'reported_seq': 12706, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:40.180702+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:46:40.180702+0000', 'last_peered': '2024-09-05T16:46:40.180702+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:46:40.180702+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "33'2500", 'ondisk_log_start': "33'2500", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10093, 'log_dups_size': 0, 'ondisk_log_size': 10093, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2147, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12591, 'num_write_kb': 50373, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2143}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:46:53.567 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:46:53.567 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "39'12772", 'reported_seq': 12900, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:40.183200+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:46:40.183200+0000', 'last_peered': '2024-09-05T16:46:40.183200+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:46:40.183200+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "33'2700", 'ondisk_log_start': "33'2700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10072, 'log_dups_size': 0, 'ondisk_log_size': 10072, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12772, 'num_write_kb': 51088, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2219}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:46:53.568 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:46:53.568 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "39'12660", 'reported_seq': 13051, 'reported_epoch': 44, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:51.347437+0000', 'last_change': '2024-09-05T16:46:50.437247+0000', 'last_active': '2024-09-05T16:46:51.347437+0000', 'last_peered': '2024-09-05T16:46:51.347437+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:46:51.347437+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "33'2600", 'ondisk_log_start': "33'2600", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10060, 'log_dups_size': 0, 'ondisk_log_size': 10060, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2000, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12658, 'num_write_kb': 50636, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 151, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -1998}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:46:53.568 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:46:53.568 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "39'12770", 'reported_seq': 12884, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:40.182678+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:46:40.182678+0000', 'last_peered': '2024-09-05T16:46:40.182678+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:46:40.182678+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "33'2700", 'ondisk_log_start': "33'2700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10070, 'log_dups_size': 0, 'ondisk_log_size': 10070, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26944, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 16, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12767, 'num_write_kb': 51081, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2240}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:46:53.568 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:46:53.568 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "39'12808", 'reported_seq': 12946, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:50.663154+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:46:50.663154+0000', 'last_peered': '2024-09-05T16:46:50.663154+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:46:50.663154+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "33'2800", 'ondisk_log_start': "33'2800", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10008, 'log_dups_size': 0, 'ondisk_log_size': 10008, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12806, 'num_write_kb': 51230, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -2243}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:46:53.568 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:46:53.568 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "39'13120", 'reported_seq': 13260, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:40.180757+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:46:40.180757+0000', 'last_peered': '2024-09-05T16:46:40.180757+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:46:40.180757+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "33'3100", 'ondisk_log_start': "33'3100", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20208, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2284, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13117, 'num_write_kb': 52476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2278}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:46:53.568 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:46:53.568 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "39'12758", 'reported_seq': 12870, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:40.183507+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:46:40.183507+0000', 'last_peered': '2024-09-05T16:46:40.183507+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:46:40.183507+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "33'2700", 'ondisk_log_start': "33'2700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10058, 'log_dups_size': 0, 'ondisk_log_size': 10058, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6736, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12757, 'num_write_kb': 51031, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2237}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:46:53.568 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:46:53.568 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "39'13124", 'reported_seq': 13246, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:40.180923+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:46:40.180923+0000', 'last_peered': '2024-09-05T16:46:40.180923+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:46:40.180923+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "33'3100", 'ondisk_log_start': "33'3100", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10024, 'log_dups_size': 0, 'ondisk_log_size': 10024, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13122, 'num_write_kb': 52500, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2332}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:46:53.569 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:46:53.569 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "39'12831", 'reported_seq': 12959, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:50.658445+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:46:50.658445+0000', 'last_peered': '2024-09-05T16:46:50.658445+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:46:50.658445+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "33'2800", 'ondisk_log_start': "33'2800", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10031, 'log_dups_size': 0, 'ondisk_log_size': 10031, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2310, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12829, 'num_write_kb': 51325, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -2306}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:46:53.569 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:46:53.569 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "39'12773", 'reported_seq': 12897, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:50.655396+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:46:50.655396+0000', 'last_peered': '2024-09-05T16:46:50.655396+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:46:50.655396+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "33'2700", 'ondisk_log_start': "33'2700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10073, 'log_dups_size': 0, 'ondisk_log_size': 10073, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12771, 'num_write_kb': 51092, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2218}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:46:53.569 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:46:53.996 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:46:53.996 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 22251/84 objects degraded (26489.286%), 10 pgs degraded, 10 pgs undersized","count":20},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":817,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":45,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":10},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":9},{"state_name":"active+clean","count":5},{"state_name":"peering","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":21,"data_bytes":141456,"bytes_used":2827628544,"bytes_avail":383719428096,"bytes_total":386547056640,"inactive_pgs_ratio":0.0625,"degraded_objects":22251,"degraded_total":84,"degraded_ratio":264.89285714285717,"recovering_objects_per_sec":245,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1964,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:46:54.011 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:46:54.012 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:46:54.313 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:46:54.313 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:46:54.325 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":444,"stamp":"2024-09-05T16:46:53.101357+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":141456,"num_objects":21,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":167352,"num_write_kb":621607,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4872,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2761356,"kb_used_data":1524,"kb_used_omap":14,"kb_used_meta":2759729,"kb_avail":374726004,"statfs":{"total":386547056640,"available":383719428096,"internally_reserved":0,"allocated":1560576,"data_stored":623871,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2825963063},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1964,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001617"},"pg_stats":[{"pgid":"1.f","version":"39'12904","reported_seq":17504,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-05T16:46:40.183431+0000","last_change":"2024-09-05T16:46:27.826222+0000","last_active":"2024-09-05T16:46:40.183431+0000","last_peered":"2024-09-05T16:46:40.183431+0000","last_clean":"2024-09-05T16:46:40.183431+0000","last_became_active":"2024-09-05T16:46:27.757928+0000","last_became_peered":"2024-09-05T16:46:27.757928+0000","last_unstale":"2024-09-05T16:46:40.183431+0000","last_undegraded":"2024-09-05T16:46:40.183431+0000","last_fullsized":"2024-09-05T16:46:40.183431+0000","mapping_epoch":40,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:32:35.713428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12904,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2262,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12552","reported_seq":17221,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-05T16:46:51.660993+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:46:51.660993+0000","last_peered":"2024-09-05T16:46:51.660993+0000","last_clean":"2024-09-05T16:46:51.660993+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:46:51.660993+0000","last_undegraded":"2024-09-05T16:46:51.660993+0000","last_fullsized":"2024-09-05T16:46:51.660993+0000","mapping_epoch":38,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1865,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12752","reported_seq":17357,"reported_epoch":43,"state":"peering","last_fresh":"2024-09-05T16:46:50.436402+0000","last_change":"2024-09-05T16:46:50.662459+0000","last_active":"2024-09-05T16:46:50.662459+0000","last_peered":"2024-09-05T16:46:50.436402+0000","last_clean":"2024-09-05T16:33:53.454690+0000","last_became_active":"2024-09-05T16:34:56.859218+0000","last_became_peered":"2024-09-05T16:34:56.859218+0000","last_unstale":"2024-09-05T16:46:50.436402+0000","last_undegraded":"2024-09-05T16:46:50.436402+0000","last_fullsized":"2024-09-05T16:34:56.560384+0000","mapping_epoch":44,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12751,"num_write_kb":50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2252,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,3],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12934","reported_seq":17681,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-05T16:46:51.660302+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:46:51.660302+0000","last_peered":"2024-09-05T16:46:51.660302+0000","last_clean":"2024-09-05T16:46:51.660302+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:46:51.660302+0000","last_undegraded":"2024-09-05T16:46:51.660302+0000","last_fullsized":"2024-09-05T16:46:51.660302+0000","mapping_epoch":34,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7234,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12593","reported_seq":12706,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:40.180702+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:46:40.180702+0000","last_peered":"2024-09-05T16:46:40.180702+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:46:40.180702+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12591,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2143}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12772","reported_seq":12900,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:40.183200+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:46:40.183200+0000","last_peered":"2024-09-05T16:46:40.183200+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:46:40.183200+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12772,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2219}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12660","reported_seq":13051,"reported_epoch":44,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:51.347437+0000","last_change":"2024-09-05T16:46:50.437247+0000","last_active":"2024-09-05T16:46:51.347437+0000","last_peered":"2024-09-05T16:46:51.347437+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:46:51.347437+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2000,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12658,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":151,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1998}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12777","reported_seq":12898,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-05T16:46:51.660614+0000","last_change":"2024-09-05T16:46:40.458315+0000","last_active":"2024-09-05T16:46:51.660614+0000","last_peered":"2024-09-05T16:46:51.660614+0000","last_clean":"2024-09-05T16:46:51.660614+0000","last_became_active":"2024-09-05T16:46:40.184288+0000","last_became_peered":"2024-09-05T16:46:40.184288+0000","last_unstale":"2024-09-05T16:46:51.660614+0000","last_undegraded":"2024-09-05T16:46:51.660614+0000","last_fullsized":"2024-09-05T16:46:51.660614+0000","mapping_epoch":42,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12777","last_scrub_stamp":"2024-09-05T16:46:40.458226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:40.458226+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:38:58.734210+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'12770","reported_seq":12884,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:40.182678+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:46:40.182678+0000","last_peered":"2024-09-05T16:46:40.182678+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:46:40.182678+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12767,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2240}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12808","reported_seq":12946,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:50.663154+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:46:50.663154+0000","last_peered":"2024-09-05T16:46:50.663154+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:46:50.663154+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12806,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2243}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13120","reported_seq":13260,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:40.180757+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:46:40.180757+0000","last_peered":"2024-09-05T16:46:40.180757+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:46:40.180757+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13117,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2278}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12758","reported_seq":12870,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:40.183507+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:46:40.183507+0000","last_peered":"2024-09-05T16:46:40.183507+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:46:40.183507+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2237}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12808","reported_seq":17561,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-05T16:46:51.660402+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:46:51.660402+0000","last_peered":"2024-09-05T16:46:51.660402+0000","last_clean":"2024-09-05T16:46:51.660402+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:46:51.660402+0000","last_undegraded":"2024-09-05T16:46:51.660402+0000","last_fullsized":"2024-09-05T16:46:51.660402+0000","mapping_epoch":36,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4408,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13124","reported_seq":13246,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:40.180923+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:46:40.180923+0000","last_peered":"2024-09-05T16:46:40.180923+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:46:40.180923+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13122,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2332}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12831","reported_seq":12959,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:50.658445+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:46:50.658445+0000","last_peered":"2024-09-05T16:46:50.658445+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:46:50.658445+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12829,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2306}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'12773","reported_seq":12897,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:50.655396+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:46:50.655396+0000","last_peered":"2024-09-05T16:46:50.655396+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:46:50.655396+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12771,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2218}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":141456,"num_objects":21,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":167352,"num_write_kb":621607,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4872,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":86016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084441,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722044,"kb_used_data":356,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649796,"statfs":{"total":96636764160,"available":95897391104,"internally_reserved":0,"allocated":364544,"data_stored":135127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":738979539},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986319,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":595792,"kb_used_data":440,"kb_used_omap":4,"kb_used_meta":595323,"kb_avail":93776048,"statfs":{"total":96636764160,"available":96026673152,"internally_reserved":0,"allocated":450560,"data_stored":211614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":609611474},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673125,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713216,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":93658624,"statfs":{"total":96636764160,"available":95906430976,"internally_reserved":0,"allocated":372736,"data_stored":138565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":729938373},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182556,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":730304,"kb_used_data":364,"kb_used_omap":4,"kb_used_meta":729915,"kb_avail":93641536,"statfs":{"total":96636764160,"available":95888932864,"internally_reserved":0,"allocated":372736,"data_stored":138565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":747433677},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":73728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:46:57.328 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:46:57.643 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:46:57.643 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:46:57.656 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":446,"stamp":"2024-09-05T16:46:57.102140+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":134720,"num_objects":20,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21452,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":154601,"num_write_kb":570608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3419,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2744752,"kb_used_data":1556,"kb_used_omap":14,"kb_used_meta":2743089,"kb_avail":374742608,"statfs":{"total":386547056640,"available":383736430592,"internally_reserved":0,"allocated":1593344,"data_stored":637560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2808923703},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-6736,"num_objects":-1,"num_object_clones":0,"num_object_copies":-4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1918,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1,"num_whiteouts":0,"num_read":-1,"num_read_kb":-1,"num_write":-12751,"num_write_kb":-50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-334,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001546"},"pg_stats":[{"pgid":"1.f","version":"39'12904","reported_seq":17508,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-05T16:46:51.661878+0000","last_change":"2024-09-05T16:46:27.826222+0000","last_active":"2024-09-05T16:46:51.661878+0000","last_peered":"2024-09-05T16:46:51.661878+0000","last_clean":"2024-09-05T16:46:51.661878+0000","last_became_active":"2024-09-05T16:46:27.757928+0000","last_became_peered":"2024-09-05T16:46:27.757928+0000","last_unstale":"2024-09-05T16:46:51.661878+0000","last_undegraded":"2024-09-05T16:46:51.661878+0000","last_fullsized":"2024-09-05T16:46:51.661878+0000","mapping_epoch":40,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:32:35.713428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12904,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2262,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12552","reported_seq":17221,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-05T16:46:51.660993+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:46:51.660993+0000","last_peered":"2024-09-05T16:46:51.660993+0000","last_clean":"2024-09-05T16:46:51.660993+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:46:51.660993+0000","last_undegraded":"2024-09-05T16:46:51.660993+0000","last_fullsized":"2024-09-05T16:46:51.660993+0000","mapping_epoch":38,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1865,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12752","reported_seq":17376,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-05T16:46:51.992073+0000","last_change":"2024-09-05T16:46:51.992073+0000","last_active":"2024-09-05T16:46:51.992073+0000","last_peered":"2024-09-05T16:46:51.992073+0000","last_clean":"2024-09-05T16:46:51.992073+0000","last_became_active":"2024-09-05T16:46:51.667128+0000","last_became_peered":"2024-09-05T16:46:51.667128+0000","last_unstale":"2024-09-05T16:46:51.992073+0000","last_undegraded":"2024-09-05T16:46:51.992073+0000","last_fullsized":"2024-09-05T16:46:51.992073+0000","mapping_epoch":44,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12752","last_scrub_stamp":"2024-09-05T16:46:51.991995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:51.991995+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:04:54.120031+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12934","reported_seq":17681,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-05T16:46:51.660302+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:46:51.660302+0000","last_peered":"2024-09-05T16:46:51.660302+0000","last_clean":"2024-09-05T16:46:51.660302+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:46:51.660302+0000","last_undegraded":"2024-09-05T16:46:51.660302+0000","last_fullsized":"2024-09-05T16:46:51.660302+0000","mapping_epoch":34,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7234,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12593","reported_seq":12710,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:51.659051+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:46:51.659051+0000","last_peered":"2024-09-05T16:46:51.659051+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:46:51.659051+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12591,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2143}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12772","reported_seq":12904,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:51.661169+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:46:51.661169+0000","last_peered":"2024-09-05T16:46:51.661169+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:46:51.661169+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12772,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2219}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12660","reported_seq":14651,"reported_epoch":45,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:56.346864+0000","last_change":"2024-09-05T16:46:50.437247+0000","last_active":"2024-09-05T16:46:56.346864+0000","last_peered":"2024-09-05T16:46:56.346864+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:46:56.346864+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1201,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12658,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":950,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1199}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12777","reported_seq":12898,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-05T16:46:51.660614+0000","last_change":"2024-09-05T16:46:40.458315+0000","last_active":"2024-09-05T16:46:51.660614+0000","last_peered":"2024-09-05T16:46:51.660614+0000","last_clean":"2024-09-05T16:46:51.660614+0000","last_became_active":"2024-09-05T16:46:40.184288+0000","last_became_peered":"2024-09-05T16:46:40.184288+0000","last_unstale":"2024-09-05T16:46:51.660614+0000","last_undegraded":"2024-09-05T16:46:51.660614+0000","last_fullsized":"2024-09-05T16:46:51.660614+0000","mapping_epoch":42,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12777","last_scrub_stamp":"2024-09-05T16:46:40.458226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:40.458226+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:38:58.734210+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'12770","reported_seq":12889,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:51.660389+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:46:51.660389+0000","last_peered":"2024-09-05T16:46:51.660389+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:46:51.660389+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12767,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2240}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12808","reported_seq":12948,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:51.663085+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:46:51.663085+0000","last_peered":"2024-09-05T16:46:51.663085+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:46:51.663085+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12806,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2243}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13120","reported_seq":13264,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:51.659414+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:46:51.659414+0000","last_peered":"2024-09-05T16:46:51.659414+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:46:51.659414+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13117,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2278}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12758","reported_seq":12874,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:51.661868+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:46:51.661868+0000","last_peered":"2024-09-05T16:46:51.661868+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:46:51.661868+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2237}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12808","reported_seq":17561,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-05T16:46:51.660402+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:46:51.660402+0000","last_peered":"2024-09-05T16:46:51.660402+0000","last_clean":"2024-09-05T16:46:51.660402+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:46:51.660402+0000","last_undegraded":"2024-09-05T16:46:51.660402+0000","last_fullsized":"2024-09-05T16:46:51.660402+0000","mapping_epoch":36,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4408,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13124","reported_seq":13250,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:51.659301+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:46:51.659301+0000","last_peered":"2024-09-05T16:46:51.659301+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:46:51.659301+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13122,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2332}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12831","reported_seq":12961,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:51.662869+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:46:51.662869+0000","last_peered":"2024-09-05T16:46:51.662869+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:46:51.662869+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12829,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2306}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'12773","reported_seq":12899,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:51.658835+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:46:51.658835+0000","last_peered":"2024-09-05T16:46:51.658835+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:46:51.658835+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12771,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2218}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":134720,"num_objects":20,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21452,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":154601,"num_write_kb":570608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3419,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":86016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084442,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722060,"kb_used_data":372,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649780,"statfs":{"total":96636764160,"available":95897374720,"internally_reserved":0,"allocated":380928,"data_stored":141982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":738979539},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986319,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":595792,"kb_used_data":440,"kb_used_omap":4,"kb_used_meta":595323,"kb_avail":93776048,"statfs":{"total":96636764160,"available":96026673152,"internally_reserved":0,"allocated":450560,"data_stored":211614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":609611474},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673126,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713224,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":93658616,"statfs":{"total":96636764160,"available":95906422784,"internally_reserved":0,"allocated":380928,"data_stored":141982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":729938373},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182557,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713676,"kb_used_data":372,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658164,"statfs":{"total":96636764160,"available":95905959936,"internally_reserved":0,"allocated":380928,"data_stored":141982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":730394317},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":73728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:46:57.657 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:46:57.998 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:46:57.999 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:46:58.010 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":446,"stamp":"2024-09-05T16:46:57.102140+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":134720,"num_objects":20,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21452,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":154601,"num_write_kb":570608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3419,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2744752,"kb_used_data":1556,"kb_used_omap":14,"kb_used_meta":2743089,"kb_avail":374742608,"statfs":{"total":386547056640,"available":383736430592,"internally_reserved":0,"allocated":1593344,"data_stored":637560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2808923703},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-6736,"num_objects":-1,"num_object_clones":0,"num_object_copies":-4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1918,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1,"num_whiteouts":0,"num_read":-1,"num_read_kb":-1,"num_write":-12751,"num_write_kb":-50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-334,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001546"},"pg_stats":[{"pgid":"1.f","version":"39'12904","reported_seq":17508,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-05T16:46:51.661878+0000","last_change":"2024-09-05T16:46:27.826222+0000","last_active":"2024-09-05T16:46:51.661878+0000","last_peered":"2024-09-05T16:46:51.661878+0000","last_clean":"2024-09-05T16:46:51.661878+0000","last_became_active":"2024-09-05T16:46:27.757928+0000","last_became_peered":"2024-09-05T16:46:27.757928+0000","last_unstale":"2024-09-05T16:46:51.661878+0000","last_undegraded":"2024-09-05T16:46:51.661878+0000","last_fullsized":"2024-09-05T16:46:51.661878+0000","mapping_epoch":40,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:32:35.713428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12904,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2262,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12552","reported_seq":17221,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-05T16:46:51.660993+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:46:51.660993+0000","last_peered":"2024-09-05T16:46:51.660993+0000","last_clean":"2024-09-05T16:46:51.660993+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:46:51.660993+0000","last_undegraded":"2024-09-05T16:46:51.660993+0000","last_fullsized":"2024-09-05T16:46:51.660993+0000","mapping_epoch":38,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1865,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12752","reported_seq":17376,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-05T16:46:51.992073+0000","last_change":"2024-09-05T16:46:51.992073+0000","last_active":"2024-09-05T16:46:51.992073+0000","last_peered":"2024-09-05T16:46:51.992073+0000","last_clean":"2024-09-05T16:46:51.992073+0000","last_became_active":"2024-09-05T16:46:51.667128+0000","last_became_peered":"2024-09-05T16:46:51.667128+0000","last_unstale":"2024-09-05T16:46:51.992073+0000","last_undegraded":"2024-09-05T16:46:51.992073+0000","last_fullsized":"2024-09-05T16:46:51.992073+0000","mapping_epoch":44,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12752","last_scrub_stamp":"2024-09-05T16:46:51.991995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:51.991995+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:04:54.120031+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12934","reported_seq":17681,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-05T16:46:51.660302+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:46:51.660302+0000","last_peered":"2024-09-05T16:46:51.660302+0000","last_clean":"2024-09-05T16:46:51.660302+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:46:51.660302+0000","last_undegraded":"2024-09-05T16:46:51.660302+0000","last_fullsized":"2024-09-05T16:46:51.660302+0000","mapping_epoch":34,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7234,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12593","reported_seq":12710,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:51.659051+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:46:51.659051+0000","last_peered":"2024-09-05T16:46:51.659051+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:46:51.659051+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12591,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2143}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12772","reported_seq":12904,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:51.661169+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:46:51.661169+0000","last_peered":"2024-09-05T16:46:51.661169+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:46:51.661169+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12772,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2219}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12660","reported_seq":14651,"reported_epoch":45,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:56.346864+0000","last_change":"2024-09-05T16:46:50.437247+0000","last_active":"2024-09-05T16:46:56.346864+0000","last_peered":"2024-09-05T16:46:56.346864+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:46:56.346864+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1201,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12658,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":950,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1199}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12777","reported_seq":12898,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-05T16:46:51.660614+0000","last_change":"2024-09-05T16:46:40.458315+0000","last_active":"2024-09-05T16:46:51.660614+0000","last_peered":"2024-09-05T16:46:51.660614+0000","last_clean":"2024-09-05T16:46:51.660614+0000","last_became_active":"2024-09-05T16:46:40.184288+0000","last_became_peered":"2024-09-05T16:46:40.184288+0000","last_unstale":"2024-09-05T16:46:51.660614+0000","last_undegraded":"2024-09-05T16:46:51.660614+0000","last_fullsized":"2024-09-05T16:46:51.660614+0000","mapping_epoch":42,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12777","last_scrub_stamp":"2024-09-05T16:46:40.458226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:40.458226+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:38:58.734210+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'12770","reported_seq":12889,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:51.660389+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:46:51.660389+0000","last_peered":"2024-09-05T16:46:51.660389+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:46:51.660389+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12767,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2240}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12808","reported_seq":12948,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:51.663085+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:46:51.663085+0000","last_peered":"2024-09-05T16:46:51.663085+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:46:51.663085+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12806,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2243}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13120","reported_seq":13264,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:51.659414+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:46:51.659414+0000","last_peered":"2024-09-05T16:46:51.659414+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:46:51.659414+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13117,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2278}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12758","reported_seq":12874,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:51.661868+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:46:51.661868+0000","last_peered":"2024-09-05T16:46:51.661868+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:46:51.661868+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2237}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12808","reported_seq":17561,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-05T16:46:51.660402+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:46:51.660402+0000","last_peered":"2024-09-05T16:46:51.660402+0000","last_clean":"2024-09-05T16:46:51.660402+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:46:51.660402+0000","last_undegraded":"2024-09-05T16:46:51.660402+0000","last_fullsized":"2024-09-05T16:46:51.660402+0000","mapping_epoch":36,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4408,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13124","reported_seq":13250,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:51.659301+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:46:51.659301+0000","last_peered":"2024-09-05T16:46:51.659301+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:46:51.659301+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13122,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2332}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12831","reported_seq":12961,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:51.662869+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:46:51.662869+0000","last_peered":"2024-09-05T16:46:51.662869+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:46:51.662869+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12829,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2306}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'12773","reported_seq":12899,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:51.658835+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:46:51.658835+0000","last_peered":"2024-09-05T16:46:51.658835+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:46:51.658835+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12771,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2218}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":134720,"num_objects":20,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21452,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":154601,"num_write_kb":570608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3419,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":86016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084442,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722060,"kb_used_data":372,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649780,"statfs":{"total":96636764160,"available":95897374720,"internally_reserved":0,"allocated":380928,"data_stored":141982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":738979539},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986319,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":595792,"kb_used_data":440,"kb_used_omap":4,"kb_used_meta":595323,"kb_avail":93776048,"statfs":{"total":96636764160,"available":96026673152,"internally_reserved":0,"allocated":450560,"data_stored":211614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":609611474},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673126,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713224,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":93658616,"statfs":{"total":96636764160,"available":95906422784,"internally_reserved":0,"allocated":380928,"data_stored":141982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":729938373},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182557,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713676,"kb_used_data":372,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658164,"statfs":{"total":96636764160,"available":95905959936,"internally_reserved":0,"allocated":380928,"data_stored":141982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":730394317},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":73728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:46:58.011 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:46:58.012 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "39'12593", 'reported_seq': 12710, 'reported_epoch': 45, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:51.659051+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:46:51.659051+0000', 'last_peered': '2024-09-05T16:46:51.659051+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:46:51.659051+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "33'2500", 'ondisk_log_start': "33'2500", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10093, 'log_dups_size': 0, 'ondisk_log_size': 10093, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2147, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12591, 'num_write_kb': 50373, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2143}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:46:58.012 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:46:58.012 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "39'12772", 'reported_seq': 12904, 'reported_epoch': 45, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:51.661169+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:46:51.661169+0000', 'last_peered': '2024-09-05T16:46:51.661169+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:46:51.661169+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "33'2700", 'ondisk_log_start': "33'2700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10072, 'log_dups_size': 0, 'ondisk_log_size': 10072, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12772, 'num_write_kb': 51088, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2219}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:46:58.012 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:46:58.012 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "39'12660", 'reported_seq': 14651, 'reported_epoch': 45, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:56.346864+0000', 'last_change': '2024-09-05T16:46:50.437247+0000', 'last_active': '2024-09-05T16:46:56.346864+0000', 'last_peered': '2024-09-05T16:46:56.346864+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:46:56.346864+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "33'2600", 'ondisk_log_start': "33'2600", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10060, 'log_dups_size': 0, 'ondisk_log_size': 10060, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1201, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12658, 'num_write_kb': 50636, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 950, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -1199}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:46:58.012 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:46:58.012 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "39'12770", 'reported_seq': 12889, 'reported_epoch': 45, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:51.660389+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:46:51.660389+0000', 'last_peered': '2024-09-05T16:46:51.660389+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:46:51.660389+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "33'2700", 'ondisk_log_start': "33'2700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10070, 'log_dups_size': 0, 'ondisk_log_size': 10070, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26944, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 16, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12767, 'num_write_kb': 51081, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2240}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:46:58.012 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:46:58.012 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "39'12808", 'reported_seq': 12948, 'reported_epoch': 45, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:51.663085+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:46:51.663085+0000', 'last_peered': '2024-09-05T16:46:51.663085+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:46:51.663085+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "33'2800", 'ondisk_log_start': "33'2800", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10008, 'log_dups_size': 0, 'ondisk_log_size': 10008, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12806, 'num_write_kb': 51230, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -2243}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:46:58.012 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:46:58.013 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "39'13120", 'reported_seq': 13264, 'reported_epoch': 45, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:51.659414+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:46:51.659414+0000', 'last_peered': '2024-09-05T16:46:51.659414+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:46:51.659414+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "33'3100", 'ondisk_log_start': "33'3100", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20208, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2284, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13117, 'num_write_kb': 52476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2278}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:46:58.013 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:46:58.013 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "39'12758", 'reported_seq': 12874, 'reported_epoch': 45, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:51.661868+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:46:51.661868+0000', 'last_peered': '2024-09-05T16:46:51.661868+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:46:51.661868+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "33'2700", 'ondisk_log_start': "33'2700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10058, 'log_dups_size': 0, 'ondisk_log_size': 10058, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6736, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12757, 'num_write_kb': 51031, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2237}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:46:58.013 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:46:58.013 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "39'13124", 'reported_seq': 13250, 'reported_epoch': 45, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:51.659301+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:46:51.659301+0000', 'last_peered': '2024-09-05T16:46:51.659301+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:46:51.659301+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "33'3100", 'ondisk_log_start': "33'3100", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10024, 'log_dups_size': 0, 'ondisk_log_size': 10024, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13122, 'num_write_kb': 52500, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2332}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:46:58.013 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:46:58.013 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "39'12831", 'reported_seq': 12961, 'reported_epoch': 45, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:51.662869+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:46:51.662869+0000', 'last_peered': '2024-09-05T16:46:51.662869+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:46:51.662869+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "33'2800", 'ondisk_log_start': "33'2800", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10031, 'log_dups_size': 0, 'ondisk_log_size': 10031, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2310, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12829, 'num_write_kb': 51325, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -2306}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:46:58.013 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:46:58.013 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "39'12773", 'reported_seq': 12899, 'reported_epoch': 45, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:51.658835+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:46:51.658835+0000', 'last_peered': '2024-09-05T16:46:51.658835+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:46:51.658835+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "33'2700", 'ondisk_log_start': "33'2700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10073, 'log_dups_size': 0, 'ondisk_log_size': 10073, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12771, 'num_write_kb': 51092, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2218}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:46:58.014 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:46:58.402 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:46:58.403 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 21452/80 objects degraded (26815.000%), 10 pgs degraded, 10 pgs undersized","count":20},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":821,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":45,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":10},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":9},{"state_name":"active+clean","count":6},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":20,"data_bytes":134720,"bytes_used":2810626048,"bytes_avail":383736430592,"bytes_total":386547056640,"degraded_objects":21452,"degraded_total":80,"degraded_ratio":268.14999999999998},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:46:58.416 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-09-05T16:46:58.417 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:46:58.735 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:46:58.736 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:46:58.748 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":446,"stamp":"2024-09-05T16:46:57.102140+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":134720,"num_objects":20,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21452,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":154601,"num_write_kb":570608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3419,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2744752,"kb_used_data":1556,"kb_used_omap":14,"kb_used_meta":2743089,"kb_avail":374742608,"statfs":{"total":386547056640,"available":383736430592,"internally_reserved":0,"allocated":1593344,"data_stored":637560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2808923703},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-6736,"num_objects":-1,"num_object_clones":0,"num_object_copies":-4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1918,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1,"num_whiteouts":0,"num_read":-1,"num_read_kb":-1,"num_write":-12751,"num_write_kb":-50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-334,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001546"},"pg_stats":[{"pgid":"1.f","version":"39'12904","reported_seq":17508,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-05T16:46:51.661878+0000","last_change":"2024-09-05T16:46:27.826222+0000","last_active":"2024-09-05T16:46:51.661878+0000","last_peered":"2024-09-05T16:46:51.661878+0000","last_clean":"2024-09-05T16:46:51.661878+0000","last_became_active":"2024-09-05T16:46:27.757928+0000","last_became_peered":"2024-09-05T16:46:27.757928+0000","last_unstale":"2024-09-05T16:46:51.661878+0000","last_undegraded":"2024-09-05T16:46:51.661878+0000","last_fullsized":"2024-09-05T16:46:51.661878+0000","mapping_epoch":40,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:32:35.713428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12904,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2262,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12552","reported_seq":17221,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-05T16:46:51.660993+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:46:51.660993+0000","last_peered":"2024-09-05T16:46:51.660993+0000","last_clean":"2024-09-05T16:46:51.660993+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:46:51.660993+0000","last_undegraded":"2024-09-05T16:46:51.660993+0000","last_fullsized":"2024-09-05T16:46:51.660993+0000","mapping_epoch":38,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1865,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12752","reported_seq":17376,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-05T16:46:51.992073+0000","last_change":"2024-09-05T16:46:51.992073+0000","last_active":"2024-09-05T16:46:51.992073+0000","last_peered":"2024-09-05T16:46:51.992073+0000","last_clean":"2024-09-05T16:46:51.992073+0000","last_became_active":"2024-09-05T16:46:51.667128+0000","last_became_peered":"2024-09-05T16:46:51.667128+0000","last_unstale":"2024-09-05T16:46:51.992073+0000","last_undegraded":"2024-09-05T16:46:51.992073+0000","last_fullsized":"2024-09-05T16:46:51.992073+0000","mapping_epoch":44,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12752","last_scrub_stamp":"2024-09-05T16:46:51.991995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:51.991995+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:04:54.120031+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12934","reported_seq":17681,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-05T16:46:51.660302+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:46:51.660302+0000","last_peered":"2024-09-05T16:46:51.660302+0000","last_clean":"2024-09-05T16:46:51.660302+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:46:51.660302+0000","last_undegraded":"2024-09-05T16:46:51.660302+0000","last_fullsized":"2024-09-05T16:46:51.660302+0000","mapping_epoch":34,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7234,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12593","reported_seq":12710,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:51.659051+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:46:51.659051+0000","last_peered":"2024-09-05T16:46:51.659051+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:46:51.659051+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12591,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2143}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12772","reported_seq":12904,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:51.661169+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:46:51.661169+0000","last_peered":"2024-09-05T16:46:51.661169+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:46:51.661169+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12772,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2219}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12660","reported_seq":14651,"reported_epoch":45,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:56.346864+0000","last_change":"2024-09-05T16:46:50.437247+0000","last_active":"2024-09-05T16:46:56.346864+0000","last_peered":"2024-09-05T16:46:56.346864+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:46:56.346864+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1201,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12658,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":950,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1199}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12777","reported_seq":12898,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-05T16:46:51.660614+0000","last_change":"2024-09-05T16:46:40.458315+0000","last_active":"2024-09-05T16:46:51.660614+0000","last_peered":"2024-09-05T16:46:51.660614+0000","last_clean":"2024-09-05T16:46:51.660614+0000","last_became_active":"2024-09-05T16:46:40.184288+0000","last_became_peered":"2024-09-05T16:46:40.184288+0000","last_unstale":"2024-09-05T16:46:51.660614+0000","last_undegraded":"2024-09-05T16:46:51.660614+0000","last_fullsized":"2024-09-05T16:46:51.660614+0000","mapping_epoch":42,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12777","last_scrub_stamp":"2024-09-05T16:46:40.458226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:40.458226+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:38:58.734210+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'12770","reported_seq":12889,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:51.660389+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:46:51.660389+0000","last_peered":"2024-09-05T16:46:51.660389+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:46:51.660389+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12767,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2240}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12808","reported_seq":12948,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:51.663085+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:46:51.663085+0000","last_peered":"2024-09-05T16:46:51.663085+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:46:51.663085+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12806,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2243}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13120","reported_seq":13264,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:51.659414+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:46:51.659414+0000","last_peered":"2024-09-05T16:46:51.659414+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:46:51.659414+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13117,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2278}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12758","reported_seq":12874,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:51.661868+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:46:51.661868+0000","last_peered":"2024-09-05T16:46:51.661868+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:46:51.661868+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2237}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12808","reported_seq":17561,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-05T16:46:51.660402+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:46:51.660402+0000","last_peered":"2024-09-05T16:46:51.660402+0000","last_clean":"2024-09-05T16:46:51.660402+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:46:51.660402+0000","last_undegraded":"2024-09-05T16:46:51.660402+0000","last_fullsized":"2024-09-05T16:46:51.660402+0000","mapping_epoch":36,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4408,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13124","reported_seq":13250,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:51.659301+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:46:51.659301+0000","last_peered":"2024-09-05T16:46:51.659301+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:46:51.659301+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13122,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2332}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12831","reported_seq":12961,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:51.662869+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:46:51.662869+0000","last_peered":"2024-09-05T16:46:51.662869+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:46:51.662869+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12829,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2306}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'12773","reported_seq":12899,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:51.658835+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:46:51.658835+0000","last_peered":"2024-09-05T16:46:51.658835+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:46:51.658835+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12771,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2218}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":134720,"num_objects":20,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21452,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":154601,"num_write_kb":570608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3419,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":86016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084442,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722060,"kb_used_data":372,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649780,"statfs":{"total":96636764160,"available":95897374720,"internally_reserved":0,"allocated":380928,"data_stored":141982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":738979539},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986319,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":595792,"kb_used_data":440,"kb_used_omap":4,"kb_used_meta":595323,"kb_avail":93776048,"statfs":{"total":96636764160,"available":96026673152,"internally_reserved":0,"allocated":450560,"data_stored":211614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":609611474},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673126,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713224,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":93658616,"statfs":{"total":96636764160,"available":95906422784,"internally_reserved":0,"allocated":380928,"data_stored":141982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":729938373},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182557,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713676,"kb_used_data":372,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658164,"statfs":{"total":96636764160,"available":95905959936,"internally_reserved":0,"allocated":380928,"data_stored":141982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":730394317},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":73728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:47:01.750 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:47:02.082 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:47:02.082 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:47:02.091 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":448,"stamp":"2024-09-05T16:47:01.102889+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":134720,"num_objects":20,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21452,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":154601,"num_write_kb":570608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3419,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2736424,"kb_used_data":1556,"kb_used_omap":14,"kb_used_meta":2734769,"kb_avail":374750936,"statfs":{"total":386547056640,"available":383744958464,"internally_reserved":0,"allocated":1593344,"data_stored":637560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2800404023},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-6736,"num_objects":-1,"num_object_clones":0,"num_object_copies":-4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-932,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1,"num_whiteouts":0,"num_read":-1,"num_read_kb":-1,"num_write":-12751,"num_write_kb":-50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1320,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001918"},"pg_stats":[{"pgid":"1.f","version":"39'12904","reported_seq":17508,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-05T16:46:51.661878+0000","last_change":"2024-09-05T16:46:27.826222+0000","last_active":"2024-09-05T16:46:51.661878+0000","last_peered":"2024-09-05T16:46:51.661878+0000","last_clean":"2024-09-05T16:46:51.661878+0000","last_became_active":"2024-09-05T16:46:27.757928+0000","last_became_peered":"2024-09-05T16:46:27.757928+0000","last_unstale":"2024-09-05T16:46:51.661878+0000","last_undegraded":"2024-09-05T16:46:51.661878+0000","last_fullsized":"2024-09-05T16:46:51.661878+0000","mapping_epoch":40,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:32:35.713428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12904,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2262,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12552","reported_seq":17221,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-05T16:46:51.660993+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:46:51.660993+0000","last_peered":"2024-09-05T16:46:51.660993+0000","last_clean":"2024-09-05T16:46:51.660993+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:46:51.660993+0000","last_undegraded":"2024-09-05T16:46:51.660993+0000","last_fullsized":"2024-09-05T16:46:51.660993+0000","mapping_epoch":38,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1865,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12752","reported_seq":17376,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-05T16:46:51.992073+0000","last_change":"2024-09-05T16:46:51.992073+0000","last_active":"2024-09-05T16:46:51.992073+0000","last_peered":"2024-09-05T16:46:51.992073+0000","last_clean":"2024-09-05T16:46:51.992073+0000","last_became_active":"2024-09-05T16:46:51.667128+0000","last_became_peered":"2024-09-05T16:46:51.667128+0000","last_unstale":"2024-09-05T16:46:51.992073+0000","last_undegraded":"2024-09-05T16:46:51.992073+0000","last_fullsized":"2024-09-05T16:46:51.992073+0000","mapping_epoch":44,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12752","last_scrub_stamp":"2024-09-05T16:46:51.991995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:51.991995+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:04:54.120031+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12934","reported_seq":17681,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-05T16:46:51.660302+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:46:51.660302+0000","last_peered":"2024-09-05T16:46:51.660302+0000","last_clean":"2024-09-05T16:46:51.660302+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:46:51.660302+0000","last_undegraded":"2024-09-05T16:46:51.660302+0000","last_fullsized":"2024-09-05T16:46:51.660302+0000","mapping_epoch":34,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7234,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12593","reported_seq":12710,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:51.659051+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:46:51.659051+0000","last_peered":"2024-09-05T16:46:51.659051+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:46:51.659051+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12591,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2143}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12772","reported_seq":12904,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:51.661169+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:46:51.661169+0000","last_peered":"2024-09-05T16:46:51.661169+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:46:51.661169+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12772,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2219}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12660","reported_seq":14651,"reported_epoch":45,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:56.346864+0000","last_change":"2024-09-05T16:46:50.437247+0000","last_active":"2024-09-05T16:46:56.346864+0000","last_peered":"2024-09-05T16:46:56.346864+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:46:56.346864+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1201,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12658,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":950,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1199}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12777","reported_seq":12898,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-05T16:46:51.660614+0000","last_change":"2024-09-05T16:46:40.458315+0000","last_active":"2024-09-05T16:46:51.660614+0000","last_peered":"2024-09-05T16:46:51.660614+0000","last_clean":"2024-09-05T16:46:51.660614+0000","last_became_active":"2024-09-05T16:46:40.184288+0000","last_became_peered":"2024-09-05T16:46:40.184288+0000","last_unstale":"2024-09-05T16:46:51.660614+0000","last_undegraded":"2024-09-05T16:46:51.660614+0000","last_fullsized":"2024-09-05T16:46:51.660614+0000","mapping_epoch":42,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12777","last_scrub_stamp":"2024-09-05T16:46:40.458226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:40.458226+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:38:58.734210+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'12770","reported_seq":12889,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:51.660389+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:46:51.660389+0000","last_peered":"2024-09-05T16:46:51.660389+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:46:51.660389+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12767,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2240}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12808","reported_seq":12948,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:51.663085+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:46:51.663085+0000","last_peered":"2024-09-05T16:46:51.663085+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:46:51.663085+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12806,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2243}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13120","reported_seq":13265,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:51.660518+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:46:51.660518+0000","last_peered":"2024-09-05T16:46:51.660518+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:46:51.660518+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13117,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2278}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12758","reported_seq":12874,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:51.661868+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:46:51.661868+0000","last_peered":"2024-09-05T16:46:51.661868+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:46:51.661868+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2237}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12808","reported_seq":17561,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-05T16:46:51.660402+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:46:51.660402+0000","last_peered":"2024-09-05T16:46:51.660402+0000","last_clean":"2024-09-05T16:46:51.660402+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:46:51.660402+0000","last_undegraded":"2024-09-05T16:46:51.660402+0000","last_fullsized":"2024-09-05T16:46:51.660402+0000","mapping_epoch":36,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4408,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13124","reported_seq":13250,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:51.659301+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:46:51.659301+0000","last_peered":"2024-09-05T16:46:51.659301+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:46:51.659301+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13122,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2332}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12831","reported_seq":12961,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:51.662869+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:46:51.662869+0000","last_peered":"2024-09-05T16:46:51.662869+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:46:51.662869+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12829,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2306}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'12773","reported_seq":12899,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:51.658835+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:46:51.658835+0000","last_peered":"2024-09-05T16:46:51.658835+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:46:51.658835+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12771,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2218}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":134720,"num_objects":20,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21452,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":154601,"num_write_kb":570608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3419,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":86016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084443,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722060,"kb_used_data":372,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649780,"statfs":{"total":96636764160,"available":95897374720,"internally_reserved":0,"allocated":380928,"data_stored":141982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":738979539},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986320,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":587464,"kb_used_data":440,"kb_used_omap":4,"kb_used_meta":587003,"kb_avail":93784376,"statfs":{"total":96636764160,"available":96035201024,"internally_reserved":0,"allocated":450560,"data_stored":211614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":601091794},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673127,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713224,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":93658616,"statfs":{"total":96636764160,"available":95906422784,"internally_reserved":0,"allocated":380928,"data_stored":141982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":729938373},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182557,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713676,"kb_used_data":372,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658164,"statfs":{"total":96636764160,"available":95905959936,"internally_reserved":0,"allocated":380928,"data_stored":141982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":730394317},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":73728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:47:02.093 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:47:02.477 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:47:02.477 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:47:02.491 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":449,"stamp":"2024-09-05T16:47:02.168108+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":134720,"num_objects":20,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20422,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":154601,"num_write_kb":570608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4449,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2736424,"kb_used_data":1556,"kb_used_omap":14,"kb_used_meta":2734769,"kb_avail":374750936,"statfs":{"total":386547056640,"available":383744958464,"internally_reserved":0,"allocated":1593344,"data_stored":637560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2800404023},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-6736,"num_objects":-1,"num_object_clones":0,"num_object_copies":-4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1829,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1,"num_whiteouts":0,"num_read":-1,"num_read_kb":-1,"num_write":-12751,"num_write_kb":-50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-423,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.511139"},"pg_stats":[{"pgid":"1.f","version":"39'12904","reported_seq":17508,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-05T16:46:51.661878+0000","last_change":"2024-09-05T16:46:27.826222+0000","last_active":"2024-09-05T16:46:51.661878+0000","last_peered":"2024-09-05T16:46:51.661878+0000","last_clean":"2024-09-05T16:46:51.661878+0000","last_became_active":"2024-09-05T16:46:27.757928+0000","last_became_peered":"2024-09-05T16:46:27.757928+0000","last_unstale":"2024-09-05T16:46:51.661878+0000","last_undegraded":"2024-09-05T16:46:51.661878+0000","last_fullsized":"2024-09-05T16:46:51.661878+0000","mapping_epoch":40,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:32:35.713428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12904,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2262,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12552","reported_seq":17221,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-05T16:46:51.660993+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:46:51.660993+0000","last_peered":"2024-09-05T16:46:51.660993+0000","last_clean":"2024-09-05T16:46:51.660993+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:46:51.660993+0000","last_undegraded":"2024-09-05T16:46:51.660993+0000","last_fullsized":"2024-09-05T16:46:51.660993+0000","mapping_epoch":38,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1865,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12752","reported_seq":17376,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-05T16:46:51.992073+0000","last_change":"2024-09-05T16:46:51.992073+0000","last_active":"2024-09-05T16:46:51.992073+0000","last_peered":"2024-09-05T16:46:51.992073+0000","last_clean":"2024-09-05T16:46:51.992073+0000","last_became_active":"2024-09-05T16:46:51.667128+0000","last_became_peered":"2024-09-05T16:46:51.667128+0000","last_unstale":"2024-09-05T16:46:51.992073+0000","last_undegraded":"2024-09-05T16:46:51.992073+0000","last_fullsized":"2024-09-05T16:46:51.992073+0000","mapping_epoch":44,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12752","last_scrub_stamp":"2024-09-05T16:46:51.991995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:51.991995+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:04:54.120031+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12934","reported_seq":17681,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-05T16:46:51.660302+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:46:51.660302+0000","last_peered":"2024-09-05T16:46:51.660302+0000","last_clean":"2024-09-05T16:46:51.660302+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:46:51.660302+0000","last_undegraded":"2024-09-05T16:46:51.660302+0000","last_fullsized":"2024-09-05T16:46:51.660302+0000","mapping_epoch":34,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7234,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12593","reported_seq":12710,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:51.659051+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:46:51.659051+0000","last_peered":"2024-09-05T16:46:51.659051+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:46:51.659051+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12591,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2143}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12772","reported_seq":12904,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:51.661169+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:46:51.661169+0000","last_peered":"2024-09-05T16:46:51.661169+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:46:51.661169+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12772,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2219}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12660","reported_seq":16711,"reported_epoch":45,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:01.346649+0000","last_change":"2024-09-05T16:46:50.437247+0000","last_active":"2024-09-05T16:47:01.346649+0000","last_peered":"2024-09-05T16:47:01.346649+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:47:01.346649+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":171,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12658,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1980,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-169}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12777","reported_seq":12898,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-05T16:46:51.660614+0000","last_change":"2024-09-05T16:46:40.458315+0000","last_active":"2024-09-05T16:46:51.660614+0000","last_peered":"2024-09-05T16:46:51.660614+0000","last_clean":"2024-09-05T16:46:51.660614+0000","last_became_active":"2024-09-05T16:46:40.184288+0000","last_became_peered":"2024-09-05T16:46:40.184288+0000","last_unstale":"2024-09-05T16:46:51.660614+0000","last_undegraded":"2024-09-05T16:46:51.660614+0000","last_fullsized":"2024-09-05T16:46:51.660614+0000","mapping_epoch":42,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12777","last_scrub_stamp":"2024-09-05T16:46:40.458226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:40.458226+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:38:58.734210+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'12770","reported_seq":12889,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:51.660389+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:46:51.660389+0000","last_peered":"2024-09-05T16:46:51.660389+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:46:51.660389+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12767,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2240}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12808","reported_seq":12948,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:51.663085+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:46:51.663085+0000","last_peered":"2024-09-05T16:46:51.663085+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:46:51.663085+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12806,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2243}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13120","reported_seq":13265,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:51.660518+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:46:51.660518+0000","last_peered":"2024-09-05T16:46:51.660518+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:46:51.660518+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13117,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2278}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12758","reported_seq":12874,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:51.661868+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:46:51.661868+0000","last_peered":"2024-09-05T16:46:51.661868+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:46:51.661868+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2237}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12808","reported_seq":17561,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-05T16:46:51.660402+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:46:51.660402+0000","last_peered":"2024-09-05T16:46:51.660402+0000","last_clean":"2024-09-05T16:46:51.660402+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:46:51.660402+0000","last_undegraded":"2024-09-05T16:46:51.660402+0000","last_fullsized":"2024-09-05T16:46:51.660402+0000","mapping_epoch":36,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4408,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13124","reported_seq":13250,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:51.659301+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:46:51.659301+0000","last_peered":"2024-09-05T16:46:51.659301+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:46:51.659301+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13122,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2332}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12831","reported_seq":12961,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:51.662869+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:46:51.662869+0000","last_peered":"2024-09-05T16:46:51.662869+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:46:51.662869+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12829,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2306}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'12773","reported_seq":12899,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:51.658835+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:46:51.658835+0000","last_peered":"2024-09-05T16:46:51.658835+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:46:51.658835+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12771,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2218}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":134720,"num_objects":20,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20422,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":154601,"num_write_kb":570608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4449,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":86016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084443,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722060,"kb_used_data":372,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649780,"statfs":{"total":96636764160,"available":95897374720,"internally_reserved":0,"allocated":380928,"data_stored":141982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":738979539},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986320,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":587464,"kb_used_data":440,"kb_used_omap":4,"kb_used_meta":587003,"kb_avail":93784376,"statfs":{"total":96636764160,"available":96035201024,"internally_reserved":0,"allocated":450560,"data_stored":211614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":601091794},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673127,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713224,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":93658616,"statfs":{"total":96636764160,"available":95906422784,"internally_reserved":0,"allocated":380928,"data_stored":141982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":729938373},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182558,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713676,"kb_used_data":372,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658164,"statfs":{"total":96636764160,"available":95905959936,"internally_reserved":0,"allocated":380928,"data_stored":141982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":730394317},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":73728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:47:02.492 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:47:02.492 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "39'12593", 'reported_seq': 12710, 'reported_epoch': 45, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:51.659051+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:46:51.659051+0000', 'last_peered': '2024-09-05T16:46:51.659051+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:46:51.659051+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "33'2500", 'ondisk_log_start': "33'2500", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10093, 'log_dups_size': 0, 'ondisk_log_size': 10093, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2147, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12591, 'num_write_kb': 50373, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2143}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:47:02.492 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:47:02.492 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "39'12772", 'reported_seq': 12904, 'reported_epoch': 45, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:51.661169+0000', 'last_change': '2024-09-05T16:34:56.860530+0000', 'last_active': '2024-09-05T16:46:51.661169+0000', 'last_peered': '2024-09-05T16:46:51.661169+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:46:51.661169+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "33'2700", 'ondisk_log_start': "33'2700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10072, 'log_dups_size': 0, 'ondisk_log_size': 10072, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12772, 'num_write_kb': 51088, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2219}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:47:02.492 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:47:02.492 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "39'12660", 'reported_seq': 16711, 'reported_epoch': 45, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:47:01.346649+0000', 'last_change': '2024-09-05T16:46:50.437247+0000', 'last_active': '2024-09-05T16:47:01.346649+0000', 'last_peered': '2024-09-05T16:47:01.346649+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:47:01.346649+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "33'2600", 'ondisk_log_start': "33'2600", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10060, 'log_dups_size': 0, 'ondisk_log_size': 10060, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 171, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12658, 'num_write_kb': 50636, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1980, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -169}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:47:02.492 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:47:02.492 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "39'12770", 'reported_seq': 12889, 'reported_epoch': 45, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:51.660389+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:46:51.660389+0000', 'last_peered': '2024-09-05T16:46:51.660389+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:46:51.660389+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "33'2700", 'ondisk_log_start': "33'2700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10070, 'log_dups_size': 0, 'ondisk_log_size': 10070, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26944, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 16, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12767, 'num_write_kb': 51081, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2240}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:47:02.492 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:47:02.493 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "39'12808", 'reported_seq': 12948, 'reported_epoch': 45, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:51.663085+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:46:51.663085+0000', 'last_peered': '2024-09-05T16:46:51.663085+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:46:51.663085+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "33'2800", 'ondisk_log_start': "33'2800", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10008, 'log_dups_size': 0, 'ondisk_log_size': 10008, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12806, 'num_write_kb': 51230, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -2243}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:47:02.493 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:47:02.493 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "39'13120", 'reported_seq': 13265, 'reported_epoch': 45, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:51.660518+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:46:51.660518+0000', 'last_peered': '2024-09-05T16:46:51.660518+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:46:51.660518+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "33'3100", 'ondisk_log_start': "33'3100", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20208, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2284, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13117, 'num_write_kb': 52476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2278}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:47:02.493 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:47:02.493 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "39'12758", 'reported_seq': 12874, 'reported_epoch': 45, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:51.661868+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:46:51.661868+0000', 'last_peered': '2024-09-05T16:46:51.661868+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:46:51.661868+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "33'2700", 'ondisk_log_start': "33'2700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10058, 'log_dups_size': 0, 'ondisk_log_size': 10058, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6736, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12757, 'num_write_kb': 51031, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2237}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:47:02.493 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:47:02.493 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "39'13124", 'reported_seq': 13250, 'reported_epoch': 45, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:51.659301+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:46:51.659301+0000', 'last_peered': '2024-09-05T16:46:51.659301+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:46:51.659301+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "33'3100", 'ondisk_log_start': "33'3100", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10024, 'log_dups_size': 0, 'ondisk_log_size': 10024, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13122, 'num_write_kb': 52500, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2332}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:47:02.493 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:47:02.493 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "39'12831", 'reported_seq': 12961, 'reported_epoch': 45, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:51.662869+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:46:51.662869+0000', 'last_peered': '2024-09-05T16:46:51.662869+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:46:51.662869+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "33'2800", 'ondisk_log_start': "33'2800", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10031, 'log_dups_size': 0, 'ondisk_log_size': 10031, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2310, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12829, 'num_write_kb': 51325, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -2306}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:47:02.493 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:47:02.493 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "39'12773", 'reported_seq': 12899, 'reported_epoch': 45, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:51.658835+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:46:51.658835+0000', 'last_peered': '2024-09-05T16:46:51.658835+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:46:51.658835+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "33'2700", 'ondisk_log_start': "33'2700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10073, 'log_dups_size': 0, 'ondisk_log_size': 10073, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12771, 'num_write_kb': 51092, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2218}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:47:02.494 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:47:02.930 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:47:02.930 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 21452/80 objects degraded (26815.000%), 10 pgs degraded, 10 pgs undersized","count":20},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":826,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":46,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":9},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":9},{"state_name":"active+clean","count":6},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":20,"data_bytes":134720,"bytes_used":2802098176,"bytes_avail":383744958464,"bytes_total":386547056640,"degraded_objects":21452,"degraded_total":80,"degraded_ratio":268.14999999999998},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:47:02.945 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-09-05T16:47:02.945 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:47:03.335 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:47:03.336 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:47:03.348 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":451,"stamp":"2024-09-05T16:47:03.176779+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":134720,"num_objects":20,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20422,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":154601,"num_write_kb":570608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4449,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2744756,"kb_used_data":1564,"kb_used_omap":14,"kb_used_meta":2743089,"kb_avail":374742604,"statfs":{"total":386547056640,"available":383736426496,"internally_reserved":0,"allocated":1601536,"data_stored":640961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2808923703},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-6736,"num_objects":-1,"num_object_clones":0,"num_object_copies":-4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1829,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1,"num_whiteouts":0,"num_read":-1,"num_read_kb":-1,"num_write":-12751,"num_write_kb":-50999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-423,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.074990"},"pg_stats":[{"pgid":"1.f","version":"39'12904","reported_seq":17508,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-05T16:46:51.661878+0000","last_change":"2024-09-05T16:46:27.826222+0000","last_active":"2024-09-05T16:46:51.661878+0000","last_peered":"2024-09-05T16:46:51.661878+0000","last_clean":"2024-09-05T16:46:51.661878+0000","last_became_active":"2024-09-05T16:46:27.757928+0000","last_became_peered":"2024-09-05T16:46:27.757928+0000","last_unstale":"2024-09-05T16:46:51.661878+0000","last_undegraded":"2024-09-05T16:46:51.661878+0000","last_fullsized":"2024-09-05T16:46:51.661878+0000","mapping_epoch":40,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:32:35.713428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12904,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2262,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12552","reported_seq":17223,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-05T16:47:02.169721+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:47:02.169721+0000","last_peered":"2024-09-05T16:47:02.169721+0000","last_clean":"2024-09-05T16:47:02.169721+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:47:02.169721+0000","last_undegraded":"2024-09-05T16:47:02.169721+0000","last_fullsized":"2024-09-05T16:47:02.169721+0000","mapping_epoch":38,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1865,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12752","reported_seq":17376,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-05T16:46:51.992073+0000","last_change":"2024-09-05T16:46:51.992073+0000","last_active":"2024-09-05T16:46:51.992073+0000","last_peered":"2024-09-05T16:46:51.992073+0000","last_clean":"2024-09-05T16:46:51.992073+0000","last_became_active":"2024-09-05T16:46:51.667128+0000","last_became_peered":"2024-09-05T16:46:51.667128+0000","last_unstale":"2024-09-05T16:46:51.992073+0000","last_undegraded":"2024-09-05T16:46:51.992073+0000","last_fullsized":"2024-09-05T16:46:51.992073+0000","mapping_epoch":44,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12752","last_scrub_stamp":"2024-09-05T16:46:51.991995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:51.991995+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:04:54.120031+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12934","reported_seq":17683,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-05T16:47:02.169362+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:47:02.169362+0000","last_peered":"2024-09-05T16:47:02.169362+0000","last_clean":"2024-09-05T16:47:02.169362+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:47:02.169362+0000","last_undegraded":"2024-09-05T16:47:02.169362+0000","last_fullsized":"2024-09-05T16:47:02.169362+0000","mapping_epoch":34,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7234,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12593","reported_seq":12710,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:51.659051+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:46:51.659051+0000","last_peered":"2024-09-05T16:46:51.659051+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:46:51.659051+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12591,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2143}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12772","reported_seq":12904,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:51.661169+0000","last_change":"2024-09-05T16:34:56.860530+0000","last_active":"2024-09-05T16:46:51.661169+0000","last_peered":"2024-09-05T16:46:51.661169+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:46:51.661169+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12772,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2219}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12660","reported_seq":16711,"reported_epoch":45,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:01.346649+0000","last_change":"2024-09-05T16:46:50.437247+0000","last_active":"2024-09-05T16:47:01.346649+0000","last_peered":"2024-09-05T16:47:01.346649+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:47:01.346649+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":171,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12658,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1980,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-169}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12777","reported_seq":12900,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-05T16:47:02.169491+0000","last_change":"2024-09-05T16:46:40.458315+0000","last_active":"2024-09-05T16:47:02.169491+0000","last_peered":"2024-09-05T16:47:02.169491+0000","last_clean":"2024-09-05T16:47:02.169491+0000","last_became_active":"2024-09-05T16:46:40.184288+0000","last_became_peered":"2024-09-05T16:46:40.184288+0000","last_unstale":"2024-09-05T16:47:02.169491+0000","last_undegraded":"2024-09-05T16:47:02.169491+0000","last_fullsized":"2024-09-05T16:47:02.169491+0000","mapping_epoch":42,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12777","last_scrub_stamp":"2024-09-05T16:46:40.458226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:40.458226+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:38:58.734210+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'12770","reported_seq":12889,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:51.660389+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:46:51.660389+0000","last_peered":"2024-09-05T16:46:51.660389+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:46:51.660389+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12767,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2240}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12808","reported_seq":12948,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:51.663085+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:46:51.663085+0000","last_peered":"2024-09-05T16:46:51.663085+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:46:51.663085+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12806,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2243}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13120","reported_seq":13265,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:51.660518+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:46:51.660518+0000","last_peered":"2024-09-05T16:46:51.660518+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:46:51.660518+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13117,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2278}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12758","reported_seq":12874,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:51.661868+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:46:51.661868+0000","last_peered":"2024-09-05T16:46:51.661868+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:46:51.661868+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2237}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12808","reported_seq":17563,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-05T16:47:02.169412+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:47:02.169412+0000","last_peered":"2024-09-05T16:47:02.169412+0000","last_clean":"2024-09-05T16:47:02.169412+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:47:02.169412+0000","last_undegraded":"2024-09-05T16:47:02.169412+0000","last_fullsized":"2024-09-05T16:47:02.169412+0000","mapping_epoch":36,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4408,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13124","reported_seq":13250,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:51.659301+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:46:51.659301+0000","last_peered":"2024-09-05T16:46:51.659301+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:46:51.659301+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13122,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2332}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12831","reported_seq":12961,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:51.662869+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:46:51.662869+0000","last_peered":"2024-09-05T16:46:51.662869+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:46:51.662869+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12829,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2306}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'12773","reported_seq":12899,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:51.658835+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:46:51.658835+0000","last_peered":"2024-09-05T16:46:51.658835+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:46:51.658835+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12771,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2218}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":134720,"num_objects":20,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20422,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":154601,"num_write_kb":570608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4449,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":86016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084443,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722060,"kb_used_data":372,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649780,"statfs":{"total":96636764160,"available":95897374720,"internally_reserved":0,"allocated":380928,"data_stored":141982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":738979539},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986321,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":595796,"kb_used_data":448,"kb_used_omap":4,"kb_used_meta":595323,"kb_avail":93776044,"statfs":{"total":96636764160,"available":96026669056,"internally_reserved":0,"allocated":458752,"data_stored":215015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":609611474},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673127,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713224,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":93658616,"statfs":{"total":96636764160,"available":95906422784,"internally_reserved":0,"allocated":380928,"data_stored":141982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":729938373},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182558,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713676,"kb_used_data":372,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658164,"statfs":{"total":96636764160,"available":95905959936,"internally_reserved":0,"allocated":380928,"data_stored":141982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":730394317},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":73728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:47:06.350 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:47:06.655 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:47:06.655 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:47:06.667 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":452,"stamp":"2024-09-05T16:47:05.103580+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":134720,"num_objects":20,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":154601,"num_write_kb":570608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4630,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2744760,"kb_used_data":1572,"kb_used_omap":14,"kb_used_meta":2743089,"kb_avail":374742600,"statfs":{"total":386547056640,"available":383736422400,"internally_reserved":0,"allocated":1609728,"data_stored":644362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2808923703},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1211,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1211,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001440"},"pg_stats":[{"pgid":"1.f","version":"39'12904","reported_seq":17512,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-05T16:47:03.183174+0000","last_change":"2024-09-05T16:46:27.826222+0000","last_active":"2024-09-05T16:47:03.183174+0000","last_peered":"2024-09-05T16:47:03.183174+0000","last_clean":"2024-09-05T16:47:03.183174+0000","last_became_active":"2024-09-05T16:46:27.757928+0000","last_became_peered":"2024-09-05T16:46:27.757928+0000","last_unstale":"2024-09-05T16:47:03.183174+0000","last_undegraded":"2024-09-05T16:47:03.183174+0000","last_fullsized":"2024-09-05T16:47:03.183174+0000","mapping_epoch":40,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:32:35.713428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12904,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2262,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12552","reported_seq":17223,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-05T16:47:02.169721+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:47:02.169721+0000","last_peered":"2024-09-05T16:47:02.169721+0000","last_clean":"2024-09-05T16:47:02.169721+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:47:02.169721+0000","last_undegraded":"2024-09-05T16:47:02.169721+0000","last_fullsized":"2024-09-05T16:47:02.169721+0000","mapping_epoch":38,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1865,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12752","reported_seq":17376,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-05T16:46:51.992073+0000","last_change":"2024-09-05T16:46:51.992073+0000","last_active":"2024-09-05T16:46:51.992073+0000","last_peered":"2024-09-05T16:46:51.992073+0000","last_clean":"2024-09-05T16:46:51.992073+0000","last_became_active":"2024-09-05T16:46:51.667128+0000","last_became_peered":"2024-09-05T16:46:51.667128+0000","last_unstale":"2024-09-05T16:46:51.992073+0000","last_undegraded":"2024-09-05T16:46:51.992073+0000","last_fullsized":"2024-09-05T16:46:51.992073+0000","mapping_epoch":44,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12752","last_scrub_stamp":"2024-09-05T16:46:51.991995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:51.991995+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:04:54.120031+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12934","reported_seq":17683,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-05T16:47:02.169362+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:47:02.169362+0000","last_peered":"2024-09-05T16:47:02.169362+0000","last_clean":"2024-09-05T16:47:02.169362+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:47:02.169362+0000","last_undegraded":"2024-09-05T16:47:02.169362+0000","last_fullsized":"2024-09-05T16:47:02.169362+0000","mapping_epoch":34,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7234,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12593","reported_seq":12714,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:03.180301+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:47:03.180301+0000","last_peered":"2024-09-05T16:47:03.180301+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:47:03.180301+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12591,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2143}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12772","reported_seq":13271,"reported_epoch":47,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:03.203001+0000","last_change":"2024-09-05T16:47:02.041664+0000","last_active":"2024-09-05T16:47:03.203001+0000","last_peered":"2024-09-05T16:47:03.203001+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:47:03.203001+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12772,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":212,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2038}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12660","reported_seq":16711,"reported_epoch":45,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:01.346649+0000","last_change":"2024-09-05T16:46:50.437247+0000","last_active":"2024-09-05T16:47:01.346649+0000","last_peered":"2024-09-05T16:47:01.346649+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:47:01.346649+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":171,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12658,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1980,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-169}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12777","reported_seq":12900,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-05T16:47:02.169491+0000","last_change":"2024-09-05T16:46:40.458315+0000","last_active":"2024-09-05T16:47:02.169491+0000","last_peered":"2024-09-05T16:47:02.169491+0000","last_clean":"2024-09-05T16:47:02.169491+0000","last_became_active":"2024-09-05T16:46:40.184288+0000","last_became_peered":"2024-09-05T16:46:40.184288+0000","last_unstale":"2024-09-05T16:47:02.169491+0000","last_undegraded":"2024-09-05T16:47:02.169491+0000","last_fullsized":"2024-09-05T16:47:02.169491+0000","mapping_epoch":42,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12777","last_scrub_stamp":"2024-09-05T16:46:40.458226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:40.458226+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:38:58.734210+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'12770","reported_seq":12893,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:03.182908+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:47:03.182908+0000","last_peered":"2024-09-05T16:47:03.182908+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:47:03.182908+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12767,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2240}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12808","reported_seq":12948,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:51.663085+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:46:51.663085+0000","last_peered":"2024-09-05T16:46:51.663085+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:46:51.663085+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12806,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2243}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13120","reported_seq":13269,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:03.182464+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:47:03.182464+0000","last_peered":"2024-09-05T16:47:03.182464+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:47:03.182464+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13117,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2278}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12758","reported_seq":12878,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:03.183659+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:47:03.183659+0000","last_peered":"2024-09-05T16:47:03.183659+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:47:03.183659+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2237}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12808","reported_seq":17563,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-05T16:47:02.169412+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:47:02.169412+0000","last_peered":"2024-09-05T16:47:02.169412+0000","last_clean":"2024-09-05T16:47:02.169412+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:47:02.169412+0000","last_undegraded":"2024-09-05T16:47:02.169412+0000","last_fullsized":"2024-09-05T16:47:02.169412+0000","mapping_epoch":36,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4408,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13124","reported_seq":13254,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:03.182450+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:47:03.182450+0000","last_peered":"2024-09-05T16:47:03.182450+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:47:03.182450+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13122,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2332}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12831","reported_seq":12961,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:51.662869+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:46:51.662869+0000","last_peered":"2024-09-05T16:46:51.662869+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:46:51.662869+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12829,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2306}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'12773","reported_seq":12899,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:51.658835+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:46:51.658835+0000","last_peered":"2024-09-05T16:46:51.658835+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:46:51.658835+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12771,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2218}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":134720,"num_objects":20,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":154601,"num_write_kb":570608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4630,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":86016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084444,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722064,"kb_used_data":380,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649776,"statfs":{"total":96636764160,"available":95897370624,"internally_reserved":0,"allocated":389120,"data_stored":145383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":738979539},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986321,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":595796,"kb_used_data":448,"kb_used_omap":4,"kb_used_meta":595323,"kb_avail":93776044,"statfs":{"total":96636764160,"available":96026669056,"internally_reserved":0,"allocated":458752,"data_stored":215015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":609611474},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673127,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713224,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":93658616,"statfs":{"total":96636764160,"available":95906422784,"internally_reserved":0,"allocated":380928,"data_stored":141982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":729938373},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182558,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713676,"kb_used_data":372,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658164,"statfs":{"total":96636764160,"available":95905959936,"internally_reserved":0,"allocated":380928,"data_stored":141982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":730394317},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":73728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:47:06.668 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:47:06.969 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:47:07.051 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:47:07.052 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":452,"stamp":"2024-09-05T16:47:05.103580+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":134720,"num_objects":20,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":154601,"num_write_kb":570608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4630,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2744760,"kb_used_data":1572,"kb_used_omap":14,"kb_used_meta":2743089,"kb_avail":374742600,"statfs":{"total":386547056640,"available":383736422400,"internally_reserved":0,"allocated":1609728,"data_stored":644362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2808923703},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1211,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1211,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001440"},"pg_stats":[{"pgid":"1.f","version":"39'12904","reported_seq":17512,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-05T16:47:03.183174+0000","last_change":"2024-09-05T16:46:27.826222+0000","last_active":"2024-09-05T16:47:03.183174+0000","last_peered":"2024-09-05T16:47:03.183174+0000","last_clean":"2024-09-05T16:47:03.183174+0000","last_became_active":"2024-09-05T16:46:27.757928+0000","last_became_peered":"2024-09-05T16:46:27.757928+0000","last_unstale":"2024-09-05T16:47:03.183174+0000","last_undegraded":"2024-09-05T16:47:03.183174+0000","last_fullsized":"2024-09-05T16:47:03.183174+0000","mapping_epoch":40,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:32:35.713428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12904,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2262,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12552","reported_seq":17223,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-05T16:47:02.169721+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:47:02.169721+0000","last_peered":"2024-09-05T16:47:02.169721+0000","last_clean":"2024-09-05T16:47:02.169721+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:47:02.169721+0000","last_undegraded":"2024-09-05T16:47:02.169721+0000","last_fullsized":"2024-09-05T16:47:02.169721+0000","mapping_epoch":38,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1865,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12752","reported_seq":17376,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-05T16:46:51.992073+0000","last_change":"2024-09-05T16:46:51.992073+0000","last_active":"2024-09-05T16:46:51.992073+0000","last_peered":"2024-09-05T16:46:51.992073+0000","last_clean":"2024-09-05T16:46:51.992073+0000","last_became_active":"2024-09-05T16:46:51.667128+0000","last_became_peered":"2024-09-05T16:46:51.667128+0000","last_unstale":"2024-09-05T16:46:51.992073+0000","last_undegraded":"2024-09-05T16:46:51.992073+0000","last_fullsized":"2024-09-05T16:46:51.992073+0000","mapping_epoch":44,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12752","last_scrub_stamp":"2024-09-05T16:46:51.991995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:51.991995+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:04:54.120031+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12934","reported_seq":17683,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-05T16:47:02.169362+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:47:02.169362+0000","last_peered":"2024-09-05T16:47:02.169362+0000","last_clean":"2024-09-05T16:47:02.169362+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:47:02.169362+0000","last_undegraded":"2024-09-05T16:47:02.169362+0000","last_fullsized":"2024-09-05T16:47:02.169362+0000","mapping_epoch":34,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7234,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12593","reported_seq":12714,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:03.180301+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:47:03.180301+0000","last_peered":"2024-09-05T16:47:03.180301+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:47:03.180301+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12591,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2143}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12772","reported_seq":13271,"reported_epoch":47,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:03.203001+0000","last_change":"2024-09-05T16:47:02.041664+0000","last_active":"2024-09-05T16:47:03.203001+0000","last_peered":"2024-09-05T16:47:03.203001+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:47:03.203001+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12772,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":212,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2038}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12660","reported_seq":16711,"reported_epoch":45,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:01.346649+0000","last_change":"2024-09-05T16:46:50.437247+0000","last_active":"2024-09-05T16:47:01.346649+0000","last_peered":"2024-09-05T16:47:01.346649+0000","last_clean":"2024-09-05T16:33:53.758091+0000","last_became_active":"2024-09-05T16:34:56.894018+0000","last_became_peered":"2024-09-05T16:34:56.894018+0000","last_unstale":"2024-09-05T16:47:01.346649+0000","last_undegraded":"2024-09-05T16:34:56.592187+0000","last_fullsized":"2024-09-05T16:34:56.559436+0000","mapping_epoch":32,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":171,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12658,"num_write_kb":50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1980,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-169}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12777","reported_seq":12900,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-05T16:47:02.169491+0000","last_change":"2024-09-05T16:46:40.458315+0000","last_active":"2024-09-05T16:47:02.169491+0000","last_peered":"2024-09-05T16:47:02.169491+0000","last_clean":"2024-09-05T16:47:02.169491+0000","last_became_active":"2024-09-05T16:46:40.184288+0000","last_became_peered":"2024-09-05T16:46:40.184288+0000","last_unstale":"2024-09-05T16:47:02.169491+0000","last_undegraded":"2024-09-05T16:47:02.169491+0000","last_fullsized":"2024-09-05T16:47:02.169491+0000","mapping_epoch":42,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12777","last_scrub_stamp":"2024-09-05T16:46:40.458226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:40.458226+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:38:58.734210+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'12770","reported_seq":12893,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:03.182908+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:47:03.182908+0000","last_peered":"2024-09-05T16:47:03.182908+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:47:03.182908+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12767,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2240}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12808","reported_seq":12948,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:51.663085+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:46:51.663085+0000","last_peered":"2024-09-05T16:46:51.663085+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:46:51.663085+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12806,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2243}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13120","reported_seq":13269,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:03.182464+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:47:03.182464+0000","last_peered":"2024-09-05T16:47:03.182464+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:47:03.182464+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13117,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2278}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12758","reported_seq":12878,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:03.183659+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:47:03.183659+0000","last_peered":"2024-09-05T16:47:03.183659+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:47:03.183659+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2237}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12808","reported_seq":17563,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-05T16:47:02.169412+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:47:02.169412+0000","last_peered":"2024-09-05T16:47:02.169412+0000","last_clean":"2024-09-05T16:47:02.169412+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:47:02.169412+0000","last_undegraded":"2024-09-05T16:47:02.169412+0000","last_fullsized":"2024-09-05T16:47:02.169412+0000","mapping_epoch":36,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4408,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13124","reported_seq":13254,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:03.182450+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:47:03.182450+0000","last_peered":"2024-09-05T16:47:03.182450+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:47:03.182450+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13122,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2332}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12831","reported_seq":12961,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:51.662869+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:46:51.662869+0000","last_peered":"2024-09-05T16:46:51.662869+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:46:51.662869+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12829,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2306}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'12773","reported_seq":12899,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:46:51.658835+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:46:51.658835+0000","last_peered":"2024-09-05T16:46:51.658835+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:46:51.658835+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12771,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2218}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":134720,"num_objects":20,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":154601,"num_write_kb":570608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4630,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":86016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084444,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722064,"kb_used_data":380,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649776,"statfs":{"total":96636764160,"available":95897370624,"internally_reserved":0,"allocated":389120,"data_stored":145383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":738979539},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986321,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":595796,"kb_used_data":448,"kb_used_omap":4,"kb_used_meta":595323,"kb_avail":93776044,"statfs":{"total":96636764160,"available":96026669056,"internally_reserved":0,"allocated":458752,"data_stored":215015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":609611474},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673127,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713224,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":93658616,"statfs":{"total":96636764160,"available":95906422784,"internally_reserved":0,"allocated":380928,"data_stored":141982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":729938373},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182558,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713676,"kb_used_data":372,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658164,"statfs":{"total":96636764160,"available":95905959936,"internally_reserved":0,"allocated":380928,"data_stored":141982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":730394317},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":73728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:47:07.053 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:47:07.053 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "39'12593", 'reported_seq': 12714, 'reported_epoch': 47, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:47:03.180301+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:47:03.180301+0000', 'last_peered': '2024-09-05T16:47:03.180301+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:47:03.180301+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "33'2500", 'ondisk_log_start': "33'2500", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10093, 'log_dups_size': 0, 'ondisk_log_size': 10093, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2147, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12591, 'num_write_kb': 50373, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2143}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:47:07.053 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:47:07.053 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "39'12772", 'reported_seq': 13271, 'reported_epoch': 47, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:47:03.203001+0000', 'last_change': '2024-09-05T16:47:02.041664+0000', 'last_active': '2024-09-05T16:47:03.203001+0000', 'last_peered': '2024-09-05T16:47:03.203001+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:47:03.203001+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "33'2700", 'ondisk_log_start': "33'2700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10072, 'log_dups_size': 0, 'ondisk_log_size': 10072, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2038, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12772, 'num_write_kb': 51088, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 212, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2038}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:47:07.053 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-05T16:47:07.053 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "39'12660", 'reported_seq': 16711, 'reported_epoch': 45, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:47:01.346649+0000', 'last_change': '2024-09-05T16:46:50.437247+0000', 'last_active': '2024-09-05T16:47:01.346649+0000', 'last_peered': '2024-09-05T16:47:01.346649+0000', 'last_clean': '2024-09-05T16:33:53.758091+0000', 'last_became_active': '2024-09-05T16:34:56.894018+0000', 'last_became_peered': '2024-09-05T16:34:56.894018+0000', 'last_unstale': '2024-09-05T16:47:01.346649+0000', 'last_undegraded': '2024-09-05T16:34:56.592187+0000', 'last_fullsized': '2024-09-05T16:34:56.559436+0000', 'mapping_epoch': 32, 'log_start': "33'2600", 'ondisk_log_start': "33'2600", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10060, 'log_dups_size': 0, 'ondisk_log_size': 10060, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 171, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12658, 'num_write_kb': 50636, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1980, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -169}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:47:07.053 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:47:07.053 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "39'12770", 'reported_seq': 12893, 'reported_epoch': 47, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:47:03.182908+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:47:03.182908+0000', 'last_peered': '2024-09-05T16:47:03.182908+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:47:03.182908+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "33'2700", 'ondisk_log_start': "33'2700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10070, 'log_dups_size': 0, 'ondisk_log_size': 10070, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26944, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 16, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12767, 'num_write_kb': 51081, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2240}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:47:07.053 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:47:07.053 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "39'12808", 'reported_seq': 12948, 'reported_epoch': 45, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:51.663085+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:46:51.663085+0000', 'last_peered': '2024-09-05T16:46:51.663085+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:46:51.663085+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "33'2800", 'ondisk_log_start': "33'2800", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10008, 'log_dups_size': 0, 'ondisk_log_size': 10008, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12806, 'num_write_kb': 51230, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -2243}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:47:07.054 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:47:07.054 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "39'13120", 'reported_seq': 13269, 'reported_epoch': 47, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:47:03.182464+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:47:03.182464+0000', 'last_peered': '2024-09-05T16:47:03.182464+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:47:03.182464+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "33'3100", 'ondisk_log_start': "33'3100", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20208, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2284, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13117, 'num_write_kb': 52476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2278}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:47:07.054 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:47:07.054 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "39'12758", 'reported_seq': 12878, 'reported_epoch': 47, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:47:03.183659+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:47:03.183659+0000', 'last_peered': '2024-09-05T16:47:03.183659+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:47:03.183659+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "33'2700", 'ondisk_log_start': "33'2700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10058, 'log_dups_size': 0, 'ondisk_log_size': 10058, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6736, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12757, 'num_write_kb': 51031, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2237}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:47:07.054 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:47:07.054 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "39'13124", 'reported_seq': 13254, 'reported_epoch': 47, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:47:03.182450+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:47:03.182450+0000', 'last_peered': '2024-09-05T16:47:03.182450+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:47:03.182450+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "33'3100", 'ondisk_log_start': "33'3100", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10024, 'log_dups_size': 0, 'ondisk_log_size': 10024, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13122, 'num_write_kb': 52500, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2332}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:47:07.054 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:47:07.054 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "39'12831", 'reported_seq': 12961, 'reported_epoch': 45, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:51.662869+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:46:51.662869+0000', 'last_peered': '2024-09-05T16:46:51.662869+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:46:51.662869+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "33'2800", 'ondisk_log_start': "33'2800", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10031, 'log_dups_size': 0, 'ondisk_log_size': 10031, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2310, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12829, 'num_write_kb': 51325, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -2306}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:47:07.054 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:47:07.054 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "39'12773", 'reported_seq': 12899, 'reported_epoch': 45, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:46:51.658835+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:46:51.658835+0000', 'last_peered': '2024-09-05T16:46:51.658835+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:46:51.658835+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "33'2700", 'ondisk_log_start': "33'2700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10073, 'log_dups_size': 0, 'ondisk_log_size': 10073, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12771, 'num_write_kb': 51092, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2218}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:47:07.055 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:47:07.455 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:47:07.456 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 20070/72 objects degraded (27875.000%), 9 pgs degraded, 9 pgs undersized","count":18},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":830,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":47,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":9},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":8},{"state_name":"active+clean","count":7},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":18,"data_bytes":121248,"bytes_used":2810667008,"bytes_avail":383736389632,"bytes_total":386547056640,"degraded_objects":20070,"degraded_total":72,"degraded_ratio":278.75},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:47:07.469 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-09-05T16:47:07.469 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:47:07.770 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:47:07.771 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:47:07.783 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":453,"stamp":"2024-09-05T16:47:07.103938+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":121248,"num_objects":18,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20070,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141943,"num_write_kb":519972,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2650,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2744792,"kb_used_data":1604,"kb_used_omap":14,"kb_used_meta":2743089,"kb_avail":374742568,"statfs":{"total":386547056640,"available":383736389632,"internally_reserved":0,"allocated":1642496,"data_stored":657924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2808923703},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13472,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1382,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12658,"num_write_kb":-50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-769,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001406"},"pg_stats":[{"pgid":"1.f","version":"39'12904","reported_seq":17512,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-05T16:47:03.183174+0000","last_change":"2024-09-05T16:46:27.826222+0000","last_active":"2024-09-05T16:47:03.183174+0000","last_peered":"2024-09-05T16:47:03.183174+0000","last_clean":"2024-09-05T16:47:03.183174+0000","last_became_active":"2024-09-05T16:46:27.757928+0000","last_became_peered":"2024-09-05T16:46:27.757928+0000","last_unstale":"2024-09-05T16:47:03.183174+0000","last_undegraded":"2024-09-05T16:47:03.183174+0000","last_fullsized":"2024-09-05T16:47:03.183174+0000","mapping_epoch":40,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:32:35.713428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12904,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2262,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12552","reported_seq":17223,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-05T16:47:02.169721+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:47:02.169721+0000","last_peered":"2024-09-05T16:47:02.169721+0000","last_clean":"2024-09-05T16:47:02.169721+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:47:02.169721+0000","last_undegraded":"2024-09-05T16:47:02.169721+0000","last_fullsized":"2024-09-05T16:47:02.169721+0000","mapping_epoch":38,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1865,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12752","reported_seq":17380,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-05T16:47:03.180780+0000","last_change":"2024-09-05T16:46:51.992073+0000","last_active":"2024-09-05T16:47:03.180780+0000","last_peered":"2024-09-05T16:47:03.180780+0000","last_clean":"2024-09-05T16:47:03.180780+0000","last_became_active":"2024-09-05T16:46:51.667128+0000","last_became_peered":"2024-09-05T16:46:51.667128+0000","last_unstale":"2024-09-05T16:47:03.180780+0000","last_undegraded":"2024-09-05T16:47:03.180780+0000","last_fullsized":"2024-09-05T16:47:03.180780+0000","mapping_epoch":44,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12752","last_scrub_stamp":"2024-09-05T16:46:51.991995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:51.991995+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:04:54.120031+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12934","reported_seq":17683,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-05T16:47:02.169362+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:47:02.169362+0000","last_peered":"2024-09-05T16:47:02.169362+0000","last_clean":"2024-09-05T16:47:02.169362+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:47:02.169362+0000","last_undegraded":"2024-09-05T16:47:02.169362+0000","last_fullsized":"2024-09-05T16:47:02.169362+0000","mapping_epoch":34,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7234,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12593","reported_seq":12714,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:03.180301+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:47:03.180301+0000","last_peered":"2024-09-05T16:47:03.180301+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:47:03.180301+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12591,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2143}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12772","reported_seq":13271,"reported_epoch":47,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:03.203001+0000","last_change":"2024-09-05T16:47:02.041664+0000","last_active":"2024-09-05T16:47:03.203001+0000","last_peered":"2024-09-05T16:47:03.203001+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:47:03.203001+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12772,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":212,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2038}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12660","reported_seq":17074,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-05T16:47:03.683348+0000","last_change":"2024-09-05T16:47:03.683348+0000","last_active":"2024-09-05T16:47:03.683348+0000","last_peered":"2024-09-05T16:47:03.683348+0000","last_clean":"2024-09-05T16:47:03.683348+0000","last_became_active":"2024-09-05T16:47:03.185143+0000","last_became_peered":"2024-09-05T16:47:03.185143+0000","last_unstale":"2024-09-05T16:47:03.683348+0000","last_undegraded":"2024-09-05T16:47:03.683348+0000","last_fullsized":"2024-09-05T16:47:03.683348+0000","mapping_epoch":46,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12660","last_scrub_stamp":"2024-09-05T16:47:03.683262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:03.683262+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:42:16.533299+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12777","reported_seq":12900,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-05T16:47:02.169491+0000","last_change":"2024-09-05T16:46:40.458315+0000","last_active":"2024-09-05T16:47:02.169491+0000","last_peered":"2024-09-05T16:47:02.169491+0000","last_clean":"2024-09-05T16:47:02.169491+0000","last_became_active":"2024-09-05T16:46:40.184288+0000","last_became_peered":"2024-09-05T16:46:40.184288+0000","last_unstale":"2024-09-05T16:47:02.169491+0000","last_undegraded":"2024-09-05T16:47:02.169491+0000","last_fullsized":"2024-09-05T16:47:02.169491+0000","mapping_epoch":42,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12777","last_scrub_stamp":"2024-09-05T16:46:40.458226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:40.458226+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:38:58.734210+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'12770","reported_seq":12893,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:03.182908+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:47:03.182908+0000","last_peered":"2024-09-05T16:47:03.182908+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:47:03.182908+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12767,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2240}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12808","reported_seq":12952,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:03.180902+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:47:03.180902+0000","last_peered":"2024-09-05T16:47:03.180902+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:47:03.180902+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12806,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2243}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13120","reported_seq":13269,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:03.182464+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:47:03.182464+0000","last_peered":"2024-09-05T16:47:03.182464+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:47:03.182464+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13117,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2278}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12758","reported_seq":12878,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:03.183659+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:47:03.183659+0000","last_peered":"2024-09-05T16:47:03.183659+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:47:03.183659+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2237}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12808","reported_seq":17563,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-05T16:47:02.169412+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:47:02.169412+0000","last_peered":"2024-09-05T16:47:02.169412+0000","last_clean":"2024-09-05T16:47:02.169412+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:47:02.169412+0000","last_undegraded":"2024-09-05T16:47:02.169412+0000","last_fullsized":"2024-09-05T16:47:02.169412+0000","mapping_epoch":36,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4408,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13124","reported_seq":13254,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:03.182450+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:47:03.182450+0000","last_peered":"2024-09-05T16:47:03.182450+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:47:03.182450+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13122,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2332}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12831","reported_seq":12966,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:03.181577+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:47:03.181577+0000","last_peered":"2024-09-05T16:47:03.181577+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:47:03.181577+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12829,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2306}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'12773","reported_seq":12903,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:03.178047+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:47:03.178047+0000","last_peered":"2024-09-05T16:47:03.178047+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:47:03.178047+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12771,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2218}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":121248,"num_objects":18,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20070,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141943,"num_write_kb":519972,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2650,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":86016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084444,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722064,"kb_used_data":380,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649776,"statfs":{"total":96636764160,"available":95897370624,"internally_reserved":0,"allocated":389120,"data_stored":145383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":738979539},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986321,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":595796,"kb_used_data":448,"kb_used_omap":4,"kb_used_meta":595323,"kb_avail":93776044,"statfs":{"total":96636764160,"available":96026669056,"internally_reserved":0,"allocated":458752,"data_stored":215015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":609611474},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673128,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713240,"kb_used_data":388,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":93658600,"statfs":{"total":96636764160,"available":95906406400,"internally_reserved":0,"allocated":397312,"data_stored":148763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":729938373},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182559,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713692,"kb_used_data":388,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658148,"statfs":{"total":96636764160,"available":95905943552,"internally_reserved":0,"allocated":397312,"data_stored":148763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":730394317},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":73728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:47:10.785 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:47:11.086 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:47:11.086 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:47:11.099 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":454,"stamp":"2024-09-05T16:47:09.104309+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":121248,"num_objects":18,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19246,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141943,"num_write_kb":519972,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3474,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2736488,"kb_used_data":1620,"kb_used_omap":14,"kb_used_meta":2734769,"kb_avail":374750872,"statfs":{"total":386547056640,"available":383744892928,"internally_reserved":0,"allocated":1658880,"data_stored":664684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2800404023},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13472,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2206,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12658,"num_write_kb":-50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001420"},"pg_stats":[{"pgid":"1.f","version":"39'12904","reported_seq":17512,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-05T16:47:03.183174+0000","last_change":"2024-09-05T16:46:27.826222+0000","last_active":"2024-09-05T16:47:03.183174+0000","last_peered":"2024-09-05T16:47:03.183174+0000","last_clean":"2024-09-05T16:47:03.183174+0000","last_became_active":"2024-09-05T16:46:27.757928+0000","last_became_peered":"2024-09-05T16:46:27.757928+0000","last_unstale":"2024-09-05T16:47:03.183174+0000","last_undegraded":"2024-09-05T16:47:03.183174+0000","last_fullsized":"2024-09-05T16:47:03.183174+0000","mapping_epoch":40,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:32:35.713428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12904,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2262,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12552","reported_seq":17225,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-05T16:47:03.182122+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:47:03.182122+0000","last_peered":"2024-09-05T16:47:03.182122+0000","last_clean":"2024-09-05T16:47:03.182122+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:47:03.182122+0000","last_undegraded":"2024-09-05T16:47:03.182122+0000","last_fullsized":"2024-09-05T16:47:03.182122+0000","mapping_epoch":38,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1865,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12752","reported_seq":17380,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-05T16:47:03.180780+0000","last_change":"2024-09-05T16:46:51.992073+0000","last_active":"2024-09-05T16:47:03.180780+0000","last_peered":"2024-09-05T16:47:03.180780+0000","last_clean":"2024-09-05T16:47:03.180780+0000","last_became_active":"2024-09-05T16:46:51.667128+0000","last_became_peered":"2024-09-05T16:46:51.667128+0000","last_unstale":"2024-09-05T16:47:03.180780+0000","last_undegraded":"2024-09-05T16:47:03.180780+0000","last_fullsized":"2024-09-05T16:47:03.180780+0000","mapping_epoch":44,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12752","last_scrub_stamp":"2024-09-05T16:46:51.991995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:51.991995+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:04:54.120031+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12934","reported_seq":17685,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-05T16:47:03.181344+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:47:03.181344+0000","last_peered":"2024-09-05T16:47:03.181344+0000","last_clean":"2024-09-05T16:47:03.181344+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:47:03.181344+0000","last_undegraded":"2024-09-05T16:47:03.181344+0000","last_fullsized":"2024-09-05T16:47:03.181344+0000","mapping_epoch":34,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7234,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12593","reported_seq":12714,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:03.180301+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:47:03.180301+0000","last_peered":"2024-09-05T16:47:03.180301+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:47:03.180301+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12591,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2143}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12772","reported_seq":14919,"reported_epoch":47,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:08.203737+0000","last_change":"2024-09-05T16:47:02.041664+0000","last_active":"2024-09-05T16:47:08.203737+0000","last_peered":"2024-09-05T16:47:08.203737+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:47:08.203737+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1214,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12772,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1036,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1214}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12660","reported_seq":17074,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-05T16:47:03.683348+0000","last_change":"2024-09-05T16:47:03.683348+0000","last_active":"2024-09-05T16:47:03.683348+0000","last_peered":"2024-09-05T16:47:03.683348+0000","last_clean":"2024-09-05T16:47:03.683348+0000","last_became_active":"2024-09-05T16:47:03.185143+0000","last_became_peered":"2024-09-05T16:47:03.185143+0000","last_unstale":"2024-09-05T16:47:03.683348+0000","last_undegraded":"2024-09-05T16:47:03.683348+0000","last_fullsized":"2024-09-05T16:47:03.683348+0000","mapping_epoch":46,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12660","last_scrub_stamp":"2024-09-05T16:47:03.683262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:03.683262+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:42:16.533299+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12777","reported_seq":12902,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-05T16:47:03.181612+0000","last_change":"2024-09-05T16:46:40.458315+0000","last_active":"2024-09-05T16:47:03.181612+0000","last_peered":"2024-09-05T16:47:03.181612+0000","last_clean":"2024-09-05T16:47:03.181612+0000","last_became_active":"2024-09-05T16:46:40.184288+0000","last_became_peered":"2024-09-05T16:46:40.184288+0000","last_unstale":"2024-09-05T16:47:03.181612+0000","last_undegraded":"2024-09-05T16:47:03.181612+0000","last_fullsized":"2024-09-05T16:47:03.181612+0000","mapping_epoch":42,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12777","last_scrub_stamp":"2024-09-05T16:46:40.458226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:40.458226+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:38:58.734210+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'12770","reported_seq":12893,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:03.182908+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:47:03.182908+0000","last_peered":"2024-09-05T16:47:03.182908+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:47:03.182908+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12767,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2240}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12808","reported_seq":12952,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:03.180902+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:47:03.180902+0000","last_peered":"2024-09-05T16:47:03.180902+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:47:03.180902+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12806,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2243}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13120","reported_seq":13269,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:03.182464+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:47:03.182464+0000","last_peered":"2024-09-05T16:47:03.182464+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:47:03.182464+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13117,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2278}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12758","reported_seq":12878,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:03.183659+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:47:03.183659+0000","last_peered":"2024-09-05T16:47:03.183659+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:47:03.183659+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2237}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12808","reported_seq":17566,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-05T16:47:03.181511+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:47:03.181511+0000","last_peered":"2024-09-05T16:47:03.181511+0000","last_clean":"2024-09-05T16:47:03.181511+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:47:03.181511+0000","last_undegraded":"2024-09-05T16:47:03.181511+0000","last_fullsized":"2024-09-05T16:47:03.181511+0000","mapping_epoch":36,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4408,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13124","reported_seq":13254,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:03.182450+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:47:03.182450+0000","last_peered":"2024-09-05T16:47:03.182450+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:47:03.182450+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13122,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2332}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12831","reported_seq":12966,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:03.181577+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:47:03.181577+0000","last_peered":"2024-09-05T16:47:03.181577+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:47:03.181577+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12829,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2306}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'12773","reported_seq":12903,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:03.178047+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:47:03.178047+0000","last_peered":"2024-09-05T16:47:03.178047+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:47:03.178047+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12771,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2218}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":121248,"num_objects":18,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19246,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141943,"num_write_kb":519972,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3474,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":86016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084445,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722076,"kb_used_data":388,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649764,"statfs":{"total":96636764160,"available":95897358336,"internally_reserved":0,"allocated":397312,"data_stored":148763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":738979539},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986322,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":587480,"kb_used_data":456,"kb_used_omap":4,"kb_used_meta":587003,"kb_avail":93784360,"statfs":{"total":96636764160,"available":96035184640,"internally_reserved":0,"allocated":466944,"data_stored":218395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":601091794},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673128,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713240,"kb_used_data":388,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":93658600,"statfs":{"total":96636764160,"available":95906406400,"internally_reserved":0,"allocated":397312,"data_stored":148763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":729938373},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182559,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713692,"kb_used_data":388,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658148,"statfs":{"total":96636764160,"available":95905943552,"internally_reserved":0,"allocated":397312,"data_stored":148763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":730394317},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":73728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:47:11.100 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:47:11.415 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:47:11.416 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:47:11.428 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":455,"stamp":"2024-09-05T16:47:11.104795+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":121248,"num_objects":18,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19246,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141943,"num_write_kb":519972,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3474,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2736488,"kb_used_data":1620,"kb_used_omap":14,"kb_used_meta":2734769,"kb_avail":374750872,"statfs":{"total":386547056640,"available":383744892928,"internally_reserved":0,"allocated":1658880,"data_stored":664684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2800404023},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13472,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1176,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12658,"num_write_kb":-50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-975,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.936687"},"pg_stats":[{"pgid":"1.f","version":"39'12904","reported_seq":17512,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-05T16:47:03.183174+0000","last_change":"2024-09-05T16:46:27.826222+0000","last_active":"2024-09-05T16:47:03.183174+0000","last_peered":"2024-09-05T16:47:03.183174+0000","last_clean":"2024-09-05T16:47:03.183174+0000","last_became_active":"2024-09-05T16:46:27.757928+0000","last_became_peered":"2024-09-05T16:46:27.757928+0000","last_unstale":"2024-09-05T16:47:03.183174+0000","last_undegraded":"2024-09-05T16:47:03.183174+0000","last_fullsized":"2024-09-05T16:47:03.183174+0000","mapping_epoch":40,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:32:35.713428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12904,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2262,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12552","reported_seq":17225,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-05T16:47:03.182122+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:47:03.182122+0000","last_peered":"2024-09-05T16:47:03.182122+0000","last_clean":"2024-09-05T16:47:03.182122+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:47:03.182122+0000","last_undegraded":"2024-09-05T16:47:03.182122+0000","last_fullsized":"2024-09-05T16:47:03.182122+0000","mapping_epoch":38,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1865,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12752","reported_seq":17380,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-05T16:47:03.180780+0000","last_change":"2024-09-05T16:46:51.992073+0000","last_active":"2024-09-05T16:47:03.180780+0000","last_peered":"2024-09-05T16:47:03.180780+0000","last_clean":"2024-09-05T16:47:03.180780+0000","last_became_active":"2024-09-05T16:46:51.667128+0000","last_became_peered":"2024-09-05T16:46:51.667128+0000","last_unstale":"2024-09-05T16:47:03.180780+0000","last_undegraded":"2024-09-05T16:47:03.180780+0000","last_fullsized":"2024-09-05T16:47:03.180780+0000","mapping_epoch":44,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12752","last_scrub_stamp":"2024-09-05T16:46:51.991995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:51.991995+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:04:54.120031+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12934","reported_seq":17685,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-05T16:47:03.181344+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:47:03.181344+0000","last_peered":"2024-09-05T16:47:03.181344+0000","last_clean":"2024-09-05T16:47:03.181344+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:47:03.181344+0000","last_undegraded":"2024-09-05T16:47:03.181344+0000","last_fullsized":"2024-09-05T16:47:03.181344+0000","mapping_epoch":34,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7234,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12593","reported_seq":12714,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:03.180301+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:47:03.180301+0000","last_peered":"2024-09-05T16:47:03.180301+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:47:03.180301+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12591,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2143}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12772","reported_seq":14919,"reported_epoch":47,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:08.203737+0000","last_change":"2024-09-05T16:47:02.041664+0000","last_active":"2024-09-05T16:47:08.203737+0000","last_peered":"2024-09-05T16:47:08.203737+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:47:08.203737+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1214,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12772,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1036,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1214}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12660","reported_seq":17074,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-05T16:47:03.683348+0000","last_change":"2024-09-05T16:47:03.683348+0000","last_active":"2024-09-05T16:47:03.683348+0000","last_peered":"2024-09-05T16:47:03.683348+0000","last_clean":"2024-09-05T16:47:03.683348+0000","last_became_active":"2024-09-05T16:47:03.185143+0000","last_became_peered":"2024-09-05T16:47:03.185143+0000","last_unstale":"2024-09-05T16:47:03.683348+0000","last_undegraded":"2024-09-05T16:47:03.683348+0000","last_fullsized":"2024-09-05T16:47:03.683348+0000","mapping_epoch":46,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12660","last_scrub_stamp":"2024-09-05T16:47:03.683262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:03.683262+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:42:16.533299+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12777","reported_seq":12902,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-05T16:47:03.181612+0000","last_change":"2024-09-05T16:46:40.458315+0000","last_active":"2024-09-05T16:47:03.181612+0000","last_peered":"2024-09-05T16:47:03.181612+0000","last_clean":"2024-09-05T16:47:03.181612+0000","last_became_active":"2024-09-05T16:46:40.184288+0000","last_became_peered":"2024-09-05T16:46:40.184288+0000","last_unstale":"2024-09-05T16:47:03.181612+0000","last_undegraded":"2024-09-05T16:47:03.181612+0000","last_fullsized":"2024-09-05T16:47:03.181612+0000","mapping_epoch":42,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12777","last_scrub_stamp":"2024-09-05T16:46:40.458226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:40.458226+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:38:58.734210+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'12770","reported_seq":12893,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:03.182908+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:47:03.182908+0000","last_peered":"2024-09-05T16:47:03.182908+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:47:03.182908+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12767,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2240}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12808","reported_seq":12952,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:03.180902+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:47:03.180902+0000","last_peered":"2024-09-05T16:47:03.180902+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:47:03.180902+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12806,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2243}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13120","reported_seq":13269,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:03.182464+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:47:03.182464+0000","last_peered":"2024-09-05T16:47:03.182464+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:47:03.182464+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13117,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2278}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12758","reported_seq":12878,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:03.183659+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:47:03.183659+0000","last_peered":"2024-09-05T16:47:03.183659+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:47:03.183659+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2237}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12808","reported_seq":17566,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-05T16:47:03.181511+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:47:03.181511+0000","last_peered":"2024-09-05T16:47:03.181511+0000","last_clean":"2024-09-05T16:47:03.181511+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:47:03.181511+0000","last_undegraded":"2024-09-05T16:47:03.181511+0000","last_fullsized":"2024-09-05T16:47:03.181511+0000","mapping_epoch":36,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4408,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13124","reported_seq":13254,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:03.182450+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:47:03.182450+0000","last_peered":"2024-09-05T16:47:03.182450+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:47:03.182450+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13122,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2332}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12831","reported_seq":12966,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:03.181577+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:47:03.181577+0000","last_peered":"2024-09-05T16:47:03.181577+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:47:03.181577+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12829,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2306}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'12773","reported_seq":12903,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:03.178047+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:47:03.178047+0000","last_peered":"2024-09-05T16:47:03.178047+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:47:03.178047+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12771,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2218}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":121248,"num_objects":18,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19246,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141943,"num_write_kb":519972,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3474,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":86016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084445,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722076,"kb_used_data":388,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649764,"statfs":{"total":96636764160,"available":95897358336,"internally_reserved":0,"allocated":397312,"data_stored":148763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":738979539},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986322,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":587480,"kb_used_data":456,"kb_used_omap":4,"kb_used_meta":587003,"kb_avail":93784360,"statfs":{"total":96636764160,"available":96035184640,"internally_reserved":0,"allocated":466944,"data_stored":218395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":601091794},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673129,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713240,"kb_used_data":388,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":93658600,"statfs":{"total":96636764160,"available":95906406400,"internally_reserved":0,"allocated":397312,"data_stored":148763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":729938373},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182559,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713692,"kb_used_data":388,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658148,"statfs":{"total":96636764160,"available":95905943552,"internally_reserved":0,"allocated":397312,"data_stored":148763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":730394317},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":73728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:47:11.429 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:47:11.429 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "39'12593", 'reported_seq': 12714, 'reported_epoch': 47, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:47:03.180301+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:47:03.180301+0000', 'last_peered': '2024-09-05T16:47:03.180301+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:47:03.180301+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "33'2500", 'ondisk_log_start': "33'2500", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10093, 'log_dups_size': 0, 'ondisk_log_size': 10093, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2147, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12591, 'num_write_kb': 50373, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2143}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:47:11.429 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:47:11.429 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "39'12772", 'reported_seq': 14919, 'reported_epoch': 47, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:47:08.203737+0000', 'last_change': '2024-09-05T16:47:02.041664+0000', 'last_active': '2024-09-05T16:47:08.203737+0000', 'last_peered': '2024-09-05T16:47:08.203737+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:47:08.203737+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "33'2700", 'ondisk_log_start': "33'2700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10072, 'log_dups_size': 0, 'ondisk_log_size': 10072, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1214, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12772, 'num_write_kb': 51088, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1036, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1214}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:47:11.429 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:47:11.429 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "39'12770", 'reported_seq': 12893, 'reported_epoch': 47, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:47:03.182908+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:47:03.182908+0000', 'last_peered': '2024-09-05T16:47:03.182908+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:47:03.182908+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "33'2700", 'ondisk_log_start': "33'2700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10070, 'log_dups_size': 0, 'ondisk_log_size': 10070, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26944, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 16, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12767, 'num_write_kb': 51081, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2240}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:47:11.429 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:47:11.430 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "39'12808", 'reported_seq': 12952, 'reported_epoch': 47, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:47:03.180902+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:47:03.180902+0000', 'last_peered': '2024-09-05T16:47:03.180902+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:47:03.180902+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "33'2800", 'ondisk_log_start': "33'2800", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10008, 'log_dups_size': 0, 'ondisk_log_size': 10008, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12806, 'num_write_kb': 51230, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -2243}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:47:11.430 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:47:11.430 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "39'13120", 'reported_seq': 13269, 'reported_epoch': 47, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:47:03.182464+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:47:03.182464+0000', 'last_peered': '2024-09-05T16:47:03.182464+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:47:03.182464+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "33'3100", 'ondisk_log_start': "33'3100", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20208, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2284, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13117, 'num_write_kb': 52476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2278}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:47:11.430 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:47:11.430 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "39'12758", 'reported_seq': 12878, 'reported_epoch': 47, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:47:03.183659+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:47:03.183659+0000', 'last_peered': '2024-09-05T16:47:03.183659+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:47:03.183659+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "33'2700", 'ondisk_log_start': "33'2700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10058, 'log_dups_size': 0, 'ondisk_log_size': 10058, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6736, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12757, 'num_write_kb': 51031, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2237}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:47:11.430 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:47:11.430 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "39'13124", 'reported_seq': 13254, 'reported_epoch': 47, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:47:03.182450+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:47:03.182450+0000', 'last_peered': '2024-09-05T16:47:03.182450+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:47:03.182450+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "33'3100", 'ondisk_log_start': "33'3100", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10024, 'log_dups_size': 0, 'ondisk_log_size': 10024, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13122, 'num_write_kb': 52500, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2332}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:47:11.430 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:47:11.430 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "39'12831", 'reported_seq': 12966, 'reported_epoch': 47, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:47:03.181577+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:47:03.181577+0000', 'last_peered': '2024-09-05T16:47:03.181577+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:47:03.181577+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "33'2800", 'ondisk_log_start': "33'2800", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10031, 'log_dups_size': 0, 'ondisk_log_size': 10031, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2310, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12829, 'num_write_kb': 51325, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -2306}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:47:11.431 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:47:11.431 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "39'12773", 'reported_seq': 12903, 'reported_epoch': 47, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:47:03.178047+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:47:03.178047+0000', 'last_peered': '2024-09-05T16:47:03.178047+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:47:03.178047+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "33'2700", 'ondisk_log_start': "33'2700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10073, 'log_dups_size': 0, 'ondisk_log_size': 10073, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12771, 'num_write_kb': 51092, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2218}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:47:11.431 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:47:11.835 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:47:11.835 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19246/72 objects degraded (26730.556%), 9 pgs degraded, 9 pgs undersized","count":18},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":835,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":47,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":9},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":8},{"state_name":"active+clean","count":7},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":18,"data_bytes":121248,"bytes_used":2802163712,"bytes_avail":383744892928,"bytes_total":386547056640,"degraded_objects":19246,"degraded_total":72,"degraded_ratio":267.30555555555554},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:47:11.848 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-09-05T16:47:11.848 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:47:12.176 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:47:12.177 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:47:12.189 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":455,"stamp":"2024-09-05T16:47:11.104795+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":121248,"num_objects":18,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19246,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141943,"num_write_kb":519972,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3474,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2736488,"kb_used_data":1620,"kb_used_omap":14,"kb_used_meta":2734769,"kb_avail":374750872,"statfs":{"total":386547056640,"available":383744892928,"internally_reserved":0,"allocated":1658880,"data_stored":664684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2800404023},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13472,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1176,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12658,"num_write_kb":-50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-975,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.936687"},"pg_stats":[{"pgid":"1.f","version":"39'12904","reported_seq":17512,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-05T16:47:03.183174+0000","last_change":"2024-09-05T16:46:27.826222+0000","last_active":"2024-09-05T16:47:03.183174+0000","last_peered":"2024-09-05T16:47:03.183174+0000","last_clean":"2024-09-05T16:47:03.183174+0000","last_became_active":"2024-09-05T16:46:27.757928+0000","last_became_peered":"2024-09-05T16:46:27.757928+0000","last_unstale":"2024-09-05T16:47:03.183174+0000","last_undegraded":"2024-09-05T16:47:03.183174+0000","last_fullsized":"2024-09-05T16:47:03.183174+0000","mapping_epoch":40,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:32:35.713428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12904,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2262,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12552","reported_seq":17225,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-05T16:47:03.182122+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:47:03.182122+0000","last_peered":"2024-09-05T16:47:03.182122+0000","last_clean":"2024-09-05T16:47:03.182122+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:47:03.182122+0000","last_undegraded":"2024-09-05T16:47:03.182122+0000","last_fullsized":"2024-09-05T16:47:03.182122+0000","mapping_epoch":38,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1865,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12752","reported_seq":17380,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-05T16:47:03.180780+0000","last_change":"2024-09-05T16:46:51.992073+0000","last_active":"2024-09-05T16:47:03.180780+0000","last_peered":"2024-09-05T16:47:03.180780+0000","last_clean":"2024-09-05T16:47:03.180780+0000","last_became_active":"2024-09-05T16:46:51.667128+0000","last_became_peered":"2024-09-05T16:46:51.667128+0000","last_unstale":"2024-09-05T16:47:03.180780+0000","last_undegraded":"2024-09-05T16:47:03.180780+0000","last_fullsized":"2024-09-05T16:47:03.180780+0000","mapping_epoch":44,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12752","last_scrub_stamp":"2024-09-05T16:46:51.991995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:51.991995+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:04:54.120031+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12934","reported_seq":17685,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-05T16:47:03.181344+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:47:03.181344+0000","last_peered":"2024-09-05T16:47:03.181344+0000","last_clean":"2024-09-05T16:47:03.181344+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:47:03.181344+0000","last_undegraded":"2024-09-05T16:47:03.181344+0000","last_fullsized":"2024-09-05T16:47:03.181344+0000","mapping_epoch":34,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7234,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12593","reported_seq":12714,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:03.180301+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:47:03.180301+0000","last_peered":"2024-09-05T16:47:03.180301+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:47:03.180301+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12591,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2143}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12772","reported_seq":14919,"reported_epoch":47,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:08.203737+0000","last_change":"2024-09-05T16:47:02.041664+0000","last_active":"2024-09-05T16:47:08.203737+0000","last_peered":"2024-09-05T16:47:08.203737+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:47:08.203737+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1214,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12772,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1036,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1214}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12660","reported_seq":17074,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-05T16:47:03.683348+0000","last_change":"2024-09-05T16:47:03.683348+0000","last_active":"2024-09-05T16:47:03.683348+0000","last_peered":"2024-09-05T16:47:03.683348+0000","last_clean":"2024-09-05T16:47:03.683348+0000","last_became_active":"2024-09-05T16:47:03.185143+0000","last_became_peered":"2024-09-05T16:47:03.185143+0000","last_unstale":"2024-09-05T16:47:03.683348+0000","last_undegraded":"2024-09-05T16:47:03.683348+0000","last_fullsized":"2024-09-05T16:47:03.683348+0000","mapping_epoch":46,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12660","last_scrub_stamp":"2024-09-05T16:47:03.683262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:03.683262+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:42:16.533299+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12777","reported_seq":12902,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-05T16:47:03.181612+0000","last_change":"2024-09-05T16:46:40.458315+0000","last_active":"2024-09-05T16:47:03.181612+0000","last_peered":"2024-09-05T16:47:03.181612+0000","last_clean":"2024-09-05T16:47:03.181612+0000","last_became_active":"2024-09-05T16:46:40.184288+0000","last_became_peered":"2024-09-05T16:46:40.184288+0000","last_unstale":"2024-09-05T16:47:03.181612+0000","last_undegraded":"2024-09-05T16:47:03.181612+0000","last_fullsized":"2024-09-05T16:47:03.181612+0000","mapping_epoch":42,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12777","last_scrub_stamp":"2024-09-05T16:46:40.458226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:40.458226+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:38:58.734210+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'12770","reported_seq":12893,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:03.182908+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:47:03.182908+0000","last_peered":"2024-09-05T16:47:03.182908+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:47:03.182908+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12767,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2240}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12808","reported_seq":12952,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:03.180902+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:47:03.180902+0000","last_peered":"2024-09-05T16:47:03.180902+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:47:03.180902+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12806,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2243}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13120","reported_seq":13269,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:03.182464+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:47:03.182464+0000","last_peered":"2024-09-05T16:47:03.182464+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:47:03.182464+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13117,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2278}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12758","reported_seq":12878,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:03.183659+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:47:03.183659+0000","last_peered":"2024-09-05T16:47:03.183659+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:47:03.183659+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2237}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12808","reported_seq":17566,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-05T16:47:03.181511+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:47:03.181511+0000","last_peered":"2024-09-05T16:47:03.181511+0000","last_clean":"2024-09-05T16:47:03.181511+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:47:03.181511+0000","last_undegraded":"2024-09-05T16:47:03.181511+0000","last_fullsized":"2024-09-05T16:47:03.181511+0000","mapping_epoch":36,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4408,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13124","reported_seq":13254,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:03.182450+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:47:03.182450+0000","last_peered":"2024-09-05T16:47:03.182450+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:47:03.182450+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13122,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2332}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12831","reported_seq":12966,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:03.181577+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:47:03.181577+0000","last_peered":"2024-09-05T16:47:03.181577+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:47:03.181577+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12829,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2306}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'12773","reported_seq":12903,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:03.178047+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:47:03.178047+0000","last_peered":"2024-09-05T16:47:03.178047+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:47:03.178047+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12771,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2218}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":121248,"num_objects":18,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19246,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141943,"num_write_kb":519972,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3474,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":86016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084445,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722076,"kb_used_data":388,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649764,"statfs":{"total":96636764160,"available":95897358336,"internally_reserved":0,"allocated":397312,"data_stored":148763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":738979539},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986322,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":587480,"kb_used_data":456,"kb_used_omap":4,"kb_used_meta":587003,"kb_avail":93784360,"statfs":{"total":96636764160,"available":96035184640,"internally_reserved":0,"allocated":466944,"data_stored":218395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":601091794},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673129,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713240,"kb_used_data":388,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":93658600,"statfs":{"total":96636764160,"available":95906406400,"internally_reserved":0,"allocated":397312,"data_stored":148763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":729938373},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182559,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713692,"kb_used_data":388,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658148,"statfs":{"total":96636764160,"available":95905943552,"internally_reserved":0,"allocated":397312,"data_stored":148763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":730394317},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":73728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:47:15.191 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:47:15.496 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:47:15.496 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:47:15.508 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":458,"stamp":"2024-09-05T16:47:15.105499+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":121248,"num_objects":18,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18188,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141943,"num_write_kb":519972,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4532,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2744808,"kb_used_data":1620,"kb_used_omap":14,"kb_used_meta":2743089,"kb_avail":374742552,"statfs":{"total":386547056640,"available":383736373248,"internally_reserved":0,"allocated":1658880,"data_stored":664684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2808923703},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13472,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2053,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12658,"num_write_kb":-50636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001918"},"pg_stats":[{"pgid":"1.f","version":"39'12904","reported_seq":17512,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-05T16:47:03.183174+0000","last_change":"2024-09-05T16:46:27.826222+0000","last_active":"2024-09-05T16:47:03.183174+0000","last_peered":"2024-09-05T16:47:03.183174+0000","last_clean":"2024-09-05T16:47:03.183174+0000","last_became_active":"2024-09-05T16:46:27.757928+0000","last_became_peered":"2024-09-05T16:46:27.757928+0000","last_unstale":"2024-09-05T16:47:03.183174+0000","last_undegraded":"2024-09-05T16:47:03.183174+0000","last_fullsized":"2024-09-05T16:47:03.183174+0000","mapping_epoch":40,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:32:35.713428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12904,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2262,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12552","reported_seq":17225,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-05T16:47:03.182122+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:47:03.182122+0000","last_peered":"2024-09-05T16:47:03.182122+0000","last_clean":"2024-09-05T16:47:03.182122+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:47:03.182122+0000","last_undegraded":"2024-09-05T16:47:03.182122+0000","last_fullsized":"2024-09-05T16:47:03.182122+0000","mapping_epoch":38,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1865,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12752","reported_seq":17380,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-05T16:47:03.180780+0000","last_change":"2024-09-05T16:46:51.992073+0000","last_active":"2024-09-05T16:47:03.180780+0000","last_peered":"2024-09-05T16:47:03.180780+0000","last_clean":"2024-09-05T16:47:03.180780+0000","last_became_active":"2024-09-05T16:46:51.667128+0000","last_became_peered":"2024-09-05T16:46:51.667128+0000","last_unstale":"2024-09-05T16:47:03.180780+0000","last_undegraded":"2024-09-05T16:47:03.180780+0000","last_fullsized":"2024-09-05T16:47:03.180780+0000","mapping_epoch":44,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12752","last_scrub_stamp":"2024-09-05T16:46:51.991995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:51.991995+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:04:54.120031+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12934","reported_seq":17685,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-05T16:47:03.181344+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:47:03.181344+0000","last_peered":"2024-09-05T16:47:03.181344+0000","last_clean":"2024-09-05T16:47:03.181344+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:47:03.181344+0000","last_undegraded":"2024-09-05T16:47:03.181344+0000","last_fullsized":"2024-09-05T16:47:03.181344+0000","mapping_epoch":34,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7234,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12593","reported_seq":12714,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:03.180301+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:47:03.180301+0000","last_peered":"2024-09-05T16:47:03.180301+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:47:03.180301+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12591,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2143}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12772","reported_seq":17035,"reported_epoch":47,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:13.204604+0000","last_change":"2024-09-05T16:47:02.041664+0000","last_active":"2024-09-05T16:47:13.204604+0000","last_peered":"2024-09-05T16:47:13.204604+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:47:13.204604+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":156,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12772,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2094,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-156}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12660","reported_seq":17074,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-05T16:47:03.683348+0000","last_change":"2024-09-05T16:47:03.683348+0000","last_active":"2024-09-05T16:47:03.683348+0000","last_peered":"2024-09-05T16:47:03.683348+0000","last_clean":"2024-09-05T16:47:03.683348+0000","last_became_active":"2024-09-05T16:47:03.185143+0000","last_became_peered":"2024-09-05T16:47:03.185143+0000","last_unstale":"2024-09-05T16:47:03.683348+0000","last_undegraded":"2024-09-05T16:47:03.683348+0000","last_fullsized":"2024-09-05T16:47:03.683348+0000","mapping_epoch":46,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12660","last_scrub_stamp":"2024-09-05T16:47:03.683262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:03.683262+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:42:16.533299+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12777","reported_seq":12902,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-05T16:47:03.181612+0000","last_change":"2024-09-05T16:46:40.458315+0000","last_active":"2024-09-05T16:47:03.181612+0000","last_peered":"2024-09-05T16:47:03.181612+0000","last_clean":"2024-09-05T16:47:03.181612+0000","last_became_active":"2024-09-05T16:46:40.184288+0000","last_became_peered":"2024-09-05T16:46:40.184288+0000","last_unstale":"2024-09-05T16:47:03.181612+0000","last_undegraded":"2024-09-05T16:47:03.181612+0000","last_fullsized":"2024-09-05T16:47:03.181612+0000","mapping_epoch":42,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12777","last_scrub_stamp":"2024-09-05T16:46:40.458226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:40.458226+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:38:58.734210+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'12770","reported_seq":12893,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:03.182908+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:47:03.182908+0000","last_peered":"2024-09-05T16:47:03.182908+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:47:03.182908+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12767,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2240}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12808","reported_seq":12952,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:03.180902+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:47:03.180902+0000","last_peered":"2024-09-05T16:47:03.180902+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:47:03.180902+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12806,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2243}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13120","reported_seq":13269,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:03.182464+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:47:03.182464+0000","last_peered":"2024-09-05T16:47:03.182464+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:47:03.182464+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13117,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2278}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12758","reported_seq":12878,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:03.183659+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:47:03.183659+0000","last_peered":"2024-09-05T16:47:03.183659+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:47:03.183659+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2237}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12808","reported_seq":17566,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-05T16:47:03.181511+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:47:03.181511+0000","last_peered":"2024-09-05T16:47:03.181511+0000","last_clean":"2024-09-05T16:47:03.181511+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:47:03.181511+0000","last_undegraded":"2024-09-05T16:47:03.181511+0000","last_fullsized":"2024-09-05T16:47:03.181511+0000","mapping_epoch":36,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4408,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13124","reported_seq":13254,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:03.182450+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:47:03.182450+0000","last_peered":"2024-09-05T16:47:03.182450+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:47:03.182450+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13122,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2332}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12831","reported_seq":12966,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:03.181577+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:47:03.181577+0000","last_peered":"2024-09-05T16:47:03.181577+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:47:03.181577+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12829,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2306}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'12773","reported_seq":12903,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:03.178047+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:47:03.178047+0000","last_peered":"2024-09-05T16:47:03.178047+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:47:03.178047+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12771,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2218}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":121248,"num_objects":18,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18188,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141943,"num_write_kb":519972,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4532,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":86016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084446,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722076,"kb_used_data":388,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649764,"statfs":{"total":96636764160,"available":95897358336,"internally_reserved":0,"allocated":397312,"data_stored":148763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":738979539},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986323,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":595800,"kb_used_data":456,"kb_used_omap":4,"kb_used_meta":595323,"kb_avail":93776040,"statfs":{"total":96636764160,"available":96026664960,"internally_reserved":0,"allocated":466944,"data_stored":218395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":609611474},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673129,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713240,"kb_used_data":388,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":93658600,"statfs":{"total":96636764160,"available":95906406400,"internally_reserved":0,"allocated":397312,"data_stored":148763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":729938373},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182560,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713692,"kb_used_data":388,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658148,"statfs":{"total":96636764160,"available":95905943552,"internally_reserved":0,"allocated":397312,"data_stored":148763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":730394317},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":73728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:47:15.509 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:47:15.850 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:47:15.851 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:47:15.863 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":459,"stamp":"2024-09-05T16:47:15.719285+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":121248,"num_objects":18,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18188,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141943,"num_write_kb":519972,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4532,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2744808,"kb_used_data":1620,"kb_used_omap":14,"kb_used_meta":2743089,"kb_avail":374742552,"statfs":{"total":386547056640,"available":383736373248,"internally_reserved":0,"allocated":1658880,"data_stored":664684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2808923703},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1882,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1882,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.615347"},"pg_stats":[{"pgid":"1.f","version":"39'12904","reported_seq":17512,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-05T16:47:03.183174+0000","last_change":"2024-09-05T16:46:27.826222+0000","last_active":"2024-09-05T16:47:03.183174+0000","last_peered":"2024-09-05T16:47:03.183174+0000","last_clean":"2024-09-05T16:47:03.183174+0000","last_became_active":"2024-09-05T16:46:27.757928+0000","last_became_peered":"2024-09-05T16:46:27.757928+0000","last_unstale":"2024-09-05T16:47:03.183174+0000","last_undegraded":"2024-09-05T16:47:03.183174+0000","last_fullsized":"2024-09-05T16:47:03.183174+0000","mapping_epoch":40,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:32:35.713428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12904,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2262,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12552","reported_seq":17225,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-05T16:47:03.182122+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:47:03.182122+0000","last_peered":"2024-09-05T16:47:03.182122+0000","last_clean":"2024-09-05T16:47:03.182122+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:47:03.182122+0000","last_undegraded":"2024-09-05T16:47:03.182122+0000","last_fullsized":"2024-09-05T16:47:03.182122+0000","mapping_epoch":38,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1865,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12752","reported_seq":17380,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-05T16:47:03.180780+0000","last_change":"2024-09-05T16:46:51.992073+0000","last_active":"2024-09-05T16:47:03.180780+0000","last_peered":"2024-09-05T16:47:03.180780+0000","last_clean":"2024-09-05T16:47:03.180780+0000","last_became_active":"2024-09-05T16:46:51.667128+0000","last_became_peered":"2024-09-05T16:46:51.667128+0000","last_unstale":"2024-09-05T16:47:03.180780+0000","last_undegraded":"2024-09-05T16:47:03.180780+0000","last_fullsized":"2024-09-05T16:47:03.180780+0000","mapping_epoch":44,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12752","last_scrub_stamp":"2024-09-05T16:46:51.991995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:51.991995+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:04:54.120031+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12934","reported_seq":17685,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-05T16:47:03.181344+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:47:03.181344+0000","last_peered":"2024-09-05T16:47:03.181344+0000","last_clean":"2024-09-05T16:47:03.181344+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:47:03.181344+0000","last_undegraded":"2024-09-05T16:47:03.181344+0000","last_fullsized":"2024-09-05T16:47:03.181344+0000","mapping_epoch":34,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7234,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12593","reported_seq":12714,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:03.180301+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:47:03.180301+0000","last_peered":"2024-09-05T16:47:03.180301+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:47:03.180301+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12591,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2143}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12772","reported_seq":17035,"reported_epoch":47,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:13.204604+0000","last_change":"2024-09-05T16:47:02.041664+0000","last_active":"2024-09-05T16:47:13.204604+0000","last_peered":"2024-09-05T16:47:13.204604+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:47:13.204604+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":156,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12772,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2094,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-156}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12660","reported_seq":17074,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-05T16:47:03.683348+0000","last_change":"2024-09-05T16:47:03.683348+0000","last_active":"2024-09-05T16:47:03.683348+0000","last_peered":"2024-09-05T16:47:03.683348+0000","last_clean":"2024-09-05T16:47:03.683348+0000","last_became_active":"2024-09-05T16:47:03.185143+0000","last_became_peered":"2024-09-05T16:47:03.185143+0000","last_unstale":"2024-09-05T16:47:03.683348+0000","last_undegraded":"2024-09-05T16:47:03.683348+0000","last_fullsized":"2024-09-05T16:47:03.683348+0000","mapping_epoch":46,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12660","last_scrub_stamp":"2024-09-05T16:47:03.683262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:03.683262+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:42:16.533299+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12777","reported_seq":12902,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-05T16:47:03.181612+0000","last_change":"2024-09-05T16:46:40.458315+0000","last_active":"2024-09-05T16:47:03.181612+0000","last_peered":"2024-09-05T16:47:03.181612+0000","last_clean":"2024-09-05T16:47:03.181612+0000","last_became_active":"2024-09-05T16:46:40.184288+0000","last_became_peered":"2024-09-05T16:46:40.184288+0000","last_unstale":"2024-09-05T16:47:03.181612+0000","last_undegraded":"2024-09-05T16:47:03.181612+0000","last_fullsized":"2024-09-05T16:47:03.181612+0000","mapping_epoch":42,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12777","last_scrub_stamp":"2024-09-05T16:46:40.458226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:40.458226+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:38:58.734210+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'12770","reported_seq":12893,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:03.182908+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:47:03.182908+0000","last_peered":"2024-09-05T16:47:03.182908+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:47:03.182908+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12767,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2240}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12808","reported_seq":12952,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:03.180902+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:47:03.180902+0000","last_peered":"2024-09-05T16:47:03.180902+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:47:03.180902+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12806,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2243}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13120","reported_seq":13269,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:03.182464+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:47:03.182464+0000","last_peered":"2024-09-05T16:47:03.182464+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:47:03.182464+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13117,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2278}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12758","reported_seq":12878,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:03.183659+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:47:03.183659+0000","last_peered":"2024-09-05T16:47:03.183659+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:47:03.183659+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2237}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12808","reported_seq":17566,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-05T16:47:03.181511+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:47:03.181511+0000","last_peered":"2024-09-05T16:47:03.181511+0000","last_clean":"2024-09-05T16:47:03.181511+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:47:03.181511+0000","last_undegraded":"2024-09-05T16:47:03.181511+0000","last_fullsized":"2024-09-05T16:47:03.181511+0000","mapping_epoch":36,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4408,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13124","reported_seq":13254,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:03.182450+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:47:03.182450+0000","last_peered":"2024-09-05T16:47:03.182450+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:47:03.182450+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13122,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2332}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12831","reported_seq":12966,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:03.181577+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:47:03.181577+0000","last_peered":"2024-09-05T16:47:03.181577+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:47:03.181577+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12829,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2306}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'12773","reported_seq":12903,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:03.178047+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:47:03.178047+0000","last_peered":"2024-09-05T16:47:03.178047+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:47:03.178047+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12771,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2218}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":121248,"num_objects":18,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18188,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141943,"num_write_kb":519972,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4532,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":86016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084446,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722076,"kb_used_data":388,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649764,"statfs":{"total":96636764160,"available":95897358336,"internally_reserved":0,"allocated":397312,"data_stored":148763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":738979539},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986323,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":595800,"kb_used_data":456,"kb_used_omap":4,"kb_used_meta":595323,"kb_avail":93776040,"statfs":{"total":96636764160,"available":96026664960,"internally_reserved":0,"allocated":466944,"data_stored":218395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":609611474},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673129,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713240,"kb_used_data":388,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":93658600,"statfs":{"total":96636764160,"available":95906406400,"internally_reserved":0,"allocated":397312,"data_stored":148763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":729938373},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182560,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713692,"kb_used_data":388,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658148,"statfs":{"total":96636764160,"available":95905943552,"internally_reserved":0,"allocated":397312,"data_stored":148763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":730394317},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":73728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:47:15.864 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:47:15.864 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "39'12593", 'reported_seq': 12714, 'reported_epoch': 47, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:47:03.180301+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:47:03.180301+0000', 'last_peered': '2024-09-05T16:47:03.180301+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:47:03.180301+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "33'2500", 'ondisk_log_start': "33'2500", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10093, 'log_dups_size': 0, 'ondisk_log_size': 10093, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2147, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12591, 'num_write_kb': 50373, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2143}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:47:15.864 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-05T16:47:15.865 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "39'12772", 'reported_seq': 17035, 'reported_epoch': 47, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:47:13.204604+0000', 'last_change': '2024-09-05T16:47:02.041664+0000', 'last_active': '2024-09-05T16:47:13.204604+0000', 'last_peered': '2024-09-05T16:47:13.204604+0000', 'last_clean': '2024-09-05T16:33:41.856813+0000', 'last_became_active': '2024-09-05T16:34:56.853420+0000', 'last_became_peered': '2024-09-05T16:34:56.853420+0000', 'last_unstale': '2024-09-05T16:47:13.204604+0000', 'last_undegraded': '2024-09-05T16:34:56.617578+0000', 'last_fullsized': '2024-09-05T16:34:56.561437+0000', 'mapping_epoch': 32, 'log_start': "33'2700", 'ondisk_log_start': "33'2700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10072, 'log_dups_size': 0, 'ondisk_log_size': 10072, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 156, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12772, 'num_write_kb': 51088, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2094, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -156}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:47:15.865 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:47:15.865 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "39'12770", 'reported_seq': 12893, 'reported_epoch': 47, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:47:03.182908+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:47:03.182908+0000', 'last_peered': '2024-09-05T16:47:03.182908+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:47:03.182908+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "33'2700", 'ondisk_log_start': "33'2700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10070, 'log_dups_size': 0, 'ondisk_log_size': 10070, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26944, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 16, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12767, 'num_write_kb': 51081, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2240}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:47:15.865 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:47:15.865 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "39'12808", 'reported_seq': 12952, 'reported_epoch': 47, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:47:03.180902+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:47:03.180902+0000', 'last_peered': '2024-09-05T16:47:03.180902+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:47:03.180902+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "33'2800", 'ondisk_log_start': "33'2800", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10008, 'log_dups_size': 0, 'ondisk_log_size': 10008, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12806, 'num_write_kb': 51230, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -2243}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:47:15.865 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:47:15.865 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "39'13120", 'reported_seq': 13269, 'reported_epoch': 47, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:47:03.182464+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:47:03.182464+0000', 'last_peered': '2024-09-05T16:47:03.182464+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:47:03.182464+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "33'3100", 'ondisk_log_start': "33'3100", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20208, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2284, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13117, 'num_write_kb': 52476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2278}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:47:15.865 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:47:15.865 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "39'12758", 'reported_seq': 12878, 'reported_epoch': 47, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:47:03.183659+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:47:03.183659+0000', 'last_peered': '2024-09-05T16:47:03.183659+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:47:03.183659+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "33'2700", 'ondisk_log_start': "33'2700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10058, 'log_dups_size': 0, 'ondisk_log_size': 10058, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6736, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12757, 'num_write_kb': 51031, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2237}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:47:15.865 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:47:15.865 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "39'13124", 'reported_seq': 13254, 'reported_epoch': 47, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:47:03.182450+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:47:03.182450+0000', 'last_peered': '2024-09-05T16:47:03.182450+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:47:03.182450+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "33'3100", 'ondisk_log_start': "33'3100", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10024, 'log_dups_size': 0, 'ondisk_log_size': 10024, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13122, 'num_write_kb': 52500, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2332}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:47:15.866 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:47:15.866 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "39'12831", 'reported_seq': 12966, 'reported_epoch': 47, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:47:03.181577+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:47:03.181577+0000', 'last_peered': '2024-09-05T16:47:03.181577+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:47:03.181577+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "33'2800", 'ondisk_log_start': "33'2800", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10031, 'log_dups_size': 0, 'ondisk_log_size': 10031, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2310, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12829, 'num_write_kb': 51325, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -2306}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:47:15.866 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:47:15.866 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "39'12773", 'reported_seq': 12903, 'reported_epoch': 47, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:47:03.178047+0000', 'last_change': '2024-09-05T16:34:56.917306+0000', 'last_active': '2024-09-05T16:47:03.178047+0000', 'last_peered': '2024-09-05T16:47:03.178047+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:47:03.178047+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "33'2700", 'ondisk_log_start': "33'2700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10073, 'log_dups_size': 0, 'ondisk_log_size': 10073, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12771, 'num_write_kb': 51092, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2218}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:47:15.868 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:47:16.279 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:47:16.279 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18188/72 objects degraded (25261.111%), 9 pgs degraded, 9 pgs undersized","count":18},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":839,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":49,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":8},{"state_name":"active+clean","count":7},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":18,"data_bytes":121248,"bytes_used":2810683392,"bytes_avail":383736373248,"bytes_total":386547056640,"degraded_objects":18188,"degraded_total":72,"degraded_ratio":252.61111111111114},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:47:16.292 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-09-05T16:47:16.292 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:47:16.595 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:47:16.595 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:47:16.608 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":459,"stamp":"2024-09-05T16:47:15.719285+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":121248,"num_objects":18,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18188,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141943,"num_write_kb":519972,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4532,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2744808,"kb_used_data":1620,"kb_used_omap":14,"kb_used_meta":2743089,"kb_avail":374742552,"statfs":{"total":386547056640,"available":383736373248,"internally_reserved":0,"allocated":1658880,"data_stored":664684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2808923703},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1882,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1882,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.615347"},"pg_stats":[{"pgid":"1.f","version":"39'12904","reported_seq":17512,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-05T16:47:03.183174+0000","last_change":"2024-09-05T16:46:27.826222+0000","last_active":"2024-09-05T16:47:03.183174+0000","last_peered":"2024-09-05T16:47:03.183174+0000","last_clean":"2024-09-05T16:47:03.183174+0000","last_became_active":"2024-09-05T16:46:27.757928+0000","last_became_peered":"2024-09-05T16:46:27.757928+0000","last_unstale":"2024-09-05T16:47:03.183174+0000","last_undegraded":"2024-09-05T16:47:03.183174+0000","last_fullsized":"2024-09-05T16:47:03.183174+0000","mapping_epoch":40,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:32:35.713428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12904,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2262,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12552","reported_seq":17225,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-05T16:47:03.182122+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:47:03.182122+0000","last_peered":"2024-09-05T16:47:03.182122+0000","last_clean":"2024-09-05T16:47:03.182122+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:47:03.182122+0000","last_undegraded":"2024-09-05T16:47:03.182122+0000","last_fullsized":"2024-09-05T16:47:03.182122+0000","mapping_epoch":38,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1865,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12752","reported_seq":17380,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-05T16:47:03.180780+0000","last_change":"2024-09-05T16:46:51.992073+0000","last_active":"2024-09-05T16:47:03.180780+0000","last_peered":"2024-09-05T16:47:03.180780+0000","last_clean":"2024-09-05T16:47:03.180780+0000","last_became_active":"2024-09-05T16:46:51.667128+0000","last_became_peered":"2024-09-05T16:46:51.667128+0000","last_unstale":"2024-09-05T16:47:03.180780+0000","last_undegraded":"2024-09-05T16:47:03.180780+0000","last_fullsized":"2024-09-05T16:47:03.180780+0000","mapping_epoch":44,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12752","last_scrub_stamp":"2024-09-05T16:46:51.991995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:51.991995+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:04:54.120031+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12934","reported_seq":17685,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-05T16:47:03.181344+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:47:03.181344+0000","last_peered":"2024-09-05T16:47:03.181344+0000","last_clean":"2024-09-05T16:47:03.181344+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:47:03.181344+0000","last_undegraded":"2024-09-05T16:47:03.181344+0000","last_fullsized":"2024-09-05T16:47:03.181344+0000","mapping_epoch":34,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7234,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12593","reported_seq":12714,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:03.180301+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:47:03.180301+0000","last_peered":"2024-09-05T16:47:03.180301+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:47:03.180301+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12591,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2143}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12772","reported_seq":17035,"reported_epoch":47,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:13.204604+0000","last_change":"2024-09-05T16:47:02.041664+0000","last_active":"2024-09-05T16:47:13.204604+0000","last_peered":"2024-09-05T16:47:13.204604+0000","last_clean":"2024-09-05T16:33:41.856813+0000","last_became_active":"2024-09-05T16:34:56.853420+0000","last_became_peered":"2024-09-05T16:34:56.853420+0000","last_unstale":"2024-09-05T16:47:13.204604+0000","last_undegraded":"2024-09-05T16:34:56.617578+0000","last_fullsized":"2024-09-05T16:34:56.561437+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":156,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12772,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2094,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-156}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12660","reported_seq":17074,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-05T16:47:03.683348+0000","last_change":"2024-09-05T16:47:03.683348+0000","last_active":"2024-09-05T16:47:03.683348+0000","last_peered":"2024-09-05T16:47:03.683348+0000","last_clean":"2024-09-05T16:47:03.683348+0000","last_became_active":"2024-09-05T16:47:03.185143+0000","last_became_peered":"2024-09-05T16:47:03.185143+0000","last_unstale":"2024-09-05T16:47:03.683348+0000","last_undegraded":"2024-09-05T16:47:03.683348+0000","last_fullsized":"2024-09-05T16:47:03.683348+0000","mapping_epoch":46,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12660","last_scrub_stamp":"2024-09-05T16:47:03.683262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:03.683262+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:42:16.533299+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12777","reported_seq":12902,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-05T16:47:03.181612+0000","last_change":"2024-09-05T16:46:40.458315+0000","last_active":"2024-09-05T16:47:03.181612+0000","last_peered":"2024-09-05T16:47:03.181612+0000","last_clean":"2024-09-05T16:47:03.181612+0000","last_became_active":"2024-09-05T16:46:40.184288+0000","last_became_peered":"2024-09-05T16:46:40.184288+0000","last_unstale":"2024-09-05T16:47:03.181612+0000","last_undegraded":"2024-09-05T16:47:03.181612+0000","last_fullsized":"2024-09-05T16:47:03.181612+0000","mapping_epoch":42,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12777","last_scrub_stamp":"2024-09-05T16:46:40.458226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:40.458226+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:38:58.734210+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'12770","reported_seq":12893,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:03.182908+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:47:03.182908+0000","last_peered":"2024-09-05T16:47:03.182908+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:47:03.182908+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12767,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2240}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12808","reported_seq":12952,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:03.180902+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:47:03.180902+0000","last_peered":"2024-09-05T16:47:03.180902+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:47:03.180902+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12806,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2243}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13120","reported_seq":13269,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:03.182464+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:47:03.182464+0000","last_peered":"2024-09-05T16:47:03.182464+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:47:03.182464+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13117,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2278}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12758","reported_seq":12878,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:03.183659+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:47:03.183659+0000","last_peered":"2024-09-05T16:47:03.183659+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:47:03.183659+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2237}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12808","reported_seq":17566,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-05T16:47:03.181511+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:47:03.181511+0000","last_peered":"2024-09-05T16:47:03.181511+0000","last_clean":"2024-09-05T16:47:03.181511+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:47:03.181511+0000","last_undegraded":"2024-09-05T16:47:03.181511+0000","last_fullsized":"2024-09-05T16:47:03.181511+0000","mapping_epoch":36,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4408,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13124","reported_seq":13254,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:03.182450+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:47:03.182450+0000","last_peered":"2024-09-05T16:47:03.182450+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:47:03.182450+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13122,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2332}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12831","reported_seq":12966,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:03.181577+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:47:03.181577+0000","last_peered":"2024-09-05T16:47:03.181577+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:47:03.181577+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12829,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2306}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'12773","reported_seq":12903,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:03.178047+0000","last_change":"2024-09-05T16:34:56.917306+0000","last_active":"2024-09-05T16:47:03.178047+0000","last_peered":"2024-09-05T16:47:03.178047+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:47:03.178047+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12771,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2218}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":121248,"num_objects":18,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18188,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141943,"num_write_kb":519972,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4532,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":86016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084446,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722076,"kb_used_data":388,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649764,"statfs":{"total":96636764160,"available":95897358336,"internally_reserved":0,"allocated":397312,"data_stored":148763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":738979539},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986323,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":595800,"kb_used_data":456,"kb_used_omap":4,"kb_used_meta":595323,"kb_avail":93776040,"statfs":{"total":96636764160,"available":96026664960,"internally_reserved":0,"allocated":466944,"data_stored":218395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":609611474},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673129,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713240,"kb_used_data":388,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":93658600,"statfs":{"total":96636764160,"available":95906406400,"internally_reserved":0,"allocated":397312,"data_stored":148763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":729938373},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182560,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713692,"kb_used_data":388,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658148,"statfs":{"total":96636764160,"available":95905943552,"internally_reserved":0,"allocated":397312,"data_stored":148763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":730394317},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":73728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:47:19.610 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:47:19.936 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:47:19.936 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:47:19.949 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":461,"stamp":"2024-09-05T16:47:19.106136+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":121248,"num_objects":18,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17666,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141943,"num_write_kb":519972,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5054,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2749012,"kb_used_data":1668,"kb_used_omap":14,"kb_used_meta":2747249,"kb_avail":374738348,"statfs":{"total":386547056640,"available":383732068352,"internally_reserved":0,"allocated":1708032,"data_stored":679977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2813183543},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1580,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1580,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001340"},"pg_stats":[{"pgid":"1.f","version":"39'12904","reported_seq":17516,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-05T16:47:15.723019+0000","last_change":"2024-09-05T16:46:27.826222+0000","last_active":"2024-09-05T16:47:15.723019+0000","last_peered":"2024-09-05T16:47:15.723019+0000","last_clean":"2024-09-05T16:47:15.723019+0000","last_became_active":"2024-09-05T16:46:27.757928+0000","last_became_peered":"2024-09-05T16:46:27.757928+0000","last_unstale":"2024-09-05T16:47:15.723019+0000","last_undegraded":"2024-09-05T16:47:15.723019+0000","last_fullsized":"2024-09-05T16:47:15.723019+0000","mapping_epoch":40,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:32:35.713428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12904,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2262,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12552","reported_seq":17229,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-05T16:47:15.727609+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:47:15.727609+0000","last_peered":"2024-09-05T16:47:15.727609+0000","last_clean":"2024-09-05T16:47:15.727609+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:47:15.727609+0000","last_undegraded":"2024-09-05T16:47:15.727609+0000","last_fullsized":"2024-09-05T16:47:15.727609+0000","mapping_epoch":38,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1865,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12752","reported_seq":17384,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-05T16:47:15.722915+0000","last_change":"2024-09-05T16:46:51.992073+0000","last_active":"2024-09-05T16:47:15.722915+0000","last_peered":"2024-09-05T16:47:15.722915+0000","last_clean":"2024-09-05T16:47:15.722915+0000","last_became_active":"2024-09-05T16:46:51.667128+0000","last_became_peered":"2024-09-05T16:46:51.667128+0000","last_unstale":"2024-09-05T16:47:15.722915+0000","last_undegraded":"2024-09-05T16:47:15.722915+0000","last_fullsized":"2024-09-05T16:47:15.722915+0000","mapping_epoch":44,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12752","last_scrub_stamp":"2024-09-05T16:46:51.991995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:51.991995+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:04:54.120031+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12934","reported_seq":17689,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-05T16:47:15.727311+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:47:15.727311+0000","last_peered":"2024-09-05T16:47:15.727311+0000","last_clean":"2024-09-05T16:47:15.727311+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:47:15.727311+0000","last_undegraded":"2024-09-05T16:47:15.727311+0000","last_fullsized":"2024-09-05T16:47:15.727311+0000","mapping_epoch":34,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7234,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12593","reported_seq":12718,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:15.723507+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:47:15.723507+0000","last_peered":"2024-09-05T16:47:15.723507+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:47:15.723507+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12591,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2143}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12772","reported_seq":17360,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-05T16:47:15.832645+0000","last_change":"2024-09-05T16:47:15.832530+0000","last_active":"2024-09-05T16:47:15.832645+0000","last_peered":"2024-09-05T16:47:15.832645+0000","last_clean":"2024-09-05T16:47:15.832645+0000","last_became_active":"2024-09-05T16:47:15.728759+0000","last_became_peered":"2024-09-05T16:47:15.728759+0000","last_unstale":"2024-09-05T16:47:15.832645+0000","last_undegraded":"2024-09-05T16:47:15.832645+0000","last_fullsized":"2024-09-05T16:47:15.832645+0000","mapping_epoch":48,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T02:39:11.515339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12772,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2250,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12660","reported_seq":17078,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-05T16:47:15.720462+0000","last_change":"2024-09-05T16:47:03.683348+0000","last_active":"2024-09-05T16:47:15.720462+0000","last_peered":"2024-09-05T16:47:15.720462+0000","last_clean":"2024-09-05T16:47:15.720462+0000","last_became_active":"2024-09-05T16:47:03.185143+0000","last_became_peered":"2024-09-05T16:47:03.185143+0000","last_unstale":"2024-09-05T16:47:15.720462+0000","last_undegraded":"2024-09-05T16:47:15.720462+0000","last_fullsized":"2024-09-05T16:47:15.720462+0000","mapping_epoch":46,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12660","last_scrub_stamp":"2024-09-05T16:47:03.683262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:03.683262+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:42:16.533299+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12777","reported_seq":12906,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-05T16:47:15.727595+0000","last_change":"2024-09-05T16:46:40.458315+0000","last_active":"2024-09-05T16:47:15.727595+0000","last_peered":"2024-09-05T16:47:15.727595+0000","last_clean":"2024-09-05T16:47:15.727595+0000","last_became_active":"2024-09-05T16:46:40.184288+0000","last_became_peered":"2024-09-05T16:46:40.184288+0000","last_unstale":"2024-09-05T16:47:15.727595+0000","last_undegraded":"2024-09-05T16:47:15.727595+0000","last_fullsized":"2024-09-05T16:47:15.727595+0000","mapping_epoch":42,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12777","last_scrub_stamp":"2024-09-05T16:46:40.458226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:40.458226+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:38:58.734210+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'12770","reported_seq":12897,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:15.723902+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:47:15.723902+0000","last_peered":"2024-09-05T16:47:15.723902+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:47:15.723902+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12767,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2240}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12808","reported_seq":12956,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:15.723029+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:47:15.723029+0000","last_peered":"2024-09-05T16:47:15.723029+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:47:15.723029+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12806,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2243}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13120","reported_seq":13273,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:15.723558+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:47:15.723558+0000","last_peered":"2024-09-05T16:47:15.723558+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:47:15.723558+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13117,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2278}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12758","reported_seq":12882,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:15.726091+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:47:15.726091+0000","last_peered":"2024-09-05T16:47:15.726091+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:47:15.726091+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2237}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12808","reported_seq":17570,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-05T16:47:15.727434+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:47:15.727434+0000","last_peered":"2024-09-05T16:47:15.727434+0000","last_clean":"2024-09-05T16:47:15.727434+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:47:15.727434+0000","last_undegraded":"2024-09-05T16:47:15.727434+0000","last_fullsized":"2024-09-05T16:47:15.727434+0000","mapping_epoch":36,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4408,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13124","reported_seq":13258,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:15.723926+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:47:15.723926+0000","last_peered":"2024-09-05T16:47:15.723926+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:47:15.723926+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13122,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2332}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12831","reported_seq":12970,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:15.723352+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:47:15.723352+0000","last_peered":"2024-09-05T16:47:15.723352+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:47:15.723352+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12829,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2306}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'12773","reported_seq":13640,"reported_epoch":49,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:16.348411+0000","last_change":"2024-09-05T16:47:13.789527+0000","last_active":"2024-09-05T16:47:16.348411+0000","last_peered":"2024-09-05T16:47:16.348411+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:47:16.348411+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1856,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12771,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":385,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1854}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":121248,"num_objects":18,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17666,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141943,"num_write_kb":519972,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5054,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":77824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084447,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722092,"kb_used_data":404,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649748,"statfs":{"total":96636764160,"available":95897341952,"internally_reserved":0,"allocated":413696,"data_stored":155470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":738979539},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986324,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":599968,"kb_used_data":464,"kb_used_omap":4,"kb_used_meta":599483,"kb_avail":93771872,"statfs":{"total":96636764160,"available":96022396928,"internally_reserved":0,"allocated":475136,"data_stored":216910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":613871314},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673130,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713256,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":93658584,"statfs":{"total":96636764160,"available":95906390016,"internally_reserved":0,"allocated":413696,"data_stored":155470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":729938373},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182561,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713696,"kb_used_data":396,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658144,"statfs":{"total":96636764160,"available":95905939456,"internally_reserved":0,"allocated":405504,"data_stored":152127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":730394317},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":65536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:47:19.951 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:47:20.296 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:47:20.297 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:47:20.309 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":461,"stamp":"2024-09-05T16:47:19.106136+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":121248,"num_objects":18,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17666,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141943,"num_write_kb":519972,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5054,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2749012,"kb_used_data":1668,"kb_used_omap":14,"kb_used_meta":2747249,"kb_avail":374738348,"statfs":{"total":386547056640,"available":383732068352,"internally_reserved":0,"allocated":1708032,"data_stored":679977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2813183543},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1580,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1580,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001340"},"pg_stats":[{"pgid":"1.f","version":"39'12904","reported_seq":17516,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-05T16:47:15.723019+0000","last_change":"2024-09-05T16:46:27.826222+0000","last_active":"2024-09-05T16:47:15.723019+0000","last_peered":"2024-09-05T16:47:15.723019+0000","last_clean":"2024-09-05T16:47:15.723019+0000","last_became_active":"2024-09-05T16:46:27.757928+0000","last_became_peered":"2024-09-05T16:46:27.757928+0000","last_unstale":"2024-09-05T16:47:15.723019+0000","last_undegraded":"2024-09-05T16:47:15.723019+0000","last_fullsized":"2024-09-05T16:47:15.723019+0000","mapping_epoch":40,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:32:35.713428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12904,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2262,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12552","reported_seq":17229,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-05T16:47:15.727609+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:47:15.727609+0000","last_peered":"2024-09-05T16:47:15.727609+0000","last_clean":"2024-09-05T16:47:15.727609+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:47:15.727609+0000","last_undegraded":"2024-09-05T16:47:15.727609+0000","last_fullsized":"2024-09-05T16:47:15.727609+0000","mapping_epoch":38,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1865,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12752","reported_seq":17384,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-05T16:47:15.722915+0000","last_change":"2024-09-05T16:46:51.992073+0000","last_active":"2024-09-05T16:47:15.722915+0000","last_peered":"2024-09-05T16:47:15.722915+0000","last_clean":"2024-09-05T16:47:15.722915+0000","last_became_active":"2024-09-05T16:46:51.667128+0000","last_became_peered":"2024-09-05T16:46:51.667128+0000","last_unstale":"2024-09-05T16:47:15.722915+0000","last_undegraded":"2024-09-05T16:47:15.722915+0000","last_fullsized":"2024-09-05T16:47:15.722915+0000","mapping_epoch":44,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12752","last_scrub_stamp":"2024-09-05T16:46:51.991995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:51.991995+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:04:54.120031+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12934","reported_seq":17689,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-05T16:47:15.727311+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:47:15.727311+0000","last_peered":"2024-09-05T16:47:15.727311+0000","last_clean":"2024-09-05T16:47:15.727311+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:47:15.727311+0000","last_undegraded":"2024-09-05T16:47:15.727311+0000","last_fullsized":"2024-09-05T16:47:15.727311+0000","mapping_epoch":34,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7234,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12593","reported_seq":12718,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:15.723507+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:47:15.723507+0000","last_peered":"2024-09-05T16:47:15.723507+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:47:15.723507+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12591,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2143}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12772","reported_seq":17360,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-05T16:47:15.832645+0000","last_change":"2024-09-05T16:47:15.832530+0000","last_active":"2024-09-05T16:47:15.832645+0000","last_peered":"2024-09-05T16:47:15.832645+0000","last_clean":"2024-09-05T16:47:15.832645+0000","last_became_active":"2024-09-05T16:47:15.728759+0000","last_became_peered":"2024-09-05T16:47:15.728759+0000","last_unstale":"2024-09-05T16:47:15.832645+0000","last_undegraded":"2024-09-05T16:47:15.832645+0000","last_fullsized":"2024-09-05T16:47:15.832645+0000","mapping_epoch":48,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T02:39:11.515339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12772,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2250,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12660","reported_seq":17078,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-05T16:47:15.720462+0000","last_change":"2024-09-05T16:47:03.683348+0000","last_active":"2024-09-05T16:47:15.720462+0000","last_peered":"2024-09-05T16:47:15.720462+0000","last_clean":"2024-09-05T16:47:15.720462+0000","last_became_active":"2024-09-05T16:47:03.185143+0000","last_became_peered":"2024-09-05T16:47:03.185143+0000","last_unstale":"2024-09-05T16:47:15.720462+0000","last_undegraded":"2024-09-05T16:47:15.720462+0000","last_fullsized":"2024-09-05T16:47:15.720462+0000","mapping_epoch":46,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12660","last_scrub_stamp":"2024-09-05T16:47:03.683262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:03.683262+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:42:16.533299+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12777","reported_seq":12906,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-05T16:47:15.727595+0000","last_change":"2024-09-05T16:46:40.458315+0000","last_active":"2024-09-05T16:47:15.727595+0000","last_peered":"2024-09-05T16:47:15.727595+0000","last_clean":"2024-09-05T16:47:15.727595+0000","last_became_active":"2024-09-05T16:46:40.184288+0000","last_became_peered":"2024-09-05T16:46:40.184288+0000","last_unstale":"2024-09-05T16:47:15.727595+0000","last_undegraded":"2024-09-05T16:47:15.727595+0000","last_fullsized":"2024-09-05T16:47:15.727595+0000","mapping_epoch":42,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12777","last_scrub_stamp":"2024-09-05T16:46:40.458226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:40.458226+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:38:58.734210+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'12770","reported_seq":12897,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:15.723902+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:47:15.723902+0000","last_peered":"2024-09-05T16:47:15.723902+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:47:15.723902+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12767,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2240}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12808","reported_seq":12956,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:15.723029+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:47:15.723029+0000","last_peered":"2024-09-05T16:47:15.723029+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:47:15.723029+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12806,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2243}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13120","reported_seq":13273,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:15.723558+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:47:15.723558+0000","last_peered":"2024-09-05T16:47:15.723558+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:47:15.723558+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13117,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2278}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12758","reported_seq":12882,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:15.726091+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:47:15.726091+0000","last_peered":"2024-09-05T16:47:15.726091+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:47:15.726091+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2237}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12808","reported_seq":17570,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-05T16:47:15.727434+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:47:15.727434+0000","last_peered":"2024-09-05T16:47:15.727434+0000","last_clean":"2024-09-05T16:47:15.727434+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:47:15.727434+0000","last_undegraded":"2024-09-05T16:47:15.727434+0000","last_fullsized":"2024-09-05T16:47:15.727434+0000","mapping_epoch":36,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4408,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13124","reported_seq":13258,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:15.723926+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:47:15.723926+0000","last_peered":"2024-09-05T16:47:15.723926+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:47:15.723926+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13122,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2332}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12831","reported_seq":12970,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:15.723352+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:47:15.723352+0000","last_peered":"2024-09-05T16:47:15.723352+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:47:15.723352+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12829,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2306}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'12773","reported_seq":13640,"reported_epoch":49,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:16.348411+0000","last_change":"2024-09-05T16:47:13.789527+0000","last_active":"2024-09-05T16:47:16.348411+0000","last_peered":"2024-09-05T16:47:16.348411+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:47:16.348411+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1856,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12771,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":385,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1854}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":121248,"num_objects":18,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17666,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141943,"num_write_kb":519972,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5054,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":77824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084447,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722092,"kb_used_data":404,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649748,"statfs":{"total":96636764160,"available":95897341952,"internally_reserved":0,"allocated":413696,"data_stored":155470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":738979539},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986324,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":599968,"kb_used_data":464,"kb_used_omap":4,"kb_used_meta":599483,"kb_avail":93771872,"statfs":{"total":96636764160,"available":96022396928,"internally_reserved":0,"allocated":475136,"data_stored":216910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":613871314},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673130,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713256,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":93658584,"statfs":{"total":96636764160,"available":95906390016,"internally_reserved":0,"allocated":413696,"data_stored":155470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":729938373},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182561,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713696,"kb_used_data":396,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658144,"statfs":{"total":96636764160,"available":95905939456,"internally_reserved":0,"allocated":405504,"data_stored":152127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":730394317},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":65536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:47:20.310 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:47:20.310 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "39'12593", 'reported_seq': 12718, 'reported_epoch': 49, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:47:15.723507+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:47:15.723507+0000', 'last_peered': '2024-09-05T16:47:15.723507+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:47:15.723507+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "33'2500", 'ondisk_log_start': "33'2500", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10093, 'log_dups_size': 0, 'ondisk_log_size': 10093, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2147, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12591, 'num_write_kb': 50373, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2143}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:47:20.310 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:47:20.310 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "39'12770", 'reported_seq': 12897, 'reported_epoch': 49, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:47:15.723902+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:47:15.723902+0000', 'last_peered': '2024-09-05T16:47:15.723902+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:47:15.723902+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "33'2700", 'ondisk_log_start': "33'2700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10070, 'log_dups_size': 0, 'ondisk_log_size': 10070, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26944, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 16, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12767, 'num_write_kb': 51081, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2240}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:47:20.310 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:47:20.310 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "39'12808", 'reported_seq': 12956, 'reported_epoch': 49, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:47:15.723029+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:47:15.723029+0000', 'last_peered': '2024-09-05T16:47:15.723029+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:47:15.723029+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "33'2800", 'ondisk_log_start': "33'2800", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10008, 'log_dups_size': 0, 'ondisk_log_size': 10008, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12806, 'num_write_kb': 51230, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -2243}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:47:20.310 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:47:20.311 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "39'13120", 'reported_seq': 13273, 'reported_epoch': 49, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:47:15.723558+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:47:15.723558+0000', 'last_peered': '2024-09-05T16:47:15.723558+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:47:15.723558+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "33'3100", 'ondisk_log_start': "33'3100", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20208, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2284, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13117, 'num_write_kb': 52476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2278}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:47:20.311 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:47:20.311 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "39'12758", 'reported_seq': 12882, 'reported_epoch': 49, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:47:15.726091+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:47:15.726091+0000', 'last_peered': '2024-09-05T16:47:15.726091+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:47:15.726091+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "33'2700", 'ondisk_log_start': "33'2700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10058, 'log_dups_size': 0, 'ondisk_log_size': 10058, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6736, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12757, 'num_write_kb': 51031, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2237}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:47:20.311 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:47:20.311 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "39'13124", 'reported_seq': 13258, 'reported_epoch': 49, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:47:15.723926+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:47:15.723926+0000', 'last_peered': '2024-09-05T16:47:15.723926+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:47:15.723926+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "33'3100", 'ondisk_log_start': "33'3100", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10024, 'log_dups_size': 0, 'ondisk_log_size': 10024, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13122, 'num_write_kb': 52500, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2332}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:47:20.311 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:47:20.311 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "39'12831", 'reported_seq': 12970, 'reported_epoch': 49, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:47:15.723352+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:47:15.723352+0000', 'last_peered': '2024-09-05T16:47:15.723352+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:47:15.723352+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "33'2800", 'ondisk_log_start': "33'2800", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10031, 'log_dups_size': 0, 'ondisk_log_size': 10031, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2310, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12829, 'num_write_kb': 51325, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -2306}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:47:20.311 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:47:20.311 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "39'12773", 'reported_seq': 13640, 'reported_epoch': 49, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:47:16.348411+0000', 'last_change': '2024-09-05T16:47:13.789527+0000', 'last_active': '2024-09-05T16:47:16.348411+0000', 'last_peered': '2024-09-05T16:47:16.348411+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:47:16.348411+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "33'2700", 'ondisk_log_start': "33'2700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10073, 'log_dups_size': 0, 'ondisk_log_size': 10073, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1856, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12771, 'num_write_kb': 51092, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 385, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1854}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:47:20.312 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:47:20.728 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:47:20.729 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 17666/72 objects degraded (24536.111%), 8 pgs degraded, 8 pgs undersized","count":16},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":843,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":49,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":8},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":18,"data_bytes":121248,"bytes_used":2814988288,"bytes_avail":383732068352,"bytes_total":386547056640,"degraded_objects":17666,"degraded_total":72,"degraded_ratio":245.36111111111114,"recovering_objects_per_sec":197,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1580,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:47:20.742 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:47:20.743 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:47:21.042 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:47:21.042 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:47:21.055 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":461,"stamp":"2024-09-05T16:47:19.106136+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":121248,"num_objects":18,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17666,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141943,"num_write_kb":519972,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5054,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2749012,"kb_used_data":1668,"kb_used_omap":14,"kb_used_meta":2747249,"kb_avail":374738348,"statfs":{"total":386547056640,"available":383732068352,"internally_reserved":0,"allocated":1708032,"data_stored":679977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2813183543},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1580,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1580,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001340"},"pg_stats":[{"pgid":"1.f","version":"39'12904","reported_seq":17516,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-05T16:47:15.723019+0000","last_change":"2024-09-05T16:46:27.826222+0000","last_active":"2024-09-05T16:47:15.723019+0000","last_peered":"2024-09-05T16:47:15.723019+0000","last_clean":"2024-09-05T16:47:15.723019+0000","last_became_active":"2024-09-05T16:46:27.757928+0000","last_became_peered":"2024-09-05T16:46:27.757928+0000","last_unstale":"2024-09-05T16:47:15.723019+0000","last_undegraded":"2024-09-05T16:47:15.723019+0000","last_fullsized":"2024-09-05T16:47:15.723019+0000","mapping_epoch":40,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:32:35.713428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12904,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2262,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12552","reported_seq":17229,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-05T16:47:15.727609+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:47:15.727609+0000","last_peered":"2024-09-05T16:47:15.727609+0000","last_clean":"2024-09-05T16:47:15.727609+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:47:15.727609+0000","last_undegraded":"2024-09-05T16:47:15.727609+0000","last_fullsized":"2024-09-05T16:47:15.727609+0000","mapping_epoch":38,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1865,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12752","reported_seq":17384,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-05T16:47:15.722915+0000","last_change":"2024-09-05T16:46:51.992073+0000","last_active":"2024-09-05T16:47:15.722915+0000","last_peered":"2024-09-05T16:47:15.722915+0000","last_clean":"2024-09-05T16:47:15.722915+0000","last_became_active":"2024-09-05T16:46:51.667128+0000","last_became_peered":"2024-09-05T16:46:51.667128+0000","last_unstale":"2024-09-05T16:47:15.722915+0000","last_undegraded":"2024-09-05T16:47:15.722915+0000","last_fullsized":"2024-09-05T16:47:15.722915+0000","mapping_epoch":44,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12752","last_scrub_stamp":"2024-09-05T16:46:51.991995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:51.991995+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:04:54.120031+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12934","reported_seq":17689,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-05T16:47:15.727311+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:47:15.727311+0000","last_peered":"2024-09-05T16:47:15.727311+0000","last_clean":"2024-09-05T16:47:15.727311+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:47:15.727311+0000","last_undegraded":"2024-09-05T16:47:15.727311+0000","last_fullsized":"2024-09-05T16:47:15.727311+0000","mapping_epoch":34,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7234,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12593","reported_seq":12718,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:15.723507+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:47:15.723507+0000","last_peered":"2024-09-05T16:47:15.723507+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:47:15.723507+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12591,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2143}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12772","reported_seq":17360,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-05T16:47:15.832645+0000","last_change":"2024-09-05T16:47:15.832530+0000","last_active":"2024-09-05T16:47:15.832645+0000","last_peered":"2024-09-05T16:47:15.832645+0000","last_clean":"2024-09-05T16:47:15.832645+0000","last_became_active":"2024-09-05T16:47:15.728759+0000","last_became_peered":"2024-09-05T16:47:15.728759+0000","last_unstale":"2024-09-05T16:47:15.832645+0000","last_undegraded":"2024-09-05T16:47:15.832645+0000","last_fullsized":"2024-09-05T16:47:15.832645+0000","mapping_epoch":48,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T02:39:11.515339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12772,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2250,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12660","reported_seq":17078,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-05T16:47:15.720462+0000","last_change":"2024-09-05T16:47:03.683348+0000","last_active":"2024-09-05T16:47:15.720462+0000","last_peered":"2024-09-05T16:47:15.720462+0000","last_clean":"2024-09-05T16:47:15.720462+0000","last_became_active":"2024-09-05T16:47:03.185143+0000","last_became_peered":"2024-09-05T16:47:03.185143+0000","last_unstale":"2024-09-05T16:47:15.720462+0000","last_undegraded":"2024-09-05T16:47:15.720462+0000","last_fullsized":"2024-09-05T16:47:15.720462+0000","mapping_epoch":46,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12660","last_scrub_stamp":"2024-09-05T16:47:03.683262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:03.683262+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:42:16.533299+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12777","reported_seq":12906,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-05T16:47:15.727595+0000","last_change":"2024-09-05T16:46:40.458315+0000","last_active":"2024-09-05T16:47:15.727595+0000","last_peered":"2024-09-05T16:47:15.727595+0000","last_clean":"2024-09-05T16:47:15.727595+0000","last_became_active":"2024-09-05T16:46:40.184288+0000","last_became_peered":"2024-09-05T16:46:40.184288+0000","last_unstale":"2024-09-05T16:47:15.727595+0000","last_undegraded":"2024-09-05T16:47:15.727595+0000","last_fullsized":"2024-09-05T16:47:15.727595+0000","mapping_epoch":42,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12777","last_scrub_stamp":"2024-09-05T16:46:40.458226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:40.458226+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:38:58.734210+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'12770","reported_seq":12897,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:15.723902+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:47:15.723902+0000","last_peered":"2024-09-05T16:47:15.723902+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:47:15.723902+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12767,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2240}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12808","reported_seq":12956,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:15.723029+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:47:15.723029+0000","last_peered":"2024-09-05T16:47:15.723029+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:47:15.723029+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12806,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2243}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13120","reported_seq":13273,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:15.723558+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:47:15.723558+0000","last_peered":"2024-09-05T16:47:15.723558+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:47:15.723558+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13117,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2278}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12758","reported_seq":12882,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:15.726091+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:47:15.726091+0000","last_peered":"2024-09-05T16:47:15.726091+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:47:15.726091+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2237}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12808","reported_seq":17570,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-05T16:47:15.727434+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:47:15.727434+0000","last_peered":"2024-09-05T16:47:15.727434+0000","last_clean":"2024-09-05T16:47:15.727434+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:47:15.727434+0000","last_undegraded":"2024-09-05T16:47:15.727434+0000","last_fullsized":"2024-09-05T16:47:15.727434+0000","mapping_epoch":36,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4408,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13124","reported_seq":13258,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:15.723926+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:47:15.723926+0000","last_peered":"2024-09-05T16:47:15.723926+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:47:15.723926+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13122,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2332}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12831","reported_seq":12970,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:15.723352+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:47:15.723352+0000","last_peered":"2024-09-05T16:47:15.723352+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:47:15.723352+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12829,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2306}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'12773","reported_seq":13640,"reported_epoch":49,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:16.348411+0000","last_change":"2024-09-05T16:47:13.789527+0000","last_active":"2024-09-05T16:47:16.348411+0000","last_peered":"2024-09-05T16:47:16.348411+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:47:16.348411+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1856,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12771,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":385,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1854}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":121248,"num_objects":18,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17666,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141943,"num_write_kb":519972,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5054,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":77824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084447,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722092,"kb_used_data":404,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649748,"statfs":{"total":96636764160,"available":95897341952,"internally_reserved":0,"allocated":413696,"data_stored":155470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":738979539},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986324,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":599968,"kb_used_data":464,"kb_used_omap":4,"kb_used_meta":599483,"kb_avail":93771872,"statfs":{"total":96636764160,"available":96022396928,"internally_reserved":0,"allocated":475136,"data_stored":216910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":613871314},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673130,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713256,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":93658584,"statfs":{"total":96636764160,"available":95906390016,"internally_reserved":0,"allocated":413696,"data_stored":155470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":729938373},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182561,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713696,"kb_used_data":396,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658144,"statfs":{"total":96636764160,"available":95905939456,"internally_reserved":0,"allocated":405504,"data_stored":152127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":730394317},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":65536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:47:24.057 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:47:24.369 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:47:24.370 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:47:24.382 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":463,"stamp":"2024-09-05T16:47:23.106918+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":121248,"num_objects":18,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16848,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141943,"num_write_kb":519972,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5872,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2740704,"kb_used_data":1676,"kb_used_omap":14,"kb_used_meta":2738929,"kb_avail":374746656,"statfs":{"total":386547056640,"available":383740575744,"internally_reserved":0,"allocated":1716224,"data_stored":683320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2804663863},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1340,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.389732"},"pg_stats":[{"pgid":"1.f","version":"39'12904","reported_seq":17516,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-05T16:47:15.723019+0000","last_change":"2024-09-05T16:46:27.826222+0000","last_active":"2024-09-05T16:47:15.723019+0000","last_peered":"2024-09-05T16:47:15.723019+0000","last_clean":"2024-09-05T16:47:15.723019+0000","last_became_active":"2024-09-05T16:46:27.757928+0000","last_became_peered":"2024-09-05T16:46:27.757928+0000","last_unstale":"2024-09-05T16:47:15.723019+0000","last_undegraded":"2024-09-05T16:47:15.723019+0000","last_fullsized":"2024-09-05T16:47:15.723019+0000","mapping_epoch":40,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:32:35.713428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12904,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2262,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12552","reported_seq":17229,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-05T16:47:15.727609+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:47:15.727609+0000","last_peered":"2024-09-05T16:47:15.727609+0000","last_clean":"2024-09-05T16:47:15.727609+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:47:15.727609+0000","last_undegraded":"2024-09-05T16:47:15.727609+0000","last_fullsized":"2024-09-05T16:47:15.727609+0000","mapping_epoch":38,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1865,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12752","reported_seq":17385,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-05T16:47:15.722967+0000","last_change":"2024-09-05T16:46:51.992073+0000","last_active":"2024-09-05T16:47:15.722967+0000","last_peered":"2024-09-05T16:47:15.722967+0000","last_clean":"2024-09-05T16:47:15.722967+0000","last_became_active":"2024-09-05T16:46:51.667128+0000","last_became_peered":"2024-09-05T16:46:51.667128+0000","last_unstale":"2024-09-05T16:47:15.722967+0000","last_undegraded":"2024-09-05T16:47:15.722967+0000","last_fullsized":"2024-09-05T16:47:15.722967+0000","mapping_epoch":44,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12752","last_scrub_stamp":"2024-09-05T16:46:51.991995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:51.991995+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:04:54.120031+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12934","reported_seq":17689,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-05T16:47:15.727311+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:47:15.727311+0000","last_peered":"2024-09-05T16:47:15.727311+0000","last_clean":"2024-09-05T16:47:15.727311+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:47:15.727311+0000","last_undegraded":"2024-09-05T16:47:15.727311+0000","last_fullsized":"2024-09-05T16:47:15.727311+0000","mapping_epoch":34,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7234,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12593","reported_seq":12718,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:15.723507+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:47:15.723507+0000","last_peered":"2024-09-05T16:47:15.723507+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:47:15.723507+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12591,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2143}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12772","reported_seq":17360,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-05T16:47:15.832645+0000","last_change":"2024-09-05T16:47:15.832530+0000","last_active":"2024-09-05T16:47:15.832645+0000","last_peered":"2024-09-05T16:47:15.832645+0000","last_clean":"2024-09-05T16:47:15.832645+0000","last_became_active":"2024-09-05T16:47:15.728759+0000","last_became_peered":"2024-09-05T16:47:15.728759+0000","last_unstale":"2024-09-05T16:47:15.832645+0000","last_undegraded":"2024-09-05T16:47:15.832645+0000","last_fullsized":"2024-09-05T16:47:15.832645+0000","mapping_epoch":48,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T02:39:11.515339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12772,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2250,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12660","reported_seq":17078,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-05T16:47:15.720462+0000","last_change":"2024-09-05T16:47:03.683348+0000","last_active":"2024-09-05T16:47:15.720462+0000","last_peered":"2024-09-05T16:47:15.720462+0000","last_clean":"2024-09-05T16:47:15.720462+0000","last_became_active":"2024-09-05T16:47:03.185143+0000","last_became_peered":"2024-09-05T16:47:03.185143+0000","last_unstale":"2024-09-05T16:47:15.720462+0000","last_undegraded":"2024-09-05T16:47:15.720462+0000","last_fullsized":"2024-09-05T16:47:15.720462+0000","mapping_epoch":46,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12660","last_scrub_stamp":"2024-09-05T16:47:03.683262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:03.683262+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:42:16.533299+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12777","reported_seq":12906,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-05T16:47:15.727595+0000","last_change":"2024-09-05T16:46:40.458315+0000","last_active":"2024-09-05T16:47:15.727595+0000","last_peered":"2024-09-05T16:47:15.727595+0000","last_clean":"2024-09-05T16:47:15.727595+0000","last_became_active":"2024-09-05T16:46:40.184288+0000","last_became_peered":"2024-09-05T16:46:40.184288+0000","last_unstale":"2024-09-05T16:47:15.727595+0000","last_undegraded":"2024-09-05T16:47:15.727595+0000","last_fullsized":"2024-09-05T16:47:15.727595+0000","mapping_epoch":42,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12777","last_scrub_stamp":"2024-09-05T16:46:40.458226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:40.458226+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:38:58.734210+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'12770","reported_seq":12897,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:15.723902+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:47:15.723902+0000","last_peered":"2024-09-05T16:47:15.723902+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:47:15.723902+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12767,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2240}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12808","reported_seq":12956,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:15.723029+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:47:15.723029+0000","last_peered":"2024-09-05T16:47:15.723029+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:47:15.723029+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12806,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2243}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13120","reported_seq":13273,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:15.723558+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:47:15.723558+0000","last_peered":"2024-09-05T16:47:15.723558+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:47:15.723558+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13117,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2278}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12758","reported_seq":12882,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:15.726091+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:47:15.726091+0000","last_peered":"2024-09-05T16:47:15.726091+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:47:15.726091+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2237}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12808","reported_seq":17570,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-05T16:47:15.727434+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:47:15.727434+0000","last_peered":"2024-09-05T16:47:15.727434+0000","last_clean":"2024-09-05T16:47:15.727434+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:47:15.727434+0000","last_undegraded":"2024-09-05T16:47:15.727434+0000","last_fullsized":"2024-09-05T16:47:15.727434+0000","mapping_epoch":36,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4408,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13124","reported_seq":13258,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:15.723926+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:47:15.723926+0000","last_peered":"2024-09-05T16:47:15.723926+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:47:15.723926+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13122,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2332}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12831","reported_seq":12970,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:15.723352+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:47:15.723352+0000","last_peered":"2024-09-05T16:47:15.723352+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:47:15.723352+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12829,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2306}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'12773","reported_seq":15276,"reported_epoch":49,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:21.351897+0000","last_change":"2024-09-05T16:47:13.789527+0000","last_active":"2024-09-05T16:47:21.351897+0000","last_peered":"2024-09-05T16:47:21.351897+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:47:21.351897+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12771,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1036}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":121248,"num_objects":18,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16848,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141943,"num_write_kb":519972,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5872,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":77824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084447,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722092,"kb_used_data":404,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649748,"statfs":{"total":96636764160,"available":95897341952,"internally_reserved":0,"allocated":413696,"data_stored":155470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":738979539},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986325,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":591648,"kb_used_data":464,"kb_used_omap":4,"kb_used_meta":591163,"kb_avail":93780192,"statfs":{"total":96636764160,"available":96030916608,"internally_reserved":0,"allocated":475136,"data_stored":216910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":605351634},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673131,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713256,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":93658584,"statfs":{"total":96636764160,"available":95906390016,"internally_reserved":0,"allocated":413696,"data_stored":155470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":729938373},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182562,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713708,"kb_used_data":404,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658132,"statfs":{"total":96636764160,"available":95905927168,"internally_reserved":0,"allocated":413696,"data_stored":155470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":730394317},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":65536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:47:24.383 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:47:24.679 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:47:24.680 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:47:24.692 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":463,"stamp":"2024-09-05T16:47:23.106918+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":121248,"num_objects":18,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16848,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141943,"num_write_kb":519972,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5872,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2740704,"kb_used_data":1676,"kb_used_omap":14,"kb_used_meta":2738929,"kb_avail":374746656,"statfs":{"total":386547056640,"available":383740575744,"internally_reserved":0,"allocated":1716224,"data_stored":683320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2804663863},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1340,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.389732"},"pg_stats":[{"pgid":"1.f","version":"39'12904","reported_seq":17516,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-05T16:47:15.723019+0000","last_change":"2024-09-05T16:46:27.826222+0000","last_active":"2024-09-05T16:47:15.723019+0000","last_peered":"2024-09-05T16:47:15.723019+0000","last_clean":"2024-09-05T16:47:15.723019+0000","last_became_active":"2024-09-05T16:46:27.757928+0000","last_became_peered":"2024-09-05T16:46:27.757928+0000","last_unstale":"2024-09-05T16:47:15.723019+0000","last_undegraded":"2024-09-05T16:47:15.723019+0000","last_fullsized":"2024-09-05T16:47:15.723019+0000","mapping_epoch":40,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:32:35.713428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12904,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2262,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12552","reported_seq":17229,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-05T16:47:15.727609+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:47:15.727609+0000","last_peered":"2024-09-05T16:47:15.727609+0000","last_clean":"2024-09-05T16:47:15.727609+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:47:15.727609+0000","last_undegraded":"2024-09-05T16:47:15.727609+0000","last_fullsized":"2024-09-05T16:47:15.727609+0000","mapping_epoch":38,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1865,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12752","reported_seq":17385,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-05T16:47:15.722967+0000","last_change":"2024-09-05T16:46:51.992073+0000","last_active":"2024-09-05T16:47:15.722967+0000","last_peered":"2024-09-05T16:47:15.722967+0000","last_clean":"2024-09-05T16:47:15.722967+0000","last_became_active":"2024-09-05T16:46:51.667128+0000","last_became_peered":"2024-09-05T16:46:51.667128+0000","last_unstale":"2024-09-05T16:47:15.722967+0000","last_undegraded":"2024-09-05T16:47:15.722967+0000","last_fullsized":"2024-09-05T16:47:15.722967+0000","mapping_epoch":44,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12752","last_scrub_stamp":"2024-09-05T16:46:51.991995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:51.991995+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:04:54.120031+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12934","reported_seq":17689,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-05T16:47:15.727311+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:47:15.727311+0000","last_peered":"2024-09-05T16:47:15.727311+0000","last_clean":"2024-09-05T16:47:15.727311+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:47:15.727311+0000","last_undegraded":"2024-09-05T16:47:15.727311+0000","last_fullsized":"2024-09-05T16:47:15.727311+0000","mapping_epoch":34,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7234,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12593","reported_seq":12718,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:15.723507+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:47:15.723507+0000","last_peered":"2024-09-05T16:47:15.723507+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:47:15.723507+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12591,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2143}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12772","reported_seq":17360,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-05T16:47:15.832645+0000","last_change":"2024-09-05T16:47:15.832530+0000","last_active":"2024-09-05T16:47:15.832645+0000","last_peered":"2024-09-05T16:47:15.832645+0000","last_clean":"2024-09-05T16:47:15.832645+0000","last_became_active":"2024-09-05T16:47:15.728759+0000","last_became_peered":"2024-09-05T16:47:15.728759+0000","last_unstale":"2024-09-05T16:47:15.832645+0000","last_undegraded":"2024-09-05T16:47:15.832645+0000","last_fullsized":"2024-09-05T16:47:15.832645+0000","mapping_epoch":48,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T02:39:11.515339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12772,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2250,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12660","reported_seq":17078,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-05T16:47:15.720462+0000","last_change":"2024-09-05T16:47:03.683348+0000","last_active":"2024-09-05T16:47:15.720462+0000","last_peered":"2024-09-05T16:47:15.720462+0000","last_clean":"2024-09-05T16:47:15.720462+0000","last_became_active":"2024-09-05T16:47:03.185143+0000","last_became_peered":"2024-09-05T16:47:03.185143+0000","last_unstale":"2024-09-05T16:47:15.720462+0000","last_undegraded":"2024-09-05T16:47:15.720462+0000","last_fullsized":"2024-09-05T16:47:15.720462+0000","mapping_epoch":46,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12660","last_scrub_stamp":"2024-09-05T16:47:03.683262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:03.683262+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:42:16.533299+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12777","reported_seq":12906,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-05T16:47:15.727595+0000","last_change":"2024-09-05T16:46:40.458315+0000","last_active":"2024-09-05T16:47:15.727595+0000","last_peered":"2024-09-05T16:47:15.727595+0000","last_clean":"2024-09-05T16:47:15.727595+0000","last_became_active":"2024-09-05T16:46:40.184288+0000","last_became_peered":"2024-09-05T16:46:40.184288+0000","last_unstale":"2024-09-05T16:47:15.727595+0000","last_undegraded":"2024-09-05T16:47:15.727595+0000","last_fullsized":"2024-09-05T16:47:15.727595+0000","mapping_epoch":42,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12777","last_scrub_stamp":"2024-09-05T16:46:40.458226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:40.458226+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:38:58.734210+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'12770","reported_seq":12897,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:15.723902+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:47:15.723902+0000","last_peered":"2024-09-05T16:47:15.723902+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:47:15.723902+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12767,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2240}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12808","reported_seq":12956,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:15.723029+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:47:15.723029+0000","last_peered":"2024-09-05T16:47:15.723029+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:47:15.723029+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12806,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2243}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13120","reported_seq":13273,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:15.723558+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:47:15.723558+0000","last_peered":"2024-09-05T16:47:15.723558+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:47:15.723558+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13117,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2278}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12758","reported_seq":12882,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:15.726091+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:47:15.726091+0000","last_peered":"2024-09-05T16:47:15.726091+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:47:15.726091+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2237}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12808","reported_seq":17570,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-05T16:47:15.727434+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:47:15.727434+0000","last_peered":"2024-09-05T16:47:15.727434+0000","last_clean":"2024-09-05T16:47:15.727434+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:47:15.727434+0000","last_undegraded":"2024-09-05T16:47:15.727434+0000","last_fullsized":"2024-09-05T16:47:15.727434+0000","mapping_epoch":36,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4408,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13124","reported_seq":13258,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:15.723926+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:47:15.723926+0000","last_peered":"2024-09-05T16:47:15.723926+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:47:15.723926+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13122,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2332}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12831","reported_seq":12970,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:15.723352+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:47:15.723352+0000","last_peered":"2024-09-05T16:47:15.723352+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:47:15.723352+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12829,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2306}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'12773","reported_seq":15276,"reported_epoch":49,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:21.351897+0000","last_change":"2024-09-05T16:47:13.789527+0000","last_active":"2024-09-05T16:47:21.351897+0000","last_peered":"2024-09-05T16:47:21.351897+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:47:21.351897+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12771,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1036}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":121248,"num_objects":18,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16848,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141943,"num_write_kb":519972,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5872,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":77824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084447,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722092,"kb_used_data":404,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649748,"statfs":{"total":96636764160,"available":95897341952,"internally_reserved":0,"allocated":413696,"data_stored":155470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":738979539},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986325,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":591648,"kb_used_data":464,"kb_used_omap":4,"kb_used_meta":591163,"kb_avail":93780192,"statfs":{"total":96636764160,"available":96030916608,"internally_reserved":0,"allocated":475136,"data_stored":216910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":605351634},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673131,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713256,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":93658584,"statfs":{"total":96636764160,"available":95906390016,"internally_reserved":0,"allocated":413696,"data_stored":155470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":729938373},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182562,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713708,"kb_used_data":404,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658132,"statfs":{"total":96636764160,"available":95905927168,"internally_reserved":0,"allocated":413696,"data_stored":155470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":730394317},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":65536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:47:24.693 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:47:24.693 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "39'12593", 'reported_seq': 12718, 'reported_epoch': 49, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:47:15.723507+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:47:15.723507+0000', 'last_peered': '2024-09-05T16:47:15.723507+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:47:15.723507+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "33'2500", 'ondisk_log_start': "33'2500", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10093, 'log_dups_size': 0, 'ondisk_log_size': 10093, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2147, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12591, 'num_write_kb': 50373, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2143}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:47:24.693 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:47:24.693 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "39'12770", 'reported_seq': 12897, 'reported_epoch': 49, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:47:15.723902+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:47:15.723902+0000', 'last_peered': '2024-09-05T16:47:15.723902+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:47:15.723902+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "33'2700", 'ondisk_log_start': "33'2700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10070, 'log_dups_size': 0, 'ondisk_log_size': 10070, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26944, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 16, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12767, 'num_write_kb': 51081, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2240}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:47:24.693 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:47:24.694 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "39'12808", 'reported_seq': 12956, 'reported_epoch': 49, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:47:15.723029+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:47:15.723029+0000', 'last_peered': '2024-09-05T16:47:15.723029+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:47:15.723029+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "33'2800", 'ondisk_log_start': "33'2800", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10008, 'log_dups_size': 0, 'ondisk_log_size': 10008, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12806, 'num_write_kb': 51230, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -2243}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:47:24.694 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:47:24.694 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "39'13120", 'reported_seq': 13273, 'reported_epoch': 49, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:47:15.723558+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:47:15.723558+0000', 'last_peered': '2024-09-05T16:47:15.723558+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:47:15.723558+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "33'3100", 'ondisk_log_start': "33'3100", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20208, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2284, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13117, 'num_write_kb': 52476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2278}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:47:24.694 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:47:24.694 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "39'12758", 'reported_seq': 12882, 'reported_epoch': 49, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:47:15.726091+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:47:15.726091+0000', 'last_peered': '2024-09-05T16:47:15.726091+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:47:15.726091+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "33'2700", 'ondisk_log_start': "33'2700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10058, 'log_dups_size': 0, 'ondisk_log_size': 10058, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6736, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12757, 'num_write_kb': 51031, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2237}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:47:24.694 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:47:24.694 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "39'13124", 'reported_seq': 13258, 'reported_epoch': 49, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:47:15.723926+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:47:15.723926+0000', 'last_peered': '2024-09-05T16:47:15.723926+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:47:15.723926+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "33'3100", 'ondisk_log_start': "33'3100", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10024, 'log_dups_size': 0, 'ondisk_log_size': 10024, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13122, 'num_write_kb': 52500, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2332}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:47:24.694 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:47:24.694 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "39'12831", 'reported_seq': 12970, 'reported_epoch': 49, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:47:15.723352+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:47:15.723352+0000', 'last_peered': '2024-09-05T16:47:15.723352+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:47:15.723352+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "33'2800", 'ondisk_log_start': "33'2800", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10031, 'log_dups_size': 0, 'ondisk_log_size': 10031, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2310, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12829, 'num_write_kb': 51325, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -2306}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:47:24.694 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-05T16:47:24.694 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "39'12773", 'reported_seq': 15276, 'reported_epoch': 49, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:47:21.351897+0000', 'last_change': '2024-09-05T16:47:13.789527+0000', 'last_active': '2024-09-05T16:47:21.351897+0000', 'last_peered': '2024-09-05T16:47:21.351897+0000', 'last_clean': '2024-09-05T16:33:52.842117+0000', 'last_became_active': '2024-09-05T16:34:56.895067+0000', 'last_became_peered': '2024-09-05T16:34:56.895067+0000', 'last_unstale': '2024-09-05T16:47:21.351897+0000', 'last_undegraded': '2024-09-05T16:34:56.584371+0000', 'last_fullsized': '2024-09-05T16:34:56.559350+0000', 'mapping_epoch': 32, 'log_start': "33'2700", 'ondisk_log_start': "33'2700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10073, 'log_dups_size': 0, 'ondisk_log_size': 10073, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1038, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12771, 'num_write_kb': 51092, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1203, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1036}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-05T16:47:24.695 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:47:25.076 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:47:25.076 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 16848/72 objects degraded (23400.000%), 8 pgs degraded, 8 pgs undersized","count":16},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":848,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":49,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":8},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":18,"data_bytes":121248,"bytes_used":2806480896,"bytes_avail":383740575744,"bytes_total":386547056640,"degraded_objects":16848,"degraded_total":72,"degraded_ratio":234,"recovering_objects_per_sec":159,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1340,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:47:25.089 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:47:25.089 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:47:25.432 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:47:25.432 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:47:25.444 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":464,"stamp":"2024-09-05T16:47:25.107274+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":121248,"num_objects":18,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16848,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141943,"num_write_kb":519972,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5872,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2740704,"kb_used_data":1676,"kb_used_omap":14,"kb_used_meta":2738929,"kb_avail":374746656,"statfs":{"total":386547056640,"available":383740575744,"internally_reserved":0,"allocated":1716224,"data_stored":683320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2804663863},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1340,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001775"},"pg_stats":[{"pgid":"1.f","version":"39'12904","reported_seq":17516,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-05T16:47:15.723019+0000","last_change":"2024-09-05T16:46:27.826222+0000","last_active":"2024-09-05T16:47:15.723019+0000","last_peered":"2024-09-05T16:47:15.723019+0000","last_clean":"2024-09-05T16:47:15.723019+0000","last_became_active":"2024-09-05T16:46:27.757928+0000","last_became_peered":"2024-09-05T16:46:27.757928+0000","last_unstale":"2024-09-05T16:47:15.723019+0000","last_undegraded":"2024-09-05T16:47:15.723019+0000","last_fullsized":"2024-09-05T16:47:15.723019+0000","mapping_epoch":40,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:32:35.713428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12904,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2262,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12552","reported_seq":17229,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-05T16:47:15.727609+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:47:15.727609+0000","last_peered":"2024-09-05T16:47:15.727609+0000","last_clean":"2024-09-05T16:47:15.727609+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:47:15.727609+0000","last_undegraded":"2024-09-05T16:47:15.727609+0000","last_fullsized":"2024-09-05T16:47:15.727609+0000","mapping_epoch":38,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1865,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12752","reported_seq":17385,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-05T16:47:15.722967+0000","last_change":"2024-09-05T16:46:51.992073+0000","last_active":"2024-09-05T16:47:15.722967+0000","last_peered":"2024-09-05T16:47:15.722967+0000","last_clean":"2024-09-05T16:47:15.722967+0000","last_became_active":"2024-09-05T16:46:51.667128+0000","last_became_peered":"2024-09-05T16:46:51.667128+0000","last_unstale":"2024-09-05T16:47:15.722967+0000","last_undegraded":"2024-09-05T16:47:15.722967+0000","last_fullsized":"2024-09-05T16:47:15.722967+0000","mapping_epoch":44,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12752","last_scrub_stamp":"2024-09-05T16:46:51.991995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:51.991995+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:04:54.120031+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12934","reported_seq":17689,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-05T16:47:15.727311+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:47:15.727311+0000","last_peered":"2024-09-05T16:47:15.727311+0000","last_clean":"2024-09-05T16:47:15.727311+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:47:15.727311+0000","last_undegraded":"2024-09-05T16:47:15.727311+0000","last_fullsized":"2024-09-05T16:47:15.727311+0000","mapping_epoch":34,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7234,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12593","reported_seq":12718,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:15.723507+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:47:15.723507+0000","last_peered":"2024-09-05T16:47:15.723507+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:47:15.723507+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12591,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2143}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12772","reported_seq":17360,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-05T16:47:15.832645+0000","last_change":"2024-09-05T16:47:15.832530+0000","last_active":"2024-09-05T16:47:15.832645+0000","last_peered":"2024-09-05T16:47:15.832645+0000","last_clean":"2024-09-05T16:47:15.832645+0000","last_became_active":"2024-09-05T16:47:15.728759+0000","last_became_peered":"2024-09-05T16:47:15.728759+0000","last_unstale":"2024-09-05T16:47:15.832645+0000","last_undegraded":"2024-09-05T16:47:15.832645+0000","last_fullsized":"2024-09-05T16:47:15.832645+0000","mapping_epoch":48,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T02:39:11.515339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12772,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2250,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12660","reported_seq":17078,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-05T16:47:15.720462+0000","last_change":"2024-09-05T16:47:03.683348+0000","last_active":"2024-09-05T16:47:15.720462+0000","last_peered":"2024-09-05T16:47:15.720462+0000","last_clean":"2024-09-05T16:47:15.720462+0000","last_became_active":"2024-09-05T16:47:03.185143+0000","last_became_peered":"2024-09-05T16:47:03.185143+0000","last_unstale":"2024-09-05T16:47:15.720462+0000","last_undegraded":"2024-09-05T16:47:15.720462+0000","last_fullsized":"2024-09-05T16:47:15.720462+0000","mapping_epoch":46,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12660","last_scrub_stamp":"2024-09-05T16:47:03.683262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:03.683262+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:42:16.533299+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12777","reported_seq":12906,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-05T16:47:15.727595+0000","last_change":"2024-09-05T16:46:40.458315+0000","last_active":"2024-09-05T16:47:15.727595+0000","last_peered":"2024-09-05T16:47:15.727595+0000","last_clean":"2024-09-05T16:47:15.727595+0000","last_became_active":"2024-09-05T16:46:40.184288+0000","last_became_peered":"2024-09-05T16:46:40.184288+0000","last_unstale":"2024-09-05T16:47:15.727595+0000","last_undegraded":"2024-09-05T16:47:15.727595+0000","last_fullsized":"2024-09-05T16:47:15.727595+0000","mapping_epoch":42,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12777","last_scrub_stamp":"2024-09-05T16:46:40.458226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:40.458226+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:38:58.734210+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'12770","reported_seq":12897,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:15.723902+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:47:15.723902+0000","last_peered":"2024-09-05T16:47:15.723902+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:47:15.723902+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12767,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2240}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12808","reported_seq":12956,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:15.723029+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:47:15.723029+0000","last_peered":"2024-09-05T16:47:15.723029+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:47:15.723029+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12806,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2243}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13120","reported_seq":13273,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:15.723558+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:47:15.723558+0000","last_peered":"2024-09-05T16:47:15.723558+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:47:15.723558+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13117,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2278}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12758","reported_seq":12883,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:15.727184+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:47:15.727184+0000","last_peered":"2024-09-05T16:47:15.727184+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:47:15.727184+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2237}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12808","reported_seq":17570,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-05T16:47:15.727434+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:47:15.727434+0000","last_peered":"2024-09-05T16:47:15.727434+0000","last_clean":"2024-09-05T16:47:15.727434+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:47:15.727434+0000","last_undegraded":"2024-09-05T16:47:15.727434+0000","last_fullsized":"2024-09-05T16:47:15.727434+0000","mapping_epoch":36,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4408,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13124","reported_seq":13258,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:15.723926+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:47:15.723926+0000","last_peered":"2024-09-05T16:47:15.723926+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:47:15.723926+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13122,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2332}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12831","reported_seq":12970,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:15.723352+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:47:15.723352+0000","last_peered":"2024-09-05T16:47:15.723352+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:47:15.723352+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12829,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2306}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'12773","reported_seq":15276,"reported_epoch":49,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:21.351897+0000","last_change":"2024-09-05T16:47:13.789527+0000","last_active":"2024-09-05T16:47:21.351897+0000","last_peered":"2024-09-05T16:47:21.351897+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:47:21.351897+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12771,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1036}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":121248,"num_objects":18,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16848,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141943,"num_write_kb":519972,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5872,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":77824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084448,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722092,"kb_used_data":404,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649748,"statfs":{"total":96636764160,"available":95897341952,"internally_reserved":0,"allocated":413696,"data_stored":155470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":738979539},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986325,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":591648,"kb_used_data":464,"kb_used_omap":4,"kb_used_meta":591163,"kb_avail":93780192,"statfs":{"total":96636764160,"available":96030916608,"internally_reserved":0,"allocated":475136,"data_stored":216910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":605351634},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673131,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713256,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":93658584,"statfs":{"total":96636764160,"available":95906390016,"internally_reserved":0,"allocated":413696,"data_stored":155470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":729938373},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182562,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713708,"kb_used_data":404,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658132,"statfs":{"total":96636764160,"available":95905927168,"internally_reserved":0,"allocated":413696,"data_stored":155470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":730394317},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":65536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:47:28.446 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:47:28.778 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:47:28.779 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:47:28.793 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":467,"stamp":"2024-09-05T16:47:27.195048+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":121248,"num_objects":18,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16848,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141943,"num_write_kb":519972,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5872,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2740704,"kb_used_data":1676,"kb_used_omap":14,"kb_used_meta":2738929,"kb_avail":374746656,"statfs":{"total":386547056640,"available":383740575744,"internally_reserved":0,"allocated":1716224,"data_stored":683320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2804663863},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-818,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":818,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.088911"},"pg_stats":[{"pgid":"1.f","version":"39'12904","reported_seq":17516,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-05T16:47:15.723019+0000","last_change":"2024-09-05T16:46:27.826222+0000","last_active":"2024-09-05T16:47:15.723019+0000","last_peered":"2024-09-05T16:47:15.723019+0000","last_clean":"2024-09-05T16:47:15.723019+0000","last_became_active":"2024-09-05T16:46:27.757928+0000","last_became_peered":"2024-09-05T16:46:27.757928+0000","last_unstale":"2024-09-05T16:47:15.723019+0000","last_undegraded":"2024-09-05T16:47:15.723019+0000","last_fullsized":"2024-09-05T16:47:15.723019+0000","mapping_epoch":40,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:32:35.713428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12904,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2262,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12552","reported_seq":17229,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-05T16:47:15.727609+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:47:15.727609+0000","last_peered":"2024-09-05T16:47:15.727609+0000","last_clean":"2024-09-05T16:47:15.727609+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:47:15.727609+0000","last_undegraded":"2024-09-05T16:47:15.727609+0000","last_fullsized":"2024-09-05T16:47:15.727609+0000","mapping_epoch":38,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1865,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12752","reported_seq":17385,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-05T16:47:15.722967+0000","last_change":"2024-09-05T16:46:51.992073+0000","last_active":"2024-09-05T16:47:15.722967+0000","last_peered":"2024-09-05T16:47:15.722967+0000","last_clean":"2024-09-05T16:47:15.722967+0000","last_became_active":"2024-09-05T16:46:51.667128+0000","last_became_peered":"2024-09-05T16:46:51.667128+0000","last_unstale":"2024-09-05T16:47:15.722967+0000","last_undegraded":"2024-09-05T16:47:15.722967+0000","last_fullsized":"2024-09-05T16:47:15.722967+0000","mapping_epoch":44,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12752","last_scrub_stamp":"2024-09-05T16:46:51.991995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:51.991995+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:04:54.120031+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12934","reported_seq":17689,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-05T16:47:15.727311+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:47:15.727311+0000","last_peered":"2024-09-05T16:47:15.727311+0000","last_clean":"2024-09-05T16:47:15.727311+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:47:15.727311+0000","last_undegraded":"2024-09-05T16:47:15.727311+0000","last_fullsized":"2024-09-05T16:47:15.727311+0000","mapping_epoch":34,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7234,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12593","reported_seq":12718,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:15.723507+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:47:15.723507+0000","last_peered":"2024-09-05T16:47:15.723507+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:47:15.723507+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12591,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2143}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12772","reported_seq":17360,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-05T16:47:15.832645+0000","last_change":"2024-09-05T16:47:15.832530+0000","last_active":"2024-09-05T16:47:15.832645+0000","last_peered":"2024-09-05T16:47:15.832645+0000","last_clean":"2024-09-05T16:47:15.832645+0000","last_became_active":"2024-09-05T16:47:15.728759+0000","last_became_peered":"2024-09-05T16:47:15.728759+0000","last_unstale":"2024-09-05T16:47:15.832645+0000","last_undegraded":"2024-09-05T16:47:15.832645+0000","last_fullsized":"2024-09-05T16:47:15.832645+0000","mapping_epoch":48,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T02:39:11.515339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12772,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2250,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12660","reported_seq":17080,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-05T16:47:26.196288+0000","last_change":"2024-09-05T16:47:03.683348+0000","last_active":"2024-09-05T16:47:26.196288+0000","last_peered":"2024-09-05T16:47:26.196288+0000","last_clean":"2024-09-05T16:47:26.196288+0000","last_became_active":"2024-09-05T16:47:03.185143+0000","last_became_peered":"2024-09-05T16:47:03.185143+0000","last_unstale":"2024-09-05T16:47:26.196288+0000","last_undegraded":"2024-09-05T16:47:26.196288+0000","last_fullsized":"2024-09-05T16:47:26.196288+0000","mapping_epoch":46,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12660","last_scrub_stamp":"2024-09-05T16:47:03.683262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:03.683262+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:42:16.533299+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12777","reported_seq":12906,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-05T16:47:15.727595+0000","last_change":"2024-09-05T16:46:40.458315+0000","last_active":"2024-09-05T16:47:15.727595+0000","last_peered":"2024-09-05T16:47:15.727595+0000","last_clean":"2024-09-05T16:47:15.727595+0000","last_became_active":"2024-09-05T16:46:40.184288+0000","last_became_peered":"2024-09-05T16:46:40.184288+0000","last_unstale":"2024-09-05T16:47:15.727595+0000","last_undegraded":"2024-09-05T16:47:15.727595+0000","last_fullsized":"2024-09-05T16:47:15.727595+0000","mapping_epoch":42,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12777","last_scrub_stamp":"2024-09-05T16:46:40.458226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:40.458226+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:38:58.734210+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'12770","reported_seq":12897,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:15.723902+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:47:15.723902+0000","last_peered":"2024-09-05T16:47:15.723902+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:47:15.723902+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12767,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2240}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12808","reported_seq":12956,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:15.723029+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:47:15.723029+0000","last_peered":"2024-09-05T16:47:15.723029+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:47:15.723029+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12806,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2243}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13120","reported_seq":13273,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:15.723558+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:47:15.723558+0000","last_peered":"2024-09-05T16:47:15.723558+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:47:15.723558+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13117,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2278}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12758","reported_seq":12883,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:15.727184+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:47:15.727184+0000","last_peered":"2024-09-05T16:47:15.727184+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:47:15.727184+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2237}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12808","reported_seq":17570,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-05T16:47:15.727434+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:47:15.727434+0000","last_peered":"2024-09-05T16:47:15.727434+0000","last_clean":"2024-09-05T16:47:15.727434+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:47:15.727434+0000","last_undegraded":"2024-09-05T16:47:15.727434+0000","last_fullsized":"2024-09-05T16:47:15.727434+0000","mapping_epoch":36,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4408,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13124","reported_seq":13258,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:15.723926+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:47:15.723926+0000","last_peered":"2024-09-05T16:47:15.723926+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:47:15.723926+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13122,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2332}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12831","reported_seq":12970,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:15.723352+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:47:15.723352+0000","last_peered":"2024-09-05T16:47:15.723352+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:47:15.723352+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12829,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2306}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'12773","reported_seq":15276,"reported_epoch":49,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:21.351897+0000","last_change":"2024-09-05T16:47:13.789527+0000","last_active":"2024-09-05T16:47:21.351897+0000","last_peered":"2024-09-05T16:47:21.351897+0000","last_clean":"2024-09-05T16:33:52.842117+0000","last_became_active":"2024-09-05T16:34:56.895067+0000","last_became_peered":"2024-09-05T16:34:56.895067+0000","last_unstale":"2024-09-05T16:47:21.351897+0000","last_undegraded":"2024-09-05T16:34:56.584371+0000","last_fullsized":"2024-09-05T16:34:56.559350+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12771,"num_write_kb":51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1036}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":121248,"num_objects":18,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16848,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141943,"num_write_kb":519972,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5872,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":77824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084448,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722092,"kb_used_data":404,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649748,"statfs":{"total":96636764160,"available":95897341952,"internally_reserved":0,"allocated":413696,"data_stored":155470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":738979539},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986325,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":591648,"kb_used_data":464,"kb_used_omap":4,"kb_used_meta":591163,"kb_avail":93780192,"statfs":{"total":96636764160,"available":96030916608,"internally_reserved":0,"allocated":475136,"data_stored":216910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":605351634},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673132,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713256,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":93658584,"statfs":{"total":96636764160,"available":95906390016,"internally_reserved":0,"allocated":413696,"data_stored":155470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":729938373},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182563,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713708,"kb_used_data":404,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658132,"statfs":{"total":96636764160,"available":95905927168,"internally_reserved":0,"allocated":413696,"data_stored":155470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":730394317},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":65536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:47:28.794 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:47:29.100 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:47:29.101 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:47:29.113 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":468,"stamp":"2024-09-05T16:47:29.107990+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":107776,"num_objects":16,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15810,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129172,"num_write_kb":468880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4669,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2749056,"kb_used_data":1700,"kb_used_omap":14,"kb_used_meta":2747249,"kb_avail":374738304,"statfs":{"total":386547056640,"available":383732023296,"internally_reserved":0,"allocated":1740800,"data_stored":688394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2813183543},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13472,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1856,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12771,"num_write_kb":-51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-385,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001391"},"pg_stats":[{"pgid":"1.f","version":"39'12904","reported_seq":17520,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-05T16:47:27.199630+0000","last_change":"2024-09-05T16:46:27.826222+0000","last_active":"2024-09-05T16:47:27.199630+0000","last_peered":"2024-09-05T16:47:27.199630+0000","last_clean":"2024-09-05T16:47:27.199630+0000","last_became_active":"2024-09-05T16:46:27.757928+0000","last_became_peered":"2024-09-05T16:46:27.757928+0000","last_unstale":"2024-09-05T16:47:27.199630+0000","last_undegraded":"2024-09-05T16:47:27.199630+0000","last_fullsized":"2024-09-05T16:47:27.199630+0000","mapping_epoch":40,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:32:35.713428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12904,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2262,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12552","reported_seq":17233,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-05T16:47:27.202260+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:47:27.202260+0000","last_peered":"2024-09-05T16:47:27.202260+0000","last_clean":"2024-09-05T16:47:27.202260+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:47:27.202260+0000","last_undegraded":"2024-09-05T16:47:27.202260+0000","last_fullsized":"2024-09-05T16:47:27.202260+0000","mapping_epoch":38,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1865,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12752","reported_seq":17385,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-05T16:47:15.722967+0000","last_change":"2024-09-05T16:46:51.992073+0000","last_active":"2024-09-05T16:47:15.722967+0000","last_peered":"2024-09-05T16:47:15.722967+0000","last_clean":"2024-09-05T16:47:15.722967+0000","last_became_active":"2024-09-05T16:46:51.667128+0000","last_became_peered":"2024-09-05T16:46:51.667128+0000","last_unstale":"2024-09-05T16:47:15.722967+0000","last_undegraded":"2024-09-05T16:47:15.722967+0000","last_fullsized":"2024-09-05T16:47:15.722967+0000","mapping_epoch":44,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12752","last_scrub_stamp":"2024-09-05T16:46:51.991995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:51.991995+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:04:54.120031+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12934","reported_seq":17693,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-05T16:47:27.201626+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:47:27.201626+0000","last_peered":"2024-09-05T16:47:27.201626+0000","last_clean":"2024-09-05T16:47:27.201626+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:47:27.201626+0000","last_undegraded":"2024-09-05T16:47:27.201626+0000","last_fullsized":"2024-09-05T16:47:27.201626+0000","mapping_epoch":34,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7234,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12593","reported_seq":12722,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:27.199110+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:47:27.199110+0000","last_peered":"2024-09-05T16:47:27.199110+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:47:27.199110+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12591,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2143}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12772","reported_seq":17364,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-05T16:47:27.201640+0000","last_change":"2024-09-05T16:47:15.832530+0000","last_active":"2024-09-05T16:47:27.201640+0000","last_peered":"2024-09-05T16:47:27.201640+0000","last_clean":"2024-09-05T16:47:27.201640+0000","last_became_active":"2024-09-05T16:47:15.728759+0000","last_became_peered":"2024-09-05T16:47:15.728759+0000","last_unstale":"2024-09-05T16:47:27.201640+0000","last_undegraded":"2024-09-05T16:47:27.201640+0000","last_fullsized":"2024-09-05T16:47:27.201640+0000","mapping_epoch":48,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T02:39:11.515339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12772,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2250,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12660","reported_seq":17080,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-05T16:47:26.196288+0000","last_change":"2024-09-05T16:47:03.683348+0000","last_active":"2024-09-05T16:47:26.196288+0000","last_peered":"2024-09-05T16:47:26.196288+0000","last_clean":"2024-09-05T16:47:26.196288+0000","last_became_active":"2024-09-05T16:47:03.185143+0000","last_became_peered":"2024-09-05T16:47:03.185143+0000","last_unstale":"2024-09-05T16:47:26.196288+0000","last_undegraded":"2024-09-05T16:47:26.196288+0000","last_fullsized":"2024-09-05T16:47:26.196288+0000","mapping_epoch":46,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12660","last_scrub_stamp":"2024-09-05T16:47:03.683262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:03.683262+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:42:16.533299+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12777","reported_seq":12910,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-05T16:47:27.202241+0000","last_change":"2024-09-05T16:46:40.458315+0000","last_active":"2024-09-05T16:47:27.202241+0000","last_peered":"2024-09-05T16:47:27.202241+0000","last_clean":"2024-09-05T16:47:27.202241+0000","last_became_active":"2024-09-05T16:46:40.184288+0000","last_became_peered":"2024-09-05T16:46:40.184288+0000","last_unstale":"2024-09-05T16:47:27.202241+0000","last_undegraded":"2024-09-05T16:47:27.202241+0000","last_fullsized":"2024-09-05T16:47:27.202241+0000","mapping_epoch":42,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12777","last_scrub_stamp":"2024-09-05T16:46:40.458226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:40.458226+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:38:58.734210+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'12770","reported_seq":12901,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:27.200583+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:47:27.200583+0000","last_peered":"2024-09-05T16:47:27.200583+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:47:27.200583+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12767,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2240}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12808","reported_seq":12956,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:15.723029+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:47:15.723029+0000","last_peered":"2024-09-05T16:47:15.723029+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:47:15.723029+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12806,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2243}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13120","reported_seq":13277,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:27.199253+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:47:27.199253+0000","last_peered":"2024-09-05T16:47:27.199253+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:47:27.199253+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13117,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2278}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12758","reported_seq":12887,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:27.199725+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:47:27.199725+0000","last_peered":"2024-09-05T16:47:27.199725+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:47:27.199725+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2237}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12808","reported_seq":17574,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-05T16:47:27.201730+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:47:27.201730+0000","last_peered":"2024-09-05T16:47:27.201730+0000","last_clean":"2024-09-05T16:47:27.201730+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:47:27.201730+0000","last_undegraded":"2024-09-05T16:47:27.201730+0000","last_fullsized":"2024-09-05T16:47:27.201730+0000","mapping_epoch":36,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4408,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13124","reported_seq":13263,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:27.200430+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:47:27.200430+0000","last_peered":"2024-09-05T16:47:27.200430+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:47:27.200430+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13122,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2332}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12831","reported_seq":12970,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:15.723352+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:47:15.723352+0000","last_peered":"2024-09-05T16:47:15.723352+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:47:15.723352+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12829,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2306}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'12773","reported_seq":12910,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-05T16:47:27.437356+0000","last_change":"2024-09-05T16:47:27.437275+0000","last_active":"2024-09-05T16:47:27.437356+0000","last_peered":"2024-09-05T16:47:27.437356+0000","last_clean":"2024-09-05T16:47:27.437356+0000","last_became_active":"2024-09-05T16:47:27.204989+0000","last_became_peered":"2024-09-05T16:47:27.204989+0000","last_unstale":"2024-09-05T16:47:27.437356+0000","last_undegraded":"2024-09-05T16:47:27.437356+0000","last_fullsized":"2024-09-05T16:47:27.437356+0000","mapping_epoch":50,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12773","last_scrub_stamp":"2024-09-05T16:47:27.437187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:27.437187+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T16:53:55.961513+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":107776,"num_objects":16,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15810,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129172,"num_write_kb":468880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4669,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":69632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084449,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722108,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649732,"statfs":{"total":96636764160,"available":95897325568,"internally_reserved":0,"allocated":430080,"data_stored":162103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":738979539},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986326,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":599984,"kb_used_data":472,"kb_used_omap":4,"kb_used_meta":599483,"kb_avail":93771856,"statfs":{"total":96636764160,"available":96022380544,"internally_reserved":0,"allocated":483328,"data_stored":215351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":613871314},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673132,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713256,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":93658584,"statfs":{"total":96636764160,"available":95906390016,"internally_reserved":0,"allocated":413696,"data_stored":155470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":729938373},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182563,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713708,"kb_used_data":404,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658132,"statfs":{"total":96636764160,"available":95905927168,"internally_reserved":0,"allocated":413696,"data_stored":155470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":730394317},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":57344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:47:29.114 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:47:29.114 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "39'12593", 'reported_seq': 12722, 'reported_epoch': 51, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:47:27.199110+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:47:27.199110+0000', 'last_peered': '2024-09-05T16:47:27.199110+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:47:27.199110+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "33'2500", 'ondisk_log_start': "33'2500", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10093, 'log_dups_size': 0, 'ondisk_log_size': 10093, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2147, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12591, 'num_write_kb': 50373, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2143}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:47:29.115 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:47:29.115 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "39'12770", 'reported_seq': 12901, 'reported_epoch': 51, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:47:27.200583+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:47:27.200583+0000', 'last_peered': '2024-09-05T16:47:27.200583+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:47:27.200583+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "33'2700", 'ondisk_log_start': "33'2700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10070, 'log_dups_size': 0, 'ondisk_log_size': 10070, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26944, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 16, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12767, 'num_write_kb': 51081, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2240}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:47:29.115 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:47:29.115 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "39'12808", 'reported_seq': 12956, 'reported_epoch': 49, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:47:15.723029+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:47:15.723029+0000', 'last_peered': '2024-09-05T16:47:15.723029+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:47:15.723029+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "33'2800", 'ondisk_log_start': "33'2800", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10008, 'log_dups_size': 0, 'ondisk_log_size': 10008, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12806, 'num_write_kb': 51230, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -2243}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:47:29.116 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:47:29.116 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "39'13120", 'reported_seq': 13277, 'reported_epoch': 51, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:47:27.199253+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:47:27.199253+0000', 'last_peered': '2024-09-05T16:47:27.199253+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:47:27.199253+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "33'3100", 'ondisk_log_start': "33'3100", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20208, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2284, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13117, 'num_write_kb': 52476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2278}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:47:29.116 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:47:29.116 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "39'12758", 'reported_seq': 12887, 'reported_epoch': 51, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:47:27.199725+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:47:27.199725+0000', 'last_peered': '2024-09-05T16:47:27.199725+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:47:27.199725+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "33'2700", 'ondisk_log_start': "33'2700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10058, 'log_dups_size': 0, 'ondisk_log_size': 10058, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6736, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12757, 'num_write_kb': 51031, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2237}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:47:29.117 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:47:29.117 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "39'13124", 'reported_seq': 13263, 'reported_epoch': 51, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:47:27.200430+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:47:27.200430+0000', 'last_peered': '2024-09-05T16:47:27.200430+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:47:27.200430+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "33'3100", 'ondisk_log_start': "33'3100", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10024, 'log_dups_size': 0, 'ondisk_log_size': 10024, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13122, 'num_write_kb': 52500, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2332}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:47:29.117 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:47:29.117 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "39'12831", 'reported_seq': 12970, 'reported_epoch': 49, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:47:15.723352+0000', 'last_change': '2024-09-05T16:34:56.875137+0000', 'last_active': '2024-09-05T16:47:15.723352+0000', 'last_peered': '2024-09-05T16:47:15.723352+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:47:15.723352+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "33'2800", 'ondisk_log_start': "33'2800", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10031, 'log_dups_size': 0, 'ondisk_log_size': 10031, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2310, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12829, 'num_write_kb': 51325, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -2306}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:47:29.118 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:47:29.506 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:47:29.506 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 15810/64 objects degraded (24703.125%), 7 pgs degraded, 7 pgs undersized","count":14},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":852,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":51,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":7},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":9},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7}],"num_pgs":16,"num_pools":1,"num_objects":16,"data_bytes":107776,"bytes_used":2815033344,"bytes_avail":383732023296,"bytes_total":386547056640,"degraded_objects":15810,"degraded_total":64,"degraded_ratio":247.03125},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:47:29.519 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-09-05T16:47:29.519 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:47:29.827 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:47:29.828 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:47:29.840 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":468,"stamp":"2024-09-05T16:47:29.107990+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":107776,"num_objects":16,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15810,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129172,"num_write_kb":468880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4669,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2749056,"kb_used_data":1700,"kb_used_omap":14,"kb_used_meta":2747249,"kb_avail":374738304,"statfs":{"total":386547056640,"available":383732023296,"internally_reserved":0,"allocated":1740800,"data_stored":688394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2813183543},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13472,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1856,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12771,"num_write_kb":-51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-385,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001391"},"pg_stats":[{"pgid":"1.f","version":"39'12904","reported_seq":17520,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-05T16:47:27.199630+0000","last_change":"2024-09-05T16:46:27.826222+0000","last_active":"2024-09-05T16:47:27.199630+0000","last_peered":"2024-09-05T16:47:27.199630+0000","last_clean":"2024-09-05T16:47:27.199630+0000","last_became_active":"2024-09-05T16:46:27.757928+0000","last_became_peered":"2024-09-05T16:46:27.757928+0000","last_unstale":"2024-09-05T16:47:27.199630+0000","last_undegraded":"2024-09-05T16:47:27.199630+0000","last_fullsized":"2024-09-05T16:47:27.199630+0000","mapping_epoch":40,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:32:35.713428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12904,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2262,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12552","reported_seq":17233,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-05T16:47:27.202260+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:47:27.202260+0000","last_peered":"2024-09-05T16:47:27.202260+0000","last_clean":"2024-09-05T16:47:27.202260+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:47:27.202260+0000","last_undegraded":"2024-09-05T16:47:27.202260+0000","last_fullsized":"2024-09-05T16:47:27.202260+0000","mapping_epoch":38,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1865,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12752","reported_seq":17385,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-05T16:47:15.722967+0000","last_change":"2024-09-05T16:46:51.992073+0000","last_active":"2024-09-05T16:47:15.722967+0000","last_peered":"2024-09-05T16:47:15.722967+0000","last_clean":"2024-09-05T16:47:15.722967+0000","last_became_active":"2024-09-05T16:46:51.667128+0000","last_became_peered":"2024-09-05T16:46:51.667128+0000","last_unstale":"2024-09-05T16:47:15.722967+0000","last_undegraded":"2024-09-05T16:47:15.722967+0000","last_fullsized":"2024-09-05T16:47:15.722967+0000","mapping_epoch":44,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12752","last_scrub_stamp":"2024-09-05T16:46:51.991995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:51.991995+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:04:54.120031+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12934","reported_seq":17693,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-05T16:47:27.201626+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:47:27.201626+0000","last_peered":"2024-09-05T16:47:27.201626+0000","last_clean":"2024-09-05T16:47:27.201626+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:47:27.201626+0000","last_undegraded":"2024-09-05T16:47:27.201626+0000","last_fullsized":"2024-09-05T16:47:27.201626+0000","mapping_epoch":34,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7234,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12593","reported_seq":12722,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:27.199110+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:47:27.199110+0000","last_peered":"2024-09-05T16:47:27.199110+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:47:27.199110+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12591,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2143}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12772","reported_seq":17364,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-05T16:47:27.201640+0000","last_change":"2024-09-05T16:47:15.832530+0000","last_active":"2024-09-05T16:47:27.201640+0000","last_peered":"2024-09-05T16:47:27.201640+0000","last_clean":"2024-09-05T16:47:27.201640+0000","last_became_active":"2024-09-05T16:47:15.728759+0000","last_became_peered":"2024-09-05T16:47:15.728759+0000","last_unstale":"2024-09-05T16:47:27.201640+0000","last_undegraded":"2024-09-05T16:47:27.201640+0000","last_fullsized":"2024-09-05T16:47:27.201640+0000","mapping_epoch":48,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T02:39:11.515339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12772,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2250,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12660","reported_seq":17080,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-05T16:47:26.196288+0000","last_change":"2024-09-05T16:47:03.683348+0000","last_active":"2024-09-05T16:47:26.196288+0000","last_peered":"2024-09-05T16:47:26.196288+0000","last_clean":"2024-09-05T16:47:26.196288+0000","last_became_active":"2024-09-05T16:47:03.185143+0000","last_became_peered":"2024-09-05T16:47:03.185143+0000","last_unstale":"2024-09-05T16:47:26.196288+0000","last_undegraded":"2024-09-05T16:47:26.196288+0000","last_fullsized":"2024-09-05T16:47:26.196288+0000","mapping_epoch":46,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12660","last_scrub_stamp":"2024-09-05T16:47:03.683262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:03.683262+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:42:16.533299+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12777","reported_seq":12910,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-05T16:47:27.202241+0000","last_change":"2024-09-05T16:46:40.458315+0000","last_active":"2024-09-05T16:47:27.202241+0000","last_peered":"2024-09-05T16:47:27.202241+0000","last_clean":"2024-09-05T16:47:27.202241+0000","last_became_active":"2024-09-05T16:46:40.184288+0000","last_became_peered":"2024-09-05T16:46:40.184288+0000","last_unstale":"2024-09-05T16:47:27.202241+0000","last_undegraded":"2024-09-05T16:47:27.202241+0000","last_fullsized":"2024-09-05T16:47:27.202241+0000","mapping_epoch":42,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12777","last_scrub_stamp":"2024-09-05T16:46:40.458226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:40.458226+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:38:58.734210+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'12770","reported_seq":12901,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:27.200583+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:47:27.200583+0000","last_peered":"2024-09-05T16:47:27.200583+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:47:27.200583+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12767,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2240}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12808","reported_seq":12956,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:15.723029+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:47:15.723029+0000","last_peered":"2024-09-05T16:47:15.723029+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:47:15.723029+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12806,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2243}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13120","reported_seq":13277,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:27.199253+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:47:27.199253+0000","last_peered":"2024-09-05T16:47:27.199253+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:47:27.199253+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13117,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2278}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12758","reported_seq":12887,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:27.199725+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:47:27.199725+0000","last_peered":"2024-09-05T16:47:27.199725+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:47:27.199725+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2237}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12808","reported_seq":17574,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-05T16:47:27.201730+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:47:27.201730+0000","last_peered":"2024-09-05T16:47:27.201730+0000","last_clean":"2024-09-05T16:47:27.201730+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:47:27.201730+0000","last_undegraded":"2024-09-05T16:47:27.201730+0000","last_fullsized":"2024-09-05T16:47:27.201730+0000","mapping_epoch":36,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4408,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13124","reported_seq":13263,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:27.200430+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:47:27.200430+0000","last_peered":"2024-09-05T16:47:27.200430+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:47:27.200430+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13122,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2332}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12831","reported_seq":12970,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:15.723352+0000","last_change":"2024-09-05T16:34:56.875137+0000","last_active":"2024-09-05T16:47:15.723352+0000","last_peered":"2024-09-05T16:47:15.723352+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:47:15.723352+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12829,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2306}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'12773","reported_seq":12910,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-05T16:47:27.437356+0000","last_change":"2024-09-05T16:47:27.437275+0000","last_active":"2024-09-05T16:47:27.437356+0000","last_peered":"2024-09-05T16:47:27.437356+0000","last_clean":"2024-09-05T16:47:27.437356+0000","last_became_active":"2024-09-05T16:47:27.204989+0000","last_became_peered":"2024-09-05T16:47:27.204989+0000","last_unstale":"2024-09-05T16:47:27.437356+0000","last_undegraded":"2024-09-05T16:47:27.437356+0000","last_fullsized":"2024-09-05T16:47:27.437356+0000","mapping_epoch":50,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12773","last_scrub_stamp":"2024-09-05T16:47:27.437187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:27.437187+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T16:53:55.961513+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":107776,"num_objects":16,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15810,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129172,"num_write_kb":468880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4669,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":69632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084449,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722108,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649732,"statfs":{"total":96636764160,"available":95897325568,"internally_reserved":0,"allocated":430080,"data_stored":162103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":738979539},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986326,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":599984,"kb_used_data":472,"kb_used_omap":4,"kb_used_meta":599483,"kb_avail":93771856,"statfs":{"total":96636764160,"available":96022380544,"internally_reserved":0,"allocated":483328,"data_stored":215351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":613871314},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673132,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713256,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":93658584,"statfs":{"total":96636764160,"available":95906390016,"internally_reserved":0,"allocated":413696,"data_stored":155470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":729938373},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182563,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713708,"kb_used_data":404,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658132,"statfs":{"total":96636764160,"available":95905927168,"internally_reserved":0,"allocated":413696,"data_stored":155470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":730394317},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":57344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:47:32.842 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:47:33.174 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:47:33.174 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:47:33.185 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":470,"stamp":"2024-09-05T16:47:33.108720+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":107776,"num_objects":16,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14968,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129172,"num_write_kb":468880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5511,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2758808,"kb_used_data":1732,"kb_used_omap":14,"kb_used_meta":2756977,"kb_avail":374728552,"statfs":{"total":386547056640,"available":383722037248,"internally_reserved":0,"allocated":1773568,"data_stored":701660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2823145015},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13472,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1880,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12771,"num_write_kb":-51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-361,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001445"},"pg_stats":[{"pgid":"1.f","version":"39'12904","reported_seq":17520,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-05T16:47:27.199630+0000","last_change":"2024-09-05T16:46:27.826222+0000","last_active":"2024-09-05T16:47:27.199630+0000","last_peered":"2024-09-05T16:47:27.199630+0000","last_clean":"2024-09-05T16:47:27.199630+0000","last_became_active":"2024-09-05T16:46:27.757928+0000","last_became_peered":"2024-09-05T16:46:27.757928+0000","last_unstale":"2024-09-05T16:47:27.199630+0000","last_undegraded":"2024-09-05T16:47:27.199630+0000","last_fullsized":"2024-09-05T16:47:27.199630+0000","mapping_epoch":40,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:32:35.713428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12904,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2262,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12552","reported_seq":17233,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-05T16:47:27.202260+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:47:27.202260+0000","last_peered":"2024-09-05T16:47:27.202260+0000","last_clean":"2024-09-05T16:47:27.202260+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:47:27.202260+0000","last_undegraded":"2024-09-05T16:47:27.202260+0000","last_fullsized":"2024-09-05T16:47:27.202260+0000","mapping_epoch":38,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1865,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12752","reported_seq":17389,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-05T16:47:27.203921+0000","last_change":"2024-09-05T16:46:51.992073+0000","last_active":"2024-09-05T16:47:27.203921+0000","last_peered":"2024-09-05T16:47:27.203921+0000","last_clean":"2024-09-05T16:47:27.203921+0000","last_became_active":"2024-09-05T16:46:51.667128+0000","last_became_peered":"2024-09-05T16:46:51.667128+0000","last_unstale":"2024-09-05T16:47:27.203921+0000","last_undegraded":"2024-09-05T16:47:27.203921+0000","last_fullsized":"2024-09-05T16:47:27.203921+0000","mapping_epoch":44,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12752","last_scrub_stamp":"2024-09-05T16:46:51.991995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:51.991995+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:04:54.120031+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12934","reported_seq":17693,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-05T16:47:27.201626+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:47:27.201626+0000","last_peered":"2024-09-05T16:47:27.201626+0000","last_clean":"2024-09-05T16:47:27.201626+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:47:27.201626+0000","last_undegraded":"2024-09-05T16:47:27.201626+0000","last_fullsized":"2024-09-05T16:47:27.201626+0000","mapping_epoch":34,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7234,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12593","reported_seq":12722,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:27.199110+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:47:27.199110+0000","last_peered":"2024-09-05T16:47:27.199110+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:47:27.199110+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12591,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2143}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12772","reported_seq":17364,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-05T16:47:27.201640+0000","last_change":"2024-09-05T16:47:15.832530+0000","last_active":"2024-09-05T16:47:27.201640+0000","last_peered":"2024-09-05T16:47:27.201640+0000","last_clean":"2024-09-05T16:47:27.201640+0000","last_became_active":"2024-09-05T16:47:15.728759+0000","last_became_peered":"2024-09-05T16:47:15.728759+0000","last_unstale":"2024-09-05T16:47:27.201640+0000","last_undegraded":"2024-09-05T16:47:27.201640+0000","last_fullsized":"2024-09-05T16:47:27.201640+0000","mapping_epoch":48,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T02:39:11.515339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12772,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2250,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12660","reported_seq":17082,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-05T16:47:27.198250+0000","last_change":"2024-09-05T16:47:03.683348+0000","last_active":"2024-09-05T16:47:27.198250+0000","last_peered":"2024-09-05T16:47:27.198250+0000","last_clean":"2024-09-05T16:47:27.198250+0000","last_became_active":"2024-09-05T16:47:03.185143+0000","last_became_peered":"2024-09-05T16:47:03.185143+0000","last_unstale":"2024-09-05T16:47:27.198250+0000","last_undegraded":"2024-09-05T16:47:27.198250+0000","last_fullsized":"2024-09-05T16:47:27.198250+0000","mapping_epoch":46,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12660","last_scrub_stamp":"2024-09-05T16:47:03.683262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:03.683262+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:42:16.533299+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12777","reported_seq":12910,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-05T16:47:27.202241+0000","last_change":"2024-09-05T16:46:40.458315+0000","last_active":"2024-09-05T16:47:27.202241+0000","last_peered":"2024-09-05T16:47:27.202241+0000","last_clean":"2024-09-05T16:47:27.202241+0000","last_became_active":"2024-09-05T16:46:40.184288+0000","last_became_peered":"2024-09-05T16:46:40.184288+0000","last_unstale":"2024-09-05T16:47:27.202241+0000","last_undegraded":"2024-09-05T16:47:27.202241+0000","last_fullsized":"2024-09-05T16:47:27.202241+0000","mapping_epoch":42,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12777","last_scrub_stamp":"2024-09-05T16:46:40.458226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:40.458226+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:38:58.734210+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'12770","reported_seq":12901,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:27.200583+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:47:27.200583+0000","last_peered":"2024-09-05T16:47:27.200583+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:47:27.200583+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12767,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2240}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12808","reported_seq":12960,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:27.203991+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:47:27.203991+0000","last_peered":"2024-09-05T16:47:27.203991+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:47:27.203991+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12806,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2243}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13120","reported_seq":13277,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:27.199253+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:47:27.199253+0000","last_peered":"2024-09-05T16:47:27.199253+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:47:27.199253+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13117,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2278}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12758","reported_seq":12887,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:27.199725+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:47:27.199725+0000","last_peered":"2024-09-05T16:47:27.199725+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:47:27.199725+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2237}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12808","reported_seq":17574,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-05T16:47:27.201730+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:47:27.201730+0000","last_peered":"2024-09-05T16:47:27.201730+0000","last_clean":"2024-09-05T16:47:27.201730+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:47:27.201730+0000","last_undegraded":"2024-09-05T16:47:27.201730+0000","last_fullsized":"2024-09-05T16:47:27.201730+0000","mapping_epoch":36,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4408,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13124","reported_seq":13263,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:27.200430+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:47:27.200430+0000","last_peered":"2024-09-05T16:47:27.200430+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:47:27.200430+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13122,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2332}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12831","reported_seq":14659,"reported_epoch":51,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:31.018028+0000","last_change":"2024-09-05T16:47:26.179484+0000","last_active":"2024-09-05T16:47:31.018028+0000","last_peered":"2024-09-05T16:47:31.018028+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:47:31.018028+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12829,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":856,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1466}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'12773","reported_seq":12910,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-05T16:47:27.437356+0000","last_change":"2024-09-05T16:47:27.437275+0000","last_active":"2024-09-05T16:47:27.437356+0000","last_peered":"2024-09-05T16:47:27.437356+0000","last_clean":"2024-09-05T16:47:27.437356+0000","last_became_active":"2024-09-05T16:47:27.204989+0000","last_became_peered":"2024-09-05T16:47:27.204989+0000","last_unstale":"2024-09-05T16:47:27.437356+0000","last_undegraded":"2024-09-05T16:47:27.437356+0000","last_fullsized":"2024-09-05T16:47:27.437356+0000","mapping_epoch":50,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12773","last_scrub_stamp":"2024-09-05T16:47:27.437187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:27.437187+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T16:53:55.961513+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":107776,"num_objects":16,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14968,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129172,"num_write_kb":468880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5511,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":69632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084449,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722108,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649732,"statfs":{"total":96636764160,"available":95897325568,"internally_reserved":0,"allocated":430080,"data_stored":162103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":738979539},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986327,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":609704,"kb_used_data":472,"kb_used_omap":4,"kb_used_meta":609211,"kb_avail":93762136,"statfs":{"total":96636764160,"available":96012427264,"internally_reserved":0,"allocated":483328,"data_stored":215351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":623832786},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673133,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713272,"kb_used_data":420,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":93658568,"statfs":{"total":96636764160,"available":95906373632,"internally_reserved":0,"allocated":430080,"data_stored":162103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":729938373},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182564,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713724,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658116,"statfs":{"total":96636764160,"available":95905910784,"internally_reserved":0,"allocated":430080,"data_stored":162103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":730394317},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":57344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:47:33.186 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:47:33.489 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:47:33.490 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:47:33.502 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":470,"stamp":"2024-09-05T16:47:33.108720+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":107776,"num_objects":16,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14968,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129172,"num_write_kb":468880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5511,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2758808,"kb_used_data":1732,"kb_used_omap":14,"kb_used_meta":2756977,"kb_avail":374728552,"statfs":{"total":386547056640,"available":383722037248,"internally_reserved":0,"allocated":1773568,"data_stored":701660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2823145015},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13472,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1880,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12771,"num_write_kb":-51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-361,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001445"},"pg_stats":[{"pgid":"1.f","version":"39'12904","reported_seq":17520,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-05T16:47:27.199630+0000","last_change":"2024-09-05T16:46:27.826222+0000","last_active":"2024-09-05T16:47:27.199630+0000","last_peered":"2024-09-05T16:47:27.199630+0000","last_clean":"2024-09-05T16:47:27.199630+0000","last_became_active":"2024-09-05T16:46:27.757928+0000","last_became_peered":"2024-09-05T16:46:27.757928+0000","last_unstale":"2024-09-05T16:47:27.199630+0000","last_undegraded":"2024-09-05T16:47:27.199630+0000","last_fullsized":"2024-09-05T16:47:27.199630+0000","mapping_epoch":40,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:32:35.713428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12904,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2262,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12552","reported_seq":17233,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-05T16:47:27.202260+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:47:27.202260+0000","last_peered":"2024-09-05T16:47:27.202260+0000","last_clean":"2024-09-05T16:47:27.202260+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:47:27.202260+0000","last_undegraded":"2024-09-05T16:47:27.202260+0000","last_fullsized":"2024-09-05T16:47:27.202260+0000","mapping_epoch":38,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1865,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12752","reported_seq":17389,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-05T16:47:27.203921+0000","last_change":"2024-09-05T16:46:51.992073+0000","last_active":"2024-09-05T16:47:27.203921+0000","last_peered":"2024-09-05T16:47:27.203921+0000","last_clean":"2024-09-05T16:47:27.203921+0000","last_became_active":"2024-09-05T16:46:51.667128+0000","last_became_peered":"2024-09-05T16:46:51.667128+0000","last_unstale":"2024-09-05T16:47:27.203921+0000","last_undegraded":"2024-09-05T16:47:27.203921+0000","last_fullsized":"2024-09-05T16:47:27.203921+0000","mapping_epoch":44,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12752","last_scrub_stamp":"2024-09-05T16:46:51.991995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:51.991995+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:04:54.120031+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12934","reported_seq":17693,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-05T16:47:27.201626+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:47:27.201626+0000","last_peered":"2024-09-05T16:47:27.201626+0000","last_clean":"2024-09-05T16:47:27.201626+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:47:27.201626+0000","last_undegraded":"2024-09-05T16:47:27.201626+0000","last_fullsized":"2024-09-05T16:47:27.201626+0000","mapping_epoch":34,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7234,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12593","reported_seq":12722,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:27.199110+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:47:27.199110+0000","last_peered":"2024-09-05T16:47:27.199110+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:47:27.199110+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12591,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2143}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12772","reported_seq":17364,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-05T16:47:27.201640+0000","last_change":"2024-09-05T16:47:15.832530+0000","last_active":"2024-09-05T16:47:27.201640+0000","last_peered":"2024-09-05T16:47:27.201640+0000","last_clean":"2024-09-05T16:47:27.201640+0000","last_became_active":"2024-09-05T16:47:15.728759+0000","last_became_peered":"2024-09-05T16:47:15.728759+0000","last_unstale":"2024-09-05T16:47:27.201640+0000","last_undegraded":"2024-09-05T16:47:27.201640+0000","last_fullsized":"2024-09-05T16:47:27.201640+0000","mapping_epoch":48,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T02:39:11.515339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12772,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2250,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12660","reported_seq":17082,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-05T16:47:27.198250+0000","last_change":"2024-09-05T16:47:03.683348+0000","last_active":"2024-09-05T16:47:27.198250+0000","last_peered":"2024-09-05T16:47:27.198250+0000","last_clean":"2024-09-05T16:47:27.198250+0000","last_became_active":"2024-09-05T16:47:03.185143+0000","last_became_peered":"2024-09-05T16:47:03.185143+0000","last_unstale":"2024-09-05T16:47:27.198250+0000","last_undegraded":"2024-09-05T16:47:27.198250+0000","last_fullsized":"2024-09-05T16:47:27.198250+0000","mapping_epoch":46,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12660","last_scrub_stamp":"2024-09-05T16:47:03.683262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:03.683262+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:42:16.533299+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12777","reported_seq":12910,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-05T16:47:27.202241+0000","last_change":"2024-09-05T16:46:40.458315+0000","last_active":"2024-09-05T16:47:27.202241+0000","last_peered":"2024-09-05T16:47:27.202241+0000","last_clean":"2024-09-05T16:47:27.202241+0000","last_became_active":"2024-09-05T16:46:40.184288+0000","last_became_peered":"2024-09-05T16:46:40.184288+0000","last_unstale":"2024-09-05T16:47:27.202241+0000","last_undegraded":"2024-09-05T16:47:27.202241+0000","last_fullsized":"2024-09-05T16:47:27.202241+0000","mapping_epoch":42,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12777","last_scrub_stamp":"2024-09-05T16:46:40.458226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:40.458226+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:38:58.734210+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'12770","reported_seq":12901,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:27.200583+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:47:27.200583+0000","last_peered":"2024-09-05T16:47:27.200583+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:47:27.200583+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12767,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2240}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12808","reported_seq":12960,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:27.203991+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:47:27.203991+0000","last_peered":"2024-09-05T16:47:27.203991+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:47:27.203991+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12806,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2243}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13120","reported_seq":13277,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:27.199253+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:47:27.199253+0000","last_peered":"2024-09-05T16:47:27.199253+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:47:27.199253+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13117,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2278}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12758","reported_seq":12887,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:27.199725+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:47:27.199725+0000","last_peered":"2024-09-05T16:47:27.199725+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:47:27.199725+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2237}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12808","reported_seq":17574,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-05T16:47:27.201730+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:47:27.201730+0000","last_peered":"2024-09-05T16:47:27.201730+0000","last_clean":"2024-09-05T16:47:27.201730+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:47:27.201730+0000","last_undegraded":"2024-09-05T16:47:27.201730+0000","last_fullsized":"2024-09-05T16:47:27.201730+0000","mapping_epoch":36,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4408,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13124","reported_seq":13263,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:27.200430+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:47:27.200430+0000","last_peered":"2024-09-05T16:47:27.200430+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:47:27.200430+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13122,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2332}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12831","reported_seq":14659,"reported_epoch":51,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:31.018028+0000","last_change":"2024-09-05T16:47:26.179484+0000","last_active":"2024-09-05T16:47:31.018028+0000","last_peered":"2024-09-05T16:47:31.018028+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:47:31.018028+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12829,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":856,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1466}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'12773","reported_seq":12910,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-05T16:47:27.437356+0000","last_change":"2024-09-05T16:47:27.437275+0000","last_active":"2024-09-05T16:47:27.437356+0000","last_peered":"2024-09-05T16:47:27.437356+0000","last_clean":"2024-09-05T16:47:27.437356+0000","last_became_active":"2024-09-05T16:47:27.204989+0000","last_became_peered":"2024-09-05T16:47:27.204989+0000","last_unstale":"2024-09-05T16:47:27.437356+0000","last_undegraded":"2024-09-05T16:47:27.437356+0000","last_fullsized":"2024-09-05T16:47:27.437356+0000","mapping_epoch":50,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12773","last_scrub_stamp":"2024-09-05T16:47:27.437187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:27.437187+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T16:53:55.961513+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":107776,"num_objects":16,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14968,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129172,"num_write_kb":468880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5511,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":69632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084449,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722108,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649732,"statfs":{"total":96636764160,"available":95897325568,"internally_reserved":0,"allocated":430080,"data_stored":162103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":738979539},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986327,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":609704,"kb_used_data":472,"kb_used_omap":4,"kb_used_meta":609211,"kb_avail":93762136,"statfs":{"total":96636764160,"available":96012427264,"internally_reserved":0,"allocated":483328,"data_stored":215351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":623832786},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673133,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713272,"kb_used_data":420,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":93658568,"statfs":{"total":96636764160,"available":95906373632,"internally_reserved":0,"allocated":430080,"data_stored":162103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":729938373},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182564,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713724,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658116,"statfs":{"total":96636764160,"available":95905910784,"internally_reserved":0,"allocated":430080,"data_stored":162103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":730394317},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":57344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:47:33.503 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:47:33.503 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "39'12593", 'reported_seq': 12722, 'reported_epoch': 51, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:47:27.199110+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:47:27.199110+0000', 'last_peered': '2024-09-05T16:47:27.199110+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:47:27.199110+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "33'2500", 'ondisk_log_start': "33'2500", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10093, 'log_dups_size': 0, 'ondisk_log_size': 10093, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2147, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12591, 'num_write_kb': 50373, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2143}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:47:33.503 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:47:33.503 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "39'12770", 'reported_seq': 12901, 'reported_epoch': 51, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:47:27.200583+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:47:27.200583+0000', 'last_peered': '2024-09-05T16:47:27.200583+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:47:27.200583+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "33'2700", 'ondisk_log_start': "33'2700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10070, 'log_dups_size': 0, 'ondisk_log_size': 10070, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26944, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 16, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12767, 'num_write_kb': 51081, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2240}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:47:33.503 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:47:33.503 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "39'12808", 'reported_seq': 12960, 'reported_epoch': 51, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:47:27.203991+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:47:27.203991+0000', 'last_peered': '2024-09-05T16:47:27.203991+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:47:27.203991+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "33'2800", 'ondisk_log_start': "33'2800", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10008, 'log_dups_size': 0, 'ondisk_log_size': 10008, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12806, 'num_write_kb': 51230, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -2243}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:47:33.503 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:47:33.503 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "39'13120", 'reported_seq': 13277, 'reported_epoch': 51, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:47:27.199253+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:47:27.199253+0000', 'last_peered': '2024-09-05T16:47:27.199253+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:47:27.199253+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "33'3100", 'ondisk_log_start': "33'3100", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20208, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2284, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13117, 'num_write_kb': 52476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2278}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:47:33.504 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:47:33.504 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "39'12758", 'reported_seq': 12887, 'reported_epoch': 51, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:47:27.199725+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:47:27.199725+0000', 'last_peered': '2024-09-05T16:47:27.199725+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:47:27.199725+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "33'2700", 'ondisk_log_start': "33'2700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10058, 'log_dups_size': 0, 'ondisk_log_size': 10058, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6736, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12757, 'num_write_kb': 51031, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2237}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:47:33.504 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:47:33.504 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "39'13124", 'reported_seq': 13263, 'reported_epoch': 51, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:47:27.200430+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:47:27.200430+0000', 'last_peered': '2024-09-05T16:47:27.200430+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:47:27.200430+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "33'3100", 'ondisk_log_start': "33'3100", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10024, 'log_dups_size': 0, 'ondisk_log_size': 10024, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13122, 'num_write_kb': 52500, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2332}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:47:33.504 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:47:33.504 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "39'12831", 'reported_seq': 14659, 'reported_epoch': 51, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:47:31.018028+0000', 'last_change': '2024-09-05T16:47:26.179484+0000', 'last_active': '2024-09-05T16:47:31.018028+0000', 'last_peered': '2024-09-05T16:47:31.018028+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:47:31.018028+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "33'2800", 'ondisk_log_start': "33'2800", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10031, 'log_dups_size': 0, 'ondisk_log_size': 10031, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1468, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12829, 'num_write_kb': 51325, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 856, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1466}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:47:33.504 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:47:33.879 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:47:33.880 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 14968/64 objects degraded (23387.500%), 7 pgs degraded, 7 pgs undersized","count":14},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":857,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":51,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":7},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":9},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":6},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":16,"data_bytes":107776,"bytes_used":2825019392,"bytes_avail":383722037248,"bytes_total":386547056640,"degraded_objects":14968,"degraded_total":64,"degraded_ratio":233.875},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:47:33.893 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-09-05T16:47:33.894 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:47:34.203 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:47:34.204 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:47:34.217 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":470,"stamp":"2024-09-05T16:47:33.108720+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":107776,"num_objects":16,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14968,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129172,"num_write_kb":468880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5511,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2758808,"kb_used_data":1732,"kb_used_omap":14,"kb_used_meta":2756977,"kb_avail":374728552,"statfs":{"total":386547056640,"available":383722037248,"internally_reserved":0,"allocated":1773568,"data_stored":701660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2823145015},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13472,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1880,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12771,"num_write_kb":-51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-361,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001445"},"pg_stats":[{"pgid":"1.f","version":"39'12904","reported_seq":17520,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-05T16:47:27.199630+0000","last_change":"2024-09-05T16:46:27.826222+0000","last_active":"2024-09-05T16:47:27.199630+0000","last_peered":"2024-09-05T16:47:27.199630+0000","last_clean":"2024-09-05T16:47:27.199630+0000","last_became_active":"2024-09-05T16:46:27.757928+0000","last_became_peered":"2024-09-05T16:46:27.757928+0000","last_unstale":"2024-09-05T16:47:27.199630+0000","last_undegraded":"2024-09-05T16:47:27.199630+0000","last_fullsized":"2024-09-05T16:47:27.199630+0000","mapping_epoch":40,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:32:35.713428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12904,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2262,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12552","reported_seq":17233,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-05T16:47:27.202260+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:47:27.202260+0000","last_peered":"2024-09-05T16:47:27.202260+0000","last_clean":"2024-09-05T16:47:27.202260+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:47:27.202260+0000","last_undegraded":"2024-09-05T16:47:27.202260+0000","last_fullsized":"2024-09-05T16:47:27.202260+0000","mapping_epoch":38,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1865,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12752","reported_seq":17389,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-05T16:47:27.203921+0000","last_change":"2024-09-05T16:46:51.992073+0000","last_active":"2024-09-05T16:47:27.203921+0000","last_peered":"2024-09-05T16:47:27.203921+0000","last_clean":"2024-09-05T16:47:27.203921+0000","last_became_active":"2024-09-05T16:46:51.667128+0000","last_became_peered":"2024-09-05T16:46:51.667128+0000","last_unstale":"2024-09-05T16:47:27.203921+0000","last_undegraded":"2024-09-05T16:47:27.203921+0000","last_fullsized":"2024-09-05T16:47:27.203921+0000","mapping_epoch":44,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12752","last_scrub_stamp":"2024-09-05T16:46:51.991995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:51.991995+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:04:54.120031+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12934","reported_seq":17693,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-05T16:47:27.201626+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:47:27.201626+0000","last_peered":"2024-09-05T16:47:27.201626+0000","last_clean":"2024-09-05T16:47:27.201626+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:47:27.201626+0000","last_undegraded":"2024-09-05T16:47:27.201626+0000","last_fullsized":"2024-09-05T16:47:27.201626+0000","mapping_epoch":34,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7234,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12593","reported_seq":12722,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:27.199110+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:47:27.199110+0000","last_peered":"2024-09-05T16:47:27.199110+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:47:27.199110+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12591,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2143}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12772","reported_seq":17364,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-05T16:47:27.201640+0000","last_change":"2024-09-05T16:47:15.832530+0000","last_active":"2024-09-05T16:47:27.201640+0000","last_peered":"2024-09-05T16:47:27.201640+0000","last_clean":"2024-09-05T16:47:27.201640+0000","last_became_active":"2024-09-05T16:47:15.728759+0000","last_became_peered":"2024-09-05T16:47:15.728759+0000","last_unstale":"2024-09-05T16:47:27.201640+0000","last_undegraded":"2024-09-05T16:47:27.201640+0000","last_fullsized":"2024-09-05T16:47:27.201640+0000","mapping_epoch":48,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T02:39:11.515339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12772,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2250,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12660","reported_seq":17082,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-05T16:47:27.198250+0000","last_change":"2024-09-05T16:47:03.683348+0000","last_active":"2024-09-05T16:47:27.198250+0000","last_peered":"2024-09-05T16:47:27.198250+0000","last_clean":"2024-09-05T16:47:27.198250+0000","last_became_active":"2024-09-05T16:47:03.185143+0000","last_became_peered":"2024-09-05T16:47:03.185143+0000","last_unstale":"2024-09-05T16:47:27.198250+0000","last_undegraded":"2024-09-05T16:47:27.198250+0000","last_fullsized":"2024-09-05T16:47:27.198250+0000","mapping_epoch":46,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12660","last_scrub_stamp":"2024-09-05T16:47:03.683262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:03.683262+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:42:16.533299+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12777","reported_seq":12910,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-05T16:47:27.202241+0000","last_change":"2024-09-05T16:46:40.458315+0000","last_active":"2024-09-05T16:47:27.202241+0000","last_peered":"2024-09-05T16:47:27.202241+0000","last_clean":"2024-09-05T16:47:27.202241+0000","last_became_active":"2024-09-05T16:46:40.184288+0000","last_became_peered":"2024-09-05T16:46:40.184288+0000","last_unstale":"2024-09-05T16:47:27.202241+0000","last_undegraded":"2024-09-05T16:47:27.202241+0000","last_fullsized":"2024-09-05T16:47:27.202241+0000","mapping_epoch":42,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12777","last_scrub_stamp":"2024-09-05T16:46:40.458226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:40.458226+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:38:58.734210+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'12770","reported_seq":12901,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:27.200583+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:47:27.200583+0000","last_peered":"2024-09-05T16:47:27.200583+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:47:27.200583+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12767,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2240}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12808","reported_seq":12960,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:27.203991+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:47:27.203991+0000","last_peered":"2024-09-05T16:47:27.203991+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:47:27.203991+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12806,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2243}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13120","reported_seq":13277,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:27.199253+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:47:27.199253+0000","last_peered":"2024-09-05T16:47:27.199253+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:47:27.199253+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13117,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2278}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12758","reported_seq":12887,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:27.199725+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:47:27.199725+0000","last_peered":"2024-09-05T16:47:27.199725+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:47:27.199725+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2237}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12808","reported_seq":17574,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-05T16:47:27.201730+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:47:27.201730+0000","last_peered":"2024-09-05T16:47:27.201730+0000","last_clean":"2024-09-05T16:47:27.201730+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:47:27.201730+0000","last_undegraded":"2024-09-05T16:47:27.201730+0000","last_fullsized":"2024-09-05T16:47:27.201730+0000","mapping_epoch":36,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4408,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13124","reported_seq":13263,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:27.200430+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:47:27.200430+0000","last_peered":"2024-09-05T16:47:27.200430+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:47:27.200430+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13122,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2332}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12831","reported_seq":14659,"reported_epoch":51,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:31.018028+0000","last_change":"2024-09-05T16:47:26.179484+0000","last_active":"2024-09-05T16:47:31.018028+0000","last_peered":"2024-09-05T16:47:31.018028+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:47:31.018028+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12829,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":856,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1466}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'12773","reported_seq":12910,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-05T16:47:27.437356+0000","last_change":"2024-09-05T16:47:27.437275+0000","last_active":"2024-09-05T16:47:27.437356+0000","last_peered":"2024-09-05T16:47:27.437356+0000","last_clean":"2024-09-05T16:47:27.437356+0000","last_became_active":"2024-09-05T16:47:27.204989+0000","last_became_peered":"2024-09-05T16:47:27.204989+0000","last_unstale":"2024-09-05T16:47:27.437356+0000","last_undegraded":"2024-09-05T16:47:27.437356+0000","last_fullsized":"2024-09-05T16:47:27.437356+0000","mapping_epoch":50,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12773","last_scrub_stamp":"2024-09-05T16:47:27.437187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:27.437187+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T16:53:55.961513+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":107776,"num_objects":16,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14968,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129172,"num_write_kb":468880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5511,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":69632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084449,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722108,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649732,"statfs":{"total":96636764160,"available":95897325568,"internally_reserved":0,"allocated":430080,"data_stored":162103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":738979539},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986327,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":609704,"kb_used_data":472,"kb_used_omap":4,"kb_used_meta":609211,"kb_avail":93762136,"statfs":{"total":96636764160,"available":96012427264,"internally_reserved":0,"allocated":483328,"data_stored":215351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":623832786},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673133,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713272,"kb_used_data":420,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":93658568,"statfs":{"total":96636764160,"available":95906373632,"internally_reserved":0,"allocated":430080,"data_stored":162103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":729938373},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182564,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713724,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658116,"statfs":{"total":96636764160,"available":95905910784,"internally_reserved":0,"allocated":430080,"data_stored":162103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":730394317},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":57344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:47:37.218 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:47:37.527 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:47:37.527 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:47:37.539 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":472,"stamp":"2024-09-05T16:47:37.109372+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":107776,"num_objects":16,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129172,"num_write_kb":468880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6424,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2758808,"kb_used_data":1732,"kb_used_omap":14,"kb_used_meta":2756977,"kb_avail":374728552,"statfs":{"total":386547056640,"available":383722037248,"internally_reserved":0,"allocated":1773568,"data_stored":701660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2823145015},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13472,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2793,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12771,"num_write_kb":-51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":552,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001708"},"pg_stats":[{"pgid":"1.f","version":"39'12904","reported_seq":17520,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-05T16:47:27.199630+0000","last_change":"2024-09-05T16:46:27.826222+0000","last_active":"2024-09-05T16:47:27.199630+0000","last_peered":"2024-09-05T16:47:27.199630+0000","last_clean":"2024-09-05T16:47:27.199630+0000","last_became_active":"2024-09-05T16:46:27.757928+0000","last_became_peered":"2024-09-05T16:46:27.757928+0000","last_unstale":"2024-09-05T16:47:27.199630+0000","last_undegraded":"2024-09-05T16:47:27.199630+0000","last_fullsized":"2024-09-05T16:47:27.199630+0000","mapping_epoch":40,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:32:35.713428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12904,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2262,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12552","reported_seq":17233,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-05T16:47:27.202260+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:47:27.202260+0000","last_peered":"2024-09-05T16:47:27.202260+0000","last_clean":"2024-09-05T16:47:27.202260+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:47:27.202260+0000","last_undegraded":"2024-09-05T16:47:27.202260+0000","last_fullsized":"2024-09-05T16:47:27.202260+0000","mapping_epoch":38,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1865,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12752","reported_seq":17389,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-05T16:47:27.203921+0000","last_change":"2024-09-05T16:46:51.992073+0000","last_active":"2024-09-05T16:47:27.203921+0000","last_peered":"2024-09-05T16:47:27.203921+0000","last_clean":"2024-09-05T16:47:27.203921+0000","last_became_active":"2024-09-05T16:46:51.667128+0000","last_became_peered":"2024-09-05T16:46:51.667128+0000","last_unstale":"2024-09-05T16:47:27.203921+0000","last_undegraded":"2024-09-05T16:47:27.203921+0000","last_fullsized":"2024-09-05T16:47:27.203921+0000","mapping_epoch":44,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12752","last_scrub_stamp":"2024-09-05T16:46:51.991995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:51.991995+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:04:54.120031+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12934","reported_seq":17693,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-05T16:47:27.201626+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:47:27.201626+0000","last_peered":"2024-09-05T16:47:27.201626+0000","last_clean":"2024-09-05T16:47:27.201626+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:47:27.201626+0000","last_undegraded":"2024-09-05T16:47:27.201626+0000","last_fullsized":"2024-09-05T16:47:27.201626+0000","mapping_epoch":34,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7234,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12593","reported_seq":12722,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:27.199110+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:47:27.199110+0000","last_peered":"2024-09-05T16:47:27.199110+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:47:27.199110+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12591,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2143}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12772","reported_seq":17364,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-05T16:47:27.201640+0000","last_change":"2024-09-05T16:47:15.832530+0000","last_active":"2024-09-05T16:47:27.201640+0000","last_peered":"2024-09-05T16:47:27.201640+0000","last_clean":"2024-09-05T16:47:27.201640+0000","last_became_active":"2024-09-05T16:47:15.728759+0000","last_became_peered":"2024-09-05T16:47:15.728759+0000","last_unstale":"2024-09-05T16:47:27.201640+0000","last_undegraded":"2024-09-05T16:47:27.201640+0000","last_fullsized":"2024-09-05T16:47:27.201640+0000","mapping_epoch":48,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T02:39:11.515339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12772,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2250,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12660","reported_seq":17082,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-05T16:47:27.198250+0000","last_change":"2024-09-05T16:47:03.683348+0000","last_active":"2024-09-05T16:47:27.198250+0000","last_peered":"2024-09-05T16:47:27.198250+0000","last_clean":"2024-09-05T16:47:27.198250+0000","last_became_active":"2024-09-05T16:47:03.185143+0000","last_became_peered":"2024-09-05T16:47:03.185143+0000","last_unstale":"2024-09-05T16:47:27.198250+0000","last_undegraded":"2024-09-05T16:47:27.198250+0000","last_fullsized":"2024-09-05T16:47:27.198250+0000","mapping_epoch":46,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12660","last_scrub_stamp":"2024-09-05T16:47:03.683262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:03.683262+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:42:16.533299+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12777","reported_seq":12910,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-05T16:47:27.202241+0000","last_change":"2024-09-05T16:46:40.458315+0000","last_active":"2024-09-05T16:47:27.202241+0000","last_peered":"2024-09-05T16:47:27.202241+0000","last_clean":"2024-09-05T16:47:27.202241+0000","last_became_active":"2024-09-05T16:46:40.184288+0000","last_became_peered":"2024-09-05T16:46:40.184288+0000","last_unstale":"2024-09-05T16:47:27.202241+0000","last_undegraded":"2024-09-05T16:47:27.202241+0000","last_fullsized":"2024-09-05T16:47:27.202241+0000","mapping_epoch":42,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12777","last_scrub_stamp":"2024-09-05T16:46:40.458226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:40.458226+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:38:58.734210+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'12770","reported_seq":12901,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:27.200583+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:47:27.200583+0000","last_peered":"2024-09-05T16:47:27.200583+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:47:27.200583+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12767,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2240}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12808","reported_seq":12960,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:27.203991+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:47:27.203991+0000","last_peered":"2024-09-05T16:47:27.203991+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:47:27.203991+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12806,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2243}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13120","reported_seq":13277,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:27.199253+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:47:27.199253+0000","last_peered":"2024-09-05T16:47:27.199253+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:47:27.199253+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13117,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2278}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12758","reported_seq":12887,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:27.199725+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:47:27.199725+0000","last_peered":"2024-09-05T16:47:27.199725+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:47:27.199725+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2237}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12808","reported_seq":17574,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-05T16:47:27.201730+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:47:27.201730+0000","last_peered":"2024-09-05T16:47:27.201730+0000","last_clean":"2024-09-05T16:47:27.201730+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:47:27.201730+0000","last_undegraded":"2024-09-05T16:47:27.201730+0000","last_fullsized":"2024-09-05T16:47:27.201730+0000","mapping_epoch":36,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4408,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13124","reported_seq":13263,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:27.200430+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:47:27.200430+0000","last_peered":"2024-09-05T16:47:27.200430+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:47:27.200430+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13122,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2332}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12831","reported_seq":16485,"reported_epoch":51,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:36.020459+0000","last_change":"2024-09-05T16:47:26.179484+0000","last_active":"2024-09-05T16:47:36.020459+0000","last_peered":"2024-09-05T16:47:36.020459+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:47:36.020459+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":555,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12829,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1769,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-553}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'12773","reported_seq":12910,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-05T16:47:27.437356+0000","last_change":"2024-09-05T16:47:27.437275+0000","last_active":"2024-09-05T16:47:27.437356+0000","last_peered":"2024-09-05T16:47:27.437356+0000","last_clean":"2024-09-05T16:47:27.437356+0000","last_became_active":"2024-09-05T16:47:27.204989+0000","last_became_peered":"2024-09-05T16:47:27.204989+0000","last_unstale":"2024-09-05T16:47:27.437356+0000","last_undegraded":"2024-09-05T16:47:27.437356+0000","last_fullsized":"2024-09-05T16:47:27.437356+0000","mapping_epoch":50,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12773","last_scrub_stamp":"2024-09-05T16:47:27.437187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:27.437187+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T16:53:55.961513+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":107776,"num_objects":16,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129172,"num_write_kb":468880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6424,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":69632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084450,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722108,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649732,"statfs":{"total":96636764160,"available":95897325568,"internally_reserved":0,"allocated":430080,"data_stored":162103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":738979539},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986327,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":609704,"kb_used_data":472,"kb_used_omap":4,"kb_used_meta":609211,"kb_avail":93762136,"statfs":{"total":96636764160,"available":96012427264,"internally_reserved":0,"allocated":483328,"data_stored":215351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":623832786},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673134,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713272,"kb_used_data":420,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":93658568,"statfs":{"total":96636764160,"available":95906373632,"internally_reserved":0,"allocated":430080,"data_stored":162103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":729938373},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182565,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713724,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658116,"statfs":{"total":96636764160,"available":95905910784,"internally_reserved":0,"allocated":430080,"data_stored":162103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":730394317},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":57344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:47:37.541 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:47:37.844 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:47:37.844 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:47:37.856 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":472,"stamp":"2024-09-05T16:47:37.109372+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":107776,"num_objects":16,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129172,"num_write_kb":468880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6424,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2758808,"kb_used_data":1732,"kb_used_omap":14,"kb_used_meta":2756977,"kb_avail":374728552,"statfs":{"total":386547056640,"available":383722037248,"internally_reserved":0,"allocated":1773568,"data_stored":701660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2823145015},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13472,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2793,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12771,"num_write_kb":-51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":552,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001708"},"pg_stats":[{"pgid":"1.f","version":"39'12904","reported_seq":17520,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-05T16:47:27.199630+0000","last_change":"2024-09-05T16:46:27.826222+0000","last_active":"2024-09-05T16:47:27.199630+0000","last_peered":"2024-09-05T16:47:27.199630+0000","last_clean":"2024-09-05T16:47:27.199630+0000","last_became_active":"2024-09-05T16:46:27.757928+0000","last_became_peered":"2024-09-05T16:46:27.757928+0000","last_unstale":"2024-09-05T16:47:27.199630+0000","last_undegraded":"2024-09-05T16:47:27.199630+0000","last_fullsized":"2024-09-05T16:47:27.199630+0000","mapping_epoch":40,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:32:35.713428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12904,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2262,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12552","reported_seq":17233,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-05T16:47:27.202260+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:47:27.202260+0000","last_peered":"2024-09-05T16:47:27.202260+0000","last_clean":"2024-09-05T16:47:27.202260+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:47:27.202260+0000","last_undegraded":"2024-09-05T16:47:27.202260+0000","last_fullsized":"2024-09-05T16:47:27.202260+0000","mapping_epoch":38,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1865,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12752","reported_seq":17389,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-05T16:47:27.203921+0000","last_change":"2024-09-05T16:46:51.992073+0000","last_active":"2024-09-05T16:47:27.203921+0000","last_peered":"2024-09-05T16:47:27.203921+0000","last_clean":"2024-09-05T16:47:27.203921+0000","last_became_active":"2024-09-05T16:46:51.667128+0000","last_became_peered":"2024-09-05T16:46:51.667128+0000","last_unstale":"2024-09-05T16:47:27.203921+0000","last_undegraded":"2024-09-05T16:47:27.203921+0000","last_fullsized":"2024-09-05T16:47:27.203921+0000","mapping_epoch":44,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12752","last_scrub_stamp":"2024-09-05T16:46:51.991995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:51.991995+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:04:54.120031+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12934","reported_seq":17693,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-05T16:47:27.201626+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:47:27.201626+0000","last_peered":"2024-09-05T16:47:27.201626+0000","last_clean":"2024-09-05T16:47:27.201626+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:47:27.201626+0000","last_undegraded":"2024-09-05T16:47:27.201626+0000","last_fullsized":"2024-09-05T16:47:27.201626+0000","mapping_epoch":34,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7234,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12593","reported_seq":12722,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:27.199110+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:47:27.199110+0000","last_peered":"2024-09-05T16:47:27.199110+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:47:27.199110+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12591,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2143}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12772","reported_seq":17364,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-05T16:47:27.201640+0000","last_change":"2024-09-05T16:47:15.832530+0000","last_active":"2024-09-05T16:47:27.201640+0000","last_peered":"2024-09-05T16:47:27.201640+0000","last_clean":"2024-09-05T16:47:27.201640+0000","last_became_active":"2024-09-05T16:47:15.728759+0000","last_became_peered":"2024-09-05T16:47:15.728759+0000","last_unstale":"2024-09-05T16:47:27.201640+0000","last_undegraded":"2024-09-05T16:47:27.201640+0000","last_fullsized":"2024-09-05T16:47:27.201640+0000","mapping_epoch":48,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T02:39:11.515339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12772,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2250,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12660","reported_seq":17082,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-05T16:47:27.198250+0000","last_change":"2024-09-05T16:47:03.683348+0000","last_active":"2024-09-05T16:47:27.198250+0000","last_peered":"2024-09-05T16:47:27.198250+0000","last_clean":"2024-09-05T16:47:27.198250+0000","last_became_active":"2024-09-05T16:47:03.185143+0000","last_became_peered":"2024-09-05T16:47:03.185143+0000","last_unstale":"2024-09-05T16:47:27.198250+0000","last_undegraded":"2024-09-05T16:47:27.198250+0000","last_fullsized":"2024-09-05T16:47:27.198250+0000","mapping_epoch":46,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12660","last_scrub_stamp":"2024-09-05T16:47:03.683262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:03.683262+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:42:16.533299+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12777","reported_seq":12910,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-05T16:47:27.202241+0000","last_change":"2024-09-05T16:46:40.458315+0000","last_active":"2024-09-05T16:47:27.202241+0000","last_peered":"2024-09-05T16:47:27.202241+0000","last_clean":"2024-09-05T16:47:27.202241+0000","last_became_active":"2024-09-05T16:46:40.184288+0000","last_became_peered":"2024-09-05T16:46:40.184288+0000","last_unstale":"2024-09-05T16:47:27.202241+0000","last_undegraded":"2024-09-05T16:47:27.202241+0000","last_fullsized":"2024-09-05T16:47:27.202241+0000","mapping_epoch":42,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12777","last_scrub_stamp":"2024-09-05T16:46:40.458226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:40.458226+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:38:58.734210+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'12770","reported_seq":12901,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:27.200583+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:47:27.200583+0000","last_peered":"2024-09-05T16:47:27.200583+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:47:27.200583+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12767,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2240}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12808","reported_seq":12960,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:27.203991+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:47:27.203991+0000","last_peered":"2024-09-05T16:47:27.203991+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:47:27.203991+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12806,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2243}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13120","reported_seq":13277,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:27.199253+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:47:27.199253+0000","last_peered":"2024-09-05T16:47:27.199253+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:47:27.199253+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13117,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2278}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12758","reported_seq":12887,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:27.199725+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:47:27.199725+0000","last_peered":"2024-09-05T16:47:27.199725+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:47:27.199725+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2237}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12808","reported_seq":17574,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-05T16:47:27.201730+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:47:27.201730+0000","last_peered":"2024-09-05T16:47:27.201730+0000","last_clean":"2024-09-05T16:47:27.201730+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:47:27.201730+0000","last_undegraded":"2024-09-05T16:47:27.201730+0000","last_fullsized":"2024-09-05T16:47:27.201730+0000","mapping_epoch":36,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4408,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13124","reported_seq":13263,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:27.200430+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:47:27.200430+0000","last_peered":"2024-09-05T16:47:27.200430+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:47:27.200430+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13122,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2332}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12831","reported_seq":16485,"reported_epoch":51,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:36.020459+0000","last_change":"2024-09-05T16:47:26.179484+0000","last_active":"2024-09-05T16:47:36.020459+0000","last_peered":"2024-09-05T16:47:36.020459+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:47:36.020459+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":555,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12829,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1769,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-553}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'12773","reported_seq":12910,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-05T16:47:27.437356+0000","last_change":"2024-09-05T16:47:27.437275+0000","last_active":"2024-09-05T16:47:27.437356+0000","last_peered":"2024-09-05T16:47:27.437356+0000","last_clean":"2024-09-05T16:47:27.437356+0000","last_became_active":"2024-09-05T16:47:27.204989+0000","last_became_peered":"2024-09-05T16:47:27.204989+0000","last_unstale":"2024-09-05T16:47:27.437356+0000","last_undegraded":"2024-09-05T16:47:27.437356+0000","last_fullsized":"2024-09-05T16:47:27.437356+0000","mapping_epoch":50,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12773","last_scrub_stamp":"2024-09-05T16:47:27.437187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:27.437187+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T16:53:55.961513+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":107776,"num_objects":16,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129172,"num_write_kb":468880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6424,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":69632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084450,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722108,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649732,"statfs":{"total":96636764160,"available":95897325568,"internally_reserved":0,"allocated":430080,"data_stored":162103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":738979539},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986327,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":609704,"kb_used_data":472,"kb_used_omap":4,"kb_used_meta":609211,"kb_avail":93762136,"statfs":{"total":96636764160,"available":96012427264,"internally_reserved":0,"allocated":483328,"data_stored":215351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":623832786},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673134,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713272,"kb_used_data":420,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":93658568,"statfs":{"total":96636764160,"available":95906373632,"internally_reserved":0,"allocated":430080,"data_stored":162103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":729938373},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182565,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713724,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658116,"statfs":{"total":96636764160,"available":95905910784,"internally_reserved":0,"allocated":430080,"data_stored":162103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":730394317},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":57344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:47:37.857 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:47:37.857 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "39'12593", 'reported_seq': 12722, 'reported_epoch': 51, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:47:27.199110+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:47:27.199110+0000', 'last_peered': '2024-09-05T16:47:27.199110+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:47:27.199110+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "33'2500", 'ondisk_log_start': "33'2500", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10093, 'log_dups_size': 0, 'ondisk_log_size': 10093, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2147, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12591, 'num_write_kb': 50373, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2143}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:47:37.857 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:47:37.857 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "39'12770", 'reported_seq': 12901, 'reported_epoch': 51, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:47:27.200583+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:47:27.200583+0000', 'last_peered': '2024-09-05T16:47:27.200583+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:47:27.200583+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "33'2700", 'ondisk_log_start': "33'2700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10070, 'log_dups_size': 0, 'ondisk_log_size': 10070, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26944, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 16, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12767, 'num_write_kb': 51081, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2240}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:47:37.858 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:47:37.858 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "39'12808", 'reported_seq': 12960, 'reported_epoch': 51, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:47:27.203991+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:47:27.203991+0000', 'last_peered': '2024-09-05T16:47:27.203991+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:47:27.203991+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "33'2800", 'ondisk_log_start': "33'2800", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10008, 'log_dups_size': 0, 'ondisk_log_size': 10008, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12806, 'num_write_kb': 51230, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -2243}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:47:37.858 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:47:37.858 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "39'13120", 'reported_seq': 13277, 'reported_epoch': 51, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:47:27.199253+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:47:27.199253+0000', 'last_peered': '2024-09-05T16:47:27.199253+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:47:27.199253+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "33'3100", 'ondisk_log_start': "33'3100", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20208, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2284, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13117, 'num_write_kb': 52476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2278}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:47:37.858 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:47:37.858 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "39'12758", 'reported_seq': 12887, 'reported_epoch': 51, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:47:27.199725+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:47:27.199725+0000', 'last_peered': '2024-09-05T16:47:27.199725+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:47:27.199725+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "33'2700", 'ondisk_log_start': "33'2700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10058, 'log_dups_size': 0, 'ondisk_log_size': 10058, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6736, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12757, 'num_write_kb': 51031, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2237}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:47:37.858 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:47:37.858 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "39'13124", 'reported_seq': 13263, 'reported_epoch': 51, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:47:27.200430+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:47:27.200430+0000', 'last_peered': '2024-09-05T16:47:27.200430+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:47:27.200430+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "33'3100", 'ondisk_log_start': "33'3100", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10024, 'log_dups_size': 0, 'ondisk_log_size': 10024, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13122, 'num_write_kb': 52500, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2332}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:47:37.858 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:47:37.858 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "39'12831", 'reported_seq': 16485, 'reported_epoch': 51, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:47:36.020459+0000', 'last_change': '2024-09-05T16:47:26.179484+0000', 'last_active': '2024-09-05T16:47:36.020459+0000', 'last_peered': '2024-09-05T16:47:36.020459+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:47:36.020459+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "33'2800", 'ondisk_log_start': "33'2800", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10031, 'log_dups_size': 0, 'ondisk_log_size': 10031, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 555, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12829, 'num_write_kb': 51325, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1769, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -553}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:47:37.859 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:47:38.235 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:47:38.236 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 14055/64 objects degraded (21960.938%), 7 pgs degraded, 7 pgs undersized","count":14},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":861,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":51,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":7},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":9},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":6},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":16,"data_bytes":107776,"bytes_used":2825019392,"bytes_avail":383722037248,"bytes_total":386547056640,"degraded_objects":14055,"degraded_total":64,"degraded_ratio":219.609375,"recovering_objects_per_sec":55,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":552,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:47:38.249 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:47:38.249 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:47:38.561 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:47:38.562 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:47:38.575 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":472,"stamp":"2024-09-05T16:47:37.109372+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":107776,"num_objects":16,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129172,"num_write_kb":468880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6424,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2758808,"kb_used_data":1732,"kb_used_omap":14,"kb_used_meta":2756977,"kb_avail":374728552,"statfs":{"total":386547056640,"available":383722037248,"internally_reserved":0,"allocated":1773568,"data_stored":701660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2823145015},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13472,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2793,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12771,"num_write_kb":-51092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":552,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001708"},"pg_stats":[{"pgid":"1.f","version":"39'12904","reported_seq":17520,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-05T16:47:27.199630+0000","last_change":"2024-09-05T16:46:27.826222+0000","last_active":"2024-09-05T16:47:27.199630+0000","last_peered":"2024-09-05T16:47:27.199630+0000","last_clean":"2024-09-05T16:47:27.199630+0000","last_became_active":"2024-09-05T16:46:27.757928+0000","last_became_peered":"2024-09-05T16:46:27.757928+0000","last_unstale":"2024-09-05T16:47:27.199630+0000","last_undegraded":"2024-09-05T16:47:27.199630+0000","last_fullsized":"2024-09-05T16:47:27.199630+0000","mapping_epoch":40,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:32:35.713428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12904,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2262,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12552","reported_seq":17233,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-05T16:47:27.202260+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:47:27.202260+0000","last_peered":"2024-09-05T16:47:27.202260+0000","last_clean":"2024-09-05T16:47:27.202260+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:47:27.202260+0000","last_undegraded":"2024-09-05T16:47:27.202260+0000","last_fullsized":"2024-09-05T16:47:27.202260+0000","mapping_epoch":38,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1865,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12752","reported_seq":17389,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-05T16:47:27.203921+0000","last_change":"2024-09-05T16:46:51.992073+0000","last_active":"2024-09-05T16:47:27.203921+0000","last_peered":"2024-09-05T16:47:27.203921+0000","last_clean":"2024-09-05T16:47:27.203921+0000","last_became_active":"2024-09-05T16:46:51.667128+0000","last_became_peered":"2024-09-05T16:46:51.667128+0000","last_unstale":"2024-09-05T16:47:27.203921+0000","last_undegraded":"2024-09-05T16:47:27.203921+0000","last_fullsized":"2024-09-05T16:47:27.203921+0000","mapping_epoch":44,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12752","last_scrub_stamp":"2024-09-05T16:46:51.991995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:51.991995+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:04:54.120031+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12934","reported_seq":17693,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-05T16:47:27.201626+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:47:27.201626+0000","last_peered":"2024-09-05T16:47:27.201626+0000","last_clean":"2024-09-05T16:47:27.201626+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:47:27.201626+0000","last_undegraded":"2024-09-05T16:47:27.201626+0000","last_fullsized":"2024-09-05T16:47:27.201626+0000","mapping_epoch":34,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7234,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12593","reported_seq":12722,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:27.199110+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:47:27.199110+0000","last_peered":"2024-09-05T16:47:27.199110+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:47:27.199110+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12591,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2143}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12772","reported_seq":17364,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-05T16:47:27.201640+0000","last_change":"2024-09-05T16:47:15.832530+0000","last_active":"2024-09-05T16:47:27.201640+0000","last_peered":"2024-09-05T16:47:27.201640+0000","last_clean":"2024-09-05T16:47:27.201640+0000","last_became_active":"2024-09-05T16:47:15.728759+0000","last_became_peered":"2024-09-05T16:47:15.728759+0000","last_unstale":"2024-09-05T16:47:27.201640+0000","last_undegraded":"2024-09-05T16:47:27.201640+0000","last_fullsized":"2024-09-05T16:47:27.201640+0000","mapping_epoch":48,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T02:39:11.515339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12772,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2250,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12660","reported_seq":17082,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-05T16:47:27.198250+0000","last_change":"2024-09-05T16:47:03.683348+0000","last_active":"2024-09-05T16:47:27.198250+0000","last_peered":"2024-09-05T16:47:27.198250+0000","last_clean":"2024-09-05T16:47:27.198250+0000","last_became_active":"2024-09-05T16:47:03.185143+0000","last_became_peered":"2024-09-05T16:47:03.185143+0000","last_unstale":"2024-09-05T16:47:27.198250+0000","last_undegraded":"2024-09-05T16:47:27.198250+0000","last_fullsized":"2024-09-05T16:47:27.198250+0000","mapping_epoch":46,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12660","last_scrub_stamp":"2024-09-05T16:47:03.683262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:03.683262+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:42:16.533299+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12777","reported_seq":12910,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-05T16:47:27.202241+0000","last_change":"2024-09-05T16:46:40.458315+0000","last_active":"2024-09-05T16:47:27.202241+0000","last_peered":"2024-09-05T16:47:27.202241+0000","last_clean":"2024-09-05T16:47:27.202241+0000","last_became_active":"2024-09-05T16:46:40.184288+0000","last_became_peered":"2024-09-05T16:46:40.184288+0000","last_unstale":"2024-09-05T16:47:27.202241+0000","last_undegraded":"2024-09-05T16:47:27.202241+0000","last_fullsized":"2024-09-05T16:47:27.202241+0000","mapping_epoch":42,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12777","last_scrub_stamp":"2024-09-05T16:46:40.458226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:40.458226+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:38:58.734210+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'12770","reported_seq":12901,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:27.200583+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:47:27.200583+0000","last_peered":"2024-09-05T16:47:27.200583+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:47:27.200583+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12767,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2240}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12808","reported_seq":12960,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:27.203991+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:47:27.203991+0000","last_peered":"2024-09-05T16:47:27.203991+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:47:27.203991+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12806,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2243}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13120","reported_seq":13277,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:27.199253+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:47:27.199253+0000","last_peered":"2024-09-05T16:47:27.199253+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:47:27.199253+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13117,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2278}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12758","reported_seq":12887,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:27.199725+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:47:27.199725+0000","last_peered":"2024-09-05T16:47:27.199725+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:47:27.199725+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2237}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12808","reported_seq":17574,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-05T16:47:27.201730+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:47:27.201730+0000","last_peered":"2024-09-05T16:47:27.201730+0000","last_clean":"2024-09-05T16:47:27.201730+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:47:27.201730+0000","last_undegraded":"2024-09-05T16:47:27.201730+0000","last_fullsized":"2024-09-05T16:47:27.201730+0000","mapping_epoch":36,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4408,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13124","reported_seq":13263,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:27.200430+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:47:27.200430+0000","last_peered":"2024-09-05T16:47:27.200430+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:47:27.200430+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13122,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2332}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12831","reported_seq":16485,"reported_epoch":51,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:36.020459+0000","last_change":"2024-09-05T16:47:26.179484+0000","last_active":"2024-09-05T16:47:36.020459+0000","last_peered":"2024-09-05T16:47:36.020459+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:47:36.020459+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":555,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12829,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1769,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-553}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'12773","reported_seq":12910,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-05T16:47:27.437356+0000","last_change":"2024-09-05T16:47:27.437275+0000","last_active":"2024-09-05T16:47:27.437356+0000","last_peered":"2024-09-05T16:47:27.437356+0000","last_clean":"2024-09-05T16:47:27.437356+0000","last_became_active":"2024-09-05T16:47:27.204989+0000","last_became_peered":"2024-09-05T16:47:27.204989+0000","last_unstale":"2024-09-05T16:47:27.437356+0000","last_undegraded":"2024-09-05T16:47:27.437356+0000","last_fullsized":"2024-09-05T16:47:27.437356+0000","mapping_epoch":50,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12773","last_scrub_stamp":"2024-09-05T16:47:27.437187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:27.437187+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T16:53:55.961513+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":107776,"num_objects":16,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129172,"num_write_kb":468880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6424,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":69632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084450,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722108,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649732,"statfs":{"total":96636764160,"available":95897325568,"internally_reserved":0,"allocated":430080,"data_stored":162103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":738979539},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986327,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":609704,"kb_used_data":472,"kb_used_omap":4,"kb_used_meta":609211,"kb_avail":93762136,"statfs":{"total":96636764160,"available":96012427264,"internally_reserved":0,"allocated":483328,"data_stored":215351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":623832786},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673134,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713272,"kb_used_data":420,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":93658568,"statfs":{"total":96636764160,"available":95906373632,"internally_reserved":0,"allocated":430080,"data_stored":162103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":729938373},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182565,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713724,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658116,"statfs":{"total":96636764160,"available":95905910784,"internally_reserved":0,"allocated":430080,"data_stored":162103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":730394317},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":57344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:47:41.578 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:47:41.900 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:47:41.901 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:47:41.913 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":476,"stamp":"2024-09-05T16:47:41.110100+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":107776,"num_objects":16,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129172,"num_write_kb":468880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6424,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2767136,"kb_used_data":1740,"kb_used_omap":14,"kb_used_meta":2765297,"kb_avail":374720224,"statfs":{"total":386547056640,"available":383713509376,"internally_reserved":0,"allocated":1781760,"data_stored":704950,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2831664695},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-913,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":913,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001380"},"pg_stats":[{"pgid":"1.f","version":"39'12904","reported_seq":17520,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-05T16:47:27.199630+0000","last_change":"2024-09-05T16:46:27.826222+0000","last_active":"2024-09-05T16:47:27.199630+0000","last_peered":"2024-09-05T16:47:27.199630+0000","last_clean":"2024-09-05T16:47:27.199630+0000","last_became_active":"2024-09-05T16:46:27.757928+0000","last_became_peered":"2024-09-05T16:46:27.757928+0000","last_unstale":"2024-09-05T16:47:27.199630+0000","last_undegraded":"2024-09-05T16:47:27.199630+0000","last_fullsized":"2024-09-05T16:47:27.199630+0000","mapping_epoch":40,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:32:35.713428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12904,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2262,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12552","reported_seq":17233,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-05T16:47:27.202260+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:47:27.202260+0000","last_peered":"2024-09-05T16:47:27.202260+0000","last_clean":"2024-09-05T16:47:27.202260+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:47:27.202260+0000","last_undegraded":"2024-09-05T16:47:27.202260+0000","last_fullsized":"2024-09-05T16:47:27.202260+0000","mapping_epoch":38,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1865,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12752","reported_seq":17393,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-05T16:47:40.183661+0000","last_change":"2024-09-05T16:46:51.992073+0000","last_active":"2024-09-05T16:47:40.183661+0000","last_peered":"2024-09-05T16:47:40.183661+0000","last_clean":"2024-09-05T16:47:40.183661+0000","last_became_active":"2024-09-05T16:46:51.667128+0000","last_became_peered":"2024-09-05T16:46:51.667128+0000","last_unstale":"2024-09-05T16:47:40.183661+0000","last_undegraded":"2024-09-05T16:47:40.183661+0000","last_fullsized":"2024-09-05T16:47:40.183661+0000","mapping_epoch":44,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12752","last_scrub_stamp":"2024-09-05T16:46:51.991995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:51.991995+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:04:54.120031+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12934","reported_seq":17693,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-05T16:47:27.201626+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:47:27.201626+0000","last_peered":"2024-09-05T16:47:27.201626+0000","last_clean":"2024-09-05T16:47:27.201626+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:47:27.201626+0000","last_undegraded":"2024-09-05T16:47:27.201626+0000","last_fullsized":"2024-09-05T16:47:27.201626+0000","mapping_epoch":34,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7234,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12593","reported_seq":12722,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:27.199110+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:47:27.199110+0000","last_peered":"2024-09-05T16:47:27.199110+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:47:27.199110+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12591,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2143}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12772","reported_seq":17364,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-05T16:47:27.201640+0000","last_change":"2024-09-05T16:47:15.832530+0000","last_active":"2024-09-05T16:47:27.201640+0000","last_peered":"2024-09-05T16:47:27.201640+0000","last_clean":"2024-09-05T16:47:27.201640+0000","last_became_active":"2024-09-05T16:47:15.728759+0000","last_became_peered":"2024-09-05T16:47:15.728759+0000","last_unstale":"2024-09-05T16:47:27.201640+0000","last_undegraded":"2024-09-05T16:47:27.201640+0000","last_fullsized":"2024-09-05T16:47:27.201640+0000","mapping_epoch":48,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T02:39:11.515339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12772,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2250,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12660","reported_seq":17082,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-05T16:47:27.198250+0000","last_change":"2024-09-05T16:47:03.683348+0000","last_active":"2024-09-05T16:47:27.198250+0000","last_peered":"2024-09-05T16:47:27.198250+0000","last_clean":"2024-09-05T16:47:27.198250+0000","last_became_active":"2024-09-05T16:47:03.185143+0000","last_became_peered":"2024-09-05T16:47:03.185143+0000","last_unstale":"2024-09-05T16:47:27.198250+0000","last_undegraded":"2024-09-05T16:47:27.198250+0000","last_fullsized":"2024-09-05T16:47:27.198250+0000","mapping_epoch":46,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12660","last_scrub_stamp":"2024-09-05T16:47:03.683262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:03.683262+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:42:16.533299+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12777","reported_seq":12910,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-05T16:47:27.202241+0000","last_change":"2024-09-05T16:46:40.458315+0000","last_active":"2024-09-05T16:47:27.202241+0000","last_peered":"2024-09-05T16:47:27.202241+0000","last_clean":"2024-09-05T16:47:27.202241+0000","last_became_active":"2024-09-05T16:46:40.184288+0000","last_became_peered":"2024-09-05T16:46:40.184288+0000","last_unstale":"2024-09-05T16:47:27.202241+0000","last_undegraded":"2024-09-05T16:47:27.202241+0000","last_fullsized":"2024-09-05T16:47:27.202241+0000","mapping_epoch":42,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12777","last_scrub_stamp":"2024-09-05T16:46:40.458226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:40.458226+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:38:58.734210+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'12770","reported_seq":12901,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:27.200583+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:47:27.200583+0000","last_peered":"2024-09-05T16:47:27.200583+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:47:27.200583+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12767,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2240}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12808","reported_seq":12964,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:40.183779+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:47:40.183779+0000","last_peered":"2024-09-05T16:47:40.183779+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:47:40.183779+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12806,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2243}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13120","reported_seq":13277,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:27.199253+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:47:27.199253+0000","last_peered":"2024-09-05T16:47:27.199253+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:47:27.199253+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13117,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2278}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12758","reported_seq":12887,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:27.199725+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:47:27.199725+0000","last_peered":"2024-09-05T16:47:27.199725+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:47:27.199725+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2237}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12808","reported_seq":17574,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-05T16:47:27.201730+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:47:27.201730+0000","last_peered":"2024-09-05T16:47:27.201730+0000","last_clean":"2024-09-05T16:47:27.201730+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:47:27.201730+0000","last_undegraded":"2024-09-05T16:47:27.201730+0000","last_fullsized":"2024-09-05T16:47:27.201730+0000","mapping_epoch":36,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4408,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13124","reported_seq":13263,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:27.200430+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:47:27.200430+0000","last_peered":"2024-09-05T16:47:27.200430+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:47:27.200430+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13122,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2332}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12831","reported_seq":16485,"reported_epoch":51,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:36.020459+0000","last_change":"2024-09-05T16:47:26.179484+0000","last_active":"2024-09-05T16:47:36.020459+0000","last_peered":"2024-09-05T16:47:36.020459+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:47:36.020459+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":555,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12829,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1769,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-553}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'12773","reported_seq":12910,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-05T16:47:27.437356+0000","last_change":"2024-09-05T16:47:27.437275+0000","last_active":"2024-09-05T16:47:27.437356+0000","last_peered":"2024-09-05T16:47:27.437356+0000","last_clean":"2024-09-05T16:47:27.437356+0000","last_became_active":"2024-09-05T16:47:27.204989+0000","last_became_peered":"2024-09-05T16:47:27.204989+0000","last_unstale":"2024-09-05T16:47:27.437356+0000","last_undegraded":"2024-09-05T16:47:27.437356+0000","last_fullsized":"2024-09-05T16:47:27.437356+0000","mapping_epoch":50,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12773","last_scrub_stamp":"2024-09-05T16:47:27.437187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:27.437187+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T16:53:55.961513+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":107776,"num_objects":16,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129172,"num_write_kb":468880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6424,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":69632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084451,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722108,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649732,"statfs":{"total":96636764160,"available":95897325568,"internally_reserved":0,"allocated":430080,"data_stored":162103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":738979539},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986328,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":618024,"kb_used_data":472,"kb_used_omap":4,"kb_used_meta":617531,"kb_avail":93753816,"statfs":{"total":96636764160,"available":96003907584,"internally_reserved":0,"allocated":483328,"data_stored":215351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":632352466},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673135,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713280,"kb_used_data":428,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":93658560,"statfs":{"total":96636764160,"available":95906365440,"internally_reserved":0,"allocated":438272,"data_stored":165393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":729938373},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182565,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713724,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658116,"statfs":{"total":96636764160,"available":95905910784,"internally_reserved":0,"allocated":430080,"data_stored":162103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":730394317},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":57344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:47:41.914 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:47:42.246 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:47:42.247 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:47:42.259 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":476,"stamp":"2024-09-05T16:47:41.110100+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":107776,"num_objects":16,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129172,"num_write_kb":468880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6424,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2767136,"kb_used_data":1740,"kb_used_omap":14,"kb_used_meta":2765297,"kb_avail":374720224,"statfs":{"total":386547056640,"available":383713509376,"internally_reserved":0,"allocated":1781760,"data_stored":704950,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2831664695},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-913,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":913,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001380"},"pg_stats":[{"pgid":"1.f","version":"39'12904","reported_seq":17520,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-05T16:47:27.199630+0000","last_change":"2024-09-05T16:46:27.826222+0000","last_active":"2024-09-05T16:47:27.199630+0000","last_peered":"2024-09-05T16:47:27.199630+0000","last_clean":"2024-09-05T16:47:27.199630+0000","last_became_active":"2024-09-05T16:46:27.757928+0000","last_became_peered":"2024-09-05T16:46:27.757928+0000","last_unstale":"2024-09-05T16:47:27.199630+0000","last_undegraded":"2024-09-05T16:47:27.199630+0000","last_fullsized":"2024-09-05T16:47:27.199630+0000","mapping_epoch":40,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:32:35.713428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12904,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2262,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12552","reported_seq":17233,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-05T16:47:27.202260+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:47:27.202260+0000","last_peered":"2024-09-05T16:47:27.202260+0000","last_clean":"2024-09-05T16:47:27.202260+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:47:27.202260+0000","last_undegraded":"2024-09-05T16:47:27.202260+0000","last_fullsized":"2024-09-05T16:47:27.202260+0000","mapping_epoch":38,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1865,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12752","reported_seq":17393,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-05T16:47:40.183661+0000","last_change":"2024-09-05T16:46:51.992073+0000","last_active":"2024-09-05T16:47:40.183661+0000","last_peered":"2024-09-05T16:47:40.183661+0000","last_clean":"2024-09-05T16:47:40.183661+0000","last_became_active":"2024-09-05T16:46:51.667128+0000","last_became_peered":"2024-09-05T16:46:51.667128+0000","last_unstale":"2024-09-05T16:47:40.183661+0000","last_undegraded":"2024-09-05T16:47:40.183661+0000","last_fullsized":"2024-09-05T16:47:40.183661+0000","mapping_epoch":44,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12752","last_scrub_stamp":"2024-09-05T16:46:51.991995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:51.991995+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:04:54.120031+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12934","reported_seq":17693,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-05T16:47:27.201626+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:47:27.201626+0000","last_peered":"2024-09-05T16:47:27.201626+0000","last_clean":"2024-09-05T16:47:27.201626+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:47:27.201626+0000","last_undegraded":"2024-09-05T16:47:27.201626+0000","last_fullsized":"2024-09-05T16:47:27.201626+0000","mapping_epoch":34,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7234,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12593","reported_seq":12722,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:27.199110+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:47:27.199110+0000","last_peered":"2024-09-05T16:47:27.199110+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:47:27.199110+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12591,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2143}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12772","reported_seq":17364,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-05T16:47:27.201640+0000","last_change":"2024-09-05T16:47:15.832530+0000","last_active":"2024-09-05T16:47:27.201640+0000","last_peered":"2024-09-05T16:47:27.201640+0000","last_clean":"2024-09-05T16:47:27.201640+0000","last_became_active":"2024-09-05T16:47:15.728759+0000","last_became_peered":"2024-09-05T16:47:15.728759+0000","last_unstale":"2024-09-05T16:47:27.201640+0000","last_undegraded":"2024-09-05T16:47:27.201640+0000","last_fullsized":"2024-09-05T16:47:27.201640+0000","mapping_epoch":48,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T02:39:11.515339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12772,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2250,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12660","reported_seq":17082,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-05T16:47:27.198250+0000","last_change":"2024-09-05T16:47:03.683348+0000","last_active":"2024-09-05T16:47:27.198250+0000","last_peered":"2024-09-05T16:47:27.198250+0000","last_clean":"2024-09-05T16:47:27.198250+0000","last_became_active":"2024-09-05T16:47:03.185143+0000","last_became_peered":"2024-09-05T16:47:03.185143+0000","last_unstale":"2024-09-05T16:47:27.198250+0000","last_undegraded":"2024-09-05T16:47:27.198250+0000","last_fullsized":"2024-09-05T16:47:27.198250+0000","mapping_epoch":46,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12660","last_scrub_stamp":"2024-09-05T16:47:03.683262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:03.683262+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:42:16.533299+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12777","reported_seq":12910,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-05T16:47:27.202241+0000","last_change":"2024-09-05T16:46:40.458315+0000","last_active":"2024-09-05T16:47:27.202241+0000","last_peered":"2024-09-05T16:47:27.202241+0000","last_clean":"2024-09-05T16:47:27.202241+0000","last_became_active":"2024-09-05T16:46:40.184288+0000","last_became_peered":"2024-09-05T16:46:40.184288+0000","last_unstale":"2024-09-05T16:47:27.202241+0000","last_undegraded":"2024-09-05T16:47:27.202241+0000","last_fullsized":"2024-09-05T16:47:27.202241+0000","mapping_epoch":42,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12777","last_scrub_stamp":"2024-09-05T16:46:40.458226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:40.458226+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:38:58.734210+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'12770","reported_seq":12901,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:27.200583+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:47:27.200583+0000","last_peered":"2024-09-05T16:47:27.200583+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:47:27.200583+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12767,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2240}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12808","reported_seq":12964,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:40.183779+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:47:40.183779+0000","last_peered":"2024-09-05T16:47:40.183779+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:47:40.183779+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12806,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2243}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13120","reported_seq":13277,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:27.199253+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:47:27.199253+0000","last_peered":"2024-09-05T16:47:27.199253+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:47:27.199253+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13117,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2278}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12758","reported_seq":12887,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:27.199725+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:47:27.199725+0000","last_peered":"2024-09-05T16:47:27.199725+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:47:27.199725+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2237}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12808","reported_seq":17574,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-05T16:47:27.201730+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:47:27.201730+0000","last_peered":"2024-09-05T16:47:27.201730+0000","last_clean":"2024-09-05T16:47:27.201730+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:47:27.201730+0000","last_undegraded":"2024-09-05T16:47:27.201730+0000","last_fullsized":"2024-09-05T16:47:27.201730+0000","mapping_epoch":36,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4408,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13124","reported_seq":13263,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:27.200430+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:47:27.200430+0000","last_peered":"2024-09-05T16:47:27.200430+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:47:27.200430+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13122,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2332}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12831","reported_seq":16485,"reported_epoch":51,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:36.020459+0000","last_change":"2024-09-05T16:47:26.179484+0000","last_active":"2024-09-05T16:47:36.020459+0000","last_peered":"2024-09-05T16:47:36.020459+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:47:36.020459+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":555,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12829,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1769,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-553}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'12773","reported_seq":12910,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-05T16:47:27.437356+0000","last_change":"2024-09-05T16:47:27.437275+0000","last_active":"2024-09-05T16:47:27.437356+0000","last_peered":"2024-09-05T16:47:27.437356+0000","last_clean":"2024-09-05T16:47:27.437356+0000","last_became_active":"2024-09-05T16:47:27.204989+0000","last_became_peered":"2024-09-05T16:47:27.204989+0000","last_unstale":"2024-09-05T16:47:27.437356+0000","last_undegraded":"2024-09-05T16:47:27.437356+0000","last_fullsized":"2024-09-05T16:47:27.437356+0000","mapping_epoch":50,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12773","last_scrub_stamp":"2024-09-05T16:47:27.437187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:27.437187+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T16:53:55.961513+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":107776,"num_objects":16,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129172,"num_write_kb":468880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6424,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":69632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084451,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722108,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649732,"statfs":{"total":96636764160,"available":95897325568,"internally_reserved":0,"allocated":430080,"data_stored":162103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":738979539},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986328,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":618024,"kb_used_data":472,"kb_used_omap":4,"kb_used_meta":617531,"kb_avail":93753816,"statfs":{"total":96636764160,"available":96003907584,"internally_reserved":0,"allocated":483328,"data_stored":215351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":632352466},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673135,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713280,"kb_used_data":428,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":93658560,"statfs":{"total":96636764160,"available":95906365440,"internally_reserved":0,"allocated":438272,"data_stored":165393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":729938373},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182565,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713724,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658116,"statfs":{"total":96636764160,"available":95905910784,"internally_reserved":0,"allocated":430080,"data_stored":162103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":730394317},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":57344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:47:42.260 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:47:42.260 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "39'12593", 'reported_seq': 12722, 'reported_epoch': 51, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:47:27.199110+0000', 'last_change': '2024-09-05T16:34:56.863669+0000', 'last_active': '2024-09-05T16:47:27.199110+0000', 'last_peered': '2024-09-05T16:47:27.199110+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:47:27.199110+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "33'2500", 'ondisk_log_start': "33'2500", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10093, 'log_dups_size': 0, 'ondisk_log_size': 10093, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2147, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12591, 'num_write_kb': 50373, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2143}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:47:42.260 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:47:42.260 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "39'12770", 'reported_seq': 12901, 'reported_epoch': 51, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:47:27.200583+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:47:27.200583+0000', 'last_peered': '2024-09-05T16:47:27.200583+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:47:27.200583+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "33'2700", 'ondisk_log_start': "33'2700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10070, 'log_dups_size': 0, 'ondisk_log_size': 10070, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26944, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 16, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12767, 'num_write_kb': 51081, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2240}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:47:42.260 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:47:42.260 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "39'12808", 'reported_seq': 12964, 'reported_epoch': 53, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:47:40.183779+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:47:40.183779+0000', 'last_peered': '2024-09-05T16:47:40.183779+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:47:40.183779+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "33'2800", 'ondisk_log_start': "33'2800", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10008, 'log_dups_size': 0, 'ondisk_log_size': 10008, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12806, 'num_write_kb': 51230, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -2243}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:47:42.260 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:47:42.260 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "39'13120", 'reported_seq': 13277, 'reported_epoch': 51, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:47:27.199253+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:47:27.199253+0000', 'last_peered': '2024-09-05T16:47:27.199253+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:47:27.199253+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "33'3100", 'ondisk_log_start': "33'3100", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20208, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2284, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13117, 'num_write_kb': 52476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2278}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:47:42.260 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:47:42.260 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "39'12758", 'reported_seq': 12887, 'reported_epoch': 51, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:47:27.199725+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:47:27.199725+0000', 'last_peered': '2024-09-05T16:47:27.199725+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:47:27.199725+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "33'2700", 'ondisk_log_start': "33'2700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10058, 'log_dups_size': 0, 'ondisk_log_size': 10058, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6736, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12757, 'num_write_kb': 51031, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2237}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:47:42.261 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:47:42.261 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "39'13124", 'reported_seq': 13263, 'reported_epoch': 51, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:47:27.200430+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:47:27.200430+0000', 'last_peered': '2024-09-05T16:47:27.200430+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:47:27.200430+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "33'3100", 'ondisk_log_start': "33'3100", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10024, 'log_dups_size': 0, 'ondisk_log_size': 10024, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13122, 'num_write_kb': 52500, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2332}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:47:42.261 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-05T16:47:42.261 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "39'12831", 'reported_seq': 16485, 'reported_epoch': 51, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:47:36.020459+0000', 'last_change': '2024-09-05T16:47:26.179484+0000', 'last_active': '2024-09-05T16:47:36.020459+0000', 'last_peered': '2024-09-05T16:47:36.020459+0000', 'last_clean': '2024-09-05T16:33:53.657773+0000', 'last_became_active': '2024-09-05T16:34:56.857565+0000', 'last_became_peered': '2024-09-05T16:34:56.857565+0000', 'last_unstale': '2024-09-05T16:47:36.020459+0000', 'last_undegraded': '2024-09-05T16:34:56.599248+0000', 'last_fullsized': '2024-09-05T16:34:56.560504+0000', 'mapping_epoch': 32, 'log_start': "33'2800", 'ondisk_log_start': "33'2800", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10031, 'log_dups_size': 0, 'ondisk_log_size': 10031, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 555, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12829, 'num_write_kb': 51325, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1769, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -553}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:47:42.261 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:47:42.642 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:47:42.642 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 14055/64 objects degraded (21960.938%), 7 pgs degraded, 7 pgs undersized","count":14},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":865,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":53,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":6},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":9},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":6},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":16,"data_bytes":107776,"bytes_used":2833547264,"bytes_avail":383713509376,"bytes_total":386547056640,"degraded_objects":14055,"degraded_total":64,"degraded_ratio":219.609375,"recovering_objects_per_sec":114,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":913,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:47:42.655 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:47:42.655 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:47:42.968 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:47:42.969 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:47:42.982 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":476,"stamp":"2024-09-05T16:47:41.110100+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":107776,"num_objects":16,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129172,"num_write_kb":468880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6424,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2767136,"kb_used_data":1740,"kb_used_omap":14,"kb_used_meta":2765297,"kb_avail":374720224,"statfs":{"total":386547056640,"available":383713509376,"internally_reserved":0,"allocated":1781760,"data_stored":704950,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2831664695},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-913,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":913,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001380"},"pg_stats":[{"pgid":"1.f","version":"39'12904","reported_seq":17520,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-05T16:47:27.199630+0000","last_change":"2024-09-05T16:46:27.826222+0000","last_active":"2024-09-05T16:47:27.199630+0000","last_peered":"2024-09-05T16:47:27.199630+0000","last_clean":"2024-09-05T16:47:27.199630+0000","last_became_active":"2024-09-05T16:46:27.757928+0000","last_became_peered":"2024-09-05T16:46:27.757928+0000","last_unstale":"2024-09-05T16:47:27.199630+0000","last_undegraded":"2024-09-05T16:47:27.199630+0000","last_fullsized":"2024-09-05T16:47:27.199630+0000","mapping_epoch":40,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:32:35.713428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12904,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2262,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12552","reported_seq":17233,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-05T16:47:27.202260+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:47:27.202260+0000","last_peered":"2024-09-05T16:47:27.202260+0000","last_clean":"2024-09-05T16:47:27.202260+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:47:27.202260+0000","last_undegraded":"2024-09-05T16:47:27.202260+0000","last_fullsized":"2024-09-05T16:47:27.202260+0000","mapping_epoch":38,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1865,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12752","reported_seq":17393,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-05T16:47:40.183661+0000","last_change":"2024-09-05T16:46:51.992073+0000","last_active":"2024-09-05T16:47:40.183661+0000","last_peered":"2024-09-05T16:47:40.183661+0000","last_clean":"2024-09-05T16:47:40.183661+0000","last_became_active":"2024-09-05T16:46:51.667128+0000","last_became_peered":"2024-09-05T16:46:51.667128+0000","last_unstale":"2024-09-05T16:47:40.183661+0000","last_undegraded":"2024-09-05T16:47:40.183661+0000","last_fullsized":"2024-09-05T16:47:40.183661+0000","mapping_epoch":44,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12752","last_scrub_stamp":"2024-09-05T16:46:51.991995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:51.991995+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:04:54.120031+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12934","reported_seq":17693,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-05T16:47:27.201626+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:47:27.201626+0000","last_peered":"2024-09-05T16:47:27.201626+0000","last_clean":"2024-09-05T16:47:27.201626+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:47:27.201626+0000","last_undegraded":"2024-09-05T16:47:27.201626+0000","last_fullsized":"2024-09-05T16:47:27.201626+0000","mapping_epoch":34,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7234,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12593","reported_seq":12722,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:27.199110+0000","last_change":"2024-09-05T16:34:56.863669+0000","last_active":"2024-09-05T16:47:27.199110+0000","last_peered":"2024-09-05T16:47:27.199110+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:47:27.199110+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12591,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2143}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12772","reported_seq":17364,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-05T16:47:27.201640+0000","last_change":"2024-09-05T16:47:15.832530+0000","last_active":"2024-09-05T16:47:27.201640+0000","last_peered":"2024-09-05T16:47:27.201640+0000","last_clean":"2024-09-05T16:47:27.201640+0000","last_became_active":"2024-09-05T16:47:15.728759+0000","last_became_peered":"2024-09-05T16:47:15.728759+0000","last_unstale":"2024-09-05T16:47:27.201640+0000","last_undegraded":"2024-09-05T16:47:27.201640+0000","last_fullsized":"2024-09-05T16:47:27.201640+0000","mapping_epoch":48,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T02:39:11.515339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12772,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2250,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12660","reported_seq":17082,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-05T16:47:27.198250+0000","last_change":"2024-09-05T16:47:03.683348+0000","last_active":"2024-09-05T16:47:27.198250+0000","last_peered":"2024-09-05T16:47:27.198250+0000","last_clean":"2024-09-05T16:47:27.198250+0000","last_became_active":"2024-09-05T16:47:03.185143+0000","last_became_peered":"2024-09-05T16:47:03.185143+0000","last_unstale":"2024-09-05T16:47:27.198250+0000","last_undegraded":"2024-09-05T16:47:27.198250+0000","last_fullsized":"2024-09-05T16:47:27.198250+0000","mapping_epoch":46,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12660","last_scrub_stamp":"2024-09-05T16:47:03.683262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:03.683262+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:42:16.533299+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12777","reported_seq":12910,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-05T16:47:27.202241+0000","last_change":"2024-09-05T16:46:40.458315+0000","last_active":"2024-09-05T16:47:27.202241+0000","last_peered":"2024-09-05T16:47:27.202241+0000","last_clean":"2024-09-05T16:47:27.202241+0000","last_became_active":"2024-09-05T16:46:40.184288+0000","last_became_peered":"2024-09-05T16:46:40.184288+0000","last_unstale":"2024-09-05T16:47:27.202241+0000","last_undegraded":"2024-09-05T16:47:27.202241+0000","last_fullsized":"2024-09-05T16:47:27.202241+0000","mapping_epoch":42,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12777","last_scrub_stamp":"2024-09-05T16:46:40.458226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:40.458226+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:38:58.734210+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'12770","reported_seq":12901,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:27.200583+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:47:27.200583+0000","last_peered":"2024-09-05T16:47:27.200583+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:47:27.200583+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12767,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2240}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12808","reported_seq":12964,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:40.183779+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:47:40.183779+0000","last_peered":"2024-09-05T16:47:40.183779+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:47:40.183779+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12806,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2243}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13120","reported_seq":13277,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:27.199253+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:47:27.199253+0000","last_peered":"2024-09-05T16:47:27.199253+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:47:27.199253+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13117,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2278}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12758","reported_seq":12887,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:27.199725+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:47:27.199725+0000","last_peered":"2024-09-05T16:47:27.199725+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:47:27.199725+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2237}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12808","reported_seq":17574,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-05T16:47:27.201730+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:47:27.201730+0000","last_peered":"2024-09-05T16:47:27.201730+0000","last_clean":"2024-09-05T16:47:27.201730+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:47:27.201730+0000","last_undegraded":"2024-09-05T16:47:27.201730+0000","last_fullsized":"2024-09-05T16:47:27.201730+0000","mapping_epoch":36,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4408,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13124","reported_seq":13263,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:27.200430+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:47:27.200430+0000","last_peered":"2024-09-05T16:47:27.200430+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:47:27.200430+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13122,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2332}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12831","reported_seq":16485,"reported_epoch":51,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:36.020459+0000","last_change":"2024-09-05T16:47:26.179484+0000","last_active":"2024-09-05T16:47:36.020459+0000","last_peered":"2024-09-05T16:47:36.020459+0000","last_clean":"2024-09-05T16:33:53.657773+0000","last_became_active":"2024-09-05T16:34:56.857565+0000","last_became_peered":"2024-09-05T16:34:56.857565+0000","last_unstale":"2024-09-05T16:47:36.020459+0000","last_undegraded":"2024-09-05T16:34:56.599248+0000","last_fullsized":"2024-09-05T16:34:56.560504+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":555,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12829,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1769,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-553}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"39'12773","reported_seq":12910,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-05T16:47:27.437356+0000","last_change":"2024-09-05T16:47:27.437275+0000","last_active":"2024-09-05T16:47:27.437356+0000","last_peered":"2024-09-05T16:47:27.437356+0000","last_clean":"2024-09-05T16:47:27.437356+0000","last_became_active":"2024-09-05T16:47:27.204989+0000","last_became_peered":"2024-09-05T16:47:27.204989+0000","last_unstale":"2024-09-05T16:47:27.437356+0000","last_undegraded":"2024-09-05T16:47:27.437356+0000","last_fullsized":"2024-09-05T16:47:27.437356+0000","mapping_epoch":50,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12773","last_scrub_stamp":"2024-09-05T16:47:27.437187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:27.437187+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T16:53:55.961513+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":107776,"num_objects":16,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129172,"num_write_kb":468880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6424,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":69632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084451,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722108,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649732,"statfs":{"total":96636764160,"available":95897325568,"internally_reserved":0,"allocated":430080,"data_stored":162103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":738979539},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986328,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":618024,"kb_used_data":472,"kb_used_omap":4,"kb_used_meta":617531,"kb_avail":93753816,"statfs":{"total":96636764160,"available":96003907584,"internally_reserved":0,"allocated":483328,"data_stored":215351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":632352466},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673135,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713280,"kb_used_data":428,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":93658560,"statfs":{"total":96636764160,"available":95906365440,"internally_reserved":0,"allocated":438272,"data_stored":165393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":729938373},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182565,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713724,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658116,"statfs":{"total":96636764160,"available":95905910784,"internally_reserved":0,"allocated":430080,"data_stored":162103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":730394317},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":57344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:47:45.984 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:47:46.291 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:47:46.291 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:47:46.303 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":478,"stamp":"2024-09-05T16:47:45.110735+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":94304,"num_objects":14,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12761,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116343,"num_write_kb":417555,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5394,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2758856,"kb_used_data":1780,"kb_used_omap":14,"kb_used_meta":2756977,"kb_avail":374728504,"statfs":{"total":386547056640,"available":383721988096,"internally_reserved":0,"allocated":1822720,"data_stored":716435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2823145015},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13472,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12829,"num_write_kb":-51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1030,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001363"},"pg_stats":[{"pgid":"1.f","version":"39'12904","reported_seq":17524,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-05T16:47:40.182534+0000","last_change":"2024-09-05T16:46:27.826222+0000","last_active":"2024-09-05T16:47:40.182534+0000","last_peered":"2024-09-05T16:47:40.182534+0000","last_clean":"2024-09-05T16:47:40.182534+0000","last_became_active":"2024-09-05T16:46:27.757928+0000","last_became_peered":"2024-09-05T16:46:27.757928+0000","last_unstale":"2024-09-05T16:47:40.182534+0000","last_undegraded":"2024-09-05T16:47:40.182534+0000","last_fullsized":"2024-09-05T16:47:40.182534+0000","mapping_epoch":40,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:32:35.713428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12904,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2262,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12552","reported_seq":17237,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-05T16:47:40.185571+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:47:40.185571+0000","last_peered":"2024-09-05T16:47:40.185571+0000","last_clean":"2024-09-05T16:47:40.185571+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:47:40.185571+0000","last_undegraded":"2024-09-05T16:47:40.185571+0000","last_fullsized":"2024-09-05T16:47:40.185571+0000","mapping_epoch":38,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1865,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12752","reported_seq":17393,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-05T16:47:40.183661+0000","last_change":"2024-09-05T16:46:51.992073+0000","last_active":"2024-09-05T16:47:40.183661+0000","last_peered":"2024-09-05T16:47:40.183661+0000","last_clean":"2024-09-05T16:47:40.183661+0000","last_became_active":"2024-09-05T16:46:51.667128+0000","last_became_peered":"2024-09-05T16:46:51.667128+0000","last_unstale":"2024-09-05T16:47:40.183661+0000","last_undegraded":"2024-09-05T16:47:40.183661+0000","last_fullsized":"2024-09-05T16:47:40.183661+0000","mapping_epoch":44,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12752","last_scrub_stamp":"2024-09-05T16:46:51.991995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:51.991995+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:04:54.120031+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12934","reported_seq":17698,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-05T16:47:40.185167+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:47:40.185167+0000","last_peered":"2024-09-05T16:47:40.185167+0000","last_clean":"2024-09-05T16:47:40.185167+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:47:40.185167+0000","last_undegraded":"2024-09-05T16:47:40.185167+0000","last_fullsized":"2024-09-05T16:47:40.185167+0000","mapping_epoch":34,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7234,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12593","reported_seq":14205,"reported_epoch":53,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:43.204834+0000","last_change":"2024-09-05T16:47:38.413103+0000","last_active":"2024-09-05T16:47:43.204834+0000","last_peered":"2024-09-05T16:47:43.204834+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:47:43.204834+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1408,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12591,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":758,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1406}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12772","reported_seq":17368,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-05T16:47:40.182854+0000","last_change":"2024-09-05T16:47:15.832530+0000","last_active":"2024-09-05T16:47:40.182854+0000","last_peered":"2024-09-05T16:47:40.182854+0000","last_clean":"2024-09-05T16:47:40.182854+0000","last_became_active":"2024-09-05T16:47:15.728759+0000","last_became_peered":"2024-09-05T16:47:15.728759+0000","last_unstale":"2024-09-05T16:47:40.182854+0000","last_undegraded":"2024-09-05T16:47:40.182854+0000","last_fullsized":"2024-09-05T16:47:40.182854+0000","mapping_epoch":48,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T02:39:11.515339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12772,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2250,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12660","reported_seq":17086,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-05T16:47:40.182469+0000","last_change":"2024-09-05T16:47:03.683348+0000","last_active":"2024-09-05T16:47:40.182469+0000","last_peered":"2024-09-05T16:47:40.182469+0000","last_clean":"2024-09-05T16:47:40.182469+0000","last_became_active":"2024-09-05T16:47:03.185143+0000","last_became_peered":"2024-09-05T16:47:03.185143+0000","last_unstale":"2024-09-05T16:47:40.182469+0000","last_undegraded":"2024-09-05T16:47:40.182469+0000","last_fullsized":"2024-09-05T16:47:40.182469+0000","mapping_epoch":46,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12660","last_scrub_stamp":"2024-09-05T16:47:03.683262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:03.683262+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:42:16.533299+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12777","reported_seq":12914,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-05T16:47:40.185310+0000","last_change":"2024-09-05T16:46:40.458315+0000","last_active":"2024-09-05T16:47:40.185310+0000","last_peered":"2024-09-05T16:47:40.185310+0000","last_clean":"2024-09-05T16:47:40.185310+0000","last_became_active":"2024-09-05T16:46:40.184288+0000","last_became_peered":"2024-09-05T16:46:40.184288+0000","last_unstale":"2024-09-05T16:47:40.185310+0000","last_undegraded":"2024-09-05T16:47:40.185310+0000","last_fullsized":"2024-09-05T16:47:40.185310+0000","mapping_epoch":42,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12777","last_scrub_stamp":"2024-09-05T16:46:40.458226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:40.458226+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:38:58.734210+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'12770","reported_seq":12905,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:40.182572+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:47:40.182572+0000","last_peered":"2024-09-05T16:47:40.182572+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:47:40.182572+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12767,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2240}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12808","reported_seq":12964,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:40.183779+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:47:40.183779+0000","last_peered":"2024-09-05T16:47:40.183779+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:47:40.183779+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12806,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2243}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13120","reported_seq":13281,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:40.183093+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:47:40.183093+0000","last_peered":"2024-09-05T16:47:40.183093+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:47:40.183093+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13117,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2278}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12758","reported_seq":12891,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:40.182662+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:47:40.182662+0000","last_peered":"2024-09-05T16:47:40.182662+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:47:40.182662+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2237}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12808","reported_seq":17578,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-05T16:47:40.185220+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:47:40.185220+0000","last_peered":"2024-09-05T16:47:40.185220+0000","last_clean":"2024-09-05T16:47:40.185220+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:47:40.185220+0000","last_undegraded":"2024-09-05T16:47:40.185220+0000","last_fullsized":"2024-09-05T16:47:40.185220+0000","mapping_epoch":36,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4408,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13124","reported_seq":13267,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:40.184445+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:47:40.184445+0000","last_peered":"2024-09-05T16:47:40.184445+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:47:40.184445+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13122,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2332}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12831","reported_seq":12972,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-05T16:47:40.451467+0000","last_change":"2024-09-05T16:47:40.451380+0000","last_active":"2024-09-05T16:47:40.451467+0000","last_peered":"2024-09-05T16:47:40.451467+0000","last_clean":"2024-09-05T16:47:40.451467+0000","last_became_active":"2024-09-05T16:47:40.189447+0000","last_became_peered":"2024-09-05T16:47:40.189447+0000","last_unstale":"2024-09-05T16:47:40.451467+0000","last_undegraded":"2024-09-05T16:47:40.451467+0000","last_fullsized":"2024-09-05T16:47:40.451467+0000","mapping_epoch":52,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12831","last_scrub_stamp":"2024-09-05T16:47:40.451295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:40.451295+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:46:20.946333+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12773","reported_seq":12914,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-05T16:47:40.185391+0000","last_change":"2024-09-05T16:47:27.437275+0000","last_active":"2024-09-05T16:47:40.185391+0000","last_peered":"2024-09-05T16:47:40.185391+0000","last_clean":"2024-09-05T16:47:40.185391+0000","last_became_active":"2024-09-05T16:47:27.204989+0000","last_became_peered":"2024-09-05T16:47:27.204989+0000","last_unstale":"2024-09-05T16:47:40.185391+0000","last_undegraded":"2024-09-05T16:47:40.185391+0000","last_fullsized":"2024-09-05T16:47:40.185391+0000","mapping_epoch":50,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12773","last_scrub_stamp":"2024-09-05T16:47:27.437187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:27.437187+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T16:53:55.961513+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":94304,"num_objects":14,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12761,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116343,"num_write_kb":417555,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5394,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":61440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084452,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722124,"kb_used_data":436,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649716,"statfs":{"total":96636764160,"available":95897309184,"internally_reserved":0,"allocated":446464,"data_stored":168662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":738979539},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986329,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":609712,"kb_used_data":480,"kb_used_omap":4,"kb_used_meta":609211,"kb_avail":93762128,"statfs":{"total":96636764160,"available":96012419072,"internally_reserved":0,"allocated":491520,"data_stored":213718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":623832786},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673135,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713280,"kb_used_data":428,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":93658560,"statfs":{"total":96636764160,"available":95906365440,"internally_reserved":0,"allocated":438272,"data_stored":165393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":729938373},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182566,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713740,"kb_used_data":436,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658100,"statfs":{"total":96636764160,"available":95905894400,"internally_reserved":0,"allocated":446464,"data_stored":168662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":730394317},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":49152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:47:46.304 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:47:46.625 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:47:46.626 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:47:46.638 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":478,"stamp":"2024-09-05T16:47:45.110735+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":94304,"num_objects":14,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12761,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116343,"num_write_kb":417555,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5394,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2758856,"kb_used_data":1780,"kb_used_omap":14,"kb_used_meta":2756977,"kb_avail":374728504,"statfs":{"total":386547056640,"available":383721988096,"internally_reserved":0,"allocated":1822720,"data_stored":716435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2823145015},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13472,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12829,"num_write_kb":-51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1030,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001363"},"pg_stats":[{"pgid":"1.f","version":"39'12904","reported_seq":17524,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-05T16:47:40.182534+0000","last_change":"2024-09-05T16:46:27.826222+0000","last_active":"2024-09-05T16:47:40.182534+0000","last_peered":"2024-09-05T16:47:40.182534+0000","last_clean":"2024-09-05T16:47:40.182534+0000","last_became_active":"2024-09-05T16:46:27.757928+0000","last_became_peered":"2024-09-05T16:46:27.757928+0000","last_unstale":"2024-09-05T16:47:40.182534+0000","last_undegraded":"2024-09-05T16:47:40.182534+0000","last_fullsized":"2024-09-05T16:47:40.182534+0000","mapping_epoch":40,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:32:35.713428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12904,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2262,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12552","reported_seq":17237,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-05T16:47:40.185571+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:47:40.185571+0000","last_peered":"2024-09-05T16:47:40.185571+0000","last_clean":"2024-09-05T16:47:40.185571+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:47:40.185571+0000","last_undegraded":"2024-09-05T16:47:40.185571+0000","last_fullsized":"2024-09-05T16:47:40.185571+0000","mapping_epoch":38,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1865,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12752","reported_seq":17393,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-05T16:47:40.183661+0000","last_change":"2024-09-05T16:46:51.992073+0000","last_active":"2024-09-05T16:47:40.183661+0000","last_peered":"2024-09-05T16:47:40.183661+0000","last_clean":"2024-09-05T16:47:40.183661+0000","last_became_active":"2024-09-05T16:46:51.667128+0000","last_became_peered":"2024-09-05T16:46:51.667128+0000","last_unstale":"2024-09-05T16:47:40.183661+0000","last_undegraded":"2024-09-05T16:47:40.183661+0000","last_fullsized":"2024-09-05T16:47:40.183661+0000","mapping_epoch":44,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12752","last_scrub_stamp":"2024-09-05T16:46:51.991995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:51.991995+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:04:54.120031+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12934","reported_seq":17698,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-05T16:47:40.185167+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:47:40.185167+0000","last_peered":"2024-09-05T16:47:40.185167+0000","last_clean":"2024-09-05T16:47:40.185167+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:47:40.185167+0000","last_undegraded":"2024-09-05T16:47:40.185167+0000","last_fullsized":"2024-09-05T16:47:40.185167+0000","mapping_epoch":34,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7234,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12593","reported_seq":14205,"reported_epoch":53,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:43.204834+0000","last_change":"2024-09-05T16:47:38.413103+0000","last_active":"2024-09-05T16:47:43.204834+0000","last_peered":"2024-09-05T16:47:43.204834+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:47:43.204834+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1408,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12591,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":758,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1406}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12772","reported_seq":17368,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-05T16:47:40.182854+0000","last_change":"2024-09-05T16:47:15.832530+0000","last_active":"2024-09-05T16:47:40.182854+0000","last_peered":"2024-09-05T16:47:40.182854+0000","last_clean":"2024-09-05T16:47:40.182854+0000","last_became_active":"2024-09-05T16:47:15.728759+0000","last_became_peered":"2024-09-05T16:47:15.728759+0000","last_unstale":"2024-09-05T16:47:40.182854+0000","last_undegraded":"2024-09-05T16:47:40.182854+0000","last_fullsized":"2024-09-05T16:47:40.182854+0000","mapping_epoch":48,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T02:39:11.515339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12772,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2250,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12660","reported_seq":17086,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-05T16:47:40.182469+0000","last_change":"2024-09-05T16:47:03.683348+0000","last_active":"2024-09-05T16:47:40.182469+0000","last_peered":"2024-09-05T16:47:40.182469+0000","last_clean":"2024-09-05T16:47:40.182469+0000","last_became_active":"2024-09-05T16:47:03.185143+0000","last_became_peered":"2024-09-05T16:47:03.185143+0000","last_unstale":"2024-09-05T16:47:40.182469+0000","last_undegraded":"2024-09-05T16:47:40.182469+0000","last_fullsized":"2024-09-05T16:47:40.182469+0000","mapping_epoch":46,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12660","last_scrub_stamp":"2024-09-05T16:47:03.683262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:03.683262+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:42:16.533299+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12777","reported_seq":12914,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-05T16:47:40.185310+0000","last_change":"2024-09-05T16:46:40.458315+0000","last_active":"2024-09-05T16:47:40.185310+0000","last_peered":"2024-09-05T16:47:40.185310+0000","last_clean":"2024-09-05T16:47:40.185310+0000","last_became_active":"2024-09-05T16:46:40.184288+0000","last_became_peered":"2024-09-05T16:46:40.184288+0000","last_unstale":"2024-09-05T16:47:40.185310+0000","last_undegraded":"2024-09-05T16:47:40.185310+0000","last_fullsized":"2024-09-05T16:47:40.185310+0000","mapping_epoch":42,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12777","last_scrub_stamp":"2024-09-05T16:46:40.458226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:40.458226+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:38:58.734210+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'12770","reported_seq":12905,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:40.182572+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:47:40.182572+0000","last_peered":"2024-09-05T16:47:40.182572+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:47:40.182572+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12767,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2240}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12808","reported_seq":12964,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:40.183779+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:47:40.183779+0000","last_peered":"2024-09-05T16:47:40.183779+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:47:40.183779+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12806,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2243}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13120","reported_seq":13281,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:40.183093+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:47:40.183093+0000","last_peered":"2024-09-05T16:47:40.183093+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:47:40.183093+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13117,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2278}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12758","reported_seq":12891,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:40.182662+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:47:40.182662+0000","last_peered":"2024-09-05T16:47:40.182662+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:47:40.182662+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2237}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12808","reported_seq":17578,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-05T16:47:40.185220+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:47:40.185220+0000","last_peered":"2024-09-05T16:47:40.185220+0000","last_clean":"2024-09-05T16:47:40.185220+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:47:40.185220+0000","last_undegraded":"2024-09-05T16:47:40.185220+0000","last_fullsized":"2024-09-05T16:47:40.185220+0000","mapping_epoch":36,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4408,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13124","reported_seq":13267,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:40.184445+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:47:40.184445+0000","last_peered":"2024-09-05T16:47:40.184445+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:47:40.184445+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13122,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2332}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12831","reported_seq":12972,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-05T16:47:40.451467+0000","last_change":"2024-09-05T16:47:40.451380+0000","last_active":"2024-09-05T16:47:40.451467+0000","last_peered":"2024-09-05T16:47:40.451467+0000","last_clean":"2024-09-05T16:47:40.451467+0000","last_became_active":"2024-09-05T16:47:40.189447+0000","last_became_peered":"2024-09-05T16:47:40.189447+0000","last_unstale":"2024-09-05T16:47:40.451467+0000","last_undegraded":"2024-09-05T16:47:40.451467+0000","last_fullsized":"2024-09-05T16:47:40.451467+0000","mapping_epoch":52,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12831","last_scrub_stamp":"2024-09-05T16:47:40.451295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:40.451295+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:46:20.946333+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12773","reported_seq":12914,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-05T16:47:40.185391+0000","last_change":"2024-09-05T16:47:27.437275+0000","last_active":"2024-09-05T16:47:40.185391+0000","last_peered":"2024-09-05T16:47:40.185391+0000","last_clean":"2024-09-05T16:47:40.185391+0000","last_became_active":"2024-09-05T16:47:27.204989+0000","last_became_peered":"2024-09-05T16:47:27.204989+0000","last_unstale":"2024-09-05T16:47:40.185391+0000","last_undegraded":"2024-09-05T16:47:40.185391+0000","last_fullsized":"2024-09-05T16:47:40.185391+0000","mapping_epoch":50,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12773","last_scrub_stamp":"2024-09-05T16:47:27.437187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:27.437187+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T16:53:55.961513+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":94304,"num_objects":14,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12761,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116343,"num_write_kb":417555,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5394,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":61440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084452,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722124,"kb_used_data":436,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649716,"statfs":{"total":96636764160,"available":95897309184,"internally_reserved":0,"allocated":446464,"data_stored":168662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":738979539},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986329,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":609712,"kb_used_data":480,"kb_used_omap":4,"kb_used_meta":609211,"kb_avail":93762128,"statfs":{"total":96636764160,"available":96012419072,"internally_reserved":0,"allocated":491520,"data_stored":213718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":623832786},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673135,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713280,"kb_used_data":428,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":93658560,"statfs":{"total":96636764160,"available":95906365440,"internally_reserved":0,"allocated":438272,"data_stored":165393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":729938373},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182566,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713740,"kb_used_data":436,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658100,"statfs":{"total":96636764160,"available":95905894400,"internally_reserved":0,"allocated":446464,"data_stored":168662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":730394317},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":49152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:47:46.639 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:47:46.639 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "39'12593", 'reported_seq': 14205, 'reported_epoch': 53, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:47:43.204834+0000', 'last_change': '2024-09-05T16:47:38.413103+0000', 'last_active': '2024-09-05T16:47:43.204834+0000', 'last_peered': '2024-09-05T16:47:43.204834+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:47:43.204834+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "33'2500", 'ondisk_log_start': "33'2500", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10093, 'log_dups_size': 0, 'ondisk_log_size': 10093, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1408, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12591, 'num_write_kb': 50373, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 758, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1406}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:47:46.639 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:47:46.639 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "39'12770", 'reported_seq': 12905, 'reported_epoch': 53, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:47:40.182572+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:47:40.182572+0000', 'last_peered': '2024-09-05T16:47:40.182572+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:47:40.182572+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "33'2700", 'ondisk_log_start': "33'2700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10070, 'log_dups_size': 0, 'ondisk_log_size': 10070, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26944, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 16, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12767, 'num_write_kb': 51081, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2240}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:47:46.639 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:47:46.639 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "39'12808", 'reported_seq': 12964, 'reported_epoch': 53, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:47:40.183779+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:47:40.183779+0000', 'last_peered': '2024-09-05T16:47:40.183779+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:47:40.183779+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "33'2800", 'ondisk_log_start': "33'2800", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10008, 'log_dups_size': 0, 'ondisk_log_size': 10008, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12806, 'num_write_kb': 51230, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -2243}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:47:46.639 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:47:46.639 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "39'13120", 'reported_seq': 13281, 'reported_epoch': 53, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:47:40.183093+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:47:40.183093+0000', 'last_peered': '2024-09-05T16:47:40.183093+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:47:40.183093+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "33'3100", 'ondisk_log_start': "33'3100", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20208, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2284, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13117, 'num_write_kb': 52476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2278}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:47:46.640 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:47:46.640 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "39'12758", 'reported_seq': 12891, 'reported_epoch': 53, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:47:40.182662+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:47:40.182662+0000', 'last_peered': '2024-09-05T16:47:40.182662+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:47:40.182662+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "33'2700", 'ondisk_log_start': "33'2700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10058, 'log_dups_size': 0, 'ondisk_log_size': 10058, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6736, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12757, 'num_write_kb': 51031, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2237}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:47:46.640 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:47:46.640 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "39'13124", 'reported_seq': 13267, 'reported_epoch': 53, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:47:40.184445+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:47:40.184445+0000', 'last_peered': '2024-09-05T16:47:40.184445+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:47:40.184445+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "33'3100", 'ondisk_log_start': "33'3100", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10024, 'log_dups_size': 0, 'ondisk_log_size': 10024, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13122, 'num_write_kb': 52500, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2332}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:47:46.640 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:47:47.031 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:47:47.031 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 12761/56 objects degraded (22787.500%), 6 pgs degraded, 6 pgs undersized","count":12},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":870,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":53,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":6},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":10},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":5},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":14,"data_bytes":94304,"bytes_used":2825068544,"bytes_avail":383721988096,"bytes_total":386547056640,"degraded_objects":12761,"degraded_total":56,"degraded_ratio":227.875},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:47:47.044 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-09-05T16:47:47.044 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:47:47.366 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:47:47.367 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:47:47.379 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":479,"stamp":"2024-09-05T16:47:47.111114+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":94304,"num_objects":14,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12761,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116343,"num_write_kb":417555,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5394,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2758868,"kb_used_data":1788,"kb_used_omap":14,"kb_used_meta":2756977,"kb_avail":374728492,"statfs":{"total":386547056640,"available":383721975808,"internally_reserved":0,"allocated":1830912,"data_stored":719704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2823145015},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13472,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12829,"num_write_kb":-51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1030,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001406"},"pg_stats":[{"pgid":"1.f","version":"39'12904","reported_seq":17524,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-05T16:47:40.182534+0000","last_change":"2024-09-05T16:46:27.826222+0000","last_active":"2024-09-05T16:47:40.182534+0000","last_peered":"2024-09-05T16:47:40.182534+0000","last_clean":"2024-09-05T16:47:40.182534+0000","last_became_active":"2024-09-05T16:46:27.757928+0000","last_became_peered":"2024-09-05T16:46:27.757928+0000","last_unstale":"2024-09-05T16:47:40.182534+0000","last_undegraded":"2024-09-05T16:47:40.182534+0000","last_fullsized":"2024-09-05T16:47:40.182534+0000","mapping_epoch":40,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:32:35.713428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12904,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2262,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12552","reported_seq":17237,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-05T16:47:40.185571+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:47:40.185571+0000","last_peered":"2024-09-05T16:47:40.185571+0000","last_clean":"2024-09-05T16:47:40.185571+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:47:40.185571+0000","last_undegraded":"2024-09-05T16:47:40.185571+0000","last_fullsized":"2024-09-05T16:47:40.185571+0000","mapping_epoch":38,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1865,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12752","reported_seq":17393,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-05T16:47:40.183661+0000","last_change":"2024-09-05T16:46:51.992073+0000","last_active":"2024-09-05T16:47:40.183661+0000","last_peered":"2024-09-05T16:47:40.183661+0000","last_clean":"2024-09-05T16:47:40.183661+0000","last_became_active":"2024-09-05T16:46:51.667128+0000","last_became_peered":"2024-09-05T16:46:51.667128+0000","last_unstale":"2024-09-05T16:47:40.183661+0000","last_undegraded":"2024-09-05T16:47:40.183661+0000","last_fullsized":"2024-09-05T16:47:40.183661+0000","mapping_epoch":44,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12752","last_scrub_stamp":"2024-09-05T16:46:51.991995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:51.991995+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:04:54.120031+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12934","reported_seq":17698,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-05T16:47:40.185167+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:47:40.185167+0000","last_peered":"2024-09-05T16:47:40.185167+0000","last_clean":"2024-09-05T16:47:40.185167+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:47:40.185167+0000","last_undegraded":"2024-09-05T16:47:40.185167+0000","last_fullsized":"2024-09-05T16:47:40.185167+0000","mapping_epoch":34,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7234,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12593","reported_seq":14205,"reported_epoch":53,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:43.204834+0000","last_change":"2024-09-05T16:47:38.413103+0000","last_active":"2024-09-05T16:47:43.204834+0000","last_peered":"2024-09-05T16:47:43.204834+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:47:43.204834+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1408,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12591,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":758,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1406}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12772","reported_seq":17368,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-05T16:47:40.182854+0000","last_change":"2024-09-05T16:47:15.832530+0000","last_active":"2024-09-05T16:47:40.182854+0000","last_peered":"2024-09-05T16:47:40.182854+0000","last_clean":"2024-09-05T16:47:40.182854+0000","last_became_active":"2024-09-05T16:47:15.728759+0000","last_became_peered":"2024-09-05T16:47:15.728759+0000","last_unstale":"2024-09-05T16:47:40.182854+0000","last_undegraded":"2024-09-05T16:47:40.182854+0000","last_fullsized":"2024-09-05T16:47:40.182854+0000","mapping_epoch":48,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T02:39:11.515339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12772,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2250,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12660","reported_seq":17086,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-05T16:47:40.182469+0000","last_change":"2024-09-05T16:47:03.683348+0000","last_active":"2024-09-05T16:47:40.182469+0000","last_peered":"2024-09-05T16:47:40.182469+0000","last_clean":"2024-09-05T16:47:40.182469+0000","last_became_active":"2024-09-05T16:47:03.185143+0000","last_became_peered":"2024-09-05T16:47:03.185143+0000","last_unstale":"2024-09-05T16:47:40.182469+0000","last_undegraded":"2024-09-05T16:47:40.182469+0000","last_fullsized":"2024-09-05T16:47:40.182469+0000","mapping_epoch":46,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12660","last_scrub_stamp":"2024-09-05T16:47:03.683262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:03.683262+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:42:16.533299+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12777","reported_seq":12914,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-05T16:47:40.185310+0000","last_change":"2024-09-05T16:46:40.458315+0000","last_active":"2024-09-05T16:47:40.185310+0000","last_peered":"2024-09-05T16:47:40.185310+0000","last_clean":"2024-09-05T16:47:40.185310+0000","last_became_active":"2024-09-05T16:46:40.184288+0000","last_became_peered":"2024-09-05T16:46:40.184288+0000","last_unstale":"2024-09-05T16:47:40.185310+0000","last_undegraded":"2024-09-05T16:47:40.185310+0000","last_fullsized":"2024-09-05T16:47:40.185310+0000","mapping_epoch":42,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12777","last_scrub_stamp":"2024-09-05T16:46:40.458226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:40.458226+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:38:58.734210+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'12770","reported_seq":12905,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:40.182572+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:47:40.182572+0000","last_peered":"2024-09-05T16:47:40.182572+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:47:40.182572+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12767,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2240}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12808","reported_seq":12964,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:40.183779+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:47:40.183779+0000","last_peered":"2024-09-05T16:47:40.183779+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:47:40.183779+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12806,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2243}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13120","reported_seq":13281,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:40.183093+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:47:40.183093+0000","last_peered":"2024-09-05T16:47:40.183093+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:47:40.183093+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13117,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2278}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12758","reported_seq":12891,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:40.182662+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:47:40.182662+0000","last_peered":"2024-09-05T16:47:40.182662+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:47:40.182662+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2237}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12808","reported_seq":17578,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-05T16:47:40.185220+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:47:40.185220+0000","last_peered":"2024-09-05T16:47:40.185220+0000","last_clean":"2024-09-05T16:47:40.185220+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:47:40.185220+0000","last_undegraded":"2024-09-05T16:47:40.185220+0000","last_fullsized":"2024-09-05T16:47:40.185220+0000","mapping_epoch":36,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4408,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13124","reported_seq":13267,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:40.184445+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:47:40.184445+0000","last_peered":"2024-09-05T16:47:40.184445+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:47:40.184445+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13122,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2332}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12831","reported_seq":12972,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-05T16:47:40.451467+0000","last_change":"2024-09-05T16:47:40.451380+0000","last_active":"2024-09-05T16:47:40.451467+0000","last_peered":"2024-09-05T16:47:40.451467+0000","last_clean":"2024-09-05T16:47:40.451467+0000","last_became_active":"2024-09-05T16:47:40.189447+0000","last_became_peered":"2024-09-05T16:47:40.189447+0000","last_unstale":"2024-09-05T16:47:40.451467+0000","last_undegraded":"2024-09-05T16:47:40.451467+0000","last_fullsized":"2024-09-05T16:47:40.451467+0000","mapping_epoch":52,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12831","last_scrub_stamp":"2024-09-05T16:47:40.451295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:40.451295+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:46:20.946333+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12773","reported_seq":12914,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-05T16:47:40.185391+0000","last_change":"2024-09-05T16:47:27.437275+0000","last_active":"2024-09-05T16:47:40.185391+0000","last_peered":"2024-09-05T16:47:40.185391+0000","last_clean":"2024-09-05T16:47:40.185391+0000","last_became_active":"2024-09-05T16:47:27.204989+0000","last_became_peered":"2024-09-05T16:47:27.204989+0000","last_unstale":"2024-09-05T16:47:40.185391+0000","last_undegraded":"2024-09-05T16:47:40.185391+0000","last_fullsized":"2024-09-05T16:47:40.185391+0000","mapping_epoch":50,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12773","last_scrub_stamp":"2024-09-05T16:47:27.437187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:27.437187+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T16:53:55.961513+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":94304,"num_objects":14,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12761,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116343,"num_write_kb":417555,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5394,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":61440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084452,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722124,"kb_used_data":436,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649716,"statfs":{"total":96636764160,"available":95897309184,"internally_reserved":0,"allocated":446464,"data_stored":168662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":738979539},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986329,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":609712,"kb_used_data":480,"kb_used_omap":4,"kb_used_meta":609211,"kb_avail":93762128,"statfs":{"total":96636764160,"available":96012419072,"internally_reserved":0,"allocated":491520,"data_stored":213718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":623832786},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673136,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713292,"kb_used_data":436,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":93658548,"statfs":{"total":96636764160,"available":95906353152,"internally_reserved":0,"allocated":446464,"data_stored":168662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":729938373},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182567,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713740,"kb_used_data":436,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658100,"statfs":{"total":96636764160,"available":95905894400,"internally_reserved":0,"allocated":446464,"data_stored":168662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":730394317},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":49152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:47:50.381 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:47:50.689 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:47:50.689 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:47:50.701 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":481,"stamp":"2024-09-05T16:47:50.180417+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":94304,"num_objects":14,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11780,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116343,"num_write_kb":417555,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6375,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2763028,"kb_used_data":1788,"kb_used_omap":14,"kb_used_meta":2761137,"kb_avail":374724332,"statfs":{"total":386547056640,"available":383717715968,"internally_reserved":0,"allocated":1830912,"data_stored":719704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2827404855},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13472,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2275,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12829,"num_write_kb":-51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-49,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.999221"},"pg_stats":[{"pgid":"1.f","version":"39'12904","reported_seq":17524,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-05T16:47:40.182534+0000","last_change":"2024-09-05T16:46:27.826222+0000","last_active":"2024-09-05T16:47:40.182534+0000","last_peered":"2024-09-05T16:47:40.182534+0000","last_clean":"2024-09-05T16:47:40.182534+0000","last_became_active":"2024-09-05T16:46:27.757928+0000","last_became_peered":"2024-09-05T16:46:27.757928+0000","last_unstale":"2024-09-05T16:47:40.182534+0000","last_undegraded":"2024-09-05T16:47:40.182534+0000","last_fullsized":"2024-09-05T16:47:40.182534+0000","mapping_epoch":40,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:32:35.713428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12904,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2262,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12552","reported_seq":17237,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-05T16:47:40.185571+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:47:40.185571+0000","last_peered":"2024-09-05T16:47:40.185571+0000","last_clean":"2024-09-05T16:47:40.185571+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:47:40.185571+0000","last_undegraded":"2024-09-05T16:47:40.185571+0000","last_fullsized":"2024-09-05T16:47:40.185571+0000","mapping_epoch":38,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1865,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12752","reported_seq":17393,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-05T16:47:40.183661+0000","last_change":"2024-09-05T16:46:51.992073+0000","last_active":"2024-09-05T16:47:40.183661+0000","last_peered":"2024-09-05T16:47:40.183661+0000","last_clean":"2024-09-05T16:47:40.183661+0000","last_became_active":"2024-09-05T16:46:51.667128+0000","last_became_peered":"2024-09-05T16:46:51.667128+0000","last_unstale":"2024-09-05T16:47:40.183661+0000","last_undegraded":"2024-09-05T16:47:40.183661+0000","last_fullsized":"2024-09-05T16:47:40.183661+0000","mapping_epoch":44,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12752","last_scrub_stamp":"2024-09-05T16:46:51.991995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:51.991995+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:04:54.120031+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12934","reported_seq":17698,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-05T16:47:40.185167+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:47:40.185167+0000","last_peered":"2024-09-05T16:47:40.185167+0000","last_clean":"2024-09-05T16:47:40.185167+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:47:40.185167+0000","last_undegraded":"2024-09-05T16:47:40.185167+0000","last_fullsized":"2024-09-05T16:47:40.185167+0000","mapping_epoch":34,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7234,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12593","reported_seq":16167,"reported_epoch":53,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:48.210034+0000","last_change":"2024-09-05T16:47:38.413103+0000","last_active":"2024-09-05T16:47:48.210034+0000","last_peered":"2024-09-05T16:47:48.210034+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:47:48.210034+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":427,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12591,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1739,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-425}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12772","reported_seq":17369,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-05T16:47:40.182901+0000","last_change":"2024-09-05T16:47:15.832530+0000","last_active":"2024-09-05T16:47:40.182901+0000","last_peered":"2024-09-05T16:47:40.182901+0000","last_clean":"2024-09-05T16:47:40.182901+0000","last_became_active":"2024-09-05T16:47:15.728759+0000","last_became_peered":"2024-09-05T16:47:15.728759+0000","last_unstale":"2024-09-05T16:47:40.182901+0000","last_undegraded":"2024-09-05T16:47:40.182901+0000","last_fullsized":"2024-09-05T16:47:40.182901+0000","mapping_epoch":48,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T02:39:11.515339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12772,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2250,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12660","reported_seq":17086,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-05T16:47:40.182469+0000","last_change":"2024-09-05T16:47:03.683348+0000","last_active":"2024-09-05T16:47:40.182469+0000","last_peered":"2024-09-05T16:47:40.182469+0000","last_clean":"2024-09-05T16:47:40.182469+0000","last_became_active":"2024-09-05T16:47:03.185143+0000","last_became_peered":"2024-09-05T16:47:03.185143+0000","last_unstale":"2024-09-05T16:47:40.182469+0000","last_undegraded":"2024-09-05T16:47:40.182469+0000","last_fullsized":"2024-09-05T16:47:40.182469+0000","mapping_epoch":46,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12660","last_scrub_stamp":"2024-09-05T16:47:03.683262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:03.683262+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:42:16.533299+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12777","reported_seq":12914,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-05T16:47:40.185310+0000","last_change":"2024-09-05T16:46:40.458315+0000","last_active":"2024-09-05T16:47:40.185310+0000","last_peered":"2024-09-05T16:47:40.185310+0000","last_clean":"2024-09-05T16:47:40.185310+0000","last_became_active":"2024-09-05T16:46:40.184288+0000","last_became_peered":"2024-09-05T16:46:40.184288+0000","last_unstale":"2024-09-05T16:47:40.185310+0000","last_undegraded":"2024-09-05T16:47:40.185310+0000","last_fullsized":"2024-09-05T16:47:40.185310+0000","mapping_epoch":42,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12777","last_scrub_stamp":"2024-09-05T16:46:40.458226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:40.458226+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:38:58.734210+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'12770","reported_seq":12905,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:40.182572+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:47:40.182572+0000","last_peered":"2024-09-05T16:47:40.182572+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:47:40.182572+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12767,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2240}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12808","reported_seq":12964,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:40.183779+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:47:40.183779+0000","last_peered":"2024-09-05T16:47:40.183779+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:47:40.183779+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12806,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2243}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13120","reported_seq":13281,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:40.183093+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:47:40.183093+0000","last_peered":"2024-09-05T16:47:40.183093+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:47:40.183093+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13117,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2278}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12758","reported_seq":12891,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:40.182662+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:47:40.182662+0000","last_peered":"2024-09-05T16:47:40.182662+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:47:40.182662+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2237}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12808","reported_seq":17578,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-05T16:47:40.185220+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:47:40.185220+0000","last_peered":"2024-09-05T16:47:40.185220+0000","last_clean":"2024-09-05T16:47:40.185220+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:47:40.185220+0000","last_undegraded":"2024-09-05T16:47:40.185220+0000","last_fullsized":"2024-09-05T16:47:40.185220+0000","mapping_epoch":36,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4408,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13124","reported_seq":13267,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:40.184445+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:47:40.184445+0000","last_peered":"2024-09-05T16:47:40.184445+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:47:40.184445+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13122,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2332}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12831","reported_seq":12972,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-05T16:47:40.451467+0000","last_change":"2024-09-05T16:47:40.451380+0000","last_active":"2024-09-05T16:47:40.451467+0000","last_peered":"2024-09-05T16:47:40.451467+0000","last_clean":"2024-09-05T16:47:40.451467+0000","last_became_active":"2024-09-05T16:47:40.189447+0000","last_became_peered":"2024-09-05T16:47:40.189447+0000","last_unstale":"2024-09-05T16:47:40.451467+0000","last_undegraded":"2024-09-05T16:47:40.451467+0000","last_fullsized":"2024-09-05T16:47:40.451467+0000","mapping_epoch":52,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12831","last_scrub_stamp":"2024-09-05T16:47:40.451295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:40.451295+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:46:20.946333+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12773","reported_seq":12914,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-05T16:47:40.185391+0000","last_change":"2024-09-05T16:47:27.437275+0000","last_active":"2024-09-05T16:47:40.185391+0000","last_peered":"2024-09-05T16:47:40.185391+0000","last_clean":"2024-09-05T16:47:40.185391+0000","last_became_active":"2024-09-05T16:47:27.204989+0000","last_became_peered":"2024-09-05T16:47:27.204989+0000","last_unstale":"2024-09-05T16:47:40.185391+0000","last_undegraded":"2024-09-05T16:47:40.185391+0000","last_fullsized":"2024-09-05T16:47:40.185391+0000","mapping_epoch":50,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12773","last_scrub_stamp":"2024-09-05T16:47:27.437187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:27.437187+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T16:53:55.961513+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":94304,"num_objects":14,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11780,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116343,"num_write_kb":417555,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6375,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":61440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084453,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722124,"kb_used_data":436,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649716,"statfs":{"total":96636764160,"available":95897309184,"internally_reserved":0,"allocated":446464,"data_stored":168662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":738979539},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986330,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":613872,"kb_used_data":480,"kb_used_omap":4,"kb_used_meta":613371,"kb_avail":93757968,"statfs":{"total":96636764160,"available":96008159232,"internally_reserved":0,"allocated":491520,"data_stored":213718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":628092626},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673136,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713292,"kb_used_data":436,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":93658548,"statfs":{"total":96636764160,"available":95906353152,"internally_reserved":0,"allocated":446464,"data_stored":168662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":729938373},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182567,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713740,"kb_used_data":436,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658100,"statfs":{"total":96636764160,"available":95905894400,"internally_reserved":0,"allocated":446464,"data_stored":168662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":730394317},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":49152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:47:50.702 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:47:51.045 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:47:51.045 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:47:51.058 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":481,"stamp":"2024-09-05T16:47:50.180417+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":94304,"num_objects":14,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11780,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116343,"num_write_kb":417555,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6375,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2763028,"kb_used_data":1788,"kb_used_omap":14,"kb_used_meta":2761137,"kb_avail":374724332,"statfs":{"total":386547056640,"available":383717715968,"internally_reserved":0,"allocated":1830912,"data_stored":719704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2827404855},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13472,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2275,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12829,"num_write_kb":-51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-49,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.999221"},"pg_stats":[{"pgid":"1.f","version":"39'12904","reported_seq":17524,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-05T16:47:40.182534+0000","last_change":"2024-09-05T16:46:27.826222+0000","last_active":"2024-09-05T16:47:40.182534+0000","last_peered":"2024-09-05T16:47:40.182534+0000","last_clean":"2024-09-05T16:47:40.182534+0000","last_became_active":"2024-09-05T16:46:27.757928+0000","last_became_peered":"2024-09-05T16:46:27.757928+0000","last_unstale":"2024-09-05T16:47:40.182534+0000","last_undegraded":"2024-09-05T16:47:40.182534+0000","last_fullsized":"2024-09-05T16:47:40.182534+0000","mapping_epoch":40,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:32:35.713428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12904,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2262,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12552","reported_seq":17237,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-05T16:47:40.185571+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:47:40.185571+0000","last_peered":"2024-09-05T16:47:40.185571+0000","last_clean":"2024-09-05T16:47:40.185571+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:47:40.185571+0000","last_undegraded":"2024-09-05T16:47:40.185571+0000","last_fullsized":"2024-09-05T16:47:40.185571+0000","mapping_epoch":38,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1865,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12752","reported_seq":17393,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-05T16:47:40.183661+0000","last_change":"2024-09-05T16:46:51.992073+0000","last_active":"2024-09-05T16:47:40.183661+0000","last_peered":"2024-09-05T16:47:40.183661+0000","last_clean":"2024-09-05T16:47:40.183661+0000","last_became_active":"2024-09-05T16:46:51.667128+0000","last_became_peered":"2024-09-05T16:46:51.667128+0000","last_unstale":"2024-09-05T16:47:40.183661+0000","last_undegraded":"2024-09-05T16:47:40.183661+0000","last_fullsized":"2024-09-05T16:47:40.183661+0000","mapping_epoch":44,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12752","last_scrub_stamp":"2024-09-05T16:46:51.991995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:51.991995+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:04:54.120031+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12934","reported_seq":17698,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-05T16:47:40.185167+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:47:40.185167+0000","last_peered":"2024-09-05T16:47:40.185167+0000","last_clean":"2024-09-05T16:47:40.185167+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:47:40.185167+0000","last_undegraded":"2024-09-05T16:47:40.185167+0000","last_fullsized":"2024-09-05T16:47:40.185167+0000","mapping_epoch":34,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7234,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12593","reported_seq":16167,"reported_epoch":53,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:48.210034+0000","last_change":"2024-09-05T16:47:38.413103+0000","last_active":"2024-09-05T16:47:48.210034+0000","last_peered":"2024-09-05T16:47:48.210034+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:47:48.210034+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":427,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12591,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1739,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-425}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12772","reported_seq":17369,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-05T16:47:40.182901+0000","last_change":"2024-09-05T16:47:15.832530+0000","last_active":"2024-09-05T16:47:40.182901+0000","last_peered":"2024-09-05T16:47:40.182901+0000","last_clean":"2024-09-05T16:47:40.182901+0000","last_became_active":"2024-09-05T16:47:15.728759+0000","last_became_peered":"2024-09-05T16:47:15.728759+0000","last_unstale":"2024-09-05T16:47:40.182901+0000","last_undegraded":"2024-09-05T16:47:40.182901+0000","last_fullsized":"2024-09-05T16:47:40.182901+0000","mapping_epoch":48,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T02:39:11.515339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12772,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2250,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12660","reported_seq":17086,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-05T16:47:40.182469+0000","last_change":"2024-09-05T16:47:03.683348+0000","last_active":"2024-09-05T16:47:40.182469+0000","last_peered":"2024-09-05T16:47:40.182469+0000","last_clean":"2024-09-05T16:47:40.182469+0000","last_became_active":"2024-09-05T16:47:03.185143+0000","last_became_peered":"2024-09-05T16:47:03.185143+0000","last_unstale":"2024-09-05T16:47:40.182469+0000","last_undegraded":"2024-09-05T16:47:40.182469+0000","last_fullsized":"2024-09-05T16:47:40.182469+0000","mapping_epoch":46,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12660","last_scrub_stamp":"2024-09-05T16:47:03.683262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:03.683262+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:42:16.533299+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12777","reported_seq":12914,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-05T16:47:40.185310+0000","last_change":"2024-09-05T16:46:40.458315+0000","last_active":"2024-09-05T16:47:40.185310+0000","last_peered":"2024-09-05T16:47:40.185310+0000","last_clean":"2024-09-05T16:47:40.185310+0000","last_became_active":"2024-09-05T16:46:40.184288+0000","last_became_peered":"2024-09-05T16:46:40.184288+0000","last_unstale":"2024-09-05T16:47:40.185310+0000","last_undegraded":"2024-09-05T16:47:40.185310+0000","last_fullsized":"2024-09-05T16:47:40.185310+0000","mapping_epoch":42,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12777","last_scrub_stamp":"2024-09-05T16:46:40.458226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:40.458226+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:38:58.734210+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'12770","reported_seq":12905,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:40.182572+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:47:40.182572+0000","last_peered":"2024-09-05T16:47:40.182572+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:47:40.182572+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12767,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2240}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12808","reported_seq":12964,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:40.183779+0000","last_change":"2024-09-05T16:34:56.881309+0000","last_active":"2024-09-05T16:47:40.183779+0000","last_peered":"2024-09-05T16:47:40.183779+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:47:40.183779+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12806,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2243}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13120","reported_seq":13281,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:40.183093+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:47:40.183093+0000","last_peered":"2024-09-05T16:47:40.183093+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:47:40.183093+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13117,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2278}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12758","reported_seq":12891,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:40.182662+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:47:40.182662+0000","last_peered":"2024-09-05T16:47:40.182662+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:47:40.182662+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2237}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12808","reported_seq":17578,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-05T16:47:40.185220+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:47:40.185220+0000","last_peered":"2024-09-05T16:47:40.185220+0000","last_clean":"2024-09-05T16:47:40.185220+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:47:40.185220+0000","last_undegraded":"2024-09-05T16:47:40.185220+0000","last_fullsized":"2024-09-05T16:47:40.185220+0000","mapping_epoch":36,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4408,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13124","reported_seq":13267,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:40.184445+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:47:40.184445+0000","last_peered":"2024-09-05T16:47:40.184445+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:47:40.184445+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13122,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2332}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12831","reported_seq":12972,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-05T16:47:40.451467+0000","last_change":"2024-09-05T16:47:40.451380+0000","last_active":"2024-09-05T16:47:40.451467+0000","last_peered":"2024-09-05T16:47:40.451467+0000","last_clean":"2024-09-05T16:47:40.451467+0000","last_became_active":"2024-09-05T16:47:40.189447+0000","last_became_peered":"2024-09-05T16:47:40.189447+0000","last_unstale":"2024-09-05T16:47:40.451467+0000","last_undegraded":"2024-09-05T16:47:40.451467+0000","last_fullsized":"2024-09-05T16:47:40.451467+0000","mapping_epoch":52,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12831","last_scrub_stamp":"2024-09-05T16:47:40.451295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:40.451295+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:46:20.946333+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12773","reported_seq":12914,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-05T16:47:40.185391+0000","last_change":"2024-09-05T16:47:27.437275+0000","last_active":"2024-09-05T16:47:40.185391+0000","last_peered":"2024-09-05T16:47:40.185391+0000","last_clean":"2024-09-05T16:47:40.185391+0000","last_became_active":"2024-09-05T16:47:27.204989+0000","last_became_peered":"2024-09-05T16:47:27.204989+0000","last_unstale":"2024-09-05T16:47:40.185391+0000","last_undegraded":"2024-09-05T16:47:40.185391+0000","last_fullsized":"2024-09-05T16:47:40.185391+0000","mapping_epoch":50,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12773","last_scrub_stamp":"2024-09-05T16:47:27.437187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:27.437187+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T16:53:55.961513+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":94304,"num_objects":14,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11780,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116343,"num_write_kb":417555,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6375,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":61440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084453,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722124,"kb_used_data":436,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649716,"statfs":{"total":96636764160,"available":95897309184,"internally_reserved":0,"allocated":446464,"data_stored":168662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":738979539},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986330,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":613872,"kb_used_data":480,"kb_used_omap":4,"kb_used_meta":613371,"kb_avail":93757968,"statfs":{"total":96636764160,"available":96008159232,"internally_reserved":0,"allocated":491520,"data_stored":213718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":628092626},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673136,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713292,"kb_used_data":436,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":93658548,"statfs":{"total":96636764160,"available":95906353152,"internally_reserved":0,"allocated":446464,"data_stored":168662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":729938373},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182567,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713740,"kb_used_data":436,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658100,"statfs":{"total":96636764160,"available":95905894400,"internally_reserved":0,"allocated":446464,"data_stored":168662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":730394317},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":49152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:47:51.059 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-05T16:47:51.059 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "39'12593", 'reported_seq': 16167, 'reported_epoch': 53, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:47:48.210034+0000', 'last_change': '2024-09-05T16:47:38.413103+0000', 'last_active': '2024-09-05T16:47:48.210034+0000', 'last_peered': '2024-09-05T16:47:48.210034+0000', 'last_clean': '2024-09-05T16:33:52.740957+0000', 'last_became_active': '2024-09-05T16:34:56.855551+0000', 'last_became_peered': '2024-09-05T16:34:56.855551+0000', 'last_unstale': '2024-09-05T16:47:48.210034+0000', 'last_undegraded': '2024-09-05T16:34:56.621356+0000', 'last_fullsized': '2024-09-05T16:34:56.566239+0000', 'mapping_epoch': 32, 'log_start': "33'2500", 'ondisk_log_start': "33'2500", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10093, 'log_dups_size': 0, 'ondisk_log_size': 10093, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 427, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12591, 'num_write_kb': 50373, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1739, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -425}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:47:51.059 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:47:51.059 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "39'12770", 'reported_seq': 12905, 'reported_epoch': 53, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:47:40.182572+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:47:40.182572+0000', 'last_peered': '2024-09-05T16:47:40.182572+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:47:40.182572+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "33'2700", 'ondisk_log_start': "33'2700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10070, 'log_dups_size': 0, 'ondisk_log_size': 10070, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26944, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 16, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12767, 'num_write_kb': 51081, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2240}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:47:51.059 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:47:51.059 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "39'12808", 'reported_seq': 12964, 'reported_epoch': 53, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:47:40.183779+0000', 'last_change': '2024-09-05T16:34:56.881309+0000', 'last_active': '2024-09-05T16:47:40.183779+0000', 'last_peered': '2024-09-05T16:47:40.183779+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:47:40.183779+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "33'2800", 'ondisk_log_start': "33'2800", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10008, 'log_dups_size': 0, 'ondisk_log_size': 10008, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12806, 'num_write_kb': 51230, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -2243}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:47:51.060 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:47:51.060 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "39'13120", 'reported_seq': 13281, 'reported_epoch': 53, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:47:40.183093+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:47:40.183093+0000', 'last_peered': '2024-09-05T16:47:40.183093+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:47:40.183093+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "33'3100", 'ondisk_log_start': "33'3100", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20208, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2284, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13117, 'num_write_kb': 52476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2278}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:47:51.060 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:47:51.060 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "39'12758", 'reported_seq': 12891, 'reported_epoch': 53, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:47:40.182662+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:47:40.182662+0000', 'last_peered': '2024-09-05T16:47:40.182662+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:47:40.182662+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "33'2700", 'ondisk_log_start': "33'2700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10058, 'log_dups_size': 0, 'ondisk_log_size': 10058, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6736, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12757, 'num_write_kb': 51031, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2237}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:47:51.060 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:47:51.060 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "39'13124", 'reported_seq': 13267, 'reported_epoch': 53, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:47:40.184445+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:47:40.184445+0000', 'last_peered': '2024-09-05T16:47:40.184445+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:47:40.184445+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "33'3100", 'ondisk_log_start': "33'3100", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10024, 'log_dups_size': 0, 'ondisk_log_size': 10024, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13122, 'num_write_kb': 52500, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2332}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:47:51.060 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:47:51.487 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:47:51.487 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 11609/56 objects degraded (20730.357%), 6 pgs degraded, 6 pgs undersized","count":12},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":874,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":55,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":5},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":10},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":4},{"state_name":"active+recovering+undersized+degraded+remapped","count":2}],"num_pgs":16,"num_pools":1,"num_objects":14,"data_bytes":94304,"bytes_used":2829344768,"bytes_avail":383717711872,"bytes_total":386547056640,"degraded_objects":11609,"degraded_total":56,"degraded_ratio":207.30357142857142,"recovering_objects_per_sec":12,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":122,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:47:51.501 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:47:51.501 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:47:51.846 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:47:51.846 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:47:51.858 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":483,"stamp":"2024-09-05T16:47:51.190652+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":94304,"num_objects":14,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11609,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116343,"num_write_kb":417555,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6546,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2763032,"kb_used_data":1796,"kb_used_omap":14,"kb_used_meta":2761137,"kb_avail":374724328,"statfs":{"total":386547056640,"available":383717711872,"internally_reserved":0,"allocated":1839104,"data_stored":722957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2827404855},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1891,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1891,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.080237"},"pg_stats":[{"pgid":"1.f","version":"39'12904","reported_seq":17524,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-05T16:47:40.182534+0000","last_change":"2024-09-05T16:46:27.826222+0000","last_active":"2024-09-05T16:47:40.182534+0000","last_peered":"2024-09-05T16:47:40.182534+0000","last_clean":"2024-09-05T16:47:40.182534+0000","last_became_active":"2024-09-05T16:46:27.757928+0000","last_became_peered":"2024-09-05T16:46:27.757928+0000","last_unstale":"2024-09-05T16:47:40.182534+0000","last_undegraded":"2024-09-05T16:47:40.182534+0000","last_fullsized":"2024-09-05T16:47:40.182534+0000","mapping_epoch":40,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:32:35.713428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12904,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2262,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12552","reported_seq":17237,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-05T16:47:40.185571+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:47:40.185571+0000","last_peered":"2024-09-05T16:47:40.185571+0000","last_clean":"2024-09-05T16:47:40.185571+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:47:40.185571+0000","last_undegraded":"2024-09-05T16:47:40.185571+0000","last_fullsized":"2024-09-05T16:47:40.185571+0000","mapping_epoch":38,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1865,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12752","reported_seq":17395,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-05T16:47:50.188072+0000","last_change":"2024-09-05T16:46:51.992073+0000","last_active":"2024-09-05T16:47:50.188072+0000","last_peered":"2024-09-05T16:47:50.188072+0000","last_clean":"2024-09-05T16:47:50.188072+0000","last_became_active":"2024-09-05T16:46:51.667128+0000","last_became_peered":"2024-09-05T16:46:51.667128+0000","last_unstale":"2024-09-05T16:47:50.188072+0000","last_undegraded":"2024-09-05T16:47:50.188072+0000","last_fullsized":"2024-09-05T16:47:50.188072+0000","mapping_epoch":44,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12752","last_scrub_stamp":"2024-09-05T16:46:51.991995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:51.991995+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:04:54.120031+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12934","reported_seq":17698,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-05T16:47:40.185167+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:47:40.185167+0000","last_peered":"2024-09-05T16:47:40.185167+0000","last_clean":"2024-09-05T16:47:40.185167+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:47:40.185167+0000","last_undegraded":"2024-09-05T16:47:40.185167+0000","last_fullsized":"2024-09-05T16:47:40.185167+0000","mapping_epoch":34,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7234,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12593","reported_seq":16167,"reported_epoch":53,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:48.210034+0000","last_change":"2024-09-05T16:47:38.413103+0000","last_active":"2024-09-05T16:47:48.210034+0000","last_peered":"2024-09-05T16:47:48.210034+0000","last_clean":"2024-09-05T16:33:52.740957+0000","last_became_active":"2024-09-05T16:34:56.855551+0000","last_became_peered":"2024-09-05T16:34:56.855551+0000","last_unstale":"2024-09-05T16:47:48.210034+0000","last_undegraded":"2024-09-05T16:34:56.621356+0000","last_fullsized":"2024-09-05T16:34:56.566239+0000","mapping_epoch":32,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":427,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12591,"num_write_kb":50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1739,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-425}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12772","reported_seq":17369,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-05T16:47:40.182901+0000","last_change":"2024-09-05T16:47:15.832530+0000","last_active":"2024-09-05T16:47:40.182901+0000","last_peered":"2024-09-05T16:47:40.182901+0000","last_clean":"2024-09-05T16:47:40.182901+0000","last_became_active":"2024-09-05T16:47:15.728759+0000","last_became_peered":"2024-09-05T16:47:15.728759+0000","last_unstale":"2024-09-05T16:47:40.182901+0000","last_undegraded":"2024-09-05T16:47:40.182901+0000","last_fullsized":"2024-09-05T16:47:40.182901+0000","mapping_epoch":48,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T02:39:11.515339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12772,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2250,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12660","reported_seq":17086,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-05T16:47:40.182469+0000","last_change":"2024-09-05T16:47:03.683348+0000","last_active":"2024-09-05T16:47:40.182469+0000","last_peered":"2024-09-05T16:47:40.182469+0000","last_clean":"2024-09-05T16:47:40.182469+0000","last_became_active":"2024-09-05T16:47:03.185143+0000","last_became_peered":"2024-09-05T16:47:03.185143+0000","last_unstale":"2024-09-05T16:47:40.182469+0000","last_undegraded":"2024-09-05T16:47:40.182469+0000","last_fullsized":"2024-09-05T16:47:40.182469+0000","mapping_epoch":46,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12660","last_scrub_stamp":"2024-09-05T16:47:03.683262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:03.683262+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:42:16.533299+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12777","reported_seq":12914,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-05T16:47:40.185310+0000","last_change":"2024-09-05T16:46:40.458315+0000","last_active":"2024-09-05T16:47:40.185310+0000","last_peered":"2024-09-05T16:47:40.185310+0000","last_clean":"2024-09-05T16:47:40.185310+0000","last_became_active":"2024-09-05T16:46:40.184288+0000","last_became_peered":"2024-09-05T16:46:40.184288+0000","last_unstale":"2024-09-05T16:47:40.185310+0000","last_undegraded":"2024-09-05T16:47:40.185310+0000","last_fullsized":"2024-09-05T16:47:40.185310+0000","mapping_epoch":42,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12777","last_scrub_stamp":"2024-09-05T16:46:40.458226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:40.458226+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:38:58.734210+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'12770","reported_seq":12905,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:40.182572+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:47:40.182572+0000","last_peered":"2024-09-05T16:47:40.182572+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:47:40.182572+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12767,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2240}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12808","reported_seq":13309,"reported_epoch":54,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:51.018989+0000","last_change":"2024-09-05T16:47:49.791980+0000","last_active":"2024-09-05T16:47:51.018989+0000","last_peered":"2024-09-05T16:47:51.018989+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:47:51.018989+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12806,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":190,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2074}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13120","reported_seq":13281,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:40.183093+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:47:40.183093+0000","last_peered":"2024-09-05T16:47:40.183093+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:47:40.183093+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13117,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2278}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12758","reported_seq":12891,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:40.182662+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:47:40.182662+0000","last_peered":"2024-09-05T16:47:40.182662+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:47:40.182662+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2237}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12808","reported_seq":17578,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-05T16:47:40.185220+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:47:40.185220+0000","last_peered":"2024-09-05T16:47:40.185220+0000","last_clean":"2024-09-05T16:47:40.185220+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:47:40.185220+0000","last_undegraded":"2024-09-05T16:47:40.185220+0000","last_fullsized":"2024-09-05T16:47:40.185220+0000","mapping_epoch":36,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4408,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13124","reported_seq":13267,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:40.184445+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:47:40.184445+0000","last_peered":"2024-09-05T16:47:40.184445+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:47:40.184445+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13122,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2332}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12831","reported_seq":12972,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-05T16:47:40.451467+0000","last_change":"2024-09-05T16:47:40.451380+0000","last_active":"2024-09-05T16:47:40.451467+0000","last_peered":"2024-09-05T16:47:40.451467+0000","last_clean":"2024-09-05T16:47:40.451467+0000","last_became_active":"2024-09-05T16:47:40.189447+0000","last_became_peered":"2024-09-05T16:47:40.189447+0000","last_unstale":"2024-09-05T16:47:40.451467+0000","last_undegraded":"2024-09-05T16:47:40.451467+0000","last_fullsized":"2024-09-05T16:47:40.451467+0000","mapping_epoch":52,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12831","last_scrub_stamp":"2024-09-05T16:47:40.451295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:40.451295+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:46:20.946333+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12773","reported_seq":12914,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-05T16:47:40.185391+0000","last_change":"2024-09-05T16:47:27.437275+0000","last_active":"2024-09-05T16:47:40.185391+0000","last_peered":"2024-09-05T16:47:40.185391+0000","last_clean":"2024-09-05T16:47:40.185391+0000","last_became_active":"2024-09-05T16:47:27.204989+0000","last_became_peered":"2024-09-05T16:47:27.204989+0000","last_unstale":"2024-09-05T16:47:40.185391+0000","last_undegraded":"2024-09-05T16:47:40.185391+0000","last_fullsized":"2024-09-05T16:47:40.185391+0000","mapping_epoch":50,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12773","last_scrub_stamp":"2024-09-05T16:47:27.437187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:27.437187+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T16:53:55.961513+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":94304,"num_objects":14,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11609,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116343,"num_write_kb":417555,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6546,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":61440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084453,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722124,"kb_used_data":436,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649716,"statfs":{"total":96636764160,"available":95897309184,"internally_reserved":0,"allocated":446464,"data_stored":168662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":738979539},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986330,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":613872,"kb_used_data":480,"kb_used_omap":4,"kb_used_meta":613371,"kb_avail":93757968,"statfs":{"total":96636764160,"available":96008159232,"internally_reserved":0,"allocated":491520,"data_stored":213718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":628092626},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673137,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713296,"kb_used_data":444,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":93658544,"statfs":{"total":96636764160,"available":95906349056,"internally_reserved":0,"allocated":454656,"data_stored":171915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":729938373},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182567,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713740,"kb_used_data":436,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658100,"statfs":{"total":96636764160,"available":95905894400,"internally_reserved":0,"allocated":446464,"data_stored":168662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":730394317},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":49152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:47:54.860 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:47:55.160 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:47:55.161 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:47:55.173 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":485,"stamp":"2024-09-05T16:47:55.112570+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":80832,"num_objects":12,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11182,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":103752,"num_write_kb":367182,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4807,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2754748,"kb_used_data":1828,"kb_used_omap":14,"kb_used_meta":2752817,"kb_avail":374732612,"statfs":{"total":386547056640,"available":383726194688,"internally_reserved":0,"allocated":1871872,"data_stored":730988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2818885175},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13472,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1579,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12591,"num_write_kb":-50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-587,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001456"},"pg_stats":[{"pgid":"1.f","version":"39'12904","reported_seq":17529,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-05T16:47:51.189556+0000","last_change":"2024-09-05T16:46:27.826222+0000","last_active":"2024-09-05T16:47:51.189556+0000","last_peered":"2024-09-05T16:47:51.189556+0000","last_clean":"2024-09-05T16:47:51.189556+0000","last_became_active":"2024-09-05T16:46:27.757928+0000","last_became_peered":"2024-09-05T16:46:27.757928+0000","last_unstale":"2024-09-05T16:47:51.189556+0000","last_undegraded":"2024-09-05T16:47:51.189556+0000","last_fullsized":"2024-09-05T16:47:51.189556+0000","mapping_epoch":40,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:32:35.713428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12904,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2262,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12552","reported_seq":17241,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-05T16:47:51.191535+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:47:51.191535+0000","last_peered":"2024-09-05T16:47:51.191535+0000","last_clean":"2024-09-05T16:47:51.191535+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:47:51.191535+0000","last_undegraded":"2024-09-05T16:47:51.191535+0000","last_fullsized":"2024-09-05T16:47:51.191535+0000","mapping_epoch":38,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1865,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12752","reported_seq":17395,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-05T16:47:50.188072+0000","last_change":"2024-09-05T16:46:51.992073+0000","last_active":"2024-09-05T16:47:50.188072+0000","last_peered":"2024-09-05T16:47:50.188072+0000","last_clean":"2024-09-05T16:47:50.188072+0000","last_became_active":"2024-09-05T16:46:51.667128+0000","last_became_peered":"2024-09-05T16:46:51.667128+0000","last_unstale":"2024-09-05T16:47:50.188072+0000","last_undegraded":"2024-09-05T16:47:50.188072+0000","last_fullsized":"2024-09-05T16:47:50.188072+0000","mapping_epoch":44,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12752","last_scrub_stamp":"2024-09-05T16:46:51.991995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:51.991995+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:04:54.120031+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12934","reported_seq":17702,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-05T16:47:51.192160+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:47:51.192160+0000","last_peered":"2024-09-05T16:47:51.192160+0000","last_clean":"2024-09-05T16:47:51.192160+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:47:51.192160+0000","last_undegraded":"2024-09-05T16:47:51.192160+0000","last_fullsized":"2024-09-05T16:47:51.192160+0000","mapping_epoch":34,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7234,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12593","reported_seq":17042,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-05T16:47:51.525026+0000","last_change":"2024-09-05T16:47:51.525026+0000","last_active":"2024-09-05T16:47:51.525026+0000","last_peered":"2024-09-05T16:47:51.525026+0000","last_clean":"2024-09-05T16:47:51.525026+0000","last_became_active":"2024-09-05T16:47:51.193016+0000","last_became_peered":"2024-09-05T16:47:51.193016+0000","last_unstale":"2024-09-05T16:47:51.525026+0000","last_undegraded":"2024-09-05T16:47:51.525026+0000","last_fullsized":"2024-09-05T16:47:51.525026+0000","mapping_epoch":54,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12593","last_scrub_stamp":"2024-09-05T16:47:51.524957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:51.524957+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:46:22.846859+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12772","reported_seq":17373,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-05T16:47:51.190116+0000","last_change":"2024-09-05T16:47:15.832530+0000","last_active":"2024-09-05T16:47:51.190116+0000","last_peered":"2024-09-05T16:47:51.190116+0000","last_clean":"2024-09-05T16:47:51.190116+0000","last_became_active":"2024-09-05T16:47:15.728759+0000","last_became_peered":"2024-09-05T16:47:15.728759+0000","last_unstale":"2024-09-05T16:47:51.190116+0000","last_undegraded":"2024-09-05T16:47:51.190116+0000","last_fullsized":"2024-09-05T16:47:51.190116+0000","mapping_epoch":48,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T02:39:11.515339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12772,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2250,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12660","reported_seq":17090,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-05T16:47:51.186457+0000","last_change":"2024-09-05T16:47:03.683348+0000","last_active":"2024-09-05T16:47:51.186457+0000","last_peered":"2024-09-05T16:47:51.186457+0000","last_clean":"2024-09-05T16:47:51.186457+0000","last_became_active":"2024-09-05T16:47:03.185143+0000","last_became_peered":"2024-09-05T16:47:03.185143+0000","last_unstale":"2024-09-05T16:47:51.186457+0000","last_undegraded":"2024-09-05T16:47:51.186457+0000","last_fullsized":"2024-09-05T16:47:51.186457+0000","mapping_epoch":46,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12660","last_scrub_stamp":"2024-09-05T16:47:03.683262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:03.683262+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:42:16.533299+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12777","reported_seq":12918,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-05T16:47:51.191139+0000","last_change":"2024-09-05T16:46:40.458315+0000","last_active":"2024-09-05T16:47:51.191139+0000","last_peered":"2024-09-05T16:47:51.191139+0000","last_clean":"2024-09-05T16:47:51.191139+0000","last_became_active":"2024-09-05T16:46:40.184288+0000","last_became_peered":"2024-09-05T16:46:40.184288+0000","last_unstale":"2024-09-05T16:47:51.191139+0000","last_undegraded":"2024-09-05T16:47:51.191139+0000","last_fullsized":"2024-09-05T16:47:51.191139+0000","mapping_epoch":42,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12777","last_scrub_stamp":"2024-09-05T16:46:40.458226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:40.458226+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:38:58.734210+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'12770","reported_seq":12909,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:51.189630+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:47:51.189630+0000","last_peered":"2024-09-05T16:47:51.189630+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:47:51.189630+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12767,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2240}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12808","reported_seq":13309,"reported_epoch":54,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:51.018989+0000","last_change":"2024-09-05T16:47:49.791980+0000","last_active":"2024-09-05T16:47:51.018989+0000","last_peered":"2024-09-05T16:47:51.018989+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:47:51.018989+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12806,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":190,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2074}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13120","reported_seq":13285,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:51.187991+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:47:51.187991+0000","last_peered":"2024-09-05T16:47:51.187991+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:47:51.187991+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13117,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2278}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12758","reported_seq":12895,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:51.189778+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:47:51.189778+0000","last_peered":"2024-09-05T16:47:51.189778+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:47:51.189778+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2237}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12808","reported_seq":17582,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-05T16:47:51.192174+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:47:51.192174+0000","last_peered":"2024-09-05T16:47:51.192174+0000","last_clean":"2024-09-05T16:47:51.192174+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:47:51.192174+0000","last_undegraded":"2024-09-05T16:47:51.192174+0000","last_fullsized":"2024-09-05T16:47:51.192174+0000","mapping_epoch":36,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4408,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13124","reported_seq":13271,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:51.188074+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:47:51.188074+0000","last_peered":"2024-09-05T16:47:51.188074+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:47:51.188074+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13122,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2332}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12831","reported_seq":12976,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-05T16:47:51.191044+0000","last_change":"2024-09-05T16:47:40.451380+0000","last_active":"2024-09-05T16:47:51.191044+0000","last_peered":"2024-09-05T16:47:51.191044+0000","last_clean":"2024-09-05T16:47:51.191044+0000","last_became_active":"2024-09-05T16:47:40.189447+0000","last_became_peered":"2024-09-05T16:47:40.189447+0000","last_unstale":"2024-09-05T16:47:51.191044+0000","last_undegraded":"2024-09-05T16:47:51.191044+0000","last_fullsized":"2024-09-05T16:47:51.191044+0000","mapping_epoch":52,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12831","last_scrub_stamp":"2024-09-05T16:47:40.451295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:40.451295+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:46:20.946333+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12773","reported_seq":12918,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-05T16:47:51.191240+0000","last_change":"2024-09-05T16:47:27.437275+0000","last_active":"2024-09-05T16:47:51.191240+0000","last_peered":"2024-09-05T16:47:51.191240+0000","last_clean":"2024-09-05T16:47:51.191240+0000","last_became_active":"2024-09-05T16:47:27.204989+0000","last_became_peered":"2024-09-05T16:47:27.204989+0000","last_unstale":"2024-09-05T16:47:51.191240+0000","last_undegraded":"2024-09-05T16:47:51.191240+0000","last_fullsized":"2024-09-05T16:47:51.191240+0000","mapping_epoch":50,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12773","last_scrub_stamp":"2024-09-05T16:47:27.437187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:27.437187+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T16:53:55.961513+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":80832,"num_objects":12,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11182,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":103752,"num_write_kb":367182,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4807,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":53248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084454,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722140,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649700,"statfs":{"total":96636764160,"available":95897292800,"internally_reserved":0,"allocated":462848,"data_stored":175147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":738979539},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986331,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":605568,"kb_used_data":488,"kb_used_omap":4,"kb_used_meta":605051,"kb_avail":93766272,"statfs":{"total":96636764160,"available":96016662528,"internally_reserved":0,"allocated":499712,"data_stored":212011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":619572946},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673137,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713296,"kb_used_data":444,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":93658544,"statfs":{"total":96636764160,"available":95906349056,"internally_reserved":0,"allocated":454656,"data_stored":171915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":729938373},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182568,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713744,"kb_used_data":444,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658096,"statfs":{"total":96636764160,"available":95905890304,"internally_reserved":0,"allocated":454656,"data_stored":171915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":730394317},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":40960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:47:55.174 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:47:55.468 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:47:55.469 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:47:55.482 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":485,"stamp":"2024-09-05T16:47:55.112570+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":80832,"num_objects":12,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11182,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":103752,"num_write_kb":367182,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4807,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2754748,"kb_used_data":1828,"kb_used_omap":14,"kb_used_meta":2752817,"kb_avail":374732612,"statfs":{"total":386547056640,"available":383726194688,"internally_reserved":0,"allocated":1871872,"data_stored":730988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2818885175},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13472,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1579,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12591,"num_write_kb":-50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-587,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001456"},"pg_stats":[{"pgid":"1.f","version":"39'12904","reported_seq":17529,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-05T16:47:51.189556+0000","last_change":"2024-09-05T16:46:27.826222+0000","last_active":"2024-09-05T16:47:51.189556+0000","last_peered":"2024-09-05T16:47:51.189556+0000","last_clean":"2024-09-05T16:47:51.189556+0000","last_became_active":"2024-09-05T16:46:27.757928+0000","last_became_peered":"2024-09-05T16:46:27.757928+0000","last_unstale":"2024-09-05T16:47:51.189556+0000","last_undegraded":"2024-09-05T16:47:51.189556+0000","last_fullsized":"2024-09-05T16:47:51.189556+0000","mapping_epoch":40,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:32:35.713428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12904,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2262,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12552","reported_seq":17241,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-05T16:47:51.191535+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:47:51.191535+0000","last_peered":"2024-09-05T16:47:51.191535+0000","last_clean":"2024-09-05T16:47:51.191535+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:47:51.191535+0000","last_undegraded":"2024-09-05T16:47:51.191535+0000","last_fullsized":"2024-09-05T16:47:51.191535+0000","mapping_epoch":38,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1865,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12752","reported_seq":17395,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-05T16:47:50.188072+0000","last_change":"2024-09-05T16:46:51.992073+0000","last_active":"2024-09-05T16:47:50.188072+0000","last_peered":"2024-09-05T16:47:50.188072+0000","last_clean":"2024-09-05T16:47:50.188072+0000","last_became_active":"2024-09-05T16:46:51.667128+0000","last_became_peered":"2024-09-05T16:46:51.667128+0000","last_unstale":"2024-09-05T16:47:50.188072+0000","last_undegraded":"2024-09-05T16:47:50.188072+0000","last_fullsized":"2024-09-05T16:47:50.188072+0000","mapping_epoch":44,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12752","last_scrub_stamp":"2024-09-05T16:46:51.991995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:51.991995+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:04:54.120031+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12934","reported_seq":17702,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-05T16:47:51.192160+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:47:51.192160+0000","last_peered":"2024-09-05T16:47:51.192160+0000","last_clean":"2024-09-05T16:47:51.192160+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:47:51.192160+0000","last_undegraded":"2024-09-05T16:47:51.192160+0000","last_fullsized":"2024-09-05T16:47:51.192160+0000","mapping_epoch":34,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7234,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12593","reported_seq":17042,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-05T16:47:51.525026+0000","last_change":"2024-09-05T16:47:51.525026+0000","last_active":"2024-09-05T16:47:51.525026+0000","last_peered":"2024-09-05T16:47:51.525026+0000","last_clean":"2024-09-05T16:47:51.525026+0000","last_became_active":"2024-09-05T16:47:51.193016+0000","last_became_peered":"2024-09-05T16:47:51.193016+0000","last_unstale":"2024-09-05T16:47:51.525026+0000","last_undegraded":"2024-09-05T16:47:51.525026+0000","last_fullsized":"2024-09-05T16:47:51.525026+0000","mapping_epoch":54,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12593","last_scrub_stamp":"2024-09-05T16:47:51.524957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:51.524957+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:46:22.846859+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12772","reported_seq":17373,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-05T16:47:51.190116+0000","last_change":"2024-09-05T16:47:15.832530+0000","last_active":"2024-09-05T16:47:51.190116+0000","last_peered":"2024-09-05T16:47:51.190116+0000","last_clean":"2024-09-05T16:47:51.190116+0000","last_became_active":"2024-09-05T16:47:15.728759+0000","last_became_peered":"2024-09-05T16:47:15.728759+0000","last_unstale":"2024-09-05T16:47:51.190116+0000","last_undegraded":"2024-09-05T16:47:51.190116+0000","last_fullsized":"2024-09-05T16:47:51.190116+0000","mapping_epoch":48,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T02:39:11.515339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12772,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2250,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12660","reported_seq":17090,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-05T16:47:51.186457+0000","last_change":"2024-09-05T16:47:03.683348+0000","last_active":"2024-09-05T16:47:51.186457+0000","last_peered":"2024-09-05T16:47:51.186457+0000","last_clean":"2024-09-05T16:47:51.186457+0000","last_became_active":"2024-09-05T16:47:03.185143+0000","last_became_peered":"2024-09-05T16:47:03.185143+0000","last_unstale":"2024-09-05T16:47:51.186457+0000","last_undegraded":"2024-09-05T16:47:51.186457+0000","last_fullsized":"2024-09-05T16:47:51.186457+0000","mapping_epoch":46,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12660","last_scrub_stamp":"2024-09-05T16:47:03.683262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:03.683262+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:42:16.533299+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12777","reported_seq":12918,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-05T16:47:51.191139+0000","last_change":"2024-09-05T16:46:40.458315+0000","last_active":"2024-09-05T16:47:51.191139+0000","last_peered":"2024-09-05T16:47:51.191139+0000","last_clean":"2024-09-05T16:47:51.191139+0000","last_became_active":"2024-09-05T16:46:40.184288+0000","last_became_peered":"2024-09-05T16:46:40.184288+0000","last_unstale":"2024-09-05T16:47:51.191139+0000","last_undegraded":"2024-09-05T16:47:51.191139+0000","last_fullsized":"2024-09-05T16:47:51.191139+0000","mapping_epoch":42,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12777","last_scrub_stamp":"2024-09-05T16:46:40.458226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:40.458226+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:38:58.734210+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'12770","reported_seq":12909,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:51.189630+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:47:51.189630+0000","last_peered":"2024-09-05T16:47:51.189630+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:47:51.189630+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12767,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2240}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12808","reported_seq":13309,"reported_epoch":54,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:51.018989+0000","last_change":"2024-09-05T16:47:49.791980+0000","last_active":"2024-09-05T16:47:51.018989+0000","last_peered":"2024-09-05T16:47:51.018989+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:47:51.018989+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12806,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":190,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2074}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13120","reported_seq":13285,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:51.187991+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:47:51.187991+0000","last_peered":"2024-09-05T16:47:51.187991+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:47:51.187991+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13117,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2278}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12758","reported_seq":12895,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:51.189778+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:47:51.189778+0000","last_peered":"2024-09-05T16:47:51.189778+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:47:51.189778+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2237}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12808","reported_seq":17582,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-05T16:47:51.192174+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:47:51.192174+0000","last_peered":"2024-09-05T16:47:51.192174+0000","last_clean":"2024-09-05T16:47:51.192174+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:47:51.192174+0000","last_undegraded":"2024-09-05T16:47:51.192174+0000","last_fullsized":"2024-09-05T16:47:51.192174+0000","mapping_epoch":36,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4408,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13124","reported_seq":13271,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:51.188074+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:47:51.188074+0000","last_peered":"2024-09-05T16:47:51.188074+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:47:51.188074+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13122,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2332}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12831","reported_seq":12976,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-05T16:47:51.191044+0000","last_change":"2024-09-05T16:47:40.451380+0000","last_active":"2024-09-05T16:47:51.191044+0000","last_peered":"2024-09-05T16:47:51.191044+0000","last_clean":"2024-09-05T16:47:51.191044+0000","last_became_active":"2024-09-05T16:47:40.189447+0000","last_became_peered":"2024-09-05T16:47:40.189447+0000","last_unstale":"2024-09-05T16:47:51.191044+0000","last_undegraded":"2024-09-05T16:47:51.191044+0000","last_fullsized":"2024-09-05T16:47:51.191044+0000","mapping_epoch":52,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12831","last_scrub_stamp":"2024-09-05T16:47:40.451295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:40.451295+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:46:20.946333+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12773","reported_seq":12918,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-05T16:47:51.191240+0000","last_change":"2024-09-05T16:47:27.437275+0000","last_active":"2024-09-05T16:47:51.191240+0000","last_peered":"2024-09-05T16:47:51.191240+0000","last_clean":"2024-09-05T16:47:51.191240+0000","last_became_active":"2024-09-05T16:47:27.204989+0000","last_became_peered":"2024-09-05T16:47:27.204989+0000","last_unstale":"2024-09-05T16:47:51.191240+0000","last_undegraded":"2024-09-05T16:47:51.191240+0000","last_fullsized":"2024-09-05T16:47:51.191240+0000","mapping_epoch":50,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12773","last_scrub_stamp":"2024-09-05T16:47:27.437187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:27.437187+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T16:53:55.961513+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":80832,"num_objects":12,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11182,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":103752,"num_write_kb":367182,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4807,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":53248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084454,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722140,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649700,"statfs":{"total":96636764160,"available":95897292800,"internally_reserved":0,"allocated":462848,"data_stored":175147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":738979539},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986331,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":605568,"kb_used_data":488,"kb_used_omap":4,"kb_used_meta":605051,"kb_avail":93766272,"statfs":{"total":96636764160,"available":96016662528,"internally_reserved":0,"allocated":499712,"data_stored":212011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":619572946},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673137,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713296,"kb_used_data":444,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":93658544,"statfs":{"total":96636764160,"available":95906349056,"internally_reserved":0,"allocated":454656,"data_stored":171915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":729938373},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182568,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713744,"kb_used_data":444,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658096,"statfs":{"total":96636764160,"available":95905890304,"internally_reserved":0,"allocated":454656,"data_stored":171915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":730394317},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":40960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:47:55.483 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:47:55.483 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "39'12770", 'reported_seq': 12909, 'reported_epoch': 55, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:47:51.189630+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:47:51.189630+0000', 'last_peered': '2024-09-05T16:47:51.189630+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:47:51.189630+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "33'2700", 'ondisk_log_start': "33'2700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10070, 'log_dups_size': 0, 'ondisk_log_size': 10070, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26944, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 16, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12767, 'num_write_kb': 51081, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2240}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:47:55.483 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:47:55.483 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "39'12808", 'reported_seq': 13309, 'reported_epoch': 54, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:47:51.018989+0000', 'last_change': '2024-09-05T16:47:49.791980+0000', 'last_active': '2024-09-05T16:47:51.018989+0000', 'last_peered': '2024-09-05T16:47:51.018989+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:47:51.018989+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "33'2800", 'ondisk_log_start': "33'2800", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10008, 'log_dups_size': 0, 'ondisk_log_size': 10008, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2076, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12806, 'num_write_kb': 51230, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 190, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -2074}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:47:55.483 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:47:55.483 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "39'13120", 'reported_seq': 13285, 'reported_epoch': 55, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:47:51.187991+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:47:51.187991+0000', 'last_peered': '2024-09-05T16:47:51.187991+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:47:51.187991+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "33'3100", 'ondisk_log_start': "33'3100", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20208, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2284, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13117, 'num_write_kb': 52476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2278}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:47:55.484 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:47:55.484 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "39'12758", 'reported_seq': 12895, 'reported_epoch': 55, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:47:51.189778+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:47:51.189778+0000', 'last_peered': '2024-09-05T16:47:51.189778+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:47:51.189778+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "33'2700", 'ondisk_log_start': "33'2700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10058, 'log_dups_size': 0, 'ondisk_log_size': 10058, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6736, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12757, 'num_write_kb': 51031, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2237}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:47:55.484 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:47:55.484 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "39'13124", 'reported_seq': 13271, 'reported_epoch': 55, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:47:51.188074+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:47:51.188074+0000', 'last_peered': '2024-09-05T16:47:51.188074+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:47:51.188074+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "33'3100", 'ondisk_log_start': "33'3100", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10024, 'log_dups_size': 0, 'ondisk_log_size': 10024, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13122, 'num_write_kb': 52500, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2332}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:47:55.484 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:47:55.894 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:47:55.894 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 11182/48 objects degraded (23295.833%), 5 pgs degraded, 5 pgs undersized","count":10},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":879,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":55,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":5},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":11},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":4},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":12,"data_bytes":80832,"bytes_used":2820861952,"bytes_avail":383726194688,"bytes_total":386547056640,"degraded_objects":11182,"degraded_total":48,"degraded_ratio":232.95833333333337},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:47:55.907 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-09-05T16:47:55.907 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:47:56.257 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:47:56.258 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:47:56.269 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":485,"stamp":"2024-09-05T16:47:55.112570+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":80832,"num_objects":12,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11182,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":103752,"num_write_kb":367182,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4807,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2754748,"kb_used_data":1828,"kb_used_omap":14,"kb_used_meta":2752817,"kb_avail":374732612,"statfs":{"total":386547056640,"available":383726194688,"internally_reserved":0,"allocated":1871872,"data_stored":730988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2818885175},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13472,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1579,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12591,"num_write_kb":-50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-587,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001456"},"pg_stats":[{"pgid":"1.f","version":"39'12904","reported_seq":17529,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-05T16:47:51.189556+0000","last_change":"2024-09-05T16:46:27.826222+0000","last_active":"2024-09-05T16:47:51.189556+0000","last_peered":"2024-09-05T16:47:51.189556+0000","last_clean":"2024-09-05T16:47:51.189556+0000","last_became_active":"2024-09-05T16:46:27.757928+0000","last_became_peered":"2024-09-05T16:46:27.757928+0000","last_unstale":"2024-09-05T16:47:51.189556+0000","last_undegraded":"2024-09-05T16:47:51.189556+0000","last_fullsized":"2024-09-05T16:47:51.189556+0000","mapping_epoch":40,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:32:35.713428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12904,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2262,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12552","reported_seq":17241,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-05T16:47:51.191535+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:47:51.191535+0000","last_peered":"2024-09-05T16:47:51.191535+0000","last_clean":"2024-09-05T16:47:51.191535+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:47:51.191535+0000","last_undegraded":"2024-09-05T16:47:51.191535+0000","last_fullsized":"2024-09-05T16:47:51.191535+0000","mapping_epoch":38,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1865,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12752","reported_seq":17395,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-05T16:47:50.188072+0000","last_change":"2024-09-05T16:46:51.992073+0000","last_active":"2024-09-05T16:47:50.188072+0000","last_peered":"2024-09-05T16:47:50.188072+0000","last_clean":"2024-09-05T16:47:50.188072+0000","last_became_active":"2024-09-05T16:46:51.667128+0000","last_became_peered":"2024-09-05T16:46:51.667128+0000","last_unstale":"2024-09-05T16:47:50.188072+0000","last_undegraded":"2024-09-05T16:47:50.188072+0000","last_fullsized":"2024-09-05T16:47:50.188072+0000","mapping_epoch":44,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12752","last_scrub_stamp":"2024-09-05T16:46:51.991995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:51.991995+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:04:54.120031+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12934","reported_seq":17702,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-05T16:47:51.192160+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:47:51.192160+0000","last_peered":"2024-09-05T16:47:51.192160+0000","last_clean":"2024-09-05T16:47:51.192160+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:47:51.192160+0000","last_undegraded":"2024-09-05T16:47:51.192160+0000","last_fullsized":"2024-09-05T16:47:51.192160+0000","mapping_epoch":34,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7234,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12593","reported_seq":17042,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-05T16:47:51.525026+0000","last_change":"2024-09-05T16:47:51.525026+0000","last_active":"2024-09-05T16:47:51.525026+0000","last_peered":"2024-09-05T16:47:51.525026+0000","last_clean":"2024-09-05T16:47:51.525026+0000","last_became_active":"2024-09-05T16:47:51.193016+0000","last_became_peered":"2024-09-05T16:47:51.193016+0000","last_unstale":"2024-09-05T16:47:51.525026+0000","last_undegraded":"2024-09-05T16:47:51.525026+0000","last_fullsized":"2024-09-05T16:47:51.525026+0000","mapping_epoch":54,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12593","last_scrub_stamp":"2024-09-05T16:47:51.524957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:51.524957+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:46:22.846859+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12772","reported_seq":17373,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-05T16:47:51.190116+0000","last_change":"2024-09-05T16:47:15.832530+0000","last_active":"2024-09-05T16:47:51.190116+0000","last_peered":"2024-09-05T16:47:51.190116+0000","last_clean":"2024-09-05T16:47:51.190116+0000","last_became_active":"2024-09-05T16:47:15.728759+0000","last_became_peered":"2024-09-05T16:47:15.728759+0000","last_unstale":"2024-09-05T16:47:51.190116+0000","last_undegraded":"2024-09-05T16:47:51.190116+0000","last_fullsized":"2024-09-05T16:47:51.190116+0000","mapping_epoch":48,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T02:39:11.515339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12772,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2250,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12660","reported_seq":17090,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-05T16:47:51.186457+0000","last_change":"2024-09-05T16:47:03.683348+0000","last_active":"2024-09-05T16:47:51.186457+0000","last_peered":"2024-09-05T16:47:51.186457+0000","last_clean":"2024-09-05T16:47:51.186457+0000","last_became_active":"2024-09-05T16:47:03.185143+0000","last_became_peered":"2024-09-05T16:47:03.185143+0000","last_unstale":"2024-09-05T16:47:51.186457+0000","last_undegraded":"2024-09-05T16:47:51.186457+0000","last_fullsized":"2024-09-05T16:47:51.186457+0000","mapping_epoch":46,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12660","last_scrub_stamp":"2024-09-05T16:47:03.683262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:03.683262+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:42:16.533299+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12777","reported_seq":12918,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-05T16:47:51.191139+0000","last_change":"2024-09-05T16:46:40.458315+0000","last_active":"2024-09-05T16:47:51.191139+0000","last_peered":"2024-09-05T16:47:51.191139+0000","last_clean":"2024-09-05T16:47:51.191139+0000","last_became_active":"2024-09-05T16:46:40.184288+0000","last_became_peered":"2024-09-05T16:46:40.184288+0000","last_unstale":"2024-09-05T16:47:51.191139+0000","last_undegraded":"2024-09-05T16:47:51.191139+0000","last_fullsized":"2024-09-05T16:47:51.191139+0000","mapping_epoch":42,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12777","last_scrub_stamp":"2024-09-05T16:46:40.458226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:40.458226+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:38:58.734210+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'12770","reported_seq":12909,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:51.189630+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:47:51.189630+0000","last_peered":"2024-09-05T16:47:51.189630+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:47:51.189630+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12767,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2240}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12808","reported_seq":13309,"reported_epoch":54,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:51.018989+0000","last_change":"2024-09-05T16:47:49.791980+0000","last_active":"2024-09-05T16:47:51.018989+0000","last_peered":"2024-09-05T16:47:51.018989+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:47:51.018989+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12806,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":190,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2074}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13120","reported_seq":13285,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:51.187991+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:47:51.187991+0000","last_peered":"2024-09-05T16:47:51.187991+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:47:51.187991+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13117,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2278}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12758","reported_seq":12895,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:51.189778+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:47:51.189778+0000","last_peered":"2024-09-05T16:47:51.189778+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:47:51.189778+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2237}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12808","reported_seq":17582,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-05T16:47:51.192174+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:47:51.192174+0000","last_peered":"2024-09-05T16:47:51.192174+0000","last_clean":"2024-09-05T16:47:51.192174+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:47:51.192174+0000","last_undegraded":"2024-09-05T16:47:51.192174+0000","last_fullsized":"2024-09-05T16:47:51.192174+0000","mapping_epoch":36,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4408,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13124","reported_seq":13271,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:51.188074+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:47:51.188074+0000","last_peered":"2024-09-05T16:47:51.188074+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:47:51.188074+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13122,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2332}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12831","reported_seq":12976,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-05T16:47:51.191044+0000","last_change":"2024-09-05T16:47:40.451380+0000","last_active":"2024-09-05T16:47:51.191044+0000","last_peered":"2024-09-05T16:47:51.191044+0000","last_clean":"2024-09-05T16:47:51.191044+0000","last_became_active":"2024-09-05T16:47:40.189447+0000","last_became_peered":"2024-09-05T16:47:40.189447+0000","last_unstale":"2024-09-05T16:47:51.191044+0000","last_undegraded":"2024-09-05T16:47:51.191044+0000","last_fullsized":"2024-09-05T16:47:51.191044+0000","mapping_epoch":52,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12831","last_scrub_stamp":"2024-09-05T16:47:40.451295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:40.451295+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:46:20.946333+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12773","reported_seq":12918,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-05T16:47:51.191240+0000","last_change":"2024-09-05T16:47:27.437275+0000","last_active":"2024-09-05T16:47:51.191240+0000","last_peered":"2024-09-05T16:47:51.191240+0000","last_clean":"2024-09-05T16:47:51.191240+0000","last_became_active":"2024-09-05T16:47:27.204989+0000","last_became_peered":"2024-09-05T16:47:27.204989+0000","last_unstale":"2024-09-05T16:47:51.191240+0000","last_undegraded":"2024-09-05T16:47:51.191240+0000","last_fullsized":"2024-09-05T16:47:51.191240+0000","mapping_epoch":50,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12773","last_scrub_stamp":"2024-09-05T16:47:27.437187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:27.437187+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T16:53:55.961513+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":80832,"num_objects":12,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11182,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":103752,"num_write_kb":367182,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4807,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":53248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084454,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722140,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649700,"statfs":{"total":96636764160,"available":95897292800,"internally_reserved":0,"allocated":462848,"data_stored":175147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":738979539},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986331,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":605568,"kb_used_data":488,"kb_used_omap":4,"kb_used_meta":605051,"kb_avail":93766272,"statfs":{"total":96636764160,"available":96016662528,"internally_reserved":0,"allocated":499712,"data_stored":212011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":619572946},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673137,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713296,"kb_used_data":444,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":93658544,"statfs":{"total":96636764160,"available":95906349056,"internally_reserved":0,"allocated":454656,"data_stored":171915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":729938373},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182568,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713744,"kb_used_data":444,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658096,"statfs":{"total":96636764160,"available":95905890304,"internally_reserved":0,"allocated":454656,"data_stored":171915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":730394317},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":40960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:47:59.271 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:47:59.603 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:47:59.604 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:47:59.616 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":487,"stamp":"2024-09-05T16:47:59.113196+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":80832,"num_objects":12,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10376,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":103752,"num_write_kb":367182,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5613,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2763080,"kb_used_data":1844,"kb_used_omap":14,"kb_used_meta":2761137,"kb_avail":374724280,"statfs":{"total":386547056640,"available":383717662720,"internally_reserved":0,"allocated":1888256,"data_stored":737452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2827404855},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13472,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1404,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12591,"num_write_kb":-50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-762,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.932778"},"pg_stats":[{"pgid":"1.f","version":"39'12904","reported_seq":17529,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-05T16:47:51.189556+0000","last_change":"2024-09-05T16:46:27.826222+0000","last_active":"2024-09-05T16:47:51.189556+0000","last_peered":"2024-09-05T16:47:51.189556+0000","last_clean":"2024-09-05T16:47:51.189556+0000","last_became_active":"2024-09-05T16:46:27.757928+0000","last_became_peered":"2024-09-05T16:46:27.757928+0000","last_unstale":"2024-09-05T16:47:51.189556+0000","last_undegraded":"2024-09-05T16:47:51.189556+0000","last_fullsized":"2024-09-05T16:47:51.189556+0000","mapping_epoch":40,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:32:35.713428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12904,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2262,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12552","reported_seq":17241,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-05T16:47:51.191535+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:47:51.191535+0000","last_peered":"2024-09-05T16:47:51.191535+0000","last_clean":"2024-09-05T16:47:51.191535+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:47:51.191535+0000","last_undegraded":"2024-09-05T16:47:51.191535+0000","last_fullsized":"2024-09-05T16:47:51.191535+0000","mapping_epoch":38,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1865,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12752","reported_seq":17397,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-05T16:47:51.191440+0000","last_change":"2024-09-05T16:46:51.992073+0000","last_active":"2024-09-05T16:47:51.191440+0000","last_peered":"2024-09-05T16:47:51.191440+0000","last_clean":"2024-09-05T16:47:51.191440+0000","last_became_active":"2024-09-05T16:46:51.667128+0000","last_became_peered":"2024-09-05T16:46:51.667128+0000","last_unstale":"2024-09-05T16:47:51.191440+0000","last_undegraded":"2024-09-05T16:47:51.191440+0000","last_fullsized":"2024-09-05T16:47:51.191440+0000","mapping_epoch":44,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12752","last_scrub_stamp":"2024-09-05T16:46:51.991995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:51.991995+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:04:54.120031+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12934","reported_seq":17702,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-05T16:47:51.192160+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:47:51.192160+0000","last_peered":"2024-09-05T16:47:51.192160+0000","last_clean":"2024-09-05T16:47:51.192160+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:47:51.192160+0000","last_undegraded":"2024-09-05T16:47:51.192160+0000","last_fullsized":"2024-09-05T16:47:51.192160+0000","mapping_epoch":34,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7234,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12593","reported_seq":17042,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-05T16:47:51.525026+0000","last_change":"2024-09-05T16:47:51.525026+0000","last_active":"2024-09-05T16:47:51.525026+0000","last_peered":"2024-09-05T16:47:51.525026+0000","last_clean":"2024-09-05T16:47:51.525026+0000","last_became_active":"2024-09-05T16:47:51.193016+0000","last_became_peered":"2024-09-05T16:47:51.193016+0000","last_unstale":"2024-09-05T16:47:51.525026+0000","last_undegraded":"2024-09-05T16:47:51.525026+0000","last_fullsized":"2024-09-05T16:47:51.525026+0000","mapping_epoch":54,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12593","last_scrub_stamp":"2024-09-05T16:47:51.524957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:51.524957+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:46:22.846859+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12772","reported_seq":17373,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-05T16:47:51.190116+0000","last_change":"2024-09-05T16:47:15.832530+0000","last_active":"2024-09-05T16:47:51.190116+0000","last_peered":"2024-09-05T16:47:51.190116+0000","last_clean":"2024-09-05T16:47:51.190116+0000","last_became_active":"2024-09-05T16:47:15.728759+0000","last_became_peered":"2024-09-05T16:47:15.728759+0000","last_unstale":"2024-09-05T16:47:51.190116+0000","last_undegraded":"2024-09-05T16:47:51.190116+0000","last_fullsized":"2024-09-05T16:47:51.190116+0000","mapping_epoch":48,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T02:39:11.515339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12772,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2250,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12660","reported_seq":17090,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-05T16:47:51.186457+0000","last_change":"2024-09-05T16:47:03.683348+0000","last_active":"2024-09-05T16:47:51.186457+0000","last_peered":"2024-09-05T16:47:51.186457+0000","last_clean":"2024-09-05T16:47:51.186457+0000","last_became_active":"2024-09-05T16:47:03.185143+0000","last_became_peered":"2024-09-05T16:47:03.185143+0000","last_unstale":"2024-09-05T16:47:51.186457+0000","last_undegraded":"2024-09-05T16:47:51.186457+0000","last_fullsized":"2024-09-05T16:47:51.186457+0000","mapping_epoch":46,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12660","last_scrub_stamp":"2024-09-05T16:47:03.683262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:03.683262+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:42:16.533299+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12777","reported_seq":12918,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-05T16:47:51.191139+0000","last_change":"2024-09-05T16:46:40.458315+0000","last_active":"2024-09-05T16:47:51.191139+0000","last_peered":"2024-09-05T16:47:51.191139+0000","last_clean":"2024-09-05T16:47:51.191139+0000","last_became_active":"2024-09-05T16:46:40.184288+0000","last_became_peered":"2024-09-05T16:46:40.184288+0000","last_unstale":"2024-09-05T16:47:51.191139+0000","last_undegraded":"2024-09-05T16:47:51.191139+0000","last_fullsized":"2024-09-05T16:47:51.191139+0000","mapping_epoch":42,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12777","last_scrub_stamp":"2024-09-05T16:46:40.458226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:40.458226+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:38:58.734210+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'12770","reported_seq":12909,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:51.189630+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:47:51.189630+0000","last_peered":"2024-09-05T16:47:51.189630+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:47:51.189630+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12767,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2240}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12808","reported_seq":14923,"reported_epoch":55,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:56.015193+0000","last_change":"2024-09-05T16:47:49.791980+0000","last_active":"2024-09-05T16:47:56.015193+0000","last_peered":"2024-09-05T16:47:56.015193+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:47:56.015193+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12806,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":996,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1268}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13120","reported_seq":13285,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:51.187991+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:47:51.187991+0000","last_peered":"2024-09-05T16:47:51.187991+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:47:51.187991+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13117,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2278}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12758","reported_seq":12895,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:51.189778+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:47:51.189778+0000","last_peered":"2024-09-05T16:47:51.189778+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:47:51.189778+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2237}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12808","reported_seq":17582,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-05T16:47:51.192174+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:47:51.192174+0000","last_peered":"2024-09-05T16:47:51.192174+0000","last_clean":"2024-09-05T16:47:51.192174+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:47:51.192174+0000","last_undegraded":"2024-09-05T16:47:51.192174+0000","last_fullsized":"2024-09-05T16:47:51.192174+0000","mapping_epoch":36,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4408,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13124","reported_seq":13271,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:51.188074+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:47:51.188074+0000","last_peered":"2024-09-05T16:47:51.188074+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:47:51.188074+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13122,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2332}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12831","reported_seq":12976,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-05T16:47:51.191044+0000","last_change":"2024-09-05T16:47:40.451380+0000","last_active":"2024-09-05T16:47:51.191044+0000","last_peered":"2024-09-05T16:47:51.191044+0000","last_clean":"2024-09-05T16:47:51.191044+0000","last_became_active":"2024-09-05T16:47:40.189447+0000","last_became_peered":"2024-09-05T16:47:40.189447+0000","last_unstale":"2024-09-05T16:47:51.191044+0000","last_undegraded":"2024-09-05T16:47:51.191044+0000","last_fullsized":"2024-09-05T16:47:51.191044+0000","mapping_epoch":52,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12831","last_scrub_stamp":"2024-09-05T16:47:40.451295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:40.451295+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:46:20.946333+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12773","reported_seq":12918,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-05T16:47:51.191240+0000","last_change":"2024-09-05T16:47:27.437275+0000","last_active":"2024-09-05T16:47:51.191240+0000","last_peered":"2024-09-05T16:47:51.191240+0000","last_clean":"2024-09-05T16:47:51.191240+0000","last_became_active":"2024-09-05T16:47:27.204989+0000","last_became_peered":"2024-09-05T16:47:27.204989+0000","last_unstale":"2024-09-05T16:47:51.191240+0000","last_undegraded":"2024-09-05T16:47:51.191240+0000","last_fullsized":"2024-09-05T16:47:51.191240+0000","mapping_epoch":50,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12773","last_scrub_stamp":"2024-09-05T16:47:27.437187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:27.437187+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T16:53:55.961513+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":80832,"num_objects":12,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10376,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":103752,"num_write_kb":367182,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5613,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":53248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084455,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722140,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649700,"statfs":{"total":96636764160,"available":95897292800,"internally_reserved":0,"allocated":462848,"data_stored":175147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":738979539},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986332,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":613880,"kb_used_data":488,"kb_used_omap":4,"kb_used_meta":613371,"kb_avail":93757960,"statfs":{"total":96636764160,"available":96008151040,"internally_reserved":0,"allocated":499712,"data_stored":212011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":628092626},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673138,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713304,"kb_used_data":452,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":93658536,"statfs":{"total":96636764160,"available":95906340864,"internally_reserved":0,"allocated":462848,"data_stored":175147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":729938373},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182569,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713756,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658084,"statfs":{"total":96636764160,"available":95905878016,"internally_reserved":0,"allocated":462848,"data_stored":175147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":730394317},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":40960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:47:59.617 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:47:59.939 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:47:59.940 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:47:59.952 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":487,"stamp":"2024-09-05T16:47:59.113196+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":80832,"num_objects":12,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10376,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":103752,"num_write_kb":367182,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5613,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2763080,"kb_used_data":1844,"kb_used_omap":14,"kb_used_meta":2761137,"kb_avail":374724280,"statfs":{"total":386547056640,"available":383717662720,"internally_reserved":0,"allocated":1888256,"data_stored":737452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2827404855},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13472,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1404,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12591,"num_write_kb":-50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-762,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.932778"},"pg_stats":[{"pgid":"1.f","version":"39'12904","reported_seq":17529,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-05T16:47:51.189556+0000","last_change":"2024-09-05T16:46:27.826222+0000","last_active":"2024-09-05T16:47:51.189556+0000","last_peered":"2024-09-05T16:47:51.189556+0000","last_clean":"2024-09-05T16:47:51.189556+0000","last_became_active":"2024-09-05T16:46:27.757928+0000","last_became_peered":"2024-09-05T16:46:27.757928+0000","last_unstale":"2024-09-05T16:47:51.189556+0000","last_undegraded":"2024-09-05T16:47:51.189556+0000","last_fullsized":"2024-09-05T16:47:51.189556+0000","mapping_epoch":40,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:32:35.713428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12904,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2262,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12552","reported_seq":17241,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-05T16:47:51.191535+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:47:51.191535+0000","last_peered":"2024-09-05T16:47:51.191535+0000","last_clean":"2024-09-05T16:47:51.191535+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:47:51.191535+0000","last_undegraded":"2024-09-05T16:47:51.191535+0000","last_fullsized":"2024-09-05T16:47:51.191535+0000","mapping_epoch":38,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1865,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12752","reported_seq":17397,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-05T16:47:51.191440+0000","last_change":"2024-09-05T16:46:51.992073+0000","last_active":"2024-09-05T16:47:51.191440+0000","last_peered":"2024-09-05T16:47:51.191440+0000","last_clean":"2024-09-05T16:47:51.191440+0000","last_became_active":"2024-09-05T16:46:51.667128+0000","last_became_peered":"2024-09-05T16:46:51.667128+0000","last_unstale":"2024-09-05T16:47:51.191440+0000","last_undegraded":"2024-09-05T16:47:51.191440+0000","last_fullsized":"2024-09-05T16:47:51.191440+0000","mapping_epoch":44,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12752","last_scrub_stamp":"2024-09-05T16:46:51.991995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:51.991995+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:04:54.120031+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12934","reported_seq":17702,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-05T16:47:51.192160+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:47:51.192160+0000","last_peered":"2024-09-05T16:47:51.192160+0000","last_clean":"2024-09-05T16:47:51.192160+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:47:51.192160+0000","last_undegraded":"2024-09-05T16:47:51.192160+0000","last_fullsized":"2024-09-05T16:47:51.192160+0000","mapping_epoch":34,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7234,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12593","reported_seq":17042,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-05T16:47:51.525026+0000","last_change":"2024-09-05T16:47:51.525026+0000","last_active":"2024-09-05T16:47:51.525026+0000","last_peered":"2024-09-05T16:47:51.525026+0000","last_clean":"2024-09-05T16:47:51.525026+0000","last_became_active":"2024-09-05T16:47:51.193016+0000","last_became_peered":"2024-09-05T16:47:51.193016+0000","last_unstale":"2024-09-05T16:47:51.525026+0000","last_undegraded":"2024-09-05T16:47:51.525026+0000","last_fullsized":"2024-09-05T16:47:51.525026+0000","mapping_epoch":54,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12593","last_scrub_stamp":"2024-09-05T16:47:51.524957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:51.524957+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:46:22.846859+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12772","reported_seq":17373,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-05T16:47:51.190116+0000","last_change":"2024-09-05T16:47:15.832530+0000","last_active":"2024-09-05T16:47:51.190116+0000","last_peered":"2024-09-05T16:47:51.190116+0000","last_clean":"2024-09-05T16:47:51.190116+0000","last_became_active":"2024-09-05T16:47:15.728759+0000","last_became_peered":"2024-09-05T16:47:15.728759+0000","last_unstale":"2024-09-05T16:47:51.190116+0000","last_undegraded":"2024-09-05T16:47:51.190116+0000","last_fullsized":"2024-09-05T16:47:51.190116+0000","mapping_epoch":48,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T02:39:11.515339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12772,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2250,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12660","reported_seq":17090,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-05T16:47:51.186457+0000","last_change":"2024-09-05T16:47:03.683348+0000","last_active":"2024-09-05T16:47:51.186457+0000","last_peered":"2024-09-05T16:47:51.186457+0000","last_clean":"2024-09-05T16:47:51.186457+0000","last_became_active":"2024-09-05T16:47:03.185143+0000","last_became_peered":"2024-09-05T16:47:03.185143+0000","last_unstale":"2024-09-05T16:47:51.186457+0000","last_undegraded":"2024-09-05T16:47:51.186457+0000","last_fullsized":"2024-09-05T16:47:51.186457+0000","mapping_epoch":46,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12660","last_scrub_stamp":"2024-09-05T16:47:03.683262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:03.683262+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:42:16.533299+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12777","reported_seq":12918,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-05T16:47:51.191139+0000","last_change":"2024-09-05T16:46:40.458315+0000","last_active":"2024-09-05T16:47:51.191139+0000","last_peered":"2024-09-05T16:47:51.191139+0000","last_clean":"2024-09-05T16:47:51.191139+0000","last_became_active":"2024-09-05T16:46:40.184288+0000","last_became_peered":"2024-09-05T16:46:40.184288+0000","last_unstale":"2024-09-05T16:47:51.191139+0000","last_undegraded":"2024-09-05T16:47:51.191139+0000","last_fullsized":"2024-09-05T16:47:51.191139+0000","mapping_epoch":42,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12777","last_scrub_stamp":"2024-09-05T16:46:40.458226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:40.458226+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:38:58.734210+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'12770","reported_seq":12909,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:51.189630+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:47:51.189630+0000","last_peered":"2024-09-05T16:47:51.189630+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:47:51.189630+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12767,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2240}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12808","reported_seq":14923,"reported_epoch":55,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:56.015193+0000","last_change":"2024-09-05T16:47:49.791980+0000","last_active":"2024-09-05T16:47:56.015193+0000","last_peered":"2024-09-05T16:47:56.015193+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:47:56.015193+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12806,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":996,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1268}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13120","reported_seq":13285,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:51.187991+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:47:51.187991+0000","last_peered":"2024-09-05T16:47:51.187991+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:47:51.187991+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13117,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2278}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12758","reported_seq":12895,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:51.189778+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:47:51.189778+0000","last_peered":"2024-09-05T16:47:51.189778+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:47:51.189778+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2237}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12808","reported_seq":17582,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-05T16:47:51.192174+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:47:51.192174+0000","last_peered":"2024-09-05T16:47:51.192174+0000","last_clean":"2024-09-05T16:47:51.192174+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:47:51.192174+0000","last_undegraded":"2024-09-05T16:47:51.192174+0000","last_fullsized":"2024-09-05T16:47:51.192174+0000","mapping_epoch":36,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4408,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13124","reported_seq":13271,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:51.188074+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:47:51.188074+0000","last_peered":"2024-09-05T16:47:51.188074+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:47:51.188074+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13122,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2332}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12831","reported_seq":12976,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-05T16:47:51.191044+0000","last_change":"2024-09-05T16:47:40.451380+0000","last_active":"2024-09-05T16:47:51.191044+0000","last_peered":"2024-09-05T16:47:51.191044+0000","last_clean":"2024-09-05T16:47:51.191044+0000","last_became_active":"2024-09-05T16:47:40.189447+0000","last_became_peered":"2024-09-05T16:47:40.189447+0000","last_unstale":"2024-09-05T16:47:51.191044+0000","last_undegraded":"2024-09-05T16:47:51.191044+0000","last_fullsized":"2024-09-05T16:47:51.191044+0000","mapping_epoch":52,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12831","last_scrub_stamp":"2024-09-05T16:47:40.451295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:40.451295+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:46:20.946333+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12773","reported_seq":12918,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-05T16:47:51.191240+0000","last_change":"2024-09-05T16:47:27.437275+0000","last_active":"2024-09-05T16:47:51.191240+0000","last_peered":"2024-09-05T16:47:51.191240+0000","last_clean":"2024-09-05T16:47:51.191240+0000","last_became_active":"2024-09-05T16:47:27.204989+0000","last_became_peered":"2024-09-05T16:47:27.204989+0000","last_unstale":"2024-09-05T16:47:51.191240+0000","last_undegraded":"2024-09-05T16:47:51.191240+0000","last_fullsized":"2024-09-05T16:47:51.191240+0000","mapping_epoch":50,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12773","last_scrub_stamp":"2024-09-05T16:47:27.437187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:27.437187+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T16:53:55.961513+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":80832,"num_objects":12,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10376,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":103752,"num_write_kb":367182,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5613,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":53248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084455,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722140,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649700,"statfs":{"total":96636764160,"available":95897292800,"internally_reserved":0,"allocated":462848,"data_stored":175147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":738979539},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986332,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":613880,"kb_used_data":488,"kb_used_omap":4,"kb_used_meta":613371,"kb_avail":93757960,"statfs":{"total":96636764160,"available":96008151040,"internally_reserved":0,"allocated":499712,"data_stored":212011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":628092626},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673138,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713304,"kb_used_data":452,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":93658536,"statfs":{"total":96636764160,"available":95906340864,"internally_reserved":0,"allocated":462848,"data_stored":175147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":729938373},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182569,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713756,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658084,"statfs":{"total":96636764160,"available":95905878016,"internally_reserved":0,"allocated":462848,"data_stored":175147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":730394317},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":40960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:47:59.953 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:47:59.954 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "39'12770", 'reported_seq': 12909, 'reported_epoch': 55, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:47:51.189630+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:47:51.189630+0000', 'last_peered': '2024-09-05T16:47:51.189630+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:47:51.189630+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "33'2700", 'ondisk_log_start': "33'2700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10070, 'log_dups_size': 0, 'ondisk_log_size': 10070, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26944, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 16, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12767, 'num_write_kb': 51081, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2240}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:47:59.954 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:47:59.954 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "39'12808", 'reported_seq': 14923, 'reported_epoch': 55, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:47:56.015193+0000', 'last_change': '2024-09-05T16:47:49.791980+0000', 'last_active': '2024-09-05T16:47:56.015193+0000', 'last_peered': '2024-09-05T16:47:56.015193+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:47:56.015193+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "33'2800", 'ondisk_log_start': "33'2800", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10008, 'log_dups_size': 0, 'ondisk_log_size': 10008, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1270, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12806, 'num_write_kb': 51230, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 996, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1268}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:47:59.954 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:47:59.954 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "39'13120", 'reported_seq': 13285, 'reported_epoch': 55, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:47:51.187991+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:47:51.187991+0000', 'last_peered': '2024-09-05T16:47:51.187991+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:47:51.187991+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "33'3100", 'ondisk_log_start': "33'3100", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20208, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2284, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13117, 'num_write_kb': 52476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2278}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:47:59.954 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:47:59.954 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "39'12758", 'reported_seq': 12895, 'reported_epoch': 55, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:47:51.189778+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:47:51.189778+0000', 'last_peered': '2024-09-05T16:47:51.189778+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:47:51.189778+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "33'2700", 'ondisk_log_start': "33'2700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10058, 'log_dups_size': 0, 'ondisk_log_size': 10058, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6736, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12757, 'num_write_kb': 51031, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2237}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:47:59.954 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:47:59.954 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "39'13124", 'reported_seq': 13271, 'reported_epoch': 55, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:47:51.188074+0000', 'last_change': '2024-09-05T16:34:56.912848+0000', 'last_active': '2024-09-05T16:47:51.188074+0000', 'last_peered': '2024-09-05T16:47:51.188074+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:47:51.188074+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "33'3100", 'ondisk_log_start': "33'3100", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10024, 'log_dups_size': 0, 'ondisk_log_size': 10024, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13122, 'num_write_kb': 52500, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2332}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:47:59.955 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:48:00.356 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:48:00.357 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 10376/48 objects degraded (21616.667%), 5 pgs degraded, 5 pgs undersized","count":10},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":883,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":55,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":5},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":11},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":4},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":12,"data_bytes":80832,"bytes_used":2829393920,"bytes_avail":383717662720,"bytes_total":386547056640,"degraded_objects":10376,"degraded_total":48,"degraded_ratio":216.16666666666663},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:48:00.370 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-09-05T16:48:00.370 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:48:00.726 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:48:00.727 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:48:00.739 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":487,"stamp":"2024-09-05T16:47:59.113196+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":80832,"num_objects":12,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10376,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":103752,"num_write_kb":367182,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5613,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2763080,"kb_used_data":1844,"kb_used_omap":14,"kb_used_meta":2761137,"kb_avail":374724280,"statfs":{"total":386547056640,"available":383717662720,"internally_reserved":0,"allocated":1888256,"data_stored":737452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2827404855},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13472,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1404,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12591,"num_write_kb":-50373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-762,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.932778"},"pg_stats":[{"pgid":"1.f","version":"39'12904","reported_seq":17529,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-05T16:47:51.189556+0000","last_change":"2024-09-05T16:46:27.826222+0000","last_active":"2024-09-05T16:47:51.189556+0000","last_peered":"2024-09-05T16:47:51.189556+0000","last_clean":"2024-09-05T16:47:51.189556+0000","last_became_active":"2024-09-05T16:46:27.757928+0000","last_became_peered":"2024-09-05T16:46:27.757928+0000","last_unstale":"2024-09-05T16:47:51.189556+0000","last_undegraded":"2024-09-05T16:47:51.189556+0000","last_fullsized":"2024-09-05T16:47:51.189556+0000","mapping_epoch":40,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:32:35.713428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12904,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2262,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12552","reported_seq":17241,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-05T16:47:51.191535+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:47:51.191535+0000","last_peered":"2024-09-05T16:47:51.191535+0000","last_clean":"2024-09-05T16:47:51.191535+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:47:51.191535+0000","last_undegraded":"2024-09-05T16:47:51.191535+0000","last_fullsized":"2024-09-05T16:47:51.191535+0000","mapping_epoch":38,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1865,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12752","reported_seq":17397,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-05T16:47:51.191440+0000","last_change":"2024-09-05T16:46:51.992073+0000","last_active":"2024-09-05T16:47:51.191440+0000","last_peered":"2024-09-05T16:47:51.191440+0000","last_clean":"2024-09-05T16:47:51.191440+0000","last_became_active":"2024-09-05T16:46:51.667128+0000","last_became_peered":"2024-09-05T16:46:51.667128+0000","last_unstale":"2024-09-05T16:47:51.191440+0000","last_undegraded":"2024-09-05T16:47:51.191440+0000","last_fullsized":"2024-09-05T16:47:51.191440+0000","mapping_epoch":44,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12752","last_scrub_stamp":"2024-09-05T16:46:51.991995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:51.991995+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:04:54.120031+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12934","reported_seq":17702,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-05T16:47:51.192160+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:47:51.192160+0000","last_peered":"2024-09-05T16:47:51.192160+0000","last_clean":"2024-09-05T16:47:51.192160+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:47:51.192160+0000","last_undegraded":"2024-09-05T16:47:51.192160+0000","last_fullsized":"2024-09-05T16:47:51.192160+0000","mapping_epoch":34,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7234,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12593","reported_seq":17042,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-05T16:47:51.525026+0000","last_change":"2024-09-05T16:47:51.525026+0000","last_active":"2024-09-05T16:47:51.525026+0000","last_peered":"2024-09-05T16:47:51.525026+0000","last_clean":"2024-09-05T16:47:51.525026+0000","last_became_active":"2024-09-05T16:47:51.193016+0000","last_became_peered":"2024-09-05T16:47:51.193016+0000","last_unstale":"2024-09-05T16:47:51.525026+0000","last_undegraded":"2024-09-05T16:47:51.525026+0000","last_fullsized":"2024-09-05T16:47:51.525026+0000","mapping_epoch":54,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12593","last_scrub_stamp":"2024-09-05T16:47:51.524957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:51.524957+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:46:22.846859+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12772","reported_seq":17373,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-05T16:47:51.190116+0000","last_change":"2024-09-05T16:47:15.832530+0000","last_active":"2024-09-05T16:47:51.190116+0000","last_peered":"2024-09-05T16:47:51.190116+0000","last_clean":"2024-09-05T16:47:51.190116+0000","last_became_active":"2024-09-05T16:47:15.728759+0000","last_became_peered":"2024-09-05T16:47:15.728759+0000","last_unstale":"2024-09-05T16:47:51.190116+0000","last_undegraded":"2024-09-05T16:47:51.190116+0000","last_fullsized":"2024-09-05T16:47:51.190116+0000","mapping_epoch":48,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T02:39:11.515339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12772,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2250,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12660","reported_seq":17090,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-05T16:47:51.186457+0000","last_change":"2024-09-05T16:47:03.683348+0000","last_active":"2024-09-05T16:47:51.186457+0000","last_peered":"2024-09-05T16:47:51.186457+0000","last_clean":"2024-09-05T16:47:51.186457+0000","last_became_active":"2024-09-05T16:47:03.185143+0000","last_became_peered":"2024-09-05T16:47:03.185143+0000","last_unstale":"2024-09-05T16:47:51.186457+0000","last_undegraded":"2024-09-05T16:47:51.186457+0000","last_fullsized":"2024-09-05T16:47:51.186457+0000","mapping_epoch":46,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12660","last_scrub_stamp":"2024-09-05T16:47:03.683262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:03.683262+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:42:16.533299+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12777","reported_seq":12918,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-05T16:47:51.191139+0000","last_change":"2024-09-05T16:46:40.458315+0000","last_active":"2024-09-05T16:47:51.191139+0000","last_peered":"2024-09-05T16:47:51.191139+0000","last_clean":"2024-09-05T16:47:51.191139+0000","last_became_active":"2024-09-05T16:46:40.184288+0000","last_became_peered":"2024-09-05T16:46:40.184288+0000","last_unstale":"2024-09-05T16:47:51.191139+0000","last_undegraded":"2024-09-05T16:47:51.191139+0000","last_fullsized":"2024-09-05T16:47:51.191139+0000","mapping_epoch":42,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12777","last_scrub_stamp":"2024-09-05T16:46:40.458226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:40.458226+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:38:58.734210+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'12770","reported_seq":12909,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:51.189630+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:47:51.189630+0000","last_peered":"2024-09-05T16:47:51.189630+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:47:51.189630+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12767,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2240}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12808","reported_seq":14923,"reported_epoch":55,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:56.015193+0000","last_change":"2024-09-05T16:47:49.791980+0000","last_active":"2024-09-05T16:47:56.015193+0000","last_peered":"2024-09-05T16:47:56.015193+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:47:56.015193+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12806,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":996,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1268}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13120","reported_seq":13285,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:51.187991+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:47:51.187991+0000","last_peered":"2024-09-05T16:47:51.187991+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:47:51.187991+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13117,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2278}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12758","reported_seq":12895,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:51.189778+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:47:51.189778+0000","last_peered":"2024-09-05T16:47:51.189778+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:47:51.189778+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2237}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12808","reported_seq":17582,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-05T16:47:51.192174+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:47:51.192174+0000","last_peered":"2024-09-05T16:47:51.192174+0000","last_clean":"2024-09-05T16:47:51.192174+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:47:51.192174+0000","last_undegraded":"2024-09-05T16:47:51.192174+0000","last_fullsized":"2024-09-05T16:47:51.192174+0000","mapping_epoch":36,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4408,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13124","reported_seq":13271,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:47:51.188074+0000","last_change":"2024-09-05T16:34:56.912848+0000","last_active":"2024-09-05T16:47:51.188074+0000","last_peered":"2024-09-05T16:47:51.188074+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:47:51.188074+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13122,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2332}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12831","reported_seq":12976,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-05T16:47:51.191044+0000","last_change":"2024-09-05T16:47:40.451380+0000","last_active":"2024-09-05T16:47:51.191044+0000","last_peered":"2024-09-05T16:47:51.191044+0000","last_clean":"2024-09-05T16:47:51.191044+0000","last_became_active":"2024-09-05T16:47:40.189447+0000","last_became_peered":"2024-09-05T16:47:40.189447+0000","last_unstale":"2024-09-05T16:47:51.191044+0000","last_undegraded":"2024-09-05T16:47:51.191044+0000","last_fullsized":"2024-09-05T16:47:51.191044+0000","mapping_epoch":52,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12831","last_scrub_stamp":"2024-09-05T16:47:40.451295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:40.451295+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:46:20.946333+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12773","reported_seq":12918,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-05T16:47:51.191240+0000","last_change":"2024-09-05T16:47:27.437275+0000","last_active":"2024-09-05T16:47:51.191240+0000","last_peered":"2024-09-05T16:47:51.191240+0000","last_clean":"2024-09-05T16:47:51.191240+0000","last_became_active":"2024-09-05T16:47:27.204989+0000","last_became_peered":"2024-09-05T16:47:27.204989+0000","last_unstale":"2024-09-05T16:47:51.191240+0000","last_undegraded":"2024-09-05T16:47:51.191240+0000","last_fullsized":"2024-09-05T16:47:51.191240+0000","mapping_epoch":50,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12773","last_scrub_stamp":"2024-09-05T16:47:27.437187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:27.437187+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T16:53:55.961513+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":80832,"num_objects":12,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10376,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":103752,"num_write_kb":367182,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5613,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":53248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084455,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722140,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649700,"statfs":{"total":96636764160,"available":95897292800,"internally_reserved":0,"allocated":462848,"data_stored":175147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":738979539},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986332,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":613880,"kb_used_data":488,"kb_used_omap":4,"kb_used_meta":613371,"kb_avail":93757960,"statfs":{"total":96636764160,"available":96008151040,"internally_reserved":0,"allocated":499712,"data_stored":212011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":628092626},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673138,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713304,"kb_used_data":452,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":93658536,"statfs":{"total":96636764160,"available":95906340864,"internally_reserved":0,"allocated":462848,"data_stored":175147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":729938373},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182569,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713756,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658084,"statfs":{"total":96636764160,"available":95905878016,"internally_reserved":0,"allocated":462848,"data_stored":175147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":730394317},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":40960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:48:03.741 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:48:04.046 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:48:04.046 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:48:04.058 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":491,"stamp":"2024-09-05T16:48:03.233533+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":80832,"num_objects":12,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9204,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":103752,"num_write_kb":367182,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6785,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2754768,"kb_used_data":1852,"kb_used_omap":14,"kb_used_meta":2752817,"kb_avail":374732592,"statfs":{"total":386547056640,"available":383726174208,"internally_reserved":0,"allocated":1896448,"data_stored":735692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2818885175},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1978,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1978,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.120963"},"pg_stats":[{"pgid":"1.f","version":"39'12904","reported_seq":17531,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-05T16:48:02.225658+0000","last_change":"2024-09-05T16:46:27.826222+0000","last_active":"2024-09-05T16:48:02.225658+0000","last_peered":"2024-09-05T16:48:02.225658+0000","last_clean":"2024-09-05T16:48:02.225658+0000","last_became_active":"2024-09-05T16:46:27.757928+0000","last_became_peered":"2024-09-05T16:46:27.757928+0000","last_unstale":"2024-09-05T16:48:02.225658+0000","last_undegraded":"2024-09-05T16:48:02.225658+0000","last_fullsized":"2024-09-05T16:48:02.225658+0000","mapping_epoch":40,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:32:35.713428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12904,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2262,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12552","reported_seq":17243,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-05T16:48:02.228405+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:48:02.228405+0000","last_peered":"2024-09-05T16:48:02.228405+0000","last_clean":"2024-09-05T16:48:02.228405+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:48:02.228405+0000","last_undegraded":"2024-09-05T16:48:02.228405+0000","last_fullsized":"2024-09-05T16:48:02.228405+0000","mapping_epoch":38,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1865,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12752","reported_seq":17397,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-05T16:47:51.191440+0000","last_change":"2024-09-05T16:46:51.992073+0000","last_active":"2024-09-05T16:47:51.191440+0000","last_peered":"2024-09-05T16:47:51.191440+0000","last_clean":"2024-09-05T16:47:51.191440+0000","last_became_active":"2024-09-05T16:46:51.667128+0000","last_became_peered":"2024-09-05T16:46:51.667128+0000","last_unstale":"2024-09-05T16:47:51.191440+0000","last_undegraded":"2024-09-05T16:47:51.191440+0000","last_fullsized":"2024-09-05T16:47:51.191440+0000","mapping_epoch":44,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12752","last_scrub_stamp":"2024-09-05T16:46:51.991995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:51.991995+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:04:54.120031+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12934","reported_seq":17704,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-05T16:48:02.227867+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:48:02.227867+0000","last_peered":"2024-09-05T16:48:02.227867+0000","last_clean":"2024-09-05T16:48:02.227867+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:48:02.227867+0000","last_undegraded":"2024-09-05T16:48:02.227867+0000","last_fullsized":"2024-09-05T16:48:02.227867+0000","mapping_epoch":34,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7234,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12593","reported_seq":17044,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-05T16:48:02.225421+0000","last_change":"2024-09-05T16:47:51.525026+0000","last_active":"2024-09-05T16:48:02.225421+0000","last_peered":"2024-09-05T16:48:02.225421+0000","last_clean":"2024-09-05T16:48:02.225421+0000","last_became_active":"2024-09-05T16:47:51.193016+0000","last_became_peered":"2024-09-05T16:47:51.193016+0000","last_unstale":"2024-09-05T16:48:02.225421+0000","last_undegraded":"2024-09-05T16:48:02.225421+0000","last_fullsized":"2024-09-05T16:48:02.225421+0000","mapping_epoch":54,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12593","last_scrub_stamp":"2024-09-05T16:47:51.524957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:51.524957+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:46:22.846859+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12772","reported_seq":17375,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-05T16:48:02.226551+0000","last_change":"2024-09-05T16:47:15.832530+0000","last_active":"2024-09-05T16:48:02.226551+0000","last_peered":"2024-09-05T16:48:02.226551+0000","last_clean":"2024-09-05T16:48:02.226551+0000","last_became_active":"2024-09-05T16:47:15.728759+0000","last_became_peered":"2024-09-05T16:47:15.728759+0000","last_unstale":"2024-09-05T16:48:02.226551+0000","last_undegraded":"2024-09-05T16:48:02.226551+0000","last_fullsized":"2024-09-05T16:48:02.226551+0000","mapping_epoch":48,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T02:39:11.515339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12772,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2250,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12660","reported_seq":17090,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-05T16:47:51.186457+0000","last_change":"2024-09-05T16:47:03.683348+0000","last_active":"2024-09-05T16:47:51.186457+0000","last_peered":"2024-09-05T16:47:51.186457+0000","last_clean":"2024-09-05T16:47:51.186457+0000","last_became_active":"2024-09-05T16:47:03.185143+0000","last_became_peered":"2024-09-05T16:47:03.185143+0000","last_unstale":"2024-09-05T16:47:51.186457+0000","last_undegraded":"2024-09-05T16:47:51.186457+0000","last_fullsized":"2024-09-05T16:47:51.186457+0000","mapping_epoch":46,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12660","last_scrub_stamp":"2024-09-05T16:47:03.683262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:03.683262+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:42:16.533299+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12777","reported_seq":12920,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-05T16:48:02.228165+0000","last_change":"2024-09-05T16:46:40.458315+0000","last_active":"2024-09-05T16:48:02.228165+0000","last_peered":"2024-09-05T16:48:02.228165+0000","last_clean":"2024-09-05T16:48:02.228165+0000","last_became_active":"2024-09-05T16:46:40.184288+0000","last_became_peered":"2024-09-05T16:46:40.184288+0000","last_unstale":"2024-09-05T16:48:02.228165+0000","last_undegraded":"2024-09-05T16:48:02.228165+0000","last_fullsized":"2024-09-05T16:48:02.228165+0000","mapping_epoch":42,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12777","last_scrub_stamp":"2024-09-05T16:46:40.458226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:40.458226+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:38:58.734210+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'12770","reported_seq":12911,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:48:02.225388+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:48:02.225388+0000","last_peered":"2024-09-05T16:48:02.225388+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:48:02.225388+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12767,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2240}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12808","reported_seq":16887,"reported_epoch":55,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:48:01.022535+0000","last_change":"2024-09-05T16:47:49.791980+0000","last_active":"2024-09-05T16:48:01.022535+0000","last_peered":"2024-09-05T16:48:01.022535+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:48:01.022535+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":288,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12806,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1978,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-286}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13120","reported_seq":13287,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:48:02.225594+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:48:02.225594+0000","last_peered":"2024-09-05T16:48:02.225594+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:48:02.225594+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13117,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2278}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12758","reported_seq":12897,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:48:02.225705+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:48:02.225705+0000","last_peered":"2024-09-05T16:48:02.225705+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:48:02.225705+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2237}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12808","reported_seq":17584,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-05T16:48:02.227959+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:48:02.227959+0000","last_peered":"2024-09-05T16:48:02.227959+0000","last_clean":"2024-09-05T16:48:02.227959+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:48:02.227959+0000","last_undegraded":"2024-09-05T16:48:02.227959+0000","last_fullsized":"2024-09-05T16:48:02.227959+0000","mapping_epoch":36,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4408,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13124","reported_seq":13654,"reported_epoch":56,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:48:03.206933+0000","last_change":"2024-09-05T16:48:02.084558+0000","last_active":"2024-09-05T16:48:03.206933+0000","last_peered":"2024-09-05T16:48:03.206933+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:48:03.206933+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2146,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13122,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":215,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2144}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12831","reported_seq":12978,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-05T16:48:02.228131+0000","last_change":"2024-09-05T16:47:40.451380+0000","last_active":"2024-09-05T16:48:02.228131+0000","last_peered":"2024-09-05T16:48:02.228131+0000","last_clean":"2024-09-05T16:48:02.228131+0000","last_became_active":"2024-09-05T16:47:40.189447+0000","last_became_peered":"2024-09-05T16:47:40.189447+0000","last_unstale":"2024-09-05T16:48:02.228131+0000","last_undegraded":"2024-09-05T16:48:02.228131+0000","last_fullsized":"2024-09-05T16:48:02.228131+0000","mapping_epoch":52,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12831","last_scrub_stamp":"2024-09-05T16:47:40.451295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:40.451295+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:46:20.946333+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12773","reported_seq":12920,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-05T16:48:02.228253+0000","last_change":"2024-09-05T16:47:27.437275+0000","last_active":"2024-09-05T16:48:02.228253+0000","last_peered":"2024-09-05T16:48:02.228253+0000","last_clean":"2024-09-05T16:48:02.228253+0000","last_became_active":"2024-09-05T16:47:27.204989+0000","last_became_peered":"2024-09-05T16:47:27.204989+0000","last_unstale":"2024-09-05T16:48:02.228253+0000","last_undegraded":"2024-09-05T16:48:02.228253+0000","last_fullsized":"2024-09-05T16:48:02.228253+0000","mapping_epoch":50,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12773","last_scrub_stamp":"2024-09-05T16:47:27.437187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:27.437187+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T16:53:55.961513+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":80832,"num_objects":12,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9204,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":103752,"num_write_kb":367182,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6785,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":45056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084456,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722144,"kb_used_data":460,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649696,"statfs":{"total":96636764160,"available":95897288704,"internally_reserved":0,"allocated":471040,"data_stored":178363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":738979539},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986333,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":605564,"kb_used_data":488,"kb_used_omap":4,"kb_used_meta":605051,"kb_avail":93766276,"statfs":{"total":96636764160,"available":96016666624,"internally_reserved":0,"allocated":499712,"data_stored":207035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":619572946},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673139,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713304,"kb_used_data":452,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":93658536,"statfs":{"total":96636764160,"available":95906340864,"internally_reserved":0,"allocated":462848,"data_stored":175147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":729938373},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182570,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713756,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658084,"statfs":{"total":96636764160,"available":95905878016,"internally_reserved":0,"allocated":462848,"data_stored":175147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":730394317},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:48:04.060 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:48:04.380 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:48:04.381 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:48:04.398 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":491,"stamp":"2024-09-05T16:48:03.233533+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":80832,"num_objects":12,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9204,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":103752,"num_write_kb":367182,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6785,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2754768,"kb_used_data":1852,"kb_used_omap":14,"kb_used_meta":2752817,"kb_avail":374732592,"statfs":{"total":386547056640,"available":383726174208,"internally_reserved":0,"allocated":1896448,"data_stored":735692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2818885175},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1978,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1978,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.120963"},"pg_stats":[{"pgid":"1.f","version":"39'12904","reported_seq":17531,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-05T16:48:02.225658+0000","last_change":"2024-09-05T16:46:27.826222+0000","last_active":"2024-09-05T16:48:02.225658+0000","last_peered":"2024-09-05T16:48:02.225658+0000","last_clean":"2024-09-05T16:48:02.225658+0000","last_became_active":"2024-09-05T16:46:27.757928+0000","last_became_peered":"2024-09-05T16:46:27.757928+0000","last_unstale":"2024-09-05T16:48:02.225658+0000","last_undegraded":"2024-09-05T16:48:02.225658+0000","last_fullsized":"2024-09-05T16:48:02.225658+0000","mapping_epoch":40,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:32:35.713428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12904,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2262,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12552","reported_seq":17243,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-05T16:48:02.228405+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:48:02.228405+0000","last_peered":"2024-09-05T16:48:02.228405+0000","last_clean":"2024-09-05T16:48:02.228405+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:48:02.228405+0000","last_undegraded":"2024-09-05T16:48:02.228405+0000","last_fullsized":"2024-09-05T16:48:02.228405+0000","mapping_epoch":38,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1865,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12752","reported_seq":17397,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-05T16:47:51.191440+0000","last_change":"2024-09-05T16:46:51.992073+0000","last_active":"2024-09-05T16:47:51.191440+0000","last_peered":"2024-09-05T16:47:51.191440+0000","last_clean":"2024-09-05T16:47:51.191440+0000","last_became_active":"2024-09-05T16:46:51.667128+0000","last_became_peered":"2024-09-05T16:46:51.667128+0000","last_unstale":"2024-09-05T16:47:51.191440+0000","last_undegraded":"2024-09-05T16:47:51.191440+0000","last_fullsized":"2024-09-05T16:47:51.191440+0000","mapping_epoch":44,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12752","last_scrub_stamp":"2024-09-05T16:46:51.991995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:51.991995+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:04:54.120031+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12934","reported_seq":17704,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-05T16:48:02.227867+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:48:02.227867+0000","last_peered":"2024-09-05T16:48:02.227867+0000","last_clean":"2024-09-05T16:48:02.227867+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:48:02.227867+0000","last_undegraded":"2024-09-05T16:48:02.227867+0000","last_fullsized":"2024-09-05T16:48:02.227867+0000","mapping_epoch":34,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7234,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12593","reported_seq":17044,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-05T16:48:02.225421+0000","last_change":"2024-09-05T16:47:51.525026+0000","last_active":"2024-09-05T16:48:02.225421+0000","last_peered":"2024-09-05T16:48:02.225421+0000","last_clean":"2024-09-05T16:48:02.225421+0000","last_became_active":"2024-09-05T16:47:51.193016+0000","last_became_peered":"2024-09-05T16:47:51.193016+0000","last_unstale":"2024-09-05T16:48:02.225421+0000","last_undegraded":"2024-09-05T16:48:02.225421+0000","last_fullsized":"2024-09-05T16:48:02.225421+0000","mapping_epoch":54,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12593","last_scrub_stamp":"2024-09-05T16:47:51.524957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:51.524957+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:46:22.846859+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12772","reported_seq":17375,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-05T16:48:02.226551+0000","last_change":"2024-09-05T16:47:15.832530+0000","last_active":"2024-09-05T16:48:02.226551+0000","last_peered":"2024-09-05T16:48:02.226551+0000","last_clean":"2024-09-05T16:48:02.226551+0000","last_became_active":"2024-09-05T16:47:15.728759+0000","last_became_peered":"2024-09-05T16:47:15.728759+0000","last_unstale":"2024-09-05T16:48:02.226551+0000","last_undegraded":"2024-09-05T16:48:02.226551+0000","last_fullsized":"2024-09-05T16:48:02.226551+0000","mapping_epoch":48,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T02:39:11.515339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12772,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2250,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12660","reported_seq":17090,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-05T16:47:51.186457+0000","last_change":"2024-09-05T16:47:03.683348+0000","last_active":"2024-09-05T16:47:51.186457+0000","last_peered":"2024-09-05T16:47:51.186457+0000","last_clean":"2024-09-05T16:47:51.186457+0000","last_became_active":"2024-09-05T16:47:03.185143+0000","last_became_peered":"2024-09-05T16:47:03.185143+0000","last_unstale":"2024-09-05T16:47:51.186457+0000","last_undegraded":"2024-09-05T16:47:51.186457+0000","last_fullsized":"2024-09-05T16:47:51.186457+0000","mapping_epoch":46,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12660","last_scrub_stamp":"2024-09-05T16:47:03.683262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:03.683262+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:42:16.533299+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12777","reported_seq":12920,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-05T16:48:02.228165+0000","last_change":"2024-09-05T16:46:40.458315+0000","last_active":"2024-09-05T16:48:02.228165+0000","last_peered":"2024-09-05T16:48:02.228165+0000","last_clean":"2024-09-05T16:48:02.228165+0000","last_became_active":"2024-09-05T16:46:40.184288+0000","last_became_peered":"2024-09-05T16:46:40.184288+0000","last_unstale":"2024-09-05T16:48:02.228165+0000","last_undegraded":"2024-09-05T16:48:02.228165+0000","last_fullsized":"2024-09-05T16:48:02.228165+0000","mapping_epoch":42,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12777","last_scrub_stamp":"2024-09-05T16:46:40.458226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:40.458226+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:38:58.734210+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'12770","reported_seq":12911,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:48:02.225388+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:48:02.225388+0000","last_peered":"2024-09-05T16:48:02.225388+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:48:02.225388+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12767,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2240}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12808","reported_seq":16887,"reported_epoch":55,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:48:01.022535+0000","last_change":"2024-09-05T16:47:49.791980+0000","last_active":"2024-09-05T16:48:01.022535+0000","last_peered":"2024-09-05T16:48:01.022535+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:48:01.022535+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":288,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12806,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1978,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-286}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13120","reported_seq":13287,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:48:02.225594+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:48:02.225594+0000","last_peered":"2024-09-05T16:48:02.225594+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:48:02.225594+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13117,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2278}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12758","reported_seq":12897,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:48:02.225705+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:48:02.225705+0000","last_peered":"2024-09-05T16:48:02.225705+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:48:02.225705+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2237}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12808","reported_seq":17584,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-05T16:48:02.227959+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:48:02.227959+0000","last_peered":"2024-09-05T16:48:02.227959+0000","last_clean":"2024-09-05T16:48:02.227959+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:48:02.227959+0000","last_undegraded":"2024-09-05T16:48:02.227959+0000","last_fullsized":"2024-09-05T16:48:02.227959+0000","mapping_epoch":36,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4408,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13124","reported_seq":13654,"reported_epoch":56,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:48:03.206933+0000","last_change":"2024-09-05T16:48:02.084558+0000","last_active":"2024-09-05T16:48:03.206933+0000","last_peered":"2024-09-05T16:48:03.206933+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:48:03.206933+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2146,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13122,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":215,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2144}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12831","reported_seq":12978,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-05T16:48:02.228131+0000","last_change":"2024-09-05T16:47:40.451380+0000","last_active":"2024-09-05T16:48:02.228131+0000","last_peered":"2024-09-05T16:48:02.228131+0000","last_clean":"2024-09-05T16:48:02.228131+0000","last_became_active":"2024-09-05T16:47:40.189447+0000","last_became_peered":"2024-09-05T16:47:40.189447+0000","last_unstale":"2024-09-05T16:48:02.228131+0000","last_undegraded":"2024-09-05T16:48:02.228131+0000","last_fullsized":"2024-09-05T16:48:02.228131+0000","mapping_epoch":52,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12831","last_scrub_stamp":"2024-09-05T16:47:40.451295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:40.451295+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:46:20.946333+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12773","reported_seq":12920,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-05T16:48:02.228253+0000","last_change":"2024-09-05T16:47:27.437275+0000","last_active":"2024-09-05T16:48:02.228253+0000","last_peered":"2024-09-05T16:48:02.228253+0000","last_clean":"2024-09-05T16:48:02.228253+0000","last_became_active":"2024-09-05T16:47:27.204989+0000","last_became_peered":"2024-09-05T16:47:27.204989+0000","last_unstale":"2024-09-05T16:48:02.228253+0000","last_undegraded":"2024-09-05T16:48:02.228253+0000","last_fullsized":"2024-09-05T16:48:02.228253+0000","mapping_epoch":50,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12773","last_scrub_stamp":"2024-09-05T16:47:27.437187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:27.437187+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T16:53:55.961513+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":80832,"num_objects":12,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9204,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":103752,"num_write_kb":367182,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6785,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":45056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084456,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722144,"kb_used_data":460,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649696,"statfs":{"total":96636764160,"available":95897288704,"internally_reserved":0,"allocated":471040,"data_stored":178363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":738979539},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986333,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":605564,"kb_used_data":488,"kb_used_omap":4,"kb_used_meta":605051,"kb_avail":93766276,"statfs":{"total":96636764160,"available":96016666624,"internally_reserved":0,"allocated":499712,"data_stored":207035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":619572946},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673139,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713304,"kb_used_data":452,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":93658536,"statfs":{"total":96636764160,"available":95906340864,"internally_reserved":0,"allocated":462848,"data_stored":175147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":729938373},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182570,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713756,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658084,"statfs":{"total":96636764160,"available":95905878016,"internally_reserved":0,"allocated":462848,"data_stored":175147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":730394317},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:48:04.399 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:48:04.399 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "39'12770", 'reported_seq': 12911, 'reported_epoch': 56, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:48:02.225388+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:48:02.225388+0000', 'last_peered': '2024-09-05T16:48:02.225388+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:48:02.225388+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "33'2700", 'ondisk_log_start': "33'2700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10070, 'log_dups_size': 0, 'ondisk_log_size': 10070, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26944, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 16, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12767, 'num_write_kb': 51081, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2240}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:48:04.399 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-05T16:48:04.399 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "39'12808", 'reported_seq': 16887, 'reported_epoch': 55, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:48:01.022535+0000', 'last_change': '2024-09-05T16:47:49.791980+0000', 'last_active': '2024-09-05T16:48:01.022535+0000', 'last_peered': '2024-09-05T16:48:01.022535+0000', 'last_clean': '2024-09-05T16:33:45.002311+0000', 'last_became_active': '2024-09-05T16:34:56.873917+0000', 'last_became_peered': '2024-09-05T16:34:56.873917+0000', 'last_unstale': '2024-09-05T16:48:01.022535+0000', 'last_undegraded': '2024-09-05T16:34:56.619358+0000', 'last_fullsized': '2024-09-05T16:34:56.593673+0000', 'mapping_epoch': 32, 'log_start': "33'2800", 'ondisk_log_start': "33'2800", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10008, 'log_dups_size': 0, 'ondisk_log_size': 10008, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 288, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12806, 'num_write_kb': 51230, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1978, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -286}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-05T16:48:04.399 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:48:04.399 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "39'13120", 'reported_seq': 13287, 'reported_epoch': 56, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:48:02.225594+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:48:02.225594+0000', 'last_peered': '2024-09-05T16:48:02.225594+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:48:02.225594+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "33'3100", 'ondisk_log_start': "33'3100", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20208, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2284, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13117, 'num_write_kb': 52476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2278}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:48:04.400 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:48:04.400 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "39'12758", 'reported_seq': 12897, 'reported_epoch': 56, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:48:02.225705+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:48:02.225705+0000', 'last_peered': '2024-09-05T16:48:02.225705+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:48:02.225705+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "33'2700", 'ondisk_log_start': "33'2700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10058, 'log_dups_size': 0, 'ondisk_log_size': 10058, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6736, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12757, 'num_write_kb': 51031, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2237}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:48:04.400 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:48:04.400 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "39'13124", 'reported_seq': 13654, 'reported_epoch': 56, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:48:03.206933+0000', 'last_change': '2024-09-05T16:48:02.084558+0000', 'last_active': '2024-09-05T16:48:03.206933+0000', 'last_peered': '2024-09-05T16:48:03.206933+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:48:03.206933+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "33'3100", 'ondisk_log_start': "33'3100", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10024, 'log_dups_size': 0, 'ondisk_log_size': 10024, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2146, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13122, 'num_write_kb': 52500, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 215, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2144}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:48:04.400 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:48:04.789 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:48:04.789 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 9394/48 objects degraded (19570.833%), 5 pgs degraded, 5 pgs undersized","count":10},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":888,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":57,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":4},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":11},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":4},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":12,"data_bytes":80832,"bytes_used":2820878336,"bytes_avail":383726178304,"bytes_total":386547056640,"degraded_objects":9394,"degraded_total":48,"degraded_ratio":195.70833333333337,"recovering_objects_per_sec":4,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":49,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:48:04.802 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:48:04.803 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:48:05.099 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:48:05.100 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:48:05.111 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":491,"stamp":"2024-09-05T16:48:03.233533+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":80832,"num_objects":12,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9204,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":103752,"num_write_kb":367182,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6785,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2754768,"kb_used_data":1852,"kb_used_omap":14,"kb_used_meta":2752817,"kb_avail":374732592,"statfs":{"total":386547056640,"available":383726174208,"internally_reserved":0,"allocated":1896448,"data_stored":735692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2818885175},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1978,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1978,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.120963"},"pg_stats":[{"pgid":"1.f","version":"39'12904","reported_seq":17531,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-05T16:48:02.225658+0000","last_change":"2024-09-05T16:46:27.826222+0000","last_active":"2024-09-05T16:48:02.225658+0000","last_peered":"2024-09-05T16:48:02.225658+0000","last_clean":"2024-09-05T16:48:02.225658+0000","last_became_active":"2024-09-05T16:46:27.757928+0000","last_became_peered":"2024-09-05T16:46:27.757928+0000","last_unstale":"2024-09-05T16:48:02.225658+0000","last_undegraded":"2024-09-05T16:48:02.225658+0000","last_fullsized":"2024-09-05T16:48:02.225658+0000","mapping_epoch":40,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:32:35.713428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12904,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2262,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12552","reported_seq":17243,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-05T16:48:02.228405+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:48:02.228405+0000","last_peered":"2024-09-05T16:48:02.228405+0000","last_clean":"2024-09-05T16:48:02.228405+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:48:02.228405+0000","last_undegraded":"2024-09-05T16:48:02.228405+0000","last_fullsized":"2024-09-05T16:48:02.228405+0000","mapping_epoch":38,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1865,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12752","reported_seq":17397,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-05T16:47:51.191440+0000","last_change":"2024-09-05T16:46:51.992073+0000","last_active":"2024-09-05T16:47:51.191440+0000","last_peered":"2024-09-05T16:47:51.191440+0000","last_clean":"2024-09-05T16:47:51.191440+0000","last_became_active":"2024-09-05T16:46:51.667128+0000","last_became_peered":"2024-09-05T16:46:51.667128+0000","last_unstale":"2024-09-05T16:47:51.191440+0000","last_undegraded":"2024-09-05T16:47:51.191440+0000","last_fullsized":"2024-09-05T16:47:51.191440+0000","mapping_epoch":44,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12752","last_scrub_stamp":"2024-09-05T16:46:51.991995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:51.991995+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:04:54.120031+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12934","reported_seq":17704,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-05T16:48:02.227867+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:48:02.227867+0000","last_peered":"2024-09-05T16:48:02.227867+0000","last_clean":"2024-09-05T16:48:02.227867+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:48:02.227867+0000","last_undegraded":"2024-09-05T16:48:02.227867+0000","last_fullsized":"2024-09-05T16:48:02.227867+0000","mapping_epoch":34,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7234,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12593","reported_seq":17044,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-05T16:48:02.225421+0000","last_change":"2024-09-05T16:47:51.525026+0000","last_active":"2024-09-05T16:48:02.225421+0000","last_peered":"2024-09-05T16:48:02.225421+0000","last_clean":"2024-09-05T16:48:02.225421+0000","last_became_active":"2024-09-05T16:47:51.193016+0000","last_became_peered":"2024-09-05T16:47:51.193016+0000","last_unstale":"2024-09-05T16:48:02.225421+0000","last_undegraded":"2024-09-05T16:48:02.225421+0000","last_fullsized":"2024-09-05T16:48:02.225421+0000","mapping_epoch":54,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12593","last_scrub_stamp":"2024-09-05T16:47:51.524957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:51.524957+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:46:22.846859+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12772","reported_seq":17375,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-05T16:48:02.226551+0000","last_change":"2024-09-05T16:47:15.832530+0000","last_active":"2024-09-05T16:48:02.226551+0000","last_peered":"2024-09-05T16:48:02.226551+0000","last_clean":"2024-09-05T16:48:02.226551+0000","last_became_active":"2024-09-05T16:47:15.728759+0000","last_became_peered":"2024-09-05T16:47:15.728759+0000","last_unstale":"2024-09-05T16:48:02.226551+0000","last_undegraded":"2024-09-05T16:48:02.226551+0000","last_fullsized":"2024-09-05T16:48:02.226551+0000","mapping_epoch":48,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T02:39:11.515339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12772,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2250,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12660","reported_seq":17090,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-05T16:47:51.186457+0000","last_change":"2024-09-05T16:47:03.683348+0000","last_active":"2024-09-05T16:47:51.186457+0000","last_peered":"2024-09-05T16:47:51.186457+0000","last_clean":"2024-09-05T16:47:51.186457+0000","last_became_active":"2024-09-05T16:47:03.185143+0000","last_became_peered":"2024-09-05T16:47:03.185143+0000","last_unstale":"2024-09-05T16:47:51.186457+0000","last_undegraded":"2024-09-05T16:47:51.186457+0000","last_fullsized":"2024-09-05T16:47:51.186457+0000","mapping_epoch":46,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12660","last_scrub_stamp":"2024-09-05T16:47:03.683262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:03.683262+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:42:16.533299+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12777","reported_seq":12920,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-05T16:48:02.228165+0000","last_change":"2024-09-05T16:46:40.458315+0000","last_active":"2024-09-05T16:48:02.228165+0000","last_peered":"2024-09-05T16:48:02.228165+0000","last_clean":"2024-09-05T16:48:02.228165+0000","last_became_active":"2024-09-05T16:46:40.184288+0000","last_became_peered":"2024-09-05T16:46:40.184288+0000","last_unstale":"2024-09-05T16:48:02.228165+0000","last_undegraded":"2024-09-05T16:48:02.228165+0000","last_fullsized":"2024-09-05T16:48:02.228165+0000","mapping_epoch":42,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12777","last_scrub_stamp":"2024-09-05T16:46:40.458226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:40.458226+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:38:58.734210+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'12770","reported_seq":12911,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:48:02.225388+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:48:02.225388+0000","last_peered":"2024-09-05T16:48:02.225388+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:48:02.225388+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12767,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2240}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12808","reported_seq":16887,"reported_epoch":55,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:48:01.022535+0000","last_change":"2024-09-05T16:47:49.791980+0000","last_active":"2024-09-05T16:48:01.022535+0000","last_peered":"2024-09-05T16:48:01.022535+0000","last_clean":"2024-09-05T16:33:45.002311+0000","last_became_active":"2024-09-05T16:34:56.873917+0000","last_became_peered":"2024-09-05T16:34:56.873917+0000","last_unstale":"2024-09-05T16:48:01.022535+0000","last_undegraded":"2024-09-05T16:34:56.619358+0000","last_fullsized":"2024-09-05T16:34:56.593673+0000","mapping_epoch":32,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":288,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12806,"num_write_kb":51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1978,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-286}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13120","reported_seq":13287,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:48:02.225594+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:48:02.225594+0000","last_peered":"2024-09-05T16:48:02.225594+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:48:02.225594+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13117,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2278}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12758","reported_seq":12897,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:48:02.225705+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:48:02.225705+0000","last_peered":"2024-09-05T16:48:02.225705+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:48:02.225705+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2237}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12808","reported_seq":17584,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-05T16:48:02.227959+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:48:02.227959+0000","last_peered":"2024-09-05T16:48:02.227959+0000","last_clean":"2024-09-05T16:48:02.227959+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:48:02.227959+0000","last_undegraded":"2024-09-05T16:48:02.227959+0000","last_fullsized":"2024-09-05T16:48:02.227959+0000","mapping_epoch":36,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4408,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13124","reported_seq":13654,"reported_epoch":56,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:48:03.206933+0000","last_change":"2024-09-05T16:48:02.084558+0000","last_active":"2024-09-05T16:48:03.206933+0000","last_peered":"2024-09-05T16:48:03.206933+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:48:03.206933+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2146,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13122,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":215,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2144}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12831","reported_seq":12978,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-05T16:48:02.228131+0000","last_change":"2024-09-05T16:47:40.451380+0000","last_active":"2024-09-05T16:48:02.228131+0000","last_peered":"2024-09-05T16:48:02.228131+0000","last_clean":"2024-09-05T16:48:02.228131+0000","last_became_active":"2024-09-05T16:47:40.189447+0000","last_became_peered":"2024-09-05T16:47:40.189447+0000","last_unstale":"2024-09-05T16:48:02.228131+0000","last_undegraded":"2024-09-05T16:48:02.228131+0000","last_fullsized":"2024-09-05T16:48:02.228131+0000","mapping_epoch":52,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12831","last_scrub_stamp":"2024-09-05T16:47:40.451295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:40.451295+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:46:20.946333+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12773","reported_seq":12920,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-05T16:48:02.228253+0000","last_change":"2024-09-05T16:47:27.437275+0000","last_active":"2024-09-05T16:48:02.228253+0000","last_peered":"2024-09-05T16:48:02.228253+0000","last_clean":"2024-09-05T16:48:02.228253+0000","last_became_active":"2024-09-05T16:47:27.204989+0000","last_became_peered":"2024-09-05T16:47:27.204989+0000","last_unstale":"2024-09-05T16:48:02.228253+0000","last_undegraded":"2024-09-05T16:48:02.228253+0000","last_fullsized":"2024-09-05T16:48:02.228253+0000","mapping_epoch":50,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12773","last_scrub_stamp":"2024-09-05T16:47:27.437187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:27.437187+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T16:53:55.961513+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":80832,"num_objects":12,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9204,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":103752,"num_write_kb":367182,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6785,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":45056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084456,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722144,"kb_used_data":460,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649696,"statfs":{"total":96636764160,"available":95897288704,"internally_reserved":0,"allocated":471040,"data_stored":178363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":738979539},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986333,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":605564,"kb_used_data":488,"kb_used_omap":4,"kb_used_meta":605051,"kb_avail":93766276,"statfs":{"total":96636764160,"available":96016666624,"internally_reserved":0,"allocated":499712,"data_stored":207035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":619572946},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673139,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713304,"kb_used_data":452,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":93658536,"statfs":{"total":96636764160,"available":95906340864,"internally_reserved":0,"allocated":462848,"data_stored":175147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":729938373},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182570,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713756,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658084,"statfs":{"total":96636764160,"available":95905878016,"internally_reserved":0,"allocated":462848,"data_stored":175147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":730394317},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:48:08.114 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:48:08.416 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:48:08.417 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:48:08.429 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":493,"stamp":"2024-09-05T16:48:07.114473+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":67360,"num_objects":10,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8916,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":90946,"num_write_kb":315952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4807,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2754800,"kb_used_data":1884,"kb_used_omap":14,"kb_used_meta":2752817,"kb_avail":374732560,"statfs":{"total":386547056640,"available":383726141440,"internally_reserved":0,"allocated":1929216,"data_stored":748514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2818885175},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13472,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1460,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12806,"num_write_kb":-51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-806,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001277"},"pg_stats":[{"pgid":"1.f","version":"39'12904","reported_seq":17531,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-05T16:48:02.225658+0000","last_change":"2024-09-05T16:46:27.826222+0000","last_active":"2024-09-05T16:48:02.225658+0000","last_peered":"2024-09-05T16:48:02.225658+0000","last_clean":"2024-09-05T16:48:02.225658+0000","last_became_active":"2024-09-05T16:46:27.757928+0000","last_became_peered":"2024-09-05T16:46:27.757928+0000","last_unstale":"2024-09-05T16:48:02.225658+0000","last_undegraded":"2024-09-05T16:48:02.225658+0000","last_fullsized":"2024-09-05T16:48:02.225658+0000","mapping_epoch":40,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:32:35.713428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12904,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2262,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12552","reported_seq":17243,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-05T16:48:02.228405+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:48:02.228405+0000","last_peered":"2024-09-05T16:48:02.228405+0000","last_clean":"2024-09-05T16:48:02.228405+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:48:02.228405+0000","last_undegraded":"2024-09-05T16:48:02.228405+0000","last_fullsized":"2024-09-05T16:48:02.228405+0000","mapping_epoch":38,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1865,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12752","reported_seq":17401,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-05T16:48:03.238089+0000","last_change":"2024-09-05T16:46:51.992073+0000","last_active":"2024-09-05T16:48:03.238089+0000","last_peered":"2024-09-05T16:48:03.238089+0000","last_clean":"2024-09-05T16:48:03.238089+0000","last_became_active":"2024-09-05T16:46:51.667128+0000","last_became_peered":"2024-09-05T16:46:51.667128+0000","last_unstale":"2024-09-05T16:48:03.238089+0000","last_undegraded":"2024-09-05T16:48:03.238089+0000","last_fullsized":"2024-09-05T16:48:03.238089+0000","mapping_epoch":44,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12752","last_scrub_stamp":"2024-09-05T16:46:51.991995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:51.991995+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:04:54.120031+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12934","reported_seq":17704,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-05T16:48:02.227867+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:48:02.227867+0000","last_peered":"2024-09-05T16:48:02.227867+0000","last_clean":"2024-09-05T16:48:02.227867+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:48:02.227867+0000","last_undegraded":"2024-09-05T16:48:02.227867+0000","last_fullsized":"2024-09-05T16:48:02.227867+0000","mapping_epoch":34,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7234,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12593","reported_seq":17044,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-05T16:48:02.225421+0000","last_change":"2024-09-05T16:47:51.525026+0000","last_active":"2024-09-05T16:48:02.225421+0000","last_peered":"2024-09-05T16:48:02.225421+0000","last_clean":"2024-09-05T16:48:02.225421+0000","last_became_active":"2024-09-05T16:47:51.193016+0000","last_became_peered":"2024-09-05T16:47:51.193016+0000","last_unstale":"2024-09-05T16:48:02.225421+0000","last_undegraded":"2024-09-05T16:48:02.225421+0000","last_fullsized":"2024-09-05T16:48:02.225421+0000","mapping_epoch":54,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12593","last_scrub_stamp":"2024-09-05T16:47:51.524957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:51.524957+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:46:22.846859+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12772","reported_seq":17375,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-05T16:48:02.226551+0000","last_change":"2024-09-05T16:47:15.832530+0000","last_active":"2024-09-05T16:48:02.226551+0000","last_peered":"2024-09-05T16:48:02.226551+0000","last_clean":"2024-09-05T16:48:02.226551+0000","last_became_active":"2024-09-05T16:47:15.728759+0000","last_became_peered":"2024-09-05T16:47:15.728759+0000","last_unstale":"2024-09-05T16:48:02.226551+0000","last_undegraded":"2024-09-05T16:48:02.226551+0000","last_fullsized":"2024-09-05T16:48:02.226551+0000","mapping_epoch":48,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T02:39:11.515339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12772,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2250,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12660","reported_seq":17094,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-05T16:48:03.235802+0000","last_change":"2024-09-05T16:47:03.683348+0000","last_active":"2024-09-05T16:48:03.235802+0000","last_peered":"2024-09-05T16:48:03.235802+0000","last_clean":"2024-09-05T16:48:03.235802+0000","last_became_active":"2024-09-05T16:47:03.185143+0000","last_became_peered":"2024-09-05T16:47:03.185143+0000","last_unstale":"2024-09-05T16:48:03.235802+0000","last_undegraded":"2024-09-05T16:48:03.235802+0000","last_fullsized":"2024-09-05T16:48:03.235802+0000","mapping_epoch":46,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12660","last_scrub_stamp":"2024-09-05T16:47:03.683262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:03.683262+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:42:16.533299+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12777","reported_seq":12920,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-05T16:48:02.228165+0000","last_change":"2024-09-05T16:46:40.458315+0000","last_active":"2024-09-05T16:48:02.228165+0000","last_peered":"2024-09-05T16:48:02.228165+0000","last_clean":"2024-09-05T16:48:02.228165+0000","last_became_active":"2024-09-05T16:46:40.184288+0000","last_became_peered":"2024-09-05T16:46:40.184288+0000","last_unstale":"2024-09-05T16:48:02.228165+0000","last_undegraded":"2024-09-05T16:48:02.228165+0000","last_fullsized":"2024-09-05T16:48:02.228165+0000","mapping_epoch":42,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12777","last_scrub_stamp":"2024-09-05T16:46:40.458226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:40.458226+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:38:58.734210+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'12770","reported_seq":12911,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:48:02.225388+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:48:02.225388+0000","last_peered":"2024-09-05T16:48:02.225388+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:48:02.225388+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12767,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2240}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12808","reported_seq":17484,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-05T16:48:04.288466+0000","last_change":"2024-09-05T16:48:04.288466+0000","last_active":"2024-09-05T16:48:04.288466+0000","last_peered":"2024-09-05T16:48:04.288466+0000","last_clean":"2024-09-05T16:48:04.288466+0000","last_became_active":"2024-09-05T16:48:03.239894+0000","last_became_peered":"2024-09-05T16:48:03.239894+0000","last_unstale":"2024-09-05T16:48:04.288466+0000","last_undegraded":"2024-09-05T16:48:04.288466+0000","last_fullsized":"2024-09-05T16:48:04.288466+0000","mapping_epoch":56,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12808","last_scrub_stamp":"2024-09-05T16:48:04.288388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:48:04.288388+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T21:11:32.044919+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13120","reported_seq":13287,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:48:02.225594+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:48:02.225594+0000","last_peered":"2024-09-05T16:48:02.225594+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:48:02.225594+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13117,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2278}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12758","reported_seq":12897,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:48:02.225705+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:48:02.225705+0000","last_peered":"2024-09-05T16:48:02.225705+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:48:02.225705+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2237}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12808","reported_seq":17584,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-05T16:48:02.227959+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:48:02.227959+0000","last_peered":"2024-09-05T16:48:02.227959+0000","last_clean":"2024-09-05T16:48:02.227959+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:48:02.227959+0000","last_undegraded":"2024-09-05T16:48:02.227959+0000","last_fullsized":"2024-09-05T16:48:02.227959+0000","mapping_epoch":36,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4408,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13124","reported_seq":13654,"reported_epoch":56,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:48:03.206933+0000","last_change":"2024-09-05T16:48:02.084558+0000","last_active":"2024-09-05T16:48:03.206933+0000","last_peered":"2024-09-05T16:48:03.206933+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:48:03.206933+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2146,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13122,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":215,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2144}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12831","reported_seq":12978,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-05T16:48:02.228131+0000","last_change":"2024-09-05T16:47:40.451380+0000","last_active":"2024-09-05T16:48:02.228131+0000","last_peered":"2024-09-05T16:48:02.228131+0000","last_clean":"2024-09-05T16:48:02.228131+0000","last_became_active":"2024-09-05T16:47:40.189447+0000","last_became_peered":"2024-09-05T16:47:40.189447+0000","last_unstale":"2024-09-05T16:48:02.228131+0000","last_undegraded":"2024-09-05T16:48:02.228131+0000","last_fullsized":"2024-09-05T16:48:02.228131+0000","mapping_epoch":52,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12831","last_scrub_stamp":"2024-09-05T16:47:40.451295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:40.451295+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:46:20.946333+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12773","reported_seq":12920,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-05T16:48:02.228253+0000","last_change":"2024-09-05T16:47:27.437275+0000","last_active":"2024-09-05T16:48:02.228253+0000","last_peered":"2024-09-05T16:48:02.228253+0000","last_clean":"2024-09-05T16:48:02.228253+0000","last_became_active":"2024-09-05T16:47:27.204989+0000","last_became_peered":"2024-09-05T16:47:27.204989+0000","last_unstale":"2024-09-05T16:48:02.228253+0000","last_undegraded":"2024-09-05T16:48:02.228253+0000","last_fullsized":"2024-09-05T16:48:02.228253+0000","mapping_epoch":50,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12773","last_scrub_stamp":"2024-09-05T16:47:27.437187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:27.437187+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T16:53:55.961513+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":67360,"num_objects":10,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8916,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":90946,"num_write_kb":315952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4807,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":45056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084456,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722144,"kb_used_data":460,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649696,"statfs":{"total":96636764160,"available":95897288704,"internally_reserved":0,"allocated":471040,"data_stored":178363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":738979539},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986333,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":605564,"kb_used_data":488,"kb_used_omap":4,"kb_used_meta":605051,"kb_avail":93766276,"statfs":{"total":96636764160,"available":96016666624,"internally_reserved":0,"allocated":499712,"data_stored":207035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":619572946},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673140,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713320,"kb_used_data":468,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":93658520,"statfs":{"total":96636764160,"available":95906324480,"internally_reserved":0,"allocated":479232,"data_stored":181558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":729938373},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182571,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713772,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658068,"statfs":{"total":96636764160,"available":95905861632,"internally_reserved":0,"allocated":479232,"data_stored":181558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":730394317},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:48:08.430 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:48:08.731 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:48:08.732 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:48:08.743 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":493,"stamp":"2024-09-05T16:48:07.114473+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":67360,"num_objects":10,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8916,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":90946,"num_write_kb":315952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4807,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2754800,"kb_used_data":1884,"kb_used_omap":14,"kb_used_meta":2752817,"kb_avail":374732560,"statfs":{"total":386547056640,"available":383726141440,"internally_reserved":0,"allocated":1929216,"data_stored":748514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2818885175},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13472,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1460,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12806,"num_write_kb":-51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-806,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001277"},"pg_stats":[{"pgid":"1.f","version":"39'12904","reported_seq":17531,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-05T16:48:02.225658+0000","last_change":"2024-09-05T16:46:27.826222+0000","last_active":"2024-09-05T16:48:02.225658+0000","last_peered":"2024-09-05T16:48:02.225658+0000","last_clean":"2024-09-05T16:48:02.225658+0000","last_became_active":"2024-09-05T16:46:27.757928+0000","last_became_peered":"2024-09-05T16:46:27.757928+0000","last_unstale":"2024-09-05T16:48:02.225658+0000","last_undegraded":"2024-09-05T16:48:02.225658+0000","last_fullsized":"2024-09-05T16:48:02.225658+0000","mapping_epoch":40,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:32:35.713428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12904,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2262,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12552","reported_seq":17243,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-05T16:48:02.228405+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:48:02.228405+0000","last_peered":"2024-09-05T16:48:02.228405+0000","last_clean":"2024-09-05T16:48:02.228405+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:48:02.228405+0000","last_undegraded":"2024-09-05T16:48:02.228405+0000","last_fullsized":"2024-09-05T16:48:02.228405+0000","mapping_epoch":38,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1865,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12752","reported_seq":17401,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-05T16:48:03.238089+0000","last_change":"2024-09-05T16:46:51.992073+0000","last_active":"2024-09-05T16:48:03.238089+0000","last_peered":"2024-09-05T16:48:03.238089+0000","last_clean":"2024-09-05T16:48:03.238089+0000","last_became_active":"2024-09-05T16:46:51.667128+0000","last_became_peered":"2024-09-05T16:46:51.667128+0000","last_unstale":"2024-09-05T16:48:03.238089+0000","last_undegraded":"2024-09-05T16:48:03.238089+0000","last_fullsized":"2024-09-05T16:48:03.238089+0000","mapping_epoch":44,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12752","last_scrub_stamp":"2024-09-05T16:46:51.991995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:51.991995+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:04:54.120031+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12934","reported_seq":17704,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-05T16:48:02.227867+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:48:02.227867+0000","last_peered":"2024-09-05T16:48:02.227867+0000","last_clean":"2024-09-05T16:48:02.227867+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:48:02.227867+0000","last_undegraded":"2024-09-05T16:48:02.227867+0000","last_fullsized":"2024-09-05T16:48:02.227867+0000","mapping_epoch":34,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7234,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12593","reported_seq":17044,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-05T16:48:02.225421+0000","last_change":"2024-09-05T16:47:51.525026+0000","last_active":"2024-09-05T16:48:02.225421+0000","last_peered":"2024-09-05T16:48:02.225421+0000","last_clean":"2024-09-05T16:48:02.225421+0000","last_became_active":"2024-09-05T16:47:51.193016+0000","last_became_peered":"2024-09-05T16:47:51.193016+0000","last_unstale":"2024-09-05T16:48:02.225421+0000","last_undegraded":"2024-09-05T16:48:02.225421+0000","last_fullsized":"2024-09-05T16:48:02.225421+0000","mapping_epoch":54,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12593","last_scrub_stamp":"2024-09-05T16:47:51.524957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:51.524957+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:46:22.846859+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12772","reported_seq":17375,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-05T16:48:02.226551+0000","last_change":"2024-09-05T16:47:15.832530+0000","last_active":"2024-09-05T16:48:02.226551+0000","last_peered":"2024-09-05T16:48:02.226551+0000","last_clean":"2024-09-05T16:48:02.226551+0000","last_became_active":"2024-09-05T16:47:15.728759+0000","last_became_peered":"2024-09-05T16:47:15.728759+0000","last_unstale":"2024-09-05T16:48:02.226551+0000","last_undegraded":"2024-09-05T16:48:02.226551+0000","last_fullsized":"2024-09-05T16:48:02.226551+0000","mapping_epoch":48,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T02:39:11.515339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12772,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2250,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12660","reported_seq":17094,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-05T16:48:03.235802+0000","last_change":"2024-09-05T16:47:03.683348+0000","last_active":"2024-09-05T16:48:03.235802+0000","last_peered":"2024-09-05T16:48:03.235802+0000","last_clean":"2024-09-05T16:48:03.235802+0000","last_became_active":"2024-09-05T16:47:03.185143+0000","last_became_peered":"2024-09-05T16:47:03.185143+0000","last_unstale":"2024-09-05T16:48:03.235802+0000","last_undegraded":"2024-09-05T16:48:03.235802+0000","last_fullsized":"2024-09-05T16:48:03.235802+0000","mapping_epoch":46,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12660","last_scrub_stamp":"2024-09-05T16:47:03.683262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:03.683262+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:42:16.533299+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12777","reported_seq":12920,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-05T16:48:02.228165+0000","last_change":"2024-09-05T16:46:40.458315+0000","last_active":"2024-09-05T16:48:02.228165+0000","last_peered":"2024-09-05T16:48:02.228165+0000","last_clean":"2024-09-05T16:48:02.228165+0000","last_became_active":"2024-09-05T16:46:40.184288+0000","last_became_peered":"2024-09-05T16:46:40.184288+0000","last_unstale":"2024-09-05T16:48:02.228165+0000","last_undegraded":"2024-09-05T16:48:02.228165+0000","last_fullsized":"2024-09-05T16:48:02.228165+0000","mapping_epoch":42,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12777","last_scrub_stamp":"2024-09-05T16:46:40.458226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:40.458226+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:38:58.734210+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'12770","reported_seq":12911,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:48:02.225388+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:48:02.225388+0000","last_peered":"2024-09-05T16:48:02.225388+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:48:02.225388+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12767,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2240}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12808","reported_seq":17484,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-05T16:48:04.288466+0000","last_change":"2024-09-05T16:48:04.288466+0000","last_active":"2024-09-05T16:48:04.288466+0000","last_peered":"2024-09-05T16:48:04.288466+0000","last_clean":"2024-09-05T16:48:04.288466+0000","last_became_active":"2024-09-05T16:48:03.239894+0000","last_became_peered":"2024-09-05T16:48:03.239894+0000","last_unstale":"2024-09-05T16:48:04.288466+0000","last_undegraded":"2024-09-05T16:48:04.288466+0000","last_fullsized":"2024-09-05T16:48:04.288466+0000","mapping_epoch":56,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12808","last_scrub_stamp":"2024-09-05T16:48:04.288388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:48:04.288388+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T21:11:32.044919+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13120","reported_seq":13287,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:48:02.225594+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:48:02.225594+0000","last_peered":"2024-09-05T16:48:02.225594+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:48:02.225594+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13117,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2278}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12758","reported_seq":12897,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:48:02.225705+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:48:02.225705+0000","last_peered":"2024-09-05T16:48:02.225705+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:48:02.225705+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2237}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12808","reported_seq":17584,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-05T16:48:02.227959+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:48:02.227959+0000","last_peered":"2024-09-05T16:48:02.227959+0000","last_clean":"2024-09-05T16:48:02.227959+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:48:02.227959+0000","last_undegraded":"2024-09-05T16:48:02.227959+0000","last_fullsized":"2024-09-05T16:48:02.227959+0000","mapping_epoch":36,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4408,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13124","reported_seq":13654,"reported_epoch":56,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:48:03.206933+0000","last_change":"2024-09-05T16:48:02.084558+0000","last_active":"2024-09-05T16:48:03.206933+0000","last_peered":"2024-09-05T16:48:03.206933+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:48:03.206933+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2146,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13122,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":215,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2144}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12831","reported_seq":12978,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-05T16:48:02.228131+0000","last_change":"2024-09-05T16:47:40.451380+0000","last_active":"2024-09-05T16:48:02.228131+0000","last_peered":"2024-09-05T16:48:02.228131+0000","last_clean":"2024-09-05T16:48:02.228131+0000","last_became_active":"2024-09-05T16:47:40.189447+0000","last_became_peered":"2024-09-05T16:47:40.189447+0000","last_unstale":"2024-09-05T16:48:02.228131+0000","last_undegraded":"2024-09-05T16:48:02.228131+0000","last_fullsized":"2024-09-05T16:48:02.228131+0000","mapping_epoch":52,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12831","last_scrub_stamp":"2024-09-05T16:47:40.451295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:40.451295+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:46:20.946333+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12773","reported_seq":12920,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-05T16:48:02.228253+0000","last_change":"2024-09-05T16:47:27.437275+0000","last_active":"2024-09-05T16:48:02.228253+0000","last_peered":"2024-09-05T16:48:02.228253+0000","last_clean":"2024-09-05T16:48:02.228253+0000","last_became_active":"2024-09-05T16:47:27.204989+0000","last_became_peered":"2024-09-05T16:47:27.204989+0000","last_unstale":"2024-09-05T16:48:02.228253+0000","last_undegraded":"2024-09-05T16:48:02.228253+0000","last_fullsized":"2024-09-05T16:48:02.228253+0000","mapping_epoch":50,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12773","last_scrub_stamp":"2024-09-05T16:47:27.437187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:27.437187+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T16:53:55.961513+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":67360,"num_objects":10,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8916,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":90946,"num_write_kb":315952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4807,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":45056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084456,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722144,"kb_used_data":460,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649696,"statfs":{"total":96636764160,"available":95897288704,"internally_reserved":0,"allocated":471040,"data_stored":178363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":738979539},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986333,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":605564,"kb_used_data":488,"kb_used_omap":4,"kb_used_meta":605051,"kb_avail":93766276,"statfs":{"total":96636764160,"available":96016666624,"internally_reserved":0,"allocated":499712,"data_stored":207035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":619572946},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673140,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713320,"kb_used_data":468,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":93658520,"statfs":{"total":96636764160,"available":95906324480,"internally_reserved":0,"allocated":479232,"data_stored":181558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":729938373},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182571,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713772,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658068,"statfs":{"total":96636764160,"available":95905861632,"internally_reserved":0,"allocated":479232,"data_stored":181558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":730394317},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:48:08.744 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:48:08.745 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "39'12770", 'reported_seq': 12911, 'reported_epoch': 56, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:48:02.225388+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:48:02.225388+0000', 'last_peered': '2024-09-05T16:48:02.225388+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:48:02.225388+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "33'2700", 'ondisk_log_start': "33'2700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10070, 'log_dups_size': 0, 'ondisk_log_size': 10070, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26944, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 16, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12767, 'num_write_kb': 51081, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2240}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:48:08.745 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:48:08.745 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "39'13120", 'reported_seq': 13287, 'reported_epoch': 56, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:48:02.225594+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:48:02.225594+0000', 'last_peered': '2024-09-05T16:48:02.225594+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:48:02.225594+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "33'3100", 'ondisk_log_start': "33'3100", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20208, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2284, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13117, 'num_write_kb': 52476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2278}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:48:08.745 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:48:08.745 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "39'12758", 'reported_seq': 12897, 'reported_epoch': 56, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:48:02.225705+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:48:02.225705+0000', 'last_peered': '2024-09-05T16:48:02.225705+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:48:02.225705+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "33'2700", 'ondisk_log_start': "33'2700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10058, 'log_dups_size': 0, 'ondisk_log_size': 10058, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6736, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12757, 'num_write_kb': 51031, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2237}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:48:08.745 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:48:08.745 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "39'13124", 'reported_seq': 13654, 'reported_epoch': 56, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:48:03.206933+0000', 'last_change': '2024-09-05T16:48:02.084558+0000', 'last_active': '2024-09-05T16:48:03.206933+0000', 'last_peered': '2024-09-05T16:48:03.206933+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:48:03.206933+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "33'3100", 'ondisk_log_start': "33'3100", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10024, 'log_dups_size': 0, 'ondisk_log_size': 10024, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2146, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13122, 'num_write_kb': 52500, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 215, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2144}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:48:08.745 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:48:09.134 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:48:09.134 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 8916/40 objects degraded (22290.000%), 4 pgs degraded, 4 pgs undersized","count":8},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":892,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":57,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":4},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":10,"data_bytes":67360,"bytes_used":2820915200,"bytes_avail":383726141440,"bytes_total":386547056640,"degraded_objects":8916,"degraded_total":40,"degraded_ratio":222.90000000000001},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:48:09.148 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-09-05T16:48:09.148 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:48:09.460 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:48:09.461 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:48:09.473 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":494,"stamp":"2024-09-05T16:48:09.114848+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":67360,"num_objects":10,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8124,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":90946,"num_write_kb":315952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5599,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2763136,"kb_used_data":1900,"kb_used_omap":14,"kb_used_meta":2761137,"kb_avail":374724224,"statfs":{"total":386547056640,"available":383717605376,"internally_reserved":0,"allocated":1945600,"data_stored":754904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2827404855},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13472,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12806,"num_write_kb":-51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-996,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001312"},"pg_stats":[{"pgid":"1.f","version":"39'12904","reported_seq":17533,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-05T16:48:03.237710+0000","last_change":"2024-09-05T16:46:27.826222+0000","last_active":"2024-09-05T16:48:03.237710+0000","last_peered":"2024-09-05T16:48:03.237710+0000","last_clean":"2024-09-05T16:48:03.237710+0000","last_became_active":"2024-09-05T16:46:27.757928+0000","last_became_peered":"2024-09-05T16:46:27.757928+0000","last_unstale":"2024-09-05T16:48:03.237710+0000","last_undegraded":"2024-09-05T16:48:03.237710+0000","last_fullsized":"2024-09-05T16:48:03.237710+0000","mapping_epoch":40,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:32:35.713428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12904,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2262,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12552","reported_seq":17245,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-05T16:48:03.237008+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:48:03.237008+0000","last_peered":"2024-09-05T16:48:03.237008+0000","last_clean":"2024-09-05T16:48:03.237008+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:48:03.237008+0000","last_undegraded":"2024-09-05T16:48:03.237008+0000","last_fullsized":"2024-09-05T16:48:03.237008+0000","mapping_epoch":38,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1865,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12752","reported_seq":17401,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-05T16:48:03.238089+0000","last_change":"2024-09-05T16:46:51.992073+0000","last_active":"2024-09-05T16:48:03.238089+0000","last_peered":"2024-09-05T16:48:03.238089+0000","last_clean":"2024-09-05T16:48:03.238089+0000","last_became_active":"2024-09-05T16:46:51.667128+0000","last_became_peered":"2024-09-05T16:46:51.667128+0000","last_unstale":"2024-09-05T16:48:03.238089+0000","last_undegraded":"2024-09-05T16:48:03.238089+0000","last_fullsized":"2024-09-05T16:48:03.238089+0000","mapping_epoch":44,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12752","last_scrub_stamp":"2024-09-05T16:46:51.991995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:51.991995+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:04:54.120031+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12934","reported_seq":17706,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-05T16:48:03.237150+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:48:03.237150+0000","last_peered":"2024-09-05T16:48:03.237150+0000","last_clean":"2024-09-05T16:48:03.237150+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:48:03.237150+0000","last_undegraded":"2024-09-05T16:48:03.237150+0000","last_fullsized":"2024-09-05T16:48:03.237150+0000","mapping_epoch":34,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7234,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12593","reported_seq":17046,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-05T16:48:03.235245+0000","last_change":"2024-09-05T16:47:51.525026+0000","last_active":"2024-09-05T16:48:03.235245+0000","last_peered":"2024-09-05T16:48:03.235245+0000","last_clean":"2024-09-05T16:48:03.235245+0000","last_became_active":"2024-09-05T16:47:51.193016+0000","last_became_peered":"2024-09-05T16:47:51.193016+0000","last_unstale":"2024-09-05T16:48:03.235245+0000","last_undegraded":"2024-09-05T16:48:03.235245+0000","last_fullsized":"2024-09-05T16:48:03.235245+0000","mapping_epoch":54,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12593","last_scrub_stamp":"2024-09-05T16:47:51.524957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:51.524957+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:46:22.846859+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12772","reported_seq":17377,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-05T16:48:03.235526+0000","last_change":"2024-09-05T16:47:15.832530+0000","last_active":"2024-09-05T16:48:03.235526+0000","last_peered":"2024-09-05T16:48:03.235526+0000","last_clean":"2024-09-05T16:48:03.235526+0000","last_became_active":"2024-09-05T16:47:15.728759+0000","last_became_peered":"2024-09-05T16:47:15.728759+0000","last_unstale":"2024-09-05T16:48:03.235526+0000","last_undegraded":"2024-09-05T16:48:03.235526+0000","last_fullsized":"2024-09-05T16:48:03.235526+0000","mapping_epoch":48,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T02:39:11.515339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12772,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2250,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12660","reported_seq":17094,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-05T16:48:03.235802+0000","last_change":"2024-09-05T16:47:03.683348+0000","last_active":"2024-09-05T16:48:03.235802+0000","last_peered":"2024-09-05T16:48:03.235802+0000","last_clean":"2024-09-05T16:48:03.235802+0000","last_became_active":"2024-09-05T16:47:03.185143+0000","last_became_peered":"2024-09-05T16:47:03.185143+0000","last_unstale":"2024-09-05T16:48:03.235802+0000","last_undegraded":"2024-09-05T16:48:03.235802+0000","last_fullsized":"2024-09-05T16:48:03.235802+0000","mapping_epoch":46,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12660","last_scrub_stamp":"2024-09-05T16:47:03.683262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:03.683262+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:42:16.533299+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12777","reported_seq":12922,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-05T16:48:03.236439+0000","last_change":"2024-09-05T16:46:40.458315+0000","last_active":"2024-09-05T16:48:03.236439+0000","last_peered":"2024-09-05T16:48:03.236439+0000","last_clean":"2024-09-05T16:48:03.236439+0000","last_became_active":"2024-09-05T16:46:40.184288+0000","last_became_peered":"2024-09-05T16:46:40.184288+0000","last_unstale":"2024-09-05T16:48:03.236439+0000","last_undegraded":"2024-09-05T16:48:03.236439+0000","last_fullsized":"2024-09-05T16:48:03.236439+0000","mapping_epoch":42,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12777","last_scrub_stamp":"2024-09-05T16:46:40.458226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:40.458226+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:38:58.734210+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'12770","reported_seq":12913,"reported_epoch":57,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:48:03.235498+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:48:03.235498+0000","last_peered":"2024-09-05T16:48:03.235498+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:48:03.235498+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12767,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2240}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12808","reported_seq":17484,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-05T16:48:04.288466+0000","last_change":"2024-09-05T16:48:04.288466+0000","last_active":"2024-09-05T16:48:04.288466+0000","last_peered":"2024-09-05T16:48:04.288466+0000","last_clean":"2024-09-05T16:48:04.288466+0000","last_became_active":"2024-09-05T16:48:03.239894+0000","last_became_peered":"2024-09-05T16:48:03.239894+0000","last_unstale":"2024-09-05T16:48:04.288466+0000","last_undegraded":"2024-09-05T16:48:04.288466+0000","last_fullsized":"2024-09-05T16:48:04.288466+0000","mapping_epoch":56,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12808","last_scrub_stamp":"2024-09-05T16:48:04.288388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:48:04.288388+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T21:11:32.044919+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13120","reported_seq":13289,"reported_epoch":57,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:48:03.236782+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:48:03.236782+0000","last_peered":"2024-09-05T16:48:03.236782+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:48:03.236782+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13117,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2278}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12758","reported_seq":12899,"reported_epoch":57,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:48:03.237812+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:48:03.237812+0000","last_peered":"2024-09-05T16:48:03.237812+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:48:03.237812+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2237}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12808","reported_seq":17586,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-05T16:48:03.236381+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:48:03.236381+0000","last_peered":"2024-09-05T16:48:03.236381+0000","last_clean":"2024-09-05T16:48:03.236381+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:48:03.236381+0000","last_undegraded":"2024-09-05T16:48:03.236381+0000","last_fullsized":"2024-09-05T16:48:03.236381+0000","mapping_epoch":36,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4408,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13124","reported_seq":15240,"reported_epoch":57,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:48:08.208424+0000","last_change":"2024-09-05T16:48:02.084558+0000","last_active":"2024-09-05T16:48:08.208424+0000","last_peered":"2024-09-05T16:48:08.208424+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:48:08.208424+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1354,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13122,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1007,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1352}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12831","reported_seq":12980,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-05T16:48:03.236429+0000","last_change":"2024-09-05T16:47:40.451380+0000","last_active":"2024-09-05T16:48:03.236429+0000","last_peered":"2024-09-05T16:48:03.236429+0000","last_clean":"2024-09-05T16:48:03.236429+0000","last_became_active":"2024-09-05T16:47:40.189447+0000","last_became_peered":"2024-09-05T16:47:40.189447+0000","last_unstale":"2024-09-05T16:48:03.236429+0000","last_undegraded":"2024-09-05T16:48:03.236429+0000","last_fullsized":"2024-09-05T16:48:03.236429+0000","mapping_epoch":52,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12831","last_scrub_stamp":"2024-09-05T16:47:40.451295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:40.451295+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:46:20.946333+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12773","reported_seq":12922,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-05T16:48:03.236707+0000","last_change":"2024-09-05T16:47:27.437275+0000","last_active":"2024-09-05T16:48:03.236707+0000","last_peered":"2024-09-05T16:48:03.236707+0000","last_clean":"2024-09-05T16:48:03.236707+0000","last_became_active":"2024-09-05T16:47:27.204989+0000","last_became_peered":"2024-09-05T16:47:27.204989+0000","last_unstale":"2024-09-05T16:48:03.236707+0000","last_undegraded":"2024-09-05T16:48:03.236707+0000","last_fullsized":"2024-09-05T16:48:03.236707+0000","mapping_epoch":50,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12773","last_scrub_stamp":"2024-09-05T16:47:27.437187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:27.437187+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T16:53:55.961513+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":67360,"num_objects":10,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8124,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":90946,"num_write_kb":315952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5599,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":45056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084457,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722156,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649684,"statfs":{"total":96636764160,"available":95897276416,"internally_reserved":0,"allocated":479232,"data_stored":181558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":738979539},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986334,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":613888,"kb_used_data":496,"kb_used_omap":4,"kb_used_meta":613371,"kb_avail":93757952,"statfs":{"total":96636764160,"available":96008142848,"internally_reserved":0,"allocated":507904,"data_stored":210230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":628092626},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673140,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713320,"kb_used_data":468,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":93658520,"statfs":{"total":96636764160,"available":95906324480,"internally_reserved":0,"allocated":479232,"data_stored":181558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":729938373},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182571,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713772,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658068,"statfs":{"total":96636764160,"available":95905861632,"internally_reserved":0,"allocated":479232,"data_stored":181558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":730394317},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:48:12.476 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:48:12.777 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:48:12.780 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:48:12.789 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":495,"stamp":"2024-09-05T16:48:11.115170+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":67360,"num_objects":10,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8124,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":90946,"num_write_kb":315952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5599,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2763136,"kb_used_data":1900,"kb_used_omap":14,"kb_used_meta":2761137,"kb_avail":374724224,"statfs":{"total":386547056640,"available":383717605376,"internally_reserved":0,"allocated":1945600,"data_stored":754904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2827404855},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13472,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12806,"num_write_kb":-51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-996,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.891901"},"pg_stats":[{"pgid":"1.f","version":"39'12904","reported_seq":17533,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-05T16:48:03.237710+0000","last_change":"2024-09-05T16:46:27.826222+0000","last_active":"2024-09-05T16:48:03.237710+0000","last_peered":"2024-09-05T16:48:03.237710+0000","last_clean":"2024-09-05T16:48:03.237710+0000","last_became_active":"2024-09-05T16:46:27.757928+0000","last_became_peered":"2024-09-05T16:46:27.757928+0000","last_unstale":"2024-09-05T16:48:03.237710+0000","last_undegraded":"2024-09-05T16:48:03.237710+0000","last_fullsized":"2024-09-05T16:48:03.237710+0000","mapping_epoch":40,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:32:35.713428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12904,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2262,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12552","reported_seq":17245,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-05T16:48:03.237008+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:48:03.237008+0000","last_peered":"2024-09-05T16:48:03.237008+0000","last_clean":"2024-09-05T16:48:03.237008+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:48:03.237008+0000","last_undegraded":"2024-09-05T16:48:03.237008+0000","last_fullsized":"2024-09-05T16:48:03.237008+0000","mapping_epoch":38,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1865,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12752","reported_seq":17401,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-05T16:48:03.238089+0000","last_change":"2024-09-05T16:46:51.992073+0000","last_active":"2024-09-05T16:48:03.238089+0000","last_peered":"2024-09-05T16:48:03.238089+0000","last_clean":"2024-09-05T16:48:03.238089+0000","last_became_active":"2024-09-05T16:46:51.667128+0000","last_became_peered":"2024-09-05T16:46:51.667128+0000","last_unstale":"2024-09-05T16:48:03.238089+0000","last_undegraded":"2024-09-05T16:48:03.238089+0000","last_fullsized":"2024-09-05T16:48:03.238089+0000","mapping_epoch":44,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12752","last_scrub_stamp":"2024-09-05T16:46:51.991995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:51.991995+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:04:54.120031+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12934","reported_seq":17706,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-05T16:48:03.237150+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:48:03.237150+0000","last_peered":"2024-09-05T16:48:03.237150+0000","last_clean":"2024-09-05T16:48:03.237150+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:48:03.237150+0000","last_undegraded":"2024-09-05T16:48:03.237150+0000","last_fullsized":"2024-09-05T16:48:03.237150+0000","mapping_epoch":34,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7234,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12593","reported_seq":17046,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-05T16:48:03.235245+0000","last_change":"2024-09-05T16:47:51.525026+0000","last_active":"2024-09-05T16:48:03.235245+0000","last_peered":"2024-09-05T16:48:03.235245+0000","last_clean":"2024-09-05T16:48:03.235245+0000","last_became_active":"2024-09-05T16:47:51.193016+0000","last_became_peered":"2024-09-05T16:47:51.193016+0000","last_unstale":"2024-09-05T16:48:03.235245+0000","last_undegraded":"2024-09-05T16:48:03.235245+0000","last_fullsized":"2024-09-05T16:48:03.235245+0000","mapping_epoch":54,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12593","last_scrub_stamp":"2024-09-05T16:47:51.524957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:51.524957+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:46:22.846859+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12772","reported_seq":17377,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-05T16:48:03.235526+0000","last_change":"2024-09-05T16:47:15.832530+0000","last_active":"2024-09-05T16:48:03.235526+0000","last_peered":"2024-09-05T16:48:03.235526+0000","last_clean":"2024-09-05T16:48:03.235526+0000","last_became_active":"2024-09-05T16:47:15.728759+0000","last_became_peered":"2024-09-05T16:47:15.728759+0000","last_unstale":"2024-09-05T16:48:03.235526+0000","last_undegraded":"2024-09-05T16:48:03.235526+0000","last_fullsized":"2024-09-05T16:48:03.235526+0000","mapping_epoch":48,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T02:39:11.515339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12772,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2250,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12660","reported_seq":17094,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-05T16:48:03.235802+0000","last_change":"2024-09-05T16:47:03.683348+0000","last_active":"2024-09-05T16:48:03.235802+0000","last_peered":"2024-09-05T16:48:03.235802+0000","last_clean":"2024-09-05T16:48:03.235802+0000","last_became_active":"2024-09-05T16:47:03.185143+0000","last_became_peered":"2024-09-05T16:47:03.185143+0000","last_unstale":"2024-09-05T16:48:03.235802+0000","last_undegraded":"2024-09-05T16:48:03.235802+0000","last_fullsized":"2024-09-05T16:48:03.235802+0000","mapping_epoch":46,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12660","last_scrub_stamp":"2024-09-05T16:47:03.683262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:03.683262+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:42:16.533299+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12777","reported_seq":12922,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-05T16:48:03.236439+0000","last_change":"2024-09-05T16:46:40.458315+0000","last_active":"2024-09-05T16:48:03.236439+0000","last_peered":"2024-09-05T16:48:03.236439+0000","last_clean":"2024-09-05T16:48:03.236439+0000","last_became_active":"2024-09-05T16:46:40.184288+0000","last_became_peered":"2024-09-05T16:46:40.184288+0000","last_unstale":"2024-09-05T16:48:03.236439+0000","last_undegraded":"2024-09-05T16:48:03.236439+0000","last_fullsized":"2024-09-05T16:48:03.236439+0000","mapping_epoch":42,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12777","last_scrub_stamp":"2024-09-05T16:46:40.458226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:40.458226+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:38:58.734210+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'12770","reported_seq":12913,"reported_epoch":57,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:48:03.235498+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:48:03.235498+0000","last_peered":"2024-09-05T16:48:03.235498+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:48:03.235498+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12767,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2240}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12808","reported_seq":17484,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-05T16:48:04.288466+0000","last_change":"2024-09-05T16:48:04.288466+0000","last_active":"2024-09-05T16:48:04.288466+0000","last_peered":"2024-09-05T16:48:04.288466+0000","last_clean":"2024-09-05T16:48:04.288466+0000","last_became_active":"2024-09-05T16:48:03.239894+0000","last_became_peered":"2024-09-05T16:48:03.239894+0000","last_unstale":"2024-09-05T16:48:04.288466+0000","last_undegraded":"2024-09-05T16:48:04.288466+0000","last_fullsized":"2024-09-05T16:48:04.288466+0000","mapping_epoch":56,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12808","last_scrub_stamp":"2024-09-05T16:48:04.288388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:48:04.288388+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T21:11:32.044919+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13120","reported_seq":13289,"reported_epoch":57,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:48:03.236782+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:48:03.236782+0000","last_peered":"2024-09-05T16:48:03.236782+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:48:03.236782+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13117,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2278}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12758","reported_seq":12899,"reported_epoch":57,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:48:03.237812+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:48:03.237812+0000","last_peered":"2024-09-05T16:48:03.237812+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:48:03.237812+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2237}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12808","reported_seq":17586,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-05T16:48:03.236381+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:48:03.236381+0000","last_peered":"2024-09-05T16:48:03.236381+0000","last_clean":"2024-09-05T16:48:03.236381+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:48:03.236381+0000","last_undegraded":"2024-09-05T16:48:03.236381+0000","last_fullsized":"2024-09-05T16:48:03.236381+0000","mapping_epoch":36,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4408,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13124","reported_seq":15240,"reported_epoch":57,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:48:08.208424+0000","last_change":"2024-09-05T16:48:02.084558+0000","last_active":"2024-09-05T16:48:08.208424+0000","last_peered":"2024-09-05T16:48:08.208424+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:48:08.208424+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1354,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13122,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1007,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1352}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12831","reported_seq":12980,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-05T16:48:03.236429+0000","last_change":"2024-09-05T16:47:40.451380+0000","last_active":"2024-09-05T16:48:03.236429+0000","last_peered":"2024-09-05T16:48:03.236429+0000","last_clean":"2024-09-05T16:48:03.236429+0000","last_became_active":"2024-09-05T16:47:40.189447+0000","last_became_peered":"2024-09-05T16:47:40.189447+0000","last_unstale":"2024-09-05T16:48:03.236429+0000","last_undegraded":"2024-09-05T16:48:03.236429+0000","last_fullsized":"2024-09-05T16:48:03.236429+0000","mapping_epoch":52,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12831","last_scrub_stamp":"2024-09-05T16:47:40.451295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:40.451295+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:46:20.946333+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12773","reported_seq":12922,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-05T16:48:03.236707+0000","last_change":"2024-09-05T16:47:27.437275+0000","last_active":"2024-09-05T16:48:03.236707+0000","last_peered":"2024-09-05T16:48:03.236707+0000","last_clean":"2024-09-05T16:48:03.236707+0000","last_became_active":"2024-09-05T16:47:27.204989+0000","last_became_peered":"2024-09-05T16:47:27.204989+0000","last_unstale":"2024-09-05T16:48:03.236707+0000","last_undegraded":"2024-09-05T16:48:03.236707+0000","last_fullsized":"2024-09-05T16:48:03.236707+0000","mapping_epoch":50,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12773","last_scrub_stamp":"2024-09-05T16:47:27.437187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:27.437187+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T16:53:55.961513+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":67360,"num_objects":10,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8124,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":90946,"num_write_kb":315952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5599,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":45056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084457,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722156,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649684,"statfs":{"total":96636764160,"available":95897276416,"internally_reserved":0,"allocated":479232,"data_stored":181558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":738979539},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986334,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":613888,"kb_used_data":496,"kb_used_omap":4,"kb_used_meta":613371,"kb_avail":93757952,"statfs":{"total":96636764160,"available":96008142848,"internally_reserved":0,"allocated":507904,"data_stored":210230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":628092626},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673141,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713320,"kb_used_data":468,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":93658520,"statfs":{"total":96636764160,"available":95906324480,"internally_reserved":0,"allocated":479232,"data_stored":181558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":729938373},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182571,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713772,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658068,"statfs":{"total":96636764160,"available":95905861632,"internally_reserved":0,"allocated":479232,"data_stored":181558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":730394317},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:48:12.790 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:48:13.094 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:48:13.095 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:48:13.107 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":495,"stamp":"2024-09-05T16:48:11.115170+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":67360,"num_objects":10,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8124,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":90946,"num_write_kb":315952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5599,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2763136,"kb_used_data":1900,"kb_used_omap":14,"kb_used_meta":2761137,"kb_avail":374724224,"statfs":{"total":386547056640,"available":383717605376,"internally_reserved":0,"allocated":1945600,"data_stored":754904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2827404855},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13472,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12806,"num_write_kb":-51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-996,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.891901"},"pg_stats":[{"pgid":"1.f","version":"39'12904","reported_seq":17533,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-05T16:48:03.237710+0000","last_change":"2024-09-05T16:46:27.826222+0000","last_active":"2024-09-05T16:48:03.237710+0000","last_peered":"2024-09-05T16:48:03.237710+0000","last_clean":"2024-09-05T16:48:03.237710+0000","last_became_active":"2024-09-05T16:46:27.757928+0000","last_became_peered":"2024-09-05T16:46:27.757928+0000","last_unstale":"2024-09-05T16:48:03.237710+0000","last_undegraded":"2024-09-05T16:48:03.237710+0000","last_fullsized":"2024-09-05T16:48:03.237710+0000","mapping_epoch":40,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:32:35.713428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12904,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2262,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12552","reported_seq":17245,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-05T16:48:03.237008+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:48:03.237008+0000","last_peered":"2024-09-05T16:48:03.237008+0000","last_clean":"2024-09-05T16:48:03.237008+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:48:03.237008+0000","last_undegraded":"2024-09-05T16:48:03.237008+0000","last_fullsized":"2024-09-05T16:48:03.237008+0000","mapping_epoch":38,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1865,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12752","reported_seq":17401,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-05T16:48:03.238089+0000","last_change":"2024-09-05T16:46:51.992073+0000","last_active":"2024-09-05T16:48:03.238089+0000","last_peered":"2024-09-05T16:48:03.238089+0000","last_clean":"2024-09-05T16:48:03.238089+0000","last_became_active":"2024-09-05T16:46:51.667128+0000","last_became_peered":"2024-09-05T16:46:51.667128+0000","last_unstale":"2024-09-05T16:48:03.238089+0000","last_undegraded":"2024-09-05T16:48:03.238089+0000","last_fullsized":"2024-09-05T16:48:03.238089+0000","mapping_epoch":44,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12752","last_scrub_stamp":"2024-09-05T16:46:51.991995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:51.991995+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:04:54.120031+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12934","reported_seq":17706,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-05T16:48:03.237150+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:48:03.237150+0000","last_peered":"2024-09-05T16:48:03.237150+0000","last_clean":"2024-09-05T16:48:03.237150+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:48:03.237150+0000","last_undegraded":"2024-09-05T16:48:03.237150+0000","last_fullsized":"2024-09-05T16:48:03.237150+0000","mapping_epoch":34,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7234,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12593","reported_seq":17046,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-05T16:48:03.235245+0000","last_change":"2024-09-05T16:47:51.525026+0000","last_active":"2024-09-05T16:48:03.235245+0000","last_peered":"2024-09-05T16:48:03.235245+0000","last_clean":"2024-09-05T16:48:03.235245+0000","last_became_active":"2024-09-05T16:47:51.193016+0000","last_became_peered":"2024-09-05T16:47:51.193016+0000","last_unstale":"2024-09-05T16:48:03.235245+0000","last_undegraded":"2024-09-05T16:48:03.235245+0000","last_fullsized":"2024-09-05T16:48:03.235245+0000","mapping_epoch":54,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12593","last_scrub_stamp":"2024-09-05T16:47:51.524957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:51.524957+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:46:22.846859+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12772","reported_seq":17377,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-05T16:48:03.235526+0000","last_change":"2024-09-05T16:47:15.832530+0000","last_active":"2024-09-05T16:48:03.235526+0000","last_peered":"2024-09-05T16:48:03.235526+0000","last_clean":"2024-09-05T16:48:03.235526+0000","last_became_active":"2024-09-05T16:47:15.728759+0000","last_became_peered":"2024-09-05T16:47:15.728759+0000","last_unstale":"2024-09-05T16:48:03.235526+0000","last_undegraded":"2024-09-05T16:48:03.235526+0000","last_fullsized":"2024-09-05T16:48:03.235526+0000","mapping_epoch":48,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T02:39:11.515339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12772,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2250,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12660","reported_seq":17094,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-05T16:48:03.235802+0000","last_change":"2024-09-05T16:47:03.683348+0000","last_active":"2024-09-05T16:48:03.235802+0000","last_peered":"2024-09-05T16:48:03.235802+0000","last_clean":"2024-09-05T16:48:03.235802+0000","last_became_active":"2024-09-05T16:47:03.185143+0000","last_became_peered":"2024-09-05T16:47:03.185143+0000","last_unstale":"2024-09-05T16:48:03.235802+0000","last_undegraded":"2024-09-05T16:48:03.235802+0000","last_fullsized":"2024-09-05T16:48:03.235802+0000","mapping_epoch":46,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12660","last_scrub_stamp":"2024-09-05T16:47:03.683262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:03.683262+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:42:16.533299+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12777","reported_seq":12922,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-05T16:48:03.236439+0000","last_change":"2024-09-05T16:46:40.458315+0000","last_active":"2024-09-05T16:48:03.236439+0000","last_peered":"2024-09-05T16:48:03.236439+0000","last_clean":"2024-09-05T16:48:03.236439+0000","last_became_active":"2024-09-05T16:46:40.184288+0000","last_became_peered":"2024-09-05T16:46:40.184288+0000","last_unstale":"2024-09-05T16:48:03.236439+0000","last_undegraded":"2024-09-05T16:48:03.236439+0000","last_fullsized":"2024-09-05T16:48:03.236439+0000","mapping_epoch":42,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12777","last_scrub_stamp":"2024-09-05T16:46:40.458226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:40.458226+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:38:58.734210+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'12770","reported_seq":12913,"reported_epoch":57,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:48:03.235498+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:48:03.235498+0000","last_peered":"2024-09-05T16:48:03.235498+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:48:03.235498+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12767,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2240}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12808","reported_seq":17484,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-05T16:48:04.288466+0000","last_change":"2024-09-05T16:48:04.288466+0000","last_active":"2024-09-05T16:48:04.288466+0000","last_peered":"2024-09-05T16:48:04.288466+0000","last_clean":"2024-09-05T16:48:04.288466+0000","last_became_active":"2024-09-05T16:48:03.239894+0000","last_became_peered":"2024-09-05T16:48:03.239894+0000","last_unstale":"2024-09-05T16:48:04.288466+0000","last_undegraded":"2024-09-05T16:48:04.288466+0000","last_fullsized":"2024-09-05T16:48:04.288466+0000","mapping_epoch":56,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12808","last_scrub_stamp":"2024-09-05T16:48:04.288388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:48:04.288388+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T21:11:32.044919+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13120","reported_seq":13289,"reported_epoch":57,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:48:03.236782+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:48:03.236782+0000","last_peered":"2024-09-05T16:48:03.236782+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:48:03.236782+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13117,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2278}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12758","reported_seq":12899,"reported_epoch":57,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:48:03.237812+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:48:03.237812+0000","last_peered":"2024-09-05T16:48:03.237812+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:48:03.237812+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2237}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12808","reported_seq":17586,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-05T16:48:03.236381+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:48:03.236381+0000","last_peered":"2024-09-05T16:48:03.236381+0000","last_clean":"2024-09-05T16:48:03.236381+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:48:03.236381+0000","last_undegraded":"2024-09-05T16:48:03.236381+0000","last_fullsized":"2024-09-05T16:48:03.236381+0000","mapping_epoch":36,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4408,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13124","reported_seq":15240,"reported_epoch":57,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:48:08.208424+0000","last_change":"2024-09-05T16:48:02.084558+0000","last_active":"2024-09-05T16:48:08.208424+0000","last_peered":"2024-09-05T16:48:08.208424+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:48:08.208424+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1354,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13122,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1007,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1352}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12831","reported_seq":12980,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-05T16:48:03.236429+0000","last_change":"2024-09-05T16:47:40.451380+0000","last_active":"2024-09-05T16:48:03.236429+0000","last_peered":"2024-09-05T16:48:03.236429+0000","last_clean":"2024-09-05T16:48:03.236429+0000","last_became_active":"2024-09-05T16:47:40.189447+0000","last_became_peered":"2024-09-05T16:47:40.189447+0000","last_unstale":"2024-09-05T16:48:03.236429+0000","last_undegraded":"2024-09-05T16:48:03.236429+0000","last_fullsized":"2024-09-05T16:48:03.236429+0000","mapping_epoch":52,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12831","last_scrub_stamp":"2024-09-05T16:47:40.451295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:40.451295+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:46:20.946333+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12773","reported_seq":12922,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-05T16:48:03.236707+0000","last_change":"2024-09-05T16:47:27.437275+0000","last_active":"2024-09-05T16:48:03.236707+0000","last_peered":"2024-09-05T16:48:03.236707+0000","last_clean":"2024-09-05T16:48:03.236707+0000","last_became_active":"2024-09-05T16:47:27.204989+0000","last_became_peered":"2024-09-05T16:47:27.204989+0000","last_unstale":"2024-09-05T16:48:03.236707+0000","last_undegraded":"2024-09-05T16:48:03.236707+0000","last_fullsized":"2024-09-05T16:48:03.236707+0000","mapping_epoch":50,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12773","last_scrub_stamp":"2024-09-05T16:47:27.437187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:27.437187+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T16:53:55.961513+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":67360,"num_objects":10,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8124,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":90946,"num_write_kb":315952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5599,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":45056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084457,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722156,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649684,"statfs":{"total":96636764160,"available":95897276416,"internally_reserved":0,"allocated":479232,"data_stored":181558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":738979539},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986334,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":613888,"kb_used_data":496,"kb_used_omap":4,"kb_used_meta":613371,"kb_avail":93757952,"statfs":{"total":96636764160,"available":96008142848,"internally_reserved":0,"allocated":507904,"data_stored":210230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":628092626},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673141,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713320,"kb_used_data":468,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":93658520,"statfs":{"total":96636764160,"available":95906324480,"internally_reserved":0,"allocated":479232,"data_stored":181558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":729938373},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182571,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713772,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658068,"statfs":{"total":96636764160,"available":95905861632,"internally_reserved":0,"allocated":479232,"data_stored":181558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":730394317},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:48:13.107 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:48:13.108 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "39'12770", 'reported_seq': 12913, 'reported_epoch': 57, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:48:03.235498+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:48:03.235498+0000', 'last_peered': '2024-09-05T16:48:03.235498+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:48:03.235498+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "33'2700", 'ondisk_log_start': "33'2700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10070, 'log_dups_size': 0, 'ondisk_log_size': 10070, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26944, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 16, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12767, 'num_write_kb': 51081, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2240}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:48:13.108 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:48:13.108 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "39'13120", 'reported_seq': 13289, 'reported_epoch': 57, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:48:03.236782+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:48:03.236782+0000', 'last_peered': '2024-09-05T16:48:03.236782+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:48:03.236782+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "33'3100", 'ondisk_log_start': "33'3100", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20208, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2284, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13117, 'num_write_kb': 52476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2278}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:48:13.108 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:48:13.108 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "39'12758", 'reported_seq': 12899, 'reported_epoch': 57, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:48:03.237812+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:48:03.237812+0000', 'last_peered': '2024-09-05T16:48:03.237812+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:48:03.237812+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "33'2700", 'ondisk_log_start': "33'2700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10058, 'log_dups_size': 0, 'ondisk_log_size': 10058, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6736, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12757, 'num_write_kb': 51031, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2237}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:48:13.108 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:48:13.108 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "39'13124", 'reported_seq': 15240, 'reported_epoch': 57, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:48:08.208424+0000', 'last_change': '2024-09-05T16:48:02.084558+0000', 'last_active': '2024-09-05T16:48:08.208424+0000', 'last_peered': '2024-09-05T16:48:08.208424+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:48:08.208424+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "33'3100", 'ondisk_log_start': "33'3100", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10024, 'log_dups_size': 0, 'ondisk_log_size': 10024, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1354, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13122, 'num_write_kb': 52500, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1007, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1352}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:48:13.108 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:48:13.492 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:48:13.492 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 8124/40 objects degraded (20310.000%), 4 pgs degraded, 4 pgs undersized","count":8},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":896,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":57,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":4},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":10,"data_bytes":67360,"bytes_used":2837970944,"bytes_avail":383709085696,"bytes_total":386547056640,"degraded_objects":8124,"degraded_total":40,"degraded_ratio":203.09999999999999},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:48:13.505 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-09-05T16:48:13.505 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:48:13.808 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:48:13.808 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:48:13.820 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":496,"stamp":"2024-09-05T16:48:13.115471+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":67360,"num_objects":10,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8124,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":90946,"num_write_kb":315952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5599,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2771456,"kb_used_data":1900,"kb_used_omap":14,"kb_used_meta":2769457,"kb_avail":374715904,"statfs":{"total":386547056640,"available":383709085696,"internally_reserved":0,"allocated":1945600,"data_stored":754904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2835924535},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13472,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12806,"num_write_kb":-51230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-996,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001615"},"pg_stats":[{"pgid":"1.f","version":"39'12904","reported_seq":17533,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-05T16:48:03.237710+0000","last_change":"2024-09-05T16:46:27.826222+0000","last_active":"2024-09-05T16:48:03.237710+0000","last_peered":"2024-09-05T16:48:03.237710+0000","last_clean":"2024-09-05T16:48:03.237710+0000","last_became_active":"2024-09-05T16:46:27.757928+0000","last_became_peered":"2024-09-05T16:46:27.757928+0000","last_unstale":"2024-09-05T16:48:03.237710+0000","last_undegraded":"2024-09-05T16:48:03.237710+0000","last_fullsized":"2024-09-05T16:48:03.237710+0000","mapping_epoch":40,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:32:35.713428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12904,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2262,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12552","reported_seq":17245,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-05T16:48:03.237008+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:48:03.237008+0000","last_peered":"2024-09-05T16:48:03.237008+0000","last_clean":"2024-09-05T16:48:03.237008+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:48:03.237008+0000","last_undegraded":"2024-09-05T16:48:03.237008+0000","last_fullsized":"2024-09-05T16:48:03.237008+0000","mapping_epoch":38,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1865,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12752","reported_seq":17401,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-05T16:48:03.238089+0000","last_change":"2024-09-05T16:46:51.992073+0000","last_active":"2024-09-05T16:48:03.238089+0000","last_peered":"2024-09-05T16:48:03.238089+0000","last_clean":"2024-09-05T16:48:03.238089+0000","last_became_active":"2024-09-05T16:46:51.667128+0000","last_became_peered":"2024-09-05T16:46:51.667128+0000","last_unstale":"2024-09-05T16:48:03.238089+0000","last_undegraded":"2024-09-05T16:48:03.238089+0000","last_fullsized":"2024-09-05T16:48:03.238089+0000","mapping_epoch":44,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12752","last_scrub_stamp":"2024-09-05T16:46:51.991995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:51.991995+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:04:54.120031+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12934","reported_seq":17706,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-05T16:48:03.237150+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:48:03.237150+0000","last_peered":"2024-09-05T16:48:03.237150+0000","last_clean":"2024-09-05T16:48:03.237150+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:48:03.237150+0000","last_undegraded":"2024-09-05T16:48:03.237150+0000","last_fullsized":"2024-09-05T16:48:03.237150+0000","mapping_epoch":34,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7234,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12593","reported_seq":17046,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-05T16:48:03.235245+0000","last_change":"2024-09-05T16:47:51.525026+0000","last_active":"2024-09-05T16:48:03.235245+0000","last_peered":"2024-09-05T16:48:03.235245+0000","last_clean":"2024-09-05T16:48:03.235245+0000","last_became_active":"2024-09-05T16:47:51.193016+0000","last_became_peered":"2024-09-05T16:47:51.193016+0000","last_unstale":"2024-09-05T16:48:03.235245+0000","last_undegraded":"2024-09-05T16:48:03.235245+0000","last_fullsized":"2024-09-05T16:48:03.235245+0000","mapping_epoch":54,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12593","last_scrub_stamp":"2024-09-05T16:47:51.524957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:51.524957+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:46:22.846859+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12772","reported_seq":17377,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-05T16:48:03.235526+0000","last_change":"2024-09-05T16:47:15.832530+0000","last_active":"2024-09-05T16:48:03.235526+0000","last_peered":"2024-09-05T16:48:03.235526+0000","last_clean":"2024-09-05T16:48:03.235526+0000","last_became_active":"2024-09-05T16:47:15.728759+0000","last_became_peered":"2024-09-05T16:47:15.728759+0000","last_unstale":"2024-09-05T16:48:03.235526+0000","last_undegraded":"2024-09-05T16:48:03.235526+0000","last_fullsized":"2024-09-05T16:48:03.235526+0000","mapping_epoch":48,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T02:39:11.515339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12772,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2250,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12660","reported_seq":17094,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-05T16:48:03.235802+0000","last_change":"2024-09-05T16:47:03.683348+0000","last_active":"2024-09-05T16:48:03.235802+0000","last_peered":"2024-09-05T16:48:03.235802+0000","last_clean":"2024-09-05T16:48:03.235802+0000","last_became_active":"2024-09-05T16:47:03.185143+0000","last_became_peered":"2024-09-05T16:47:03.185143+0000","last_unstale":"2024-09-05T16:48:03.235802+0000","last_undegraded":"2024-09-05T16:48:03.235802+0000","last_fullsized":"2024-09-05T16:48:03.235802+0000","mapping_epoch":46,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12660","last_scrub_stamp":"2024-09-05T16:47:03.683262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:03.683262+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:42:16.533299+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12777","reported_seq":12922,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-05T16:48:03.236439+0000","last_change":"2024-09-05T16:46:40.458315+0000","last_active":"2024-09-05T16:48:03.236439+0000","last_peered":"2024-09-05T16:48:03.236439+0000","last_clean":"2024-09-05T16:48:03.236439+0000","last_became_active":"2024-09-05T16:46:40.184288+0000","last_became_peered":"2024-09-05T16:46:40.184288+0000","last_unstale":"2024-09-05T16:48:03.236439+0000","last_undegraded":"2024-09-05T16:48:03.236439+0000","last_fullsized":"2024-09-05T16:48:03.236439+0000","mapping_epoch":42,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12777","last_scrub_stamp":"2024-09-05T16:46:40.458226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:40.458226+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:38:58.734210+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'12770","reported_seq":12913,"reported_epoch":57,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:48:03.235498+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:48:03.235498+0000","last_peered":"2024-09-05T16:48:03.235498+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:48:03.235498+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12767,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2240}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12808","reported_seq":17484,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-05T16:48:04.288466+0000","last_change":"2024-09-05T16:48:04.288466+0000","last_active":"2024-09-05T16:48:04.288466+0000","last_peered":"2024-09-05T16:48:04.288466+0000","last_clean":"2024-09-05T16:48:04.288466+0000","last_became_active":"2024-09-05T16:48:03.239894+0000","last_became_peered":"2024-09-05T16:48:03.239894+0000","last_unstale":"2024-09-05T16:48:04.288466+0000","last_undegraded":"2024-09-05T16:48:04.288466+0000","last_fullsized":"2024-09-05T16:48:04.288466+0000","mapping_epoch":56,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12808","last_scrub_stamp":"2024-09-05T16:48:04.288388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:48:04.288388+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T21:11:32.044919+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13120","reported_seq":13289,"reported_epoch":57,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:48:03.236782+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:48:03.236782+0000","last_peered":"2024-09-05T16:48:03.236782+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:48:03.236782+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13117,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2278}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12758","reported_seq":12899,"reported_epoch":57,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:48:03.237812+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:48:03.237812+0000","last_peered":"2024-09-05T16:48:03.237812+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:48:03.237812+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2237}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12808","reported_seq":17586,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-05T16:48:03.236381+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:48:03.236381+0000","last_peered":"2024-09-05T16:48:03.236381+0000","last_clean":"2024-09-05T16:48:03.236381+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:48:03.236381+0000","last_undegraded":"2024-09-05T16:48:03.236381+0000","last_fullsized":"2024-09-05T16:48:03.236381+0000","mapping_epoch":36,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4408,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13124","reported_seq":15240,"reported_epoch":57,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:48:08.208424+0000","last_change":"2024-09-05T16:48:02.084558+0000","last_active":"2024-09-05T16:48:08.208424+0000","last_peered":"2024-09-05T16:48:08.208424+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:48:08.208424+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1354,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13122,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1007,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1352}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12831","reported_seq":12980,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-05T16:48:03.236429+0000","last_change":"2024-09-05T16:47:40.451380+0000","last_active":"2024-09-05T16:48:03.236429+0000","last_peered":"2024-09-05T16:48:03.236429+0000","last_clean":"2024-09-05T16:48:03.236429+0000","last_became_active":"2024-09-05T16:47:40.189447+0000","last_became_peered":"2024-09-05T16:47:40.189447+0000","last_unstale":"2024-09-05T16:48:03.236429+0000","last_undegraded":"2024-09-05T16:48:03.236429+0000","last_fullsized":"2024-09-05T16:48:03.236429+0000","mapping_epoch":52,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12831","last_scrub_stamp":"2024-09-05T16:47:40.451295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:40.451295+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:46:20.946333+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12773","reported_seq":12922,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-05T16:48:03.236707+0000","last_change":"2024-09-05T16:47:27.437275+0000","last_active":"2024-09-05T16:48:03.236707+0000","last_peered":"2024-09-05T16:48:03.236707+0000","last_clean":"2024-09-05T16:48:03.236707+0000","last_became_active":"2024-09-05T16:47:27.204989+0000","last_became_peered":"2024-09-05T16:47:27.204989+0000","last_unstale":"2024-09-05T16:48:03.236707+0000","last_undegraded":"2024-09-05T16:48:03.236707+0000","last_fullsized":"2024-09-05T16:48:03.236707+0000","mapping_epoch":50,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12773","last_scrub_stamp":"2024-09-05T16:47:27.437187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:27.437187+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T16:53:55.961513+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":67360,"num_objects":10,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8124,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":90946,"num_write_kb":315952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5599,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":45056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084457,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722156,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649684,"statfs":{"total":96636764160,"available":95897276416,"internally_reserved":0,"allocated":479232,"data_stored":181558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":738979539},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986335,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":622208,"kb_used_data":496,"kb_used_omap":4,"kb_used_meta":621691,"kb_avail":93749632,"statfs":{"total":96636764160,"available":95999623168,"internally_reserved":0,"allocated":507904,"data_stored":210230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":636612306},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673141,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713320,"kb_used_data":468,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":93658520,"statfs":{"total":96636764160,"available":95906324480,"internally_reserved":0,"allocated":479232,"data_stored":181558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":729938373},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182572,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713772,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658068,"statfs":{"total":96636764160,"available":95905861632,"internally_reserved":0,"allocated":479232,"data_stored":181558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":730394317},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:48:16.822 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:48:17.146 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:48:17.147 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:48:17.159 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":500,"stamp":"2024-09-05T16:48:17.116137+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":67360,"num_objects":10,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7153,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":90946,"num_write_kb":315952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6570,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2771456,"kb_used_data":1900,"kb_used_omap":14,"kb_used_meta":2769457,"kb_avail":374715904,"statfs":{"total":386547056640,"available":383709085696,"internally_reserved":0,"allocated":1945600,"data_stored":754904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2835924535},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-971,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":971,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001289"},"pg_stats":[{"pgid":"1.f","version":"39'12904","reported_seq":17533,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-05T16:48:03.237710+0000","last_change":"2024-09-05T16:46:27.826222+0000","last_active":"2024-09-05T16:48:03.237710+0000","last_peered":"2024-09-05T16:48:03.237710+0000","last_clean":"2024-09-05T16:48:03.237710+0000","last_became_active":"2024-09-05T16:46:27.757928+0000","last_became_peered":"2024-09-05T16:46:27.757928+0000","last_unstale":"2024-09-05T16:48:03.237710+0000","last_undegraded":"2024-09-05T16:48:03.237710+0000","last_fullsized":"2024-09-05T16:48:03.237710+0000","mapping_epoch":40,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:32:35.713428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12904,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2262,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12552","reported_seq":17245,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-05T16:48:03.237008+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:48:03.237008+0000","last_peered":"2024-09-05T16:48:03.237008+0000","last_clean":"2024-09-05T16:48:03.237008+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:48:03.237008+0000","last_undegraded":"2024-09-05T16:48:03.237008+0000","last_fullsized":"2024-09-05T16:48:03.237008+0000","mapping_epoch":38,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1865,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12752","reported_seq":17403,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-05T16:48:15.435901+0000","last_change":"2024-09-05T16:46:51.992073+0000","last_active":"2024-09-05T16:48:15.435901+0000","last_peered":"2024-09-05T16:48:15.435901+0000","last_clean":"2024-09-05T16:48:15.435901+0000","last_became_active":"2024-09-05T16:46:51.667128+0000","last_became_peered":"2024-09-05T16:46:51.667128+0000","last_unstale":"2024-09-05T16:48:15.435901+0000","last_undegraded":"2024-09-05T16:48:15.435901+0000","last_fullsized":"2024-09-05T16:48:15.435901+0000","mapping_epoch":44,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12752","last_scrub_stamp":"2024-09-05T16:46:51.991995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:51.991995+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:04:54.120031+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12934","reported_seq":17706,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-05T16:48:03.237150+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:48:03.237150+0000","last_peered":"2024-09-05T16:48:03.237150+0000","last_clean":"2024-09-05T16:48:03.237150+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:48:03.237150+0000","last_undegraded":"2024-09-05T16:48:03.237150+0000","last_fullsized":"2024-09-05T16:48:03.237150+0000","mapping_epoch":34,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7234,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12593","reported_seq":17046,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-05T16:48:03.235245+0000","last_change":"2024-09-05T16:47:51.525026+0000","last_active":"2024-09-05T16:48:03.235245+0000","last_peered":"2024-09-05T16:48:03.235245+0000","last_clean":"2024-09-05T16:48:03.235245+0000","last_became_active":"2024-09-05T16:47:51.193016+0000","last_became_peered":"2024-09-05T16:47:51.193016+0000","last_unstale":"2024-09-05T16:48:03.235245+0000","last_undegraded":"2024-09-05T16:48:03.235245+0000","last_fullsized":"2024-09-05T16:48:03.235245+0000","mapping_epoch":54,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12593","last_scrub_stamp":"2024-09-05T16:47:51.524957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:51.524957+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:46:22.846859+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12772","reported_seq":17377,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-05T16:48:03.235526+0000","last_change":"2024-09-05T16:47:15.832530+0000","last_active":"2024-09-05T16:48:03.235526+0000","last_peered":"2024-09-05T16:48:03.235526+0000","last_clean":"2024-09-05T16:48:03.235526+0000","last_became_active":"2024-09-05T16:47:15.728759+0000","last_became_peered":"2024-09-05T16:47:15.728759+0000","last_unstale":"2024-09-05T16:48:03.235526+0000","last_undegraded":"2024-09-05T16:48:03.235526+0000","last_fullsized":"2024-09-05T16:48:03.235526+0000","mapping_epoch":48,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T02:39:11.515339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12772,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2250,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12660","reported_seq":17096,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-05T16:48:15.430442+0000","last_change":"2024-09-05T16:47:03.683348+0000","last_active":"2024-09-05T16:48:15.430442+0000","last_peered":"2024-09-05T16:48:15.430442+0000","last_clean":"2024-09-05T16:48:15.430442+0000","last_became_active":"2024-09-05T16:47:03.185143+0000","last_became_peered":"2024-09-05T16:47:03.185143+0000","last_unstale":"2024-09-05T16:48:15.430442+0000","last_undegraded":"2024-09-05T16:48:15.430442+0000","last_fullsized":"2024-09-05T16:48:15.430442+0000","mapping_epoch":46,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12660","last_scrub_stamp":"2024-09-05T16:47:03.683262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:03.683262+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:42:16.533299+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12777","reported_seq":12922,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-05T16:48:03.236439+0000","last_change":"2024-09-05T16:46:40.458315+0000","last_active":"2024-09-05T16:48:03.236439+0000","last_peered":"2024-09-05T16:48:03.236439+0000","last_clean":"2024-09-05T16:48:03.236439+0000","last_became_active":"2024-09-05T16:46:40.184288+0000","last_became_peered":"2024-09-05T16:46:40.184288+0000","last_unstale":"2024-09-05T16:48:03.236439+0000","last_undegraded":"2024-09-05T16:48:03.236439+0000","last_fullsized":"2024-09-05T16:48:03.236439+0000","mapping_epoch":42,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12777","last_scrub_stamp":"2024-09-05T16:46:40.458226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:40.458226+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:38:58.734210+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'12770","reported_seq":12913,"reported_epoch":57,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:48:03.235498+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:48:03.235498+0000","last_peered":"2024-09-05T16:48:03.235498+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:48:03.235498+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12767,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2240}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12808","reported_seq":17486,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-05T16:48:15.435905+0000","last_change":"2024-09-05T16:48:04.288466+0000","last_active":"2024-09-05T16:48:15.435905+0000","last_peered":"2024-09-05T16:48:15.435905+0000","last_clean":"2024-09-05T16:48:15.435905+0000","last_became_active":"2024-09-05T16:48:03.239894+0000","last_became_peered":"2024-09-05T16:48:03.239894+0000","last_unstale":"2024-09-05T16:48:15.435905+0000","last_undegraded":"2024-09-05T16:48:15.435905+0000","last_fullsized":"2024-09-05T16:48:15.435905+0000","mapping_epoch":56,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12808","last_scrub_stamp":"2024-09-05T16:48:04.288388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:48:04.288388+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T21:11:32.044919+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13120","reported_seq":13289,"reported_epoch":57,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:48:03.236782+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:48:03.236782+0000","last_peered":"2024-09-05T16:48:03.236782+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:48:03.236782+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13117,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2278}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12758","reported_seq":12899,"reported_epoch":57,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:48:03.237812+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:48:03.237812+0000","last_peered":"2024-09-05T16:48:03.237812+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:48:03.237812+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2237}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12808","reported_seq":17586,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-05T16:48:03.236381+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:48:03.236381+0000","last_peered":"2024-09-05T16:48:03.236381+0000","last_clean":"2024-09-05T16:48:03.236381+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:48:03.236381+0000","last_undegraded":"2024-09-05T16:48:03.236381+0000","last_fullsized":"2024-09-05T16:48:03.236381+0000","mapping_epoch":36,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4408,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13124","reported_seq":17182,"reported_epoch":57,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:48:13.209122+0000","last_change":"2024-09-05T16:48:02.084558+0000","last_active":"2024-09-05T16:48:13.209122+0000","last_peered":"2024-09-05T16:48:13.209122+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:48:13.209122+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":383,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13122,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1978,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-381}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12831","reported_seq":12980,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-05T16:48:03.236429+0000","last_change":"2024-09-05T16:47:40.451380+0000","last_active":"2024-09-05T16:48:03.236429+0000","last_peered":"2024-09-05T16:48:03.236429+0000","last_clean":"2024-09-05T16:48:03.236429+0000","last_became_active":"2024-09-05T16:47:40.189447+0000","last_became_peered":"2024-09-05T16:47:40.189447+0000","last_unstale":"2024-09-05T16:48:03.236429+0000","last_undegraded":"2024-09-05T16:48:03.236429+0000","last_fullsized":"2024-09-05T16:48:03.236429+0000","mapping_epoch":52,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12831","last_scrub_stamp":"2024-09-05T16:47:40.451295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:40.451295+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:46:20.946333+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12773","reported_seq":12922,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-05T16:48:03.236707+0000","last_change":"2024-09-05T16:47:27.437275+0000","last_active":"2024-09-05T16:48:03.236707+0000","last_peered":"2024-09-05T16:48:03.236707+0000","last_clean":"2024-09-05T16:48:03.236707+0000","last_became_active":"2024-09-05T16:47:27.204989+0000","last_became_peered":"2024-09-05T16:47:27.204989+0000","last_unstale":"2024-09-05T16:48:03.236707+0000","last_undegraded":"2024-09-05T16:48:03.236707+0000","last_fullsized":"2024-09-05T16:48:03.236707+0000","mapping_epoch":50,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12773","last_scrub_stamp":"2024-09-05T16:47:27.437187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:27.437187+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T16:53:55.961513+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":67360,"num_objects":10,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7153,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":90946,"num_write_kb":315952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6570,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":45056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084458,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722156,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649684,"statfs":{"total":96636764160,"available":95897276416,"internally_reserved":0,"allocated":479232,"data_stored":181558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":738979539},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986335,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":622208,"kb_used_data":496,"kb_used_omap":4,"kb_used_meta":621691,"kb_avail":93749632,"statfs":{"total":96636764160,"available":95999623168,"internally_reserved":0,"allocated":507904,"data_stored":210230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":636612306},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673142,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713320,"kb_used_data":468,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":93658520,"statfs":{"total":96636764160,"available":95906324480,"internally_reserved":0,"allocated":479232,"data_stored":181558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":729938373},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182573,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713772,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658068,"statfs":{"total":96636764160,"available":95905861632,"internally_reserved":0,"allocated":479232,"data_stored":181558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":730394317},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:48:17.160 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:48:17.459 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:48:17.459 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:48:17.471 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":500,"stamp":"2024-09-05T16:48:17.116137+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":67360,"num_objects":10,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7153,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":90946,"num_write_kb":315952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6570,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2771456,"kb_used_data":1900,"kb_used_omap":14,"kb_used_meta":2769457,"kb_avail":374715904,"statfs":{"total":386547056640,"available":383709085696,"internally_reserved":0,"allocated":1945600,"data_stored":754904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2835924535},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-971,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":971,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001289"},"pg_stats":[{"pgid":"1.f","version":"39'12904","reported_seq":17533,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-05T16:48:03.237710+0000","last_change":"2024-09-05T16:46:27.826222+0000","last_active":"2024-09-05T16:48:03.237710+0000","last_peered":"2024-09-05T16:48:03.237710+0000","last_clean":"2024-09-05T16:48:03.237710+0000","last_became_active":"2024-09-05T16:46:27.757928+0000","last_became_peered":"2024-09-05T16:46:27.757928+0000","last_unstale":"2024-09-05T16:48:03.237710+0000","last_undegraded":"2024-09-05T16:48:03.237710+0000","last_fullsized":"2024-09-05T16:48:03.237710+0000","mapping_epoch":40,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:32:35.713428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12904,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2262,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12552","reported_seq":17245,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-05T16:48:03.237008+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:48:03.237008+0000","last_peered":"2024-09-05T16:48:03.237008+0000","last_clean":"2024-09-05T16:48:03.237008+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:48:03.237008+0000","last_undegraded":"2024-09-05T16:48:03.237008+0000","last_fullsized":"2024-09-05T16:48:03.237008+0000","mapping_epoch":38,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1865,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12752","reported_seq":17403,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-05T16:48:15.435901+0000","last_change":"2024-09-05T16:46:51.992073+0000","last_active":"2024-09-05T16:48:15.435901+0000","last_peered":"2024-09-05T16:48:15.435901+0000","last_clean":"2024-09-05T16:48:15.435901+0000","last_became_active":"2024-09-05T16:46:51.667128+0000","last_became_peered":"2024-09-05T16:46:51.667128+0000","last_unstale":"2024-09-05T16:48:15.435901+0000","last_undegraded":"2024-09-05T16:48:15.435901+0000","last_fullsized":"2024-09-05T16:48:15.435901+0000","mapping_epoch":44,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12752","last_scrub_stamp":"2024-09-05T16:46:51.991995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:51.991995+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:04:54.120031+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12934","reported_seq":17706,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-05T16:48:03.237150+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:48:03.237150+0000","last_peered":"2024-09-05T16:48:03.237150+0000","last_clean":"2024-09-05T16:48:03.237150+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:48:03.237150+0000","last_undegraded":"2024-09-05T16:48:03.237150+0000","last_fullsized":"2024-09-05T16:48:03.237150+0000","mapping_epoch":34,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7234,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12593","reported_seq":17046,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-05T16:48:03.235245+0000","last_change":"2024-09-05T16:47:51.525026+0000","last_active":"2024-09-05T16:48:03.235245+0000","last_peered":"2024-09-05T16:48:03.235245+0000","last_clean":"2024-09-05T16:48:03.235245+0000","last_became_active":"2024-09-05T16:47:51.193016+0000","last_became_peered":"2024-09-05T16:47:51.193016+0000","last_unstale":"2024-09-05T16:48:03.235245+0000","last_undegraded":"2024-09-05T16:48:03.235245+0000","last_fullsized":"2024-09-05T16:48:03.235245+0000","mapping_epoch":54,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12593","last_scrub_stamp":"2024-09-05T16:47:51.524957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:51.524957+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:46:22.846859+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12772","reported_seq":17377,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-05T16:48:03.235526+0000","last_change":"2024-09-05T16:47:15.832530+0000","last_active":"2024-09-05T16:48:03.235526+0000","last_peered":"2024-09-05T16:48:03.235526+0000","last_clean":"2024-09-05T16:48:03.235526+0000","last_became_active":"2024-09-05T16:47:15.728759+0000","last_became_peered":"2024-09-05T16:47:15.728759+0000","last_unstale":"2024-09-05T16:48:03.235526+0000","last_undegraded":"2024-09-05T16:48:03.235526+0000","last_fullsized":"2024-09-05T16:48:03.235526+0000","mapping_epoch":48,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T02:39:11.515339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12772,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2250,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12660","reported_seq":17096,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-05T16:48:15.430442+0000","last_change":"2024-09-05T16:47:03.683348+0000","last_active":"2024-09-05T16:48:15.430442+0000","last_peered":"2024-09-05T16:48:15.430442+0000","last_clean":"2024-09-05T16:48:15.430442+0000","last_became_active":"2024-09-05T16:47:03.185143+0000","last_became_peered":"2024-09-05T16:47:03.185143+0000","last_unstale":"2024-09-05T16:48:15.430442+0000","last_undegraded":"2024-09-05T16:48:15.430442+0000","last_fullsized":"2024-09-05T16:48:15.430442+0000","mapping_epoch":46,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12660","last_scrub_stamp":"2024-09-05T16:47:03.683262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:03.683262+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:42:16.533299+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12777","reported_seq":12922,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-05T16:48:03.236439+0000","last_change":"2024-09-05T16:46:40.458315+0000","last_active":"2024-09-05T16:48:03.236439+0000","last_peered":"2024-09-05T16:48:03.236439+0000","last_clean":"2024-09-05T16:48:03.236439+0000","last_became_active":"2024-09-05T16:46:40.184288+0000","last_became_peered":"2024-09-05T16:46:40.184288+0000","last_unstale":"2024-09-05T16:48:03.236439+0000","last_undegraded":"2024-09-05T16:48:03.236439+0000","last_fullsized":"2024-09-05T16:48:03.236439+0000","mapping_epoch":42,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12777","last_scrub_stamp":"2024-09-05T16:46:40.458226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:40.458226+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:38:58.734210+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'12770","reported_seq":12913,"reported_epoch":57,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:48:03.235498+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:48:03.235498+0000","last_peered":"2024-09-05T16:48:03.235498+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:48:03.235498+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12767,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2240}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12808","reported_seq":17486,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-05T16:48:15.435905+0000","last_change":"2024-09-05T16:48:04.288466+0000","last_active":"2024-09-05T16:48:15.435905+0000","last_peered":"2024-09-05T16:48:15.435905+0000","last_clean":"2024-09-05T16:48:15.435905+0000","last_became_active":"2024-09-05T16:48:03.239894+0000","last_became_peered":"2024-09-05T16:48:03.239894+0000","last_unstale":"2024-09-05T16:48:15.435905+0000","last_undegraded":"2024-09-05T16:48:15.435905+0000","last_fullsized":"2024-09-05T16:48:15.435905+0000","mapping_epoch":56,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12808","last_scrub_stamp":"2024-09-05T16:48:04.288388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:48:04.288388+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T21:11:32.044919+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13120","reported_seq":13289,"reported_epoch":57,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:48:03.236782+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:48:03.236782+0000","last_peered":"2024-09-05T16:48:03.236782+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:48:03.236782+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13117,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2278}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12758","reported_seq":12899,"reported_epoch":57,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:48:03.237812+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:48:03.237812+0000","last_peered":"2024-09-05T16:48:03.237812+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:48:03.237812+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2237}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12808","reported_seq":17586,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-05T16:48:03.236381+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:48:03.236381+0000","last_peered":"2024-09-05T16:48:03.236381+0000","last_clean":"2024-09-05T16:48:03.236381+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:48:03.236381+0000","last_undegraded":"2024-09-05T16:48:03.236381+0000","last_fullsized":"2024-09-05T16:48:03.236381+0000","mapping_epoch":36,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4408,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13124","reported_seq":17182,"reported_epoch":57,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:48:13.209122+0000","last_change":"2024-09-05T16:48:02.084558+0000","last_active":"2024-09-05T16:48:13.209122+0000","last_peered":"2024-09-05T16:48:13.209122+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:48:13.209122+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":383,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13122,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1978,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-381}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12831","reported_seq":12980,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-05T16:48:03.236429+0000","last_change":"2024-09-05T16:47:40.451380+0000","last_active":"2024-09-05T16:48:03.236429+0000","last_peered":"2024-09-05T16:48:03.236429+0000","last_clean":"2024-09-05T16:48:03.236429+0000","last_became_active":"2024-09-05T16:47:40.189447+0000","last_became_peered":"2024-09-05T16:47:40.189447+0000","last_unstale":"2024-09-05T16:48:03.236429+0000","last_undegraded":"2024-09-05T16:48:03.236429+0000","last_fullsized":"2024-09-05T16:48:03.236429+0000","mapping_epoch":52,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12831","last_scrub_stamp":"2024-09-05T16:47:40.451295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:40.451295+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:46:20.946333+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12773","reported_seq":12922,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-05T16:48:03.236707+0000","last_change":"2024-09-05T16:47:27.437275+0000","last_active":"2024-09-05T16:48:03.236707+0000","last_peered":"2024-09-05T16:48:03.236707+0000","last_clean":"2024-09-05T16:48:03.236707+0000","last_became_active":"2024-09-05T16:47:27.204989+0000","last_became_peered":"2024-09-05T16:47:27.204989+0000","last_unstale":"2024-09-05T16:48:03.236707+0000","last_undegraded":"2024-09-05T16:48:03.236707+0000","last_fullsized":"2024-09-05T16:48:03.236707+0000","mapping_epoch":50,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12773","last_scrub_stamp":"2024-09-05T16:47:27.437187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:27.437187+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T16:53:55.961513+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":67360,"num_objects":10,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7153,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":90946,"num_write_kb":315952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6570,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":45056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084458,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722156,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649684,"statfs":{"total":96636764160,"available":95897276416,"internally_reserved":0,"allocated":479232,"data_stored":181558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":738979539},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986335,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":622208,"kb_used_data":496,"kb_used_omap":4,"kb_used_meta":621691,"kb_avail":93749632,"statfs":{"total":96636764160,"available":95999623168,"internally_reserved":0,"allocated":507904,"data_stored":210230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":636612306},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673142,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713320,"kb_used_data":468,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":93658520,"statfs":{"total":96636764160,"available":95906324480,"internally_reserved":0,"allocated":479232,"data_stored":181558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":729938373},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182573,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713772,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658068,"statfs":{"total":96636764160,"available":95905861632,"internally_reserved":0,"allocated":479232,"data_stored":181558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":730394317},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:48:17.472 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:48:17.472 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "39'12770", 'reported_seq': 12913, 'reported_epoch': 57, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:48:03.235498+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:48:03.235498+0000', 'last_peered': '2024-09-05T16:48:03.235498+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:48:03.235498+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "33'2700", 'ondisk_log_start': "33'2700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10070, 'log_dups_size': 0, 'ondisk_log_size': 10070, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26944, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 16, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12767, 'num_write_kb': 51081, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2240}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:48:17.472 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:48:17.473 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "39'13120", 'reported_seq': 13289, 'reported_epoch': 57, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:48:03.236782+0000', 'last_change': '2024-09-05T16:34:56.966897+0000', 'last_active': '2024-09-05T16:48:03.236782+0000', 'last_peered': '2024-09-05T16:48:03.236782+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:48:03.236782+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "33'3100", 'ondisk_log_start': "33'3100", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20208, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2284, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13117, 'num_write_kb': 52476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2278}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:48:17.473 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:48:17.473 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "39'12758", 'reported_seq': 12899, 'reported_epoch': 57, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:48:03.237812+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:48:03.237812+0000', 'last_peered': '2024-09-05T16:48:03.237812+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:48:03.237812+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "33'2700", 'ondisk_log_start': "33'2700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10058, 'log_dups_size': 0, 'ondisk_log_size': 10058, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6736, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12757, 'num_write_kb': 51031, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2237}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:48:17.473 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-05T16:48:17.473 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "39'13124", 'reported_seq': 17182, 'reported_epoch': 57, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:48:13.209122+0000', 'last_change': '2024-09-05T16:48:02.084558+0000', 'last_active': '2024-09-05T16:48:13.209122+0000', 'last_peered': '2024-09-05T16:48:13.209122+0000', 'last_clean': '2024-09-05T16:33:53.045479+0000', 'last_became_active': '2024-09-05T16:34:56.901469+0000', 'last_became_peered': '2024-09-05T16:34:56.901469+0000', 'last_unstale': '2024-09-05T16:48:13.209122+0000', 'last_undegraded': '2024-09-05T16:34:56.626055+0000', 'last_fullsized': '2024-09-05T16:34:56.556784+0000', 'mapping_epoch': 32, 'log_start': "33'3100", 'ondisk_log_start': "33'3100", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10024, 'log_dups_size': 0, 'ondisk_log_size': 10024, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 383, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13122, 'num_write_kb': 52500, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1978, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -381}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:48:17.473 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:48:17.855 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:48:17.855 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 7153/40 objects degraded (17882.500%), 4 pgs degraded, 4 pgs undersized","count":8},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":901,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":59,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":10,"data_bytes":67360,"bytes_used":2837970944,"bytes_avail":383709085696,"bytes_total":386547056640,"degraded_objects":7153,"degraded_total":40,"degraded_ratio":178.82499999999999,"recovering_objects_per_sec":121,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":971,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:48:17.869 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:48:17.870 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:48:18.174 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:48:18.175 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:48:18.186 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":500,"stamp":"2024-09-05T16:48:17.116137+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":67360,"num_objects":10,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7153,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":90946,"num_write_kb":315952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6570,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2771456,"kb_used_data":1900,"kb_used_omap":14,"kb_used_meta":2769457,"kb_avail":374715904,"statfs":{"total":386547056640,"available":383709085696,"internally_reserved":0,"allocated":1945600,"data_stored":754904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2835924535},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-971,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":971,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001289"},"pg_stats":[{"pgid":"1.f","version":"39'12904","reported_seq":17533,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-05T16:48:03.237710+0000","last_change":"2024-09-05T16:46:27.826222+0000","last_active":"2024-09-05T16:48:03.237710+0000","last_peered":"2024-09-05T16:48:03.237710+0000","last_clean":"2024-09-05T16:48:03.237710+0000","last_became_active":"2024-09-05T16:46:27.757928+0000","last_became_peered":"2024-09-05T16:46:27.757928+0000","last_unstale":"2024-09-05T16:48:03.237710+0000","last_undegraded":"2024-09-05T16:48:03.237710+0000","last_fullsized":"2024-09-05T16:48:03.237710+0000","mapping_epoch":40,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:32:35.713428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12904,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2262,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12552","reported_seq":17245,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-05T16:48:03.237008+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:48:03.237008+0000","last_peered":"2024-09-05T16:48:03.237008+0000","last_clean":"2024-09-05T16:48:03.237008+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:48:03.237008+0000","last_undegraded":"2024-09-05T16:48:03.237008+0000","last_fullsized":"2024-09-05T16:48:03.237008+0000","mapping_epoch":38,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1865,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12752","reported_seq":17403,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-05T16:48:15.435901+0000","last_change":"2024-09-05T16:46:51.992073+0000","last_active":"2024-09-05T16:48:15.435901+0000","last_peered":"2024-09-05T16:48:15.435901+0000","last_clean":"2024-09-05T16:48:15.435901+0000","last_became_active":"2024-09-05T16:46:51.667128+0000","last_became_peered":"2024-09-05T16:46:51.667128+0000","last_unstale":"2024-09-05T16:48:15.435901+0000","last_undegraded":"2024-09-05T16:48:15.435901+0000","last_fullsized":"2024-09-05T16:48:15.435901+0000","mapping_epoch":44,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12752","last_scrub_stamp":"2024-09-05T16:46:51.991995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:51.991995+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:04:54.120031+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12934","reported_seq":17706,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-05T16:48:03.237150+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:48:03.237150+0000","last_peered":"2024-09-05T16:48:03.237150+0000","last_clean":"2024-09-05T16:48:03.237150+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:48:03.237150+0000","last_undegraded":"2024-09-05T16:48:03.237150+0000","last_fullsized":"2024-09-05T16:48:03.237150+0000","mapping_epoch":34,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7234,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12593","reported_seq":17046,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-05T16:48:03.235245+0000","last_change":"2024-09-05T16:47:51.525026+0000","last_active":"2024-09-05T16:48:03.235245+0000","last_peered":"2024-09-05T16:48:03.235245+0000","last_clean":"2024-09-05T16:48:03.235245+0000","last_became_active":"2024-09-05T16:47:51.193016+0000","last_became_peered":"2024-09-05T16:47:51.193016+0000","last_unstale":"2024-09-05T16:48:03.235245+0000","last_undegraded":"2024-09-05T16:48:03.235245+0000","last_fullsized":"2024-09-05T16:48:03.235245+0000","mapping_epoch":54,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12593","last_scrub_stamp":"2024-09-05T16:47:51.524957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:51.524957+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:46:22.846859+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12772","reported_seq":17377,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-05T16:48:03.235526+0000","last_change":"2024-09-05T16:47:15.832530+0000","last_active":"2024-09-05T16:48:03.235526+0000","last_peered":"2024-09-05T16:48:03.235526+0000","last_clean":"2024-09-05T16:48:03.235526+0000","last_became_active":"2024-09-05T16:47:15.728759+0000","last_became_peered":"2024-09-05T16:47:15.728759+0000","last_unstale":"2024-09-05T16:48:03.235526+0000","last_undegraded":"2024-09-05T16:48:03.235526+0000","last_fullsized":"2024-09-05T16:48:03.235526+0000","mapping_epoch":48,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T02:39:11.515339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12772,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2250,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12660","reported_seq":17096,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-05T16:48:15.430442+0000","last_change":"2024-09-05T16:47:03.683348+0000","last_active":"2024-09-05T16:48:15.430442+0000","last_peered":"2024-09-05T16:48:15.430442+0000","last_clean":"2024-09-05T16:48:15.430442+0000","last_became_active":"2024-09-05T16:47:03.185143+0000","last_became_peered":"2024-09-05T16:47:03.185143+0000","last_unstale":"2024-09-05T16:48:15.430442+0000","last_undegraded":"2024-09-05T16:48:15.430442+0000","last_fullsized":"2024-09-05T16:48:15.430442+0000","mapping_epoch":46,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12660","last_scrub_stamp":"2024-09-05T16:47:03.683262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:03.683262+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:42:16.533299+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12777","reported_seq":12922,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-05T16:48:03.236439+0000","last_change":"2024-09-05T16:46:40.458315+0000","last_active":"2024-09-05T16:48:03.236439+0000","last_peered":"2024-09-05T16:48:03.236439+0000","last_clean":"2024-09-05T16:48:03.236439+0000","last_became_active":"2024-09-05T16:46:40.184288+0000","last_became_peered":"2024-09-05T16:46:40.184288+0000","last_unstale":"2024-09-05T16:48:03.236439+0000","last_undegraded":"2024-09-05T16:48:03.236439+0000","last_fullsized":"2024-09-05T16:48:03.236439+0000","mapping_epoch":42,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12777","last_scrub_stamp":"2024-09-05T16:46:40.458226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:40.458226+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:38:58.734210+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'12770","reported_seq":12913,"reported_epoch":57,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:48:03.235498+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:48:03.235498+0000","last_peered":"2024-09-05T16:48:03.235498+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:48:03.235498+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12767,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2240}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12808","reported_seq":17486,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-05T16:48:15.435905+0000","last_change":"2024-09-05T16:48:04.288466+0000","last_active":"2024-09-05T16:48:15.435905+0000","last_peered":"2024-09-05T16:48:15.435905+0000","last_clean":"2024-09-05T16:48:15.435905+0000","last_became_active":"2024-09-05T16:48:03.239894+0000","last_became_peered":"2024-09-05T16:48:03.239894+0000","last_unstale":"2024-09-05T16:48:15.435905+0000","last_undegraded":"2024-09-05T16:48:15.435905+0000","last_fullsized":"2024-09-05T16:48:15.435905+0000","mapping_epoch":56,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12808","last_scrub_stamp":"2024-09-05T16:48:04.288388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:48:04.288388+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T21:11:32.044919+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13120","reported_seq":13289,"reported_epoch":57,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:48:03.236782+0000","last_change":"2024-09-05T16:34:56.966897+0000","last_active":"2024-09-05T16:48:03.236782+0000","last_peered":"2024-09-05T16:48:03.236782+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:48:03.236782+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13117,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2278}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12758","reported_seq":12899,"reported_epoch":57,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:48:03.237812+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:48:03.237812+0000","last_peered":"2024-09-05T16:48:03.237812+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:48:03.237812+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2237}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12808","reported_seq":17586,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-05T16:48:03.236381+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:48:03.236381+0000","last_peered":"2024-09-05T16:48:03.236381+0000","last_clean":"2024-09-05T16:48:03.236381+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:48:03.236381+0000","last_undegraded":"2024-09-05T16:48:03.236381+0000","last_fullsized":"2024-09-05T16:48:03.236381+0000","mapping_epoch":36,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4408,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13124","reported_seq":17182,"reported_epoch":57,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:48:13.209122+0000","last_change":"2024-09-05T16:48:02.084558+0000","last_active":"2024-09-05T16:48:13.209122+0000","last_peered":"2024-09-05T16:48:13.209122+0000","last_clean":"2024-09-05T16:33:53.045479+0000","last_became_active":"2024-09-05T16:34:56.901469+0000","last_became_peered":"2024-09-05T16:34:56.901469+0000","last_unstale":"2024-09-05T16:48:13.209122+0000","last_undegraded":"2024-09-05T16:34:56.626055+0000","last_fullsized":"2024-09-05T16:34:56.556784+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":383,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13122,"num_write_kb":52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1978,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-381}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"39'12831","reported_seq":12980,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-05T16:48:03.236429+0000","last_change":"2024-09-05T16:47:40.451380+0000","last_active":"2024-09-05T16:48:03.236429+0000","last_peered":"2024-09-05T16:48:03.236429+0000","last_clean":"2024-09-05T16:48:03.236429+0000","last_became_active":"2024-09-05T16:47:40.189447+0000","last_became_peered":"2024-09-05T16:47:40.189447+0000","last_unstale":"2024-09-05T16:48:03.236429+0000","last_undegraded":"2024-09-05T16:48:03.236429+0000","last_fullsized":"2024-09-05T16:48:03.236429+0000","mapping_epoch":52,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12831","last_scrub_stamp":"2024-09-05T16:47:40.451295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:40.451295+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:46:20.946333+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12773","reported_seq":12922,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-05T16:48:03.236707+0000","last_change":"2024-09-05T16:47:27.437275+0000","last_active":"2024-09-05T16:48:03.236707+0000","last_peered":"2024-09-05T16:48:03.236707+0000","last_clean":"2024-09-05T16:48:03.236707+0000","last_became_active":"2024-09-05T16:47:27.204989+0000","last_became_peered":"2024-09-05T16:47:27.204989+0000","last_unstale":"2024-09-05T16:48:03.236707+0000","last_undegraded":"2024-09-05T16:48:03.236707+0000","last_fullsized":"2024-09-05T16:48:03.236707+0000","mapping_epoch":50,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12773","last_scrub_stamp":"2024-09-05T16:47:27.437187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:27.437187+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T16:53:55.961513+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":67360,"num_objects":10,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7153,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":90946,"num_write_kb":315952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6570,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":45056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084458,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722156,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649684,"statfs":{"total":96636764160,"available":95897276416,"internally_reserved":0,"allocated":479232,"data_stored":181558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":738979539},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986335,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":622208,"kb_used_data":496,"kb_used_omap":4,"kb_used_meta":621691,"kb_avail":93749632,"statfs":{"total":96636764160,"available":95999623168,"internally_reserved":0,"allocated":507904,"data_stored":210230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":636612306},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673142,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713320,"kb_used_data":468,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":93658520,"statfs":{"total":96636764160,"available":95906324480,"internally_reserved":0,"allocated":479232,"data_stored":181558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":729938373},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182573,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713772,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658068,"statfs":{"total":96636764160,"available":95905861632,"internally_reserved":0,"allocated":479232,"data_stored":181558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":730394317},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:48:21.189 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:48:21.511 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:48:21.512 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:48:21.525 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":502,"stamp":"2024-09-05T16:48:21.116739+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":53888,"num_objects":8,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6307,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":77824,"num_write_kb":263452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5055,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2763176,"kb_used_data":1936,"kb_used_omap":14,"kb_used_meta":2761137,"kb_avail":374724184,"statfs":{"total":386547056640,"available":383717564416,"internally_reserved":0,"allocated":1982464,"data_stored":761627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2827404855},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13472,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1817,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-13122,"num_write_kb":-52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-544,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001267"},"pg_stats":[{"pgid":"1.f","version":"39'12904","reported_seq":17537,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-05T16:48:16.433370+0000","last_change":"2024-09-05T16:46:27.826222+0000","last_active":"2024-09-05T16:48:16.433370+0000","last_peered":"2024-09-05T16:48:16.433370+0000","last_clean":"2024-09-05T16:48:16.433370+0000","last_became_active":"2024-09-05T16:46:27.757928+0000","last_became_peered":"2024-09-05T16:46:27.757928+0000","last_unstale":"2024-09-05T16:48:16.433370+0000","last_undegraded":"2024-09-05T16:48:16.433370+0000","last_fullsized":"2024-09-05T16:48:16.433370+0000","mapping_epoch":40,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:32:35.713428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12904,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2262,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12552","reported_seq":17250,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-05T16:48:16.430268+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:48:16.430268+0000","last_peered":"2024-09-05T16:48:16.430268+0000","last_clean":"2024-09-05T16:48:16.430268+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:48:16.430268+0000","last_undegraded":"2024-09-05T16:48:16.430268+0000","last_fullsized":"2024-09-05T16:48:16.430268+0000","mapping_epoch":38,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1865,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12752","reported_seq":17405,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-05T16:48:16.432786+0000","last_change":"2024-09-05T16:46:51.992073+0000","last_active":"2024-09-05T16:48:16.432786+0000","last_peered":"2024-09-05T16:48:16.432786+0000","last_clean":"2024-09-05T16:48:16.432786+0000","last_became_active":"2024-09-05T16:46:51.667128+0000","last_became_peered":"2024-09-05T16:46:51.667128+0000","last_unstale":"2024-09-05T16:48:16.432786+0000","last_undegraded":"2024-09-05T16:48:16.432786+0000","last_fullsized":"2024-09-05T16:48:16.432786+0000","mapping_epoch":44,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12752","last_scrub_stamp":"2024-09-05T16:46:51.991995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:51.991995+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:04:54.120031+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12934","reported_seq":17710,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-05T16:48:16.429984+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:48:16.429984+0000","last_peered":"2024-09-05T16:48:16.429984+0000","last_clean":"2024-09-05T16:48:16.429984+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:48:16.429984+0000","last_undegraded":"2024-09-05T16:48:16.429984+0000","last_fullsized":"2024-09-05T16:48:16.429984+0000","mapping_epoch":34,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7234,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12593","reported_seq":17050,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-05T16:48:16.432562+0000","last_change":"2024-09-05T16:47:51.525026+0000","last_active":"2024-09-05T16:48:16.432562+0000","last_peered":"2024-09-05T16:48:16.432562+0000","last_clean":"2024-09-05T16:48:16.432562+0000","last_became_active":"2024-09-05T16:47:51.193016+0000","last_became_peered":"2024-09-05T16:47:51.193016+0000","last_unstale":"2024-09-05T16:48:16.432562+0000","last_undegraded":"2024-09-05T16:48:16.432562+0000","last_fullsized":"2024-09-05T16:48:16.432562+0000","mapping_epoch":54,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12593","last_scrub_stamp":"2024-09-05T16:47:51.524957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:51.524957+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:46:22.846859+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12772","reported_seq":17381,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-05T16:48:16.432461+0000","last_change":"2024-09-05T16:47:15.832530+0000","last_active":"2024-09-05T16:48:16.432461+0000","last_peered":"2024-09-05T16:48:16.432461+0000","last_clean":"2024-09-05T16:48:16.432461+0000","last_became_active":"2024-09-05T16:47:15.728759+0000","last_became_peered":"2024-09-05T16:47:15.728759+0000","last_unstale":"2024-09-05T16:48:16.432461+0000","last_undegraded":"2024-09-05T16:48:16.432461+0000","last_fullsized":"2024-09-05T16:48:16.432461+0000","mapping_epoch":48,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T02:39:11.515339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12772,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2250,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12660","reported_seq":17096,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-05T16:48:15.430442+0000","last_change":"2024-09-05T16:47:03.683348+0000","last_active":"2024-09-05T16:48:15.430442+0000","last_peered":"2024-09-05T16:48:15.430442+0000","last_clean":"2024-09-05T16:48:15.430442+0000","last_became_active":"2024-09-05T16:47:03.185143+0000","last_became_peered":"2024-09-05T16:47:03.185143+0000","last_unstale":"2024-09-05T16:48:15.430442+0000","last_undegraded":"2024-09-05T16:48:15.430442+0000","last_fullsized":"2024-09-05T16:48:15.430442+0000","mapping_epoch":46,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12660","last_scrub_stamp":"2024-09-05T16:47:03.683262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:03.683262+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:42:16.533299+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12777","reported_seq":12926,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-05T16:48:16.430157+0000","last_change":"2024-09-05T16:46:40.458315+0000","last_active":"2024-09-05T16:48:16.430157+0000","last_peered":"2024-09-05T16:48:16.430157+0000","last_clean":"2024-09-05T16:48:16.430157+0000","last_became_active":"2024-09-05T16:46:40.184288+0000","last_became_peered":"2024-09-05T16:46:40.184288+0000","last_unstale":"2024-09-05T16:48:16.430157+0000","last_undegraded":"2024-09-05T16:48:16.430157+0000","last_fullsized":"2024-09-05T16:48:16.430157+0000","mapping_epoch":42,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12777","last_scrub_stamp":"2024-09-05T16:46:40.458226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:40.458226+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:38:58.734210+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'12770","reported_seq":12917,"reported_epoch":59,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:48:16.432658+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:48:16.432658+0000","last_peered":"2024-09-05T16:48:16.432658+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:48:16.432658+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12767,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2240}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12808","reported_seq":17488,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-05T16:48:16.432883+0000","last_change":"2024-09-05T16:48:04.288466+0000","last_active":"2024-09-05T16:48:16.432883+0000","last_peered":"2024-09-05T16:48:16.432883+0000","last_clean":"2024-09-05T16:48:16.432883+0000","last_became_active":"2024-09-05T16:48:03.239894+0000","last_became_peered":"2024-09-05T16:48:03.239894+0000","last_unstale":"2024-09-05T16:48:16.432883+0000","last_undegraded":"2024-09-05T16:48:16.432883+0000","last_fullsized":"2024-09-05T16:48:16.432883+0000","mapping_epoch":56,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12808","last_scrub_stamp":"2024-09-05T16:48:04.288388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:48:04.288388+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T21:11:32.044919+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13120","reported_seq":14220,"reported_epoch":59,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:48:18.210417+0000","last_change":"2024-09-05T16:48:14.899129+0000","last_active":"2024-09-05T16:48:18.210417+0000","last_peered":"2024-09-05T16:48:18.210417+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:48:18.210417+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1821,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13117,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":496,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1818}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12758","reported_seq":12903,"reported_epoch":59,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:48:16.432928+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:48:16.432928+0000","last_peered":"2024-09-05T16:48:16.432928+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:48:16.432928+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2237}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12808","reported_seq":17590,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-05T16:48:16.429984+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:48:16.429984+0000","last_peered":"2024-09-05T16:48:16.429984+0000","last_clean":"2024-09-05T16:48:16.429984+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:48:16.429984+0000","last_undegraded":"2024-09-05T16:48:16.429984+0000","last_fullsized":"2024-09-05T16:48:16.429984+0000","mapping_epoch":36,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4408,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13124","reported_seq":13261,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-05T16:48:16.582230+0000","last_change":"2024-09-05T16:48:16.582142+0000","last_active":"2024-09-05T16:48:16.582230+0000","last_peered":"2024-09-05T16:48:16.582230+0000","last_clean":"2024-09-05T16:48:16.582230+0000","last_became_active":"2024-09-05T16:48:16.435091+0000","last_became_peered":"2024-09-05T16:48:16.435091+0000","last_unstale":"2024-09-05T16:48:16.582230+0000","last_undegraded":"2024-09-05T16:48:16.582230+0000","last_fullsized":"2024-09-05T16:48:16.582230+0000","mapping_epoch":58,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'13124","last_scrub_stamp":"2024-09-05T16:48:16.582038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:48:16.582038+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T17:44:25.891565+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"39'12831","reported_seq":12984,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-05T16:48:16.430173+0000","last_change":"2024-09-05T16:47:40.451380+0000","last_active":"2024-09-05T16:48:16.430173+0000","last_peered":"2024-09-05T16:48:16.430173+0000","last_clean":"2024-09-05T16:48:16.430173+0000","last_became_active":"2024-09-05T16:47:40.189447+0000","last_became_peered":"2024-09-05T16:47:40.189447+0000","last_unstale":"2024-09-05T16:48:16.430173+0000","last_undegraded":"2024-09-05T16:48:16.430173+0000","last_fullsized":"2024-09-05T16:48:16.430173+0000","mapping_epoch":52,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12831","last_scrub_stamp":"2024-09-05T16:47:40.451295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:40.451295+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:46:20.946333+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12773","reported_seq":12926,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-05T16:48:16.430339+0000","last_change":"2024-09-05T16:47:27.437275+0000","last_active":"2024-09-05T16:48:16.430339+0000","last_peered":"2024-09-05T16:48:16.430339+0000","last_clean":"2024-09-05T16:48:16.430339+0000","last_became_active":"2024-09-05T16:47:27.204989+0000","last_became_peered":"2024-09-05T16:47:27.204989+0000","last_unstale":"2024-09-05T16:48:16.430339+0000","last_undegraded":"2024-09-05T16:48:16.430339+0000","last_fullsized":"2024-09-05T16:48:16.430339+0000","mapping_epoch":50,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12773","last_scrub_stamp":"2024-09-05T16:47:27.437187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:27.437187+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T16:53:55.961513+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":53888,"num_objects":8,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6307,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":77824,"num_write_kb":263452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5055,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084459,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":726332,"kb_used_data":484,"kb_used_omap":4,"kb_used_meta":725819,"kb_avail":93645508,"statfs":{"total":96636764160,"available":95893000192,"internally_reserved":0,"allocated":495616,"data_stored":187895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":743239379},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986336,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":609732,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":609211,"kb_avail":93762108,"statfs":{"total":96636764160,"available":96012398592,"internally_reserved":0,"allocated":512000,"data_stored":204279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":623832786},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673143,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713340,"kb_used_data":484,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":93658500,"statfs":{"total":96636764160,"available":95906304000,"internally_reserved":0,"allocated":495616,"data_stored":187895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":729938373},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182573,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713772,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658068,"statfs":{"total":96636764160,"available":95905861632,"internally_reserved":0,"allocated":479232,"data_stored":181558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":730394317},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":20480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:48:21.526 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:48:21.824 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:48:21.824 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:48:21.836 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":502,"stamp":"2024-09-05T16:48:21.116739+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":53888,"num_objects":8,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6307,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":77824,"num_write_kb":263452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5055,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2763176,"kb_used_data":1936,"kb_used_omap":14,"kb_used_meta":2761137,"kb_avail":374724184,"statfs":{"total":386547056640,"available":383717564416,"internally_reserved":0,"allocated":1982464,"data_stored":761627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2827404855},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13472,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1817,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-13122,"num_write_kb":-52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-544,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001267"},"pg_stats":[{"pgid":"1.f","version":"39'12904","reported_seq":17537,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-05T16:48:16.433370+0000","last_change":"2024-09-05T16:46:27.826222+0000","last_active":"2024-09-05T16:48:16.433370+0000","last_peered":"2024-09-05T16:48:16.433370+0000","last_clean":"2024-09-05T16:48:16.433370+0000","last_became_active":"2024-09-05T16:46:27.757928+0000","last_became_peered":"2024-09-05T16:46:27.757928+0000","last_unstale":"2024-09-05T16:48:16.433370+0000","last_undegraded":"2024-09-05T16:48:16.433370+0000","last_fullsized":"2024-09-05T16:48:16.433370+0000","mapping_epoch":40,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:32:35.713428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12904,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2262,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12552","reported_seq":17250,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-05T16:48:16.430268+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:48:16.430268+0000","last_peered":"2024-09-05T16:48:16.430268+0000","last_clean":"2024-09-05T16:48:16.430268+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:48:16.430268+0000","last_undegraded":"2024-09-05T16:48:16.430268+0000","last_fullsized":"2024-09-05T16:48:16.430268+0000","mapping_epoch":38,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1865,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12752","reported_seq":17405,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-05T16:48:16.432786+0000","last_change":"2024-09-05T16:46:51.992073+0000","last_active":"2024-09-05T16:48:16.432786+0000","last_peered":"2024-09-05T16:48:16.432786+0000","last_clean":"2024-09-05T16:48:16.432786+0000","last_became_active":"2024-09-05T16:46:51.667128+0000","last_became_peered":"2024-09-05T16:46:51.667128+0000","last_unstale":"2024-09-05T16:48:16.432786+0000","last_undegraded":"2024-09-05T16:48:16.432786+0000","last_fullsized":"2024-09-05T16:48:16.432786+0000","mapping_epoch":44,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12752","last_scrub_stamp":"2024-09-05T16:46:51.991995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:51.991995+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:04:54.120031+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12934","reported_seq":17710,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-05T16:48:16.429984+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:48:16.429984+0000","last_peered":"2024-09-05T16:48:16.429984+0000","last_clean":"2024-09-05T16:48:16.429984+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:48:16.429984+0000","last_undegraded":"2024-09-05T16:48:16.429984+0000","last_fullsized":"2024-09-05T16:48:16.429984+0000","mapping_epoch":34,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7234,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12593","reported_seq":17050,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-05T16:48:16.432562+0000","last_change":"2024-09-05T16:47:51.525026+0000","last_active":"2024-09-05T16:48:16.432562+0000","last_peered":"2024-09-05T16:48:16.432562+0000","last_clean":"2024-09-05T16:48:16.432562+0000","last_became_active":"2024-09-05T16:47:51.193016+0000","last_became_peered":"2024-09-05T16:47:51.193016+0000","last_unstale":"2024-09-05T16:48:16.432562+0000","last_undegraded":"2024-09-05T16:48:16.432562+0000","last_fullsized":"2024-09-05T16:48:16.432562+0000","mapping_epoch":54,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12593","last_scrub_stamp":"2024-09-05T16:47:51.524957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:51.524957+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:46:22.846859+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12772","reported_seq":17381,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-05T16:48:16.432461+0000","last_change":"2024-09-05T16:47:15.832530+0000","last_active":"2024-09-05T16:48:16.432461+0000","last_peered":"2024-09-05T16:48:16.432461+0000","last_clean":"2024-09-05T16:48:16.432461+0000","last_became_active":"2024-09-05T16:47:15.728759+0000","last_became_peered":"2024-09-05T16:47:15.728759+0000","last_unstale":"2024-09-05T16:48:16.432461+0000","last_undegraded":"2024-09-05T16:48:16.432461+0000","last_fullsized":"2024-09-05T16:48:16.432461+0000","mapping_epoch":48,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T02:39:11.515339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12772,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2250,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12660","reported_seq":17096,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-05T16:48:15.430442+0000","last_change":"2024-09-05T16:47:03.683348+0000","last_active":"2024-09-05T16:48:15.430442+0000","last_peered":"2024-09-05T16:48:15.430442+0000","last_clean":"2024-09-05T16:48:15.430442+0000","last_became_active":"2024-09-05T16:47:03.185143+0000","last_became_peered":"2024-09-05T16:47:03.185143+0000","last_unstale":"2024-09-05T16:48:15.430442+0000","last_undegraded":"2024-09-05T16:48:15.430442+0000","last_fullsized":"2024-09-05T16:48:15.430442+0000","mapping_epoch":46,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12660","last_scrub_stamp":"2024-09-05T16:47:03.683262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:03.683262+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:42:16.533299+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12777","reported_seq":12926,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-05T16:48:16.430157+0000","last_change":"2024-09-05T16:46:40.458315+0000","last_active":"2024-09-05T16:48:16.430157+0000","last_peered":"2024-09-05T16:48:16.430157+0000","last_clean":"2024-09-05T16:48:16.430157+0000","last_became_active":"2024-09-05T16:46:40.184288+0000","last_became_peered":"2024-09-05T16:46:40.184288+0000","last_unstale":"2024-09-05T16:48:16.430157+0000","last_undegraded":"2024-09-05T16:48:16.430157+0000","last_fullsized":"2024-09-05T16:48:16.430157+0000","mapping_epoch":42,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12777","last_scrub_stamp":"2024-09-05T16:46:40.458226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:40.458226+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:38:58.734210+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'12770","reported_seq":12917,"reported_epoch":59,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:48:16.432658+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:48:16.432658+0000","last_peered":"2024-09-05T16:48:16.432658+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:48:16.432658+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12767,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2240}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12808","reported_seq":17488,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-05T16:48:16.432883+0000","last_change":"2024-09-05T16:48:04.288466+0000","last_active":"2024-09-05T16:48:16.432883+0000","last_peered":"2024-09-05T16:48:16.432883+0000","last_clean":"2024-09-05T16:48:16.432883+0000","last_became_active":"2024-09-05T16:48:03.239894+0000","last_became_peered":"2024-09-05T16:48:03.239894+0000","last_unstale":"2024-09-05T16:48:16.432883+0000","last_undegraded":"2024-09-05T16:48:16.432883+0000","last_fullsized":"2024-09-05T16:48:16.432883+0000","mapping_epoch":56,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12808","last_scrub_stamp":"2024-09-05T16:48:04.288388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:48:04.288388+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T21:11:32.044919+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13120","reported_seq":14220,"reported_epoch":59,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:48:18.210417+0000","last_change":"2024-09-05T16:48:14.899129+0000","last_active":"2024-09-05T16:48:18.210417+0000","last_peered":"2024-09-05T16:48:18.210417+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:48:18.210417+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1821,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13117,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":496,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1818}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12758","reported_seq":12903,"reported_epoch":59,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:48:16.432928+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:48:16.432928+0000","last_peered":"2024-09-05T16:48:16.432928+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:48:16.432928+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2237}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12808","reported_seq":17590,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-05T16:48:16.429984+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:48:16.429984+0000","last_peered":"2024-09-05T16:48:16.429984+0000","last_clean":"2024-09-05T16:48:16.429984+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:48:16.429984+0000","last_undegraded":"2024-09-05T16:48:16.429984+0000","last_fullsized":"2024-09-05T16:48:16.429984+0000","mapping_epoch":36,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4408,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13124","reported_seq":13261,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-05T16:48:16.582230+0000","last_change":"2024-09-05T16:48:16.582142+0000","last_active":"2024-09-05T16:48:16.582230+0000","last_peered":"2024-09-05T16:48:16.582230+0000","last_clean":"2024-09-05T16:48:16.582230+0000","last_became_active":"2024-09-05T16:48:16.435091+0000","last_became_peered":"2024-09-05T16:48:16.435091+0000","last_unstale":"2024-09-05T16:48:16.582230+0000","last_undegraded":"2024-09-05T16:48:16.582230+0000","last_fullsized":"2024-09-05T16:48:16.582230+0000","mapping_epoch":58,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'13124","last_scrub_stamp":"2024-09-05T16:48:16.582038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:48:16.582038+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T17:44:25.891565+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"39'12831","reported_seq":12984,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-05T16:48:16.430173+0000","last_change":"2024-09-05T16:47:40.451380+0000","last_active":"2024-09-05T16:48:16.430173+0000","last_peered":"2024-09-05T16:48:16.430173+0000","last_clean":"2024-09-05T16:48:16.430173+0000","last_became_active":"2024-09-05T16:47:40.189447+0000","last_became_peered":"2024-09-05T16:47:40.189447+0000","last_unstale":"2024-09-05T16:48:16.430173+0000","last_undegraded":"2024-09-05T16:48:16.430173+0000","last_fullsized":"2024-09-05T16:48:16.430173+0000","mapping_epoch":52,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12831","last_scrub_stamp":"2024-09-05T16:47:40.451295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:40.451295+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:46:20.946333+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12773","reported_seq":12926,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-05T16:48:16.430339+0000","last_change":"2024-09-05T16:47:27.437275+0000","last_active":"2024-09-05T16:48:16.430339+0000","last_peered":"2024-09-05T16:48:16.430339+0000","last_clean":"2024-09-05T16:48:16.430339+0000","last_became_active":"2024-09-05T16:47:27.204989+0000","last_became_peered":"2024-09-05T16:47:27.204989+0000","last_unstale":"2024-09-05T16:48:16.430339+0000","last_undegraded":"2024-09-05T16:48:16.430339+0000","last_fullsized":"2024-09-05T16:48:16.430339+0000","mapping_epoch":50,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12773","last_scrub_stamp":"2024-09-05T16:47:27.437187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:27.437187+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T16:53:55.961513+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":53888,"num_objects":8,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6307,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":77824,"num_write_kb":263452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5055,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084459,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":726332,"kb_used_data":484,"kb_used_omap":4,"kb_used_meta":725819,"kb_avail":93645508,"statfs":{"total":96636764160,"available":95893000192,"internally_reserved":0,"allocated":495616,"data_stored":187895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":743239379},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986336,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":609732,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":609211,"kb_avail":93762108,"statfs":{"total":96636764160,"available":96012398592,"internally_reserved":0,"allocated":512000,"data_stored":204279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":623832786},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673143,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713340,"kb_used_data":484,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":93658500,"statfs":{"total":96636764160,"available":95906304000,"internally_reserved":0,"allocated":495616,"data_stored":187895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":729938373},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182573,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713772,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658068,"statfs":{"total":96636764160,"available":95905861632,"internally_reserved":0,"allocated":479232,"data_stored":181558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":730394317},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":20480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:48:21.837 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:48:21.837 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "39'12770", 'reported_seq': 12917, 'reported_epoch': 59, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:48:16.432658+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:48:16.432658+0000', 'last_peered': '2024-09-05T16:48:16.432658+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:48:16.432658+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "33'2700", 'ondisk_log_start': "33'2700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10070, 'log_dups_size': 0, 'ondisk_log_size': 10070, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26944, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 16, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12767, 'num_write_kb': 51081, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2240}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:48:21.837 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:48:21.838 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "39'13120", 'reported_seq': 14220, 'reported_epoch': 59, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:48:18.210417+0000', 'last_change': '2024-09-05T16:48:14.899129+0000', 'last_active': '2024-09-05T16:48:18.210417+0000', 'last_peered': '2024-09-05T16:48:18.210417+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:48:18.210417+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "33'3100", 'ondisk_log_start': "33'3100", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20208, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1821, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13117, 'num_write_kb': 52476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 496, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1818}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:48:21.838 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:48:21.838 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "39'12758", 'reported_seq': 12903, 'reported_epoch': 59, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:48:16.432928+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:48:16.432928+0000', 'last_peered': '2024-09-05T16:48:16.432928+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:48:16.432928+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "33'2700", 'ondisk_log_start': "33'2700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10058, 'log_dups_size': 0, 'ondisk_log_size': 10058, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6736, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12757, 'num_write_kb': 51031, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2237}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:48:21.838 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:48:22.214 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:48:22.214 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 6307/32 objects degraded (19709.375%), 3 pgs degraded, 3 pgs undersized","count":6},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":905,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":59,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":13},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":8,"data_bytes":53888,"bytes_used":2829492224,"bytes_avail":383717564416,"bytes_total":386547056640,"degraded_objects":6307,"degraded_total":32,"degraded_ratio":197.09375},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:48:22.227 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-09-05T16:48:22.227 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:48:22.547 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:48:22.547 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:48:22.560 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":502,"stamp":"2024-09-05T16:48:21.116739+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":53888,"num_objects":8,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6307,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":77824,"num_write_kb":263452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5055,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2763176,"kb_used_data":1936,"kb_used_omap":14,"kb_used_meta":2761137,"kb_avail":374724184,"statfs":{"total":386547056640,"available":383717564416,"internally_reserved":0,"allocated":1982464,"data_stored":761627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2827404855},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13472,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1817,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-13122,"num_write_kb":-52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-544,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001267"},"pg_stats":[{"pgid":"1.f","version":"39'12904","reported_seq":17537,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-05T16:48:16.433370+0000","last_change":"2024-09-05T16:46:27.826222+0000","last_active":"2024-09-05T16:48:16.433370+0000","last_peered":"2024-09-05T16:48:16.433370+0000","last_clean":"2024-09-05T16:48:16.433370+0000","last_became_active":"2024-09-05T16:46:27.757928+0000","last_became_peered":"2024-09-05T16:46:27.757928+0000","last_unstale":"2024-09-05T16:48:16.433370+0000","last_undegraded":"2024-09-05T16:48:16.433370+0000","last_fullsized":"2024-09-05T16:48:16.433370+0000","mapping_epoch":40,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:32:35.713428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12904,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2262,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12552","reported_seq":17250,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-05T16:48:16.430268+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:48:16.430268+0000","last_peered":"2024-09-05T16:48:16.430268+0000","last_clean":"2024-09-05T16:48:16.430268+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:48:16.430268+0000","last_undegraded":"2024-09-05T16:48:16.430268+0000","last_fullsized":"2024-09-05T16:48:16.430268+0000","mapping_epoch":38,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1865,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12752","reported_seq":17405,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-05T16:48:16.432786+0000","last_change":"2024-09-05T16:46:51.992073+0000","last_active":"2024-09-05T16:48:16.432786+0000","last_peered":"2024-09-05T16:48:16.432786+0000","last_clean":"2024-09-05T16:48:16.432786+0000","last_became_active":"2024-09-05T16:46:51.667128+0000","last_became_peered":"2024-09-05T16:46:51.667128+0000","last_unstale":"2024-09-05T16:48:16.432786+0000","last_undegraded":"2024-09-05T16:48:16.432786+0000","last_fullsized":"2024-09-05T16:48:16.432786+0000","mapping_epoch":44,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12752","last_scrub_stamp":"2024-09-05T16:46:51.991995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:51.991995+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:04:54.120031+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12934","reported_seq":17710,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-05T16:48:16.429984+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:48:16.429984+0000","last_peered":"2024-09-05T16:48:16.429984+0000","last_clean":"2024-09-05T16:48:16.429984+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:48:16.429984+0000","last_undegraded":"2024-09-05T16:48:16.429984+0000","last_fullsized":"2024-09-05T16:48:16.429984+0000","mapping_epoch":34,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7234,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12593","reported_seq":17050,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-05T16:48:16.432562+0000","last_change":"2024-09-05T16:47:51.525026+0000","last_active":"2024-09-05T16:48:16.432562+0000","last_peered":"2024-09-05T16:48:16.432562+0000","last_clean":"2024-09-05T16:48:16.432562+0000","last_became_active":"2024-09-05T16:47:51.193016+0000","last_became_peered":"2024-09-05T16:47:51.193016+0000","last_unstale":"2024-09-05T16:48:16.432562+0000","last_undegraded":"2024-09-05T16:48:16.432562+0000","last_fullsized":"2024-09-05T16:48:16.432562+0000","mapping_epoch":54,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12593","last_scrub_stamp":"2024-09-05T16:47:51.524957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:51.524957+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:46:22.846859+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12772","reported_seq":17381,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-05T16:48:16.432461+0000","last_change":"2024-09-05T16:47:15.832530+0000","last_active":"2024-09-05T16:48:16.432461+0000","last_peered":"2024-09-05T16:48:16.432461+0000","last_clean":"2024-09-05T16:48:16.432461+0000","last_became_active":"2024-09-05T16:47:15.728759+0000","last_became_peered":"2024-09-05T16:47:15.728759+0000","last_unstale":"2024-09-05T16:48:16.432461+0000","last_undegraded":"2024-09-05T16:48:16.432461+0000","last_fullsized":"2024-09-05T16:48:16.432461+0000","mapping_epoch":48,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T02:39:11.515339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12772,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2250,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12660","reported_seq":17096,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-05T16:48:15.430442+0000","last_change":"2024-09-05T16:47:03.683348+0000","last_active":"2024-09-05T16:48:15.430442+0000","last_peered":"2024-09-05T16:48:15.430442+0000","last_clean":"2024-09-05T16:48:15.430442+0000","last_became_active":"2024-09-05T16:47:03.185143+0000","last_became_peered":"2024-09-05T16:47:03.185143+0000","last_unstale":"2024-09-05T16:48:15.430442+0000","last_undegraded":"2024-09-05T16:48:15.430442+0000","last_fullsized":"2024-09-05T16:48:15.430442+0000","mapping_epoch":46,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12660","last_scrub_stamp":"2024-09-05T16:47:03.683262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:03.683262+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:42:16.533299+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12777","reported_seq":12926,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-05T16:48:16.430157+0000","last_change":"2024-09-05T16:46:40.458315+0000","last_active":"2024-09-05T16:48:16.430157+0000","last_peered":"2024-09-05T16:48:16.430157+0000","last_clean":"2024-09-05T16:48:16.430157+0000","last_became_active":"2024-09-05T16:46:40.184288+0000","last_became_peered":"2024-09-05T16:46:40.184288+0000","last_unstale":"2024-09-05T16:48:16.430157+0000","last_undegraded":"2024-09-05T16:48:16.430157+0000","last_fullsized":"2024-09-05T16:48:16.430157+0000","mapping_epoch":42,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12777","last_scrub_stamp":"2024-09-05T16:46:40.458226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:40.458226+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:38:58.734210+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'12770","reported_seq":12917,"reported_epoch":59,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:48:16.432658+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:48:16.432658+0000","last_peered":"2024-09-05T16:48:16.432658+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:48:16.432658+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12767,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2240}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12808","reported_seq":17488,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-05T16:48:16.432883+0000","last_change":"2024-09-05T16:48:04.288466+0000","last_active":"2024-09-05T16:48:16.432883+0000","last_peered":"2024-09-05T16:48:16.432883+0000","last_clean":"2024-09-05T16:48:16.432883+0000","last_became_active":"2024-09-05T16:48:03.239894+0000","last_became_peered":"2024-09-05T16:48:03.239894+0000","last_unstale":"2024-09-05T16:48:16.432883+0000","last_undegraded":"2024-09-05T16:48:16.432883+0000","last_fullsized":"2024-09-05T16:48:16.432883+0000","mapping_epoch":56,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12808","last_scrub_stamp":"2024-09-05T16:48:04.288388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:48:04.288388+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T21:11:32.044919+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13120","reported_seq":14220,"reported_epoch":59,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:48:18.210417+0000","last_change":"2024-09-05T16:48:14.899129+0000","last_active":"2024-09-05T16:48:18.210417+0000","last_peered":"2024-09-05T16:48:18.210417+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:48:18.210417+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1821,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13117,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":496,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1818}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12758","reported_seq":12903,"reported_epoch":59,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:48:16.432928+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:48:16.432928+0000","last_peered":"2024-09-05T16:48:16.432928+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:48:16.432928+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2237}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12808","reported_seq":17590,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-05T16:48:16.429984+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:48:16.429984+0000","last_peered":"2024-09-05T16:48:16.429984+0000","last_clean":"2024-09-05T16:48:16.429984+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:48:16.429984+0000","last_undegraded":"2024-09-05T16:48:16.429984+0000","last_fullsized":"2024-09-05T16:48:16.429984+0000","mapping_epoch":36,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4408,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13124","reported_seq":13261,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-05T16:48:16.582230+0000","last_change":"2024-09-05T16:48:16.582142+0000","last_active":"2024-09-05T16:48:16.582230+0000","last_peered":"2024-09-05T16:48:16.582230+0000","last_clean":"2024-09-05T16:48:16.582230+0000","last_became_active":"2024-09-05T16:48:16.435091+0000","last_became_peered":"2024-09-05T16:48:16.435091+0000","last_unstale":"2024-09-05T16:48:16.582230+0000","last_undegraded":"2024-09-05T16:48:16.582230+0000","last_fullsized":"2024-09-05T16:48:16.582230+0000","mapping_epoch":58,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'13124","last_scrub_stamp":"2024-09-05T16:48:16.582038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:48:16.582038+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T17:44:25.891565+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"39'12831","reported_seq":12984,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-05T16:48:16.430173+0000","last_change":"2024-09-05T16:47:40.451380+0000","last_active":"2024-09-05T16:48:16.430173+0000","last_peered":"2024-09-05T16:48:16.430173+0000","last_clean":"2024-09-05T16:48:16.430173+0000","last_became_active":"2024-09-05T16:47:40.189447+0000","last_became_peered":"2024-09-05T16:47:40.189447+0000","last_unstale":"2024-09-05T16:48:16.430173+0000","last_undegraded":"2024-09-05T16:48:16.430173+0000","last_fullsized":"2024-09-05T16:48:16.430173+0000","mapping_epoch":52,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12831","last_scrub_stamp":"2024-09-05T16:47:40.451295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:40.451295+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:46:20.946333+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12773","reported_seq":12926,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-05T16:48:16.430339+0000","last_change":"2024-09-05T16:47:27.437275+0000","last_active":"2024-09-05T16:48:16.430339+0000","last_peered":"2024-09-05T16:48:16.430339+0000","last_clean":"2024-09-05T16:48:16.430339+0000","last_became_active":"2024-09-05T16:47:27.204989+0000","last_became_peered":"2024-09-05T16:47:27.204989+0000","last_unstale":"2024-09-05T16:48:16.430339+0000","last_undegraded":"2024-09-05T16:48:16.430339+0000","last_fullsized":"2024-09-05T16:48:16.430339+0000","mapping_epoch":50,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12773","last_scrub_stamp":"2024-09-05T16:47:27.437187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:27.437187+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T16:53:55.961513+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":53888,"num_objects":8,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6307,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":77824,"num_write_kb":263452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5055,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084459,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":726332,"kb_used_data":484,"kb_used_omap":4,"kb_used_meta":725819,"kb_avail":93645508,"statfs":{"total":96636764160,"available":95893000192,"internally_reserved":0,"allocated":495616,"data_stored":187895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":743239379},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986336,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":609732,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":609211,"kb_avail":93762108,"statfs":{"total":96636764160,"available":96012398592,"internally_reserved":0,"allocated":512000,"data_stored":204279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":623832786},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673143,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713340,"kb_used_data":484,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":93658500,"statfs":{"total":96636764160,"available":95906304000,"internally_reserved":0,"allocated":495616,"data_stored":187895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":729938373},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182573,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713772,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658068,"statfs":{"total":96636764160,"available":95905861632,"internally_reserved":0,"allocated":479232,"data_stored":181558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":730394317},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":20480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:48:25.562 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:48:25.886 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:48:25.886 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:48:25.901 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":504,"stamp":"2024-09-05T16:48:25.117347+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":53888,"num_objects":8,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5441,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":77824,"num_write_kb":263452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5921,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2771512,"kb_used_data":1952,"kb_used_omap":14,"kb_used_meta":2769457,"kb_avail":374715848,"statfs":{"total":386547056640,"available":383709028352,"internally_reserved":0,"allocated":1998848,"data_stored":767964,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2835924535},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13472,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1712,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-13122,"num_write_kb":-52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-649,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.687857"},"pg_stats":[{"pgid":"1.f","version":"39'12904","reported_seq":17537,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-05T16:48:16.433370+0000","last_change":"2024-09-05T16:46:27.826222+0000","last_active":"2024-09-05T16:48:16.433370+0000","last_peered":"2024-09-05T16:48:16.433370+0000","last_clean":"2024-09-05T16:48:16.433370+0000","last_became_active":"2024-09-05T16:46:27.757928+0000","last_became_peered":"2024-09-05T16:46:27.757928+0000","last_unstale":"2024-09-05T16:48:16.433370+0000","last_undegraded":"2024-09-05T16:48:16.433370+0000","last_fullsized":"2024-09-05T16:48:16.433370+0000","mapping_epoch":40,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:32:35.713428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12904,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2262,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12552","reported_seq":17250,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-05T16:48:16.430268+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:48:16.430268+0000","last_peered":"2024-09-05T16:48:16.430268+0000","last_clean":"2024-09-05T16:48:16.430268+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:48:16.430268+0000","last_undegraded":"2024-09-05T16:48:16.430268+0000","last_fullsized":"2024-09-05T16:48:16.430268+0000","mapping_epoch":38,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1865,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12752","reported_seq":17405,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-05T16:48:16.432786+0000","last_change":"2024-09-05T16:46:51.992073+0000","last_active":"2024-09-05T16:48:16.432786+0000","last_peered":"2024-09-05T16:48:16.432786+0000","last_clean":"2024-09-05T16:48:16.432786+0000","last_became_active":"2024-09-05T16:46:51.667128+0000","last_became_peered":"2024-09-05T16:46:51.667128+0000","last_unstale":"2024-09-05T16:48:16.432786+0000","last_undegraded":"2024-09-05T16:48:16.432786+0000","last_fullsized":"2024-09-05T16:48:16.432786+0000","mapping_epoch":44,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12752","last_scrub_stamp":"2024-09-05T16:46:51.991995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:51.991995+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:04:54.120031+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12934","reported_seq":17710,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-05T16:48:16.429984+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:48:16.429984+0000","last_peered":"2024-09-05T16:48:16.429984+0000","last_clean":"2024-09-05T16:48:16.429984+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:48:16.429984+0000","last_undegraded":"2024-09-05T16:48:16.429984+0000","last_fullsized":"2024-09-05T16:48:16.429984+0000","mapping_epoch":34,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7234,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12593","reported_seq":17050,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-05T16:48:16.432562+0000","last_change":"2024-09-05T16:47:51.525026+0000","last_active":"2024-09-05T16:48:16.432562+0000","last_peered":"2024-09-05T16:48:16.432562+0000","last_clean":"2024-09-05T16:48:16.432562+0000","last_became_active":"2024-09-05T16:47:51.193016+0000","last_became_peered":"2024-09-05T16:47:51.193016+0000","last_unstale":"2024-09-05T16:48:16.432562+0000","last_undegraded":"2024-09-05T16:48:16.432562+0000","last_fullsized":"2024-09-05T16:48:16.432562+0000","mapping_epoch":54,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12593","last_scrub_stamp":"2024-09-05T16:47:51.524957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:51.524957+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:46:22.846859+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12772","reported_seq":17381,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-05T16:48:16.432461+0000","last_change":"2024-09-05T16:47:15.832530+0000","last_active":"2024-09-05T16:48:16.432461+0000","last_peered":"2024-09-05T16:48:16.432461+0000","last_clean":"2024-09-05T16:48:16.432461+0000","last_became_active":"2024-09-05T16:47:15.728759+0000","last_became_peered":"2024-09-05T16:47:15.728759+0000","last_unstale":"2024-09-05T16:48:16.432461+0000","last_undegraded":"2024-09-05T16:48:16.432461+0000","last_fullsized":"2024-09-05T16:48:16.432461+0000","mapping_epoch":48,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T02:39:11.515339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12772,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2250,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12660","reported_seq":17098,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-05T16:48:16.432380+0000","last_change":"2024-09-05T16:47:03.683348+0000","last_active":"2024-09-05T16:48:16.432380+0000","last_peered":"2024-09-05T16:48:16.432380+0000","last_clean":"2024-09-05T16:48:16.432380+0000","last_became_active":"2024-09-05T16:47:03.185143+0000","last_became_peered":"2024-09-05T16:47:03.185143+0000","last_unstale":"2024-09-05T16:48:16.432380+0000","last_undegraded":"2024-09-05T16:48:16.432380+0000","last_fullsized":"2024-09-05T16:48:16.432380+0000","mapping_epoch":46,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12660","last_scrub_stamp":"2024-09-05T16:47:03.683262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:03.683262+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:42:16.533299+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12777","reported_seq":12926,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-05T16:48:16.430157+0000","last_change":"2024-09-05T16:46:40.458315+0000","last_active":"2024-09-05T16:48:16.430157+0000","last_peered":"2024-09-05T16:48:16.430157+0000","last_clean":"2024-09-05T16:48:16.430157+0000","last_became_active":"2024-09-05T16:46:40.184288+0000","last_became_peered":"2024-09-05T16:46:40.184288+0000","last_unstale":"2024-09-05T16:48:16.430157+0000","last_undegraded":"2024-09-05T16:48:16.430157+0000","last_fullsized":"2024-09-05T16:48:16.430157+0000","mapping_epoch":42,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12777","last_scrub_stamp":"2024-09-05T16:46:40.458226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:40.458226+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:38:58.734210+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'12770","reported_seq":12917,"reported_epoch":59,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:48:16.432658+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:48:16.432658+0000","last_peered":"2024-09-05T16:48:16.432658+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:48:16.432658+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12767,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2240}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12808","reported_seq":17488,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-05T16:48:16.432883+0000","last_change":"2024-09-05T16:48:04.288466+0000","last_active":"2024-09-05T16:48:16.432883+0000","last_peered":"2024-09-05T16:48:16.432883+0000","last_clean":"2024-09-05T16:48:16.432883+0000","last_became_active":"2024-09-05T16:48:03.239894+0000","last_became_peered":"2024-09-05T16:48:03.239894+0000","last_unstale":"2024-09-05T16:48:16.432883+0000","last_undegraded":"2024-09-05T16:48:16.432883+0000","last_fullsized":"2024-09-05T16:48:16.432883+0000","mapping_epoch":56,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12808","last_scrub_stamp":"2024-09-05T16:48:04.288388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:48:04.288388+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T21:11:32.044919+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13120","reported_seq":15952,"reported_epoch":59,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:48:23.209534+0000","last_change":"2024-09-05T16:48:14.899129+0000","last_active":"2024-09-05T16:48:23.209534+0000","last_peered":"2024-09-05T16:48:23.209534+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:48:23.209534+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13117,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1362,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-952}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12758","reported_seq":12903,"reported_epoch":59,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:48:16.432928+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:48:16.432928+0000","last_peered":"2024-09-05T16:48:16.432928+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:48:16.432928+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2237}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12808","reported_seq":17590,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-05T16:48:16.429984+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:48:16.429984+0000","last_peered":"2024-09-05T16:48:16.429984+0000","last_clean":"2024-09-05T16:48:16.429984+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:48:16.429984+0000","last_undegraded":"2024-09-05T16:48:16.429984+0000","last_fullsized":"2024-09-05T16:48:16.429984+0000","mapping_epoch":36,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4408,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13124","reported_seq":13261,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-05T16:48:16.582230+0000","last_change":"2024-09-05T16:48:16.582142+0000","last_active":"2024-09-05T16:48:16.582230+0000","last_peered":"2024-09-05T16:48:16.582230+0000","last_clean":"2024-09-05T16:48:16.582230+0000","last_became_active":"2024-09-05T16:48:16.435091+0000","last_became_peered":"2024-09-05T16:48:16.435091+0000","last_unstale":"2024-09-05T16:48:16.582230+0000","last_undegraded":"2024-09-05T16:48:16.582230+0000","last_fullsized":"2024-09-05T16:48:16.582230+0000","mapping_epoch":58,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'13124","last_scrub_stamp":"2024-09-05T16:48:16.582038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:48:16.582038+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T17:44:25.891565+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"39'12831","reported_seq":12984,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-05T16:48:16.430173+0000","last_change":"2024-09-05T16:47:40.451380+0000","last_active":"2024-09-05T16:48:16.430173+0000","last_peered":"2024-09-05T16:48:16.430173+0000","last_clean":"2024-09-05T16:48:16.430173+0000","last_became_active":"2024-09-05T16:47:40.189447+0000","last_became_peered":"2024-09-05T16:47:40.189447+0000","last_unstale":"2024-09-05T16:48:16.430173+0000","last_undegraded":"2024-09-05T16:48:16.430173+0000","last_fullsized":"2024-09-05T16:48:16.430173+0000","mapping_epoch":52,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12831","last_scrub_stamp":"2024-09-05T16:47:40.451295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:40.451295+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:46:20.946333+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12773","reported_seq":12926,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-05T16:48:16.430339+0000","last_change":"2024-09-05T16:47:27.437275+0000","last_active":"2024-09-05T16:48:16.430339+0000","last_peered":"2024-09-05T16:48:16.430339+0000","last_clean":"2024-09-05T16:48:16.430339+0000","last_became_active":"2024-09-05T16:47:27.204989+0000","last_became_peered":"2024-09-05T16:47:27.204989+0000","last_unstale":"2024-09-05T16:48:16.430339+0000","last_undegraded":"2024-09-05T16:48:16.430339+0000","last_fullsized":"2024-09-05T16:48:16.430339+0000","mapping_epoch":50,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12773","last_scrub_stamp":"2024-09-05T16:47:27.437187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:27.437187+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T16:53:55.961513+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":53888,"num_objects":8,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5441,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":77824,"num_write_kb":263452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5921,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084460,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":726332,"kb_used_data":484,"kb_used_omap":4,"kb_used_meta":725819,"kb_avail":93645508,"statfs":{"total":96636764160,"available":95893000192,"internally_reserved":0,"allocated":495616,"data_stored":187895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":743239379},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986337,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":618052,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":617531,"kb_avail":93753788,"statfs":{"total":96636764160,"available":96003878912,"internally_reserved":0,"allocated":512000,"data_stored":204279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":632352466},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673143,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713340,"kb_used_data":484,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":93658500,"statfs":{"total":96636764160,"available":95906304000,"internally_reserved":0,"allocated":495616,"data_stored":187895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":729938373},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182574,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713788,"kb_used_data":484,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658052,"statfs":{"total":96636764160,"available":95905845248,"internally_reserved":0,"allocated":495616,"data_stored":187895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":730394317},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":20480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:48:25.902 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:48:26.223 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:48:26.224 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:48:26.236 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":504,"stamp":"2024-09-05T16:48:25.117347+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":53888,"num_objects":8,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5441,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":77824,"num_write_kb":263452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5921,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2771512,"kb_used_data":1952,"kb_used_omap":14,"kb_used_meta":2769457,"kb_avail":374715848,"statfs":{"total":386547056640,"available":383709028352,"internally_reserved":0,"allocated":1998848,"data_stored":767964,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2835924535},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13472,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1712,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-13122,"num_write_kb":-52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-649,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.687857"},"pg_stats":[{"pgid":"1.f","version":"39'12904","reported_seq":17537,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-05T16:48:16.433370+0000","last_change":"2024-09-05T16:46:27.826222+0000","last_active":"2024-09-05T16:48:16.433370+0000","last_peered":"2024-09-05T16:48:16.433370+0000","last_clean":"2024-09-05T16:48:16.433370+0000","last_became_active":"2024-09-05T16:46:27.757928+0000","last_became_peered":"2024-09-05T16:46:27.757928+0000","last_unstale":"2024-09-05T16:48:16.433370+0000","last_undegraded":"2024-09-05T16:48:16.433370+0000","last_fullsized":"2024-09-05T16:48:16.433370+0000","mapping_epoch":40,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:32:35.713428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12904,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2262,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12552","reported_seq":17250,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-05T16:48:16.430268+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:48:16.430268+0000","last_peered":"2024-09-05T16:48:16.430268+0000","last_clean":"2024-09-05T16:48:16.430268+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:48:16.430268+0000","last_undegraded":"2024-09-05T16:48:16.430268+0000","last_fullsized":"2024-09-05T16:48:16.430268+0000","mapping_epoch":38,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1865,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12752","reported_seq":17405,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-05T16:48:16.432786+0000","last_change":"2024-09-05T16:46:51.992073+0000","last_active":"2024-09-05T16:48:16.432786+0000","last_peered":"2024-09-05T16:48:16.432786+0000","last_clean":"2024-09-05T16:48:16.432786+0000","last_became_active":"2024-09-05T16:46:51.667128+0000","last_became_peered":"2024-09-05T16:46:51.667128+0000","last_unstale":"2024-09-05T16:48:16.432786+0000","last_undegraded":"2024-09-05T16:48:16.432786+0000","last_fullsized":"2024-09-05T16:48:16.432786+0000","mapping_epoch":44,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12752","last_scrub_stamp":"2024-09-05T16:46:51.991995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:51.991995+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:04:54.120031+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12934","reported_seq":17710,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-05T16:48:16.429984+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:48:16.429984+0000","last_peered":"2024-09-05T16:48:16.429984+0000","last_clean":"2024-09-05T16:48:16.429984+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:48:16.429984+0000","last_undegraded":"2024-09-05T16:48:16.429984+0000","last_fullsized":"2024-09-05T16:48:16.429984+0000","mapping_epoch":34,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7234,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12593","reported_seq":17050,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-05T16:48:16.432562+0000","last_change":"2024-09-05T16:47:51.525026+0000","last_active":"2024-09-05T16:48:16.432562+0000","last_peered":"2024-09-05T16:48:16.432562+0000","last_clean":"2024-09-05T16:48:16.432562+0000","last_became_active":"2024-09-05T16:47:51.193016+0000","last_became_peered":"2024-09-05T16:47:51.193016+0000","last_unstale":"2024-09-05T16:48:16.432562+0000","last_undegraded":"2024-09-05T16:48:16.432562+0000","last_fullsized":"2024-09-05T16:48:16.432562+0000","mapping_epoch":54,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12593","last_scrub_stamp":"2024-09-05T16:47:51.524957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:51.524957+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:46:22.846859+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12772","reported_seq":17381,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-05T16:48:16.432461+0000","last_change":"2024-09-05T16:47:15.832530+0000","last_active":"2024-09-05T16:48:16.432461+0000","last_peered":"2024-09-05T16:48:16.432461+0000","last_clean":"2024-09-05T16:48:16.432461+0000","last_became_active":"2024-09-05T16:47:15.728759+0000","last_became_peered":"2024-09-05T16:47:15.728759+0000","last_unstale":"2024-09-05T16:48:16.432461+0000","last_undegraded":"2024-09-05T16:48:16.432461+0000","last_fullsized":"2024-09-05T16:48:16.432461+0000","mapping_epoch":48,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T02:39:11.515339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12772,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2250,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12660","reported_seq":17098,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-05T16:48:16.432380+0000","last_change":"2024-09-05T16:47:03.683348+0000","last_active":"2024-09-05T16:48:16.432380+0000","last_peered":"2024-09-05T16:48:16.432380+0000","last_clean":"2024-09-05T16:48:16.432380+0000","last_became_active":"2024-09-05T16:47:03.185143+0000","last_became_peered":"2024-09-05T16:47:03.185143+0000","last_unstale":"2024-09-05T16:48:16.432380+0000","last_undegraded":"2024-09-05T16:48:16.432380+0000","last_fullsized":"2024-09-05T16:48:16.432380+0000","mapping_epoch":46,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12660","last_scrub_stamp":"2024-09-05T16:47:03.683262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:03.683262+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:42:16.533299+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12777","reported_seq":12926,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-05T16:48:16.430157+0000","last_change":"2024-09-05T16:46:40.458315+0000","last_active":"2024-09-05T16:48:16.430157+0000","last_peered":"2024-09-05T16:48:16.430157+0000","last_clean":"2024-09-05T16:48:16.430157+0000","last_became_active":"2024-09-05T16:46:40.184288+0000","last_became_peered":"2024-09-05T16:46:40.184288+0000","last_unstale":"2024-09-05T16:48:16.430157+0000","last_undegraded":"2024-09-05T16:48:16.430157+0000","last_fullsized":"2024-09-05T16:48:16.430157+0000","mapping_epoch":42,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12777","last_scrub_stamp":"2024-09-05T16:46:40.458226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:40.458226+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:38:58.734210+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'12770","reported_seq":12917,"reported_epoch":59,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:48:16.432658+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:48:16.432658+0000","last_peered":"2024-09-05T16:48:16.432658+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:48:16.432658+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12767,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2240}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12808","reported_seq":17488,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-05T16:48:16.432883+0000","last_change":"2024-09-05T16:48:04.288466+0000","last_active":"2024-09-05T16:48:16.432883+0000","last_peered":"2024-09-05T16:48:16.432883+0000","last_clean":"2024-09-05T16:48:16.432883+0000","last_became_active":"2024-09-05T16:48:03.239894+0000","last_became_peered":"2024-09-05T16:48:03.239894+0000","last_unstale":"2024-09-05T16:48:16.432883+0000","last_undegraded":"2024-09-05T16:48:16.432883+0000","last_fullsized":"2024-09-05T16:48:16.432883+0000","mapping_epoch":56,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12808","last_scrub_stamp":"2024-09-05T16:48:04.288388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:48:04.288388+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T21:11:32.044919+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13120","reported_seq":15952,"reported_epoch":59,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:48:23.209534+0000","last_change":"2024-09-05T16:48:14.899129+0000","last_active":"2024-09-05T16:48:23.209534+0000","last_peered":"2024-09-05T16:48:23.209534+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:48:23.209534+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13117,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1362,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-952}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12758","reported_seq":12903,"reported_epoch":59,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:48:16.432928+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:48:16.432928+0000","last_peered":"2024-09-05T16:48:16.432928+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:48:16.432928+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2237}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12808","reported_seq":17590,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-05T16:48:16.429984+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:48:16.429984+0000","last_peered":"2024-09-05T16:48:16.429984+0000","last_clean":"2024-09-05T16:48:16.429984+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:48:16.429984+0000","last_undegraded":"2024-09-05T16:48:16.429984+0000","last_fullsized":"2024-09-05T16:48:16.429984+0000","mapping_epoch":36,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4408,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13124","reported_seq":13261,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-05T16:48:16.582230+0000","last_change":"2024-09-05T16:48:16.582142+0000","last_active":"2024-09-05T16:48:16.582230+0000","last_peered":"2024-09-05T16:48:16.582230+0000","last_clean":"2024-09-05T16:48:16.582230+0000","last_became_active":"2024-09-05T16:48:16.435091+0000","last_became_peered":"2024-09-05T16:48:16.435091+0000","last_unstale":"2024-09-05T16:48:16.582230+0000","last_undegraded":"2024-09-05T16:48:16.582230+0000","last_fullsized":"2024-09-05T16:48:16.582230+0000","mapping_epoch":58,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'13124","last_scrub_stamp":"2024-09-05T16:48:16.582038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:48:16.582038+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T17:44:25.891565+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"39'12831","reported_seq":12984,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-05T16:48:16.430173+0000","last_change":"2024-09-05T16:47:40.451380+0000","last_active":"2024-09-05T16:48:16.430173+0000","last_peered":"2024-09-05T16:48:16.430173+0000","last_clean":"2024-09-05T16:48:16.430173+0000","last_became_active":"2024-09-05T16:47:40.189447+0000","last_became_peered":"2024-09-05T16:47:40.189447+0000","last_unstale":"2024-09-05T16:48:16.430173+0000","last_undegraded":"2024-09-05T16:48:16.430173+0000","last_fullsized":"2024-09-05T16:48:16.430173+0000","mapping_epoch":52,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12831","last_scrub_stamp":"2024-09-05T16:47:40.451295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:40.451295+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:46:20.946333+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12773","reported_seq":12926,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-05T16:48:16.430339+0000","last_change":"2024-09-05T16:47:27.437275+0000","last_active":"2024-09-05T16:48:16.430339+0000","last_peered":"2024-09-05T16:48:16.430339+0000","last_clean":"2024-09-05T16:48:16.430339+0000","last_became_active":"2024-09-05T16:47:27.204989+0000","last_became_peered":"2024-09-05T16:47:27.204989+0000","last_unstale":"2024-09-05T16:48:16.430339+0000","last_undegraded":"2024-09-05T16:48:16.430339+0000","last_fullsized":"2024-09-05T16:48:16.430339+0000","mapping_epoch":50,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12773","last_scrub_stamp":"2024-09-05T16:47:27.437187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:27.437187+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T16:53:55.961513+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":53888,"num_objects":8,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5441,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":77824,"num_write_kb":263452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5921,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084460,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":726332,"kb_used_data":484,"kb_used_omap":4,"kb_used_meta":725819,"kb_avail":93645508,"statfs":{"total":96636764160,"available":95893000192,"internally_reserved":0,"allocated":495616,"data_stored":187895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":743239379},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986337,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":618052,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":617531,"kb_avail":93753788,"statfs":{"total":96636764160,"available":96003878912,"internally_reserved":0,"allocated":512000,"data_stored":204279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":632352466},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673143,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713340,"kb_used_data":484,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":93658500,"statfs":{"total":96636764160,"available":95906304000,"internally_reserved":0,"allocated":495616,"data_stored":187895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":729938373},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182574,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713788,"kb_used_data":484,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658052,"statfs":{"total":96636764160,"available":95905845248,"internally_reserved":0,"allocated":495616,"data_stored":187895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":730394317},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":20480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:48:26.237 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:48:26.237 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "39'12770", 'reported_seq': 12917, 'reported_epoch': 59, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:48:16.432658+0000', 'last_change': '2024-09-05T16:34:56.974151+0000', 'last_active': '2024-09-05T16:48:16.432658+0000', 'last_peered': '2024-09-05T16:48:16.432658+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:48:16.432658+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "33'2700", 'ondisk_log_start': "33'2700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10070, 'log_dups_size': 0, 'ondisk_log_size': 10070, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26944, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 16, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12767, 'num_write_kb': 51081, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2240}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:48:26.237 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:48:26.237 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "39'13120", 'reported_seq': 15952, 'reported_epoch': 59, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:48:23.209534+0000', 'last_change': '2024-09-05T16:48:14.899129+0000', 'last_active': '2024-09-05T16:48:23.209534+0000', 'last_peered': '2024-09-05T16:48:23.209534+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:48:23.209534+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "33'3100", 'ondisk_log_start': "33'3100", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20208, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 955, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13117, 'num_write_kb': 52476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1362, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -952}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:48:26.237 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:48:26.237 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "39'12758", 'reported_seq': 12903, 'reported_epoch': 59, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:48:16.432928+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:48:16.432928+0000', 'last_peered': '2024-09-05T16:48:16.432928+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:48:16.432928+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "33'2700", 'ondisk_log_start': "33'2700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10058, 'log_dups_size': 0, 'ondisk_log_size': 10058, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6736, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12757, 'num_write_kb': 51031, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2237}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:48:26.238 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:48:26.640 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:48:26.640 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 5441/32 objects degraded (17003.125%), 3 pgs degraded, 3 pgs undersized","count":6},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":909,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":59,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":13},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":8,"data_bytes":53888,"bytes_used":2838028288,"bytes_avail":383709028352,"bytes_total":386547056640,"degraded_objects":5441,"degraded_total":32,"degraded_ratio":170.03125},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:48:26.653 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-09-05T16:48:26.653 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:48:26.963 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:48:26.964 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:48:26.976 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":504,"stamp":"2024-09-05T16:48:25.117347+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":53888,"num_objects":8,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5441,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":77824,"num_write_kb":263452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5921,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2771512,"kb_used_data":1952,"kb_used_omap":14,"kb_used_meta":2769457,"kb_avail":374715848,"statfs":{"total":386547056640,"available":383709028352,"internally_reserved":0,"allocated":1998848,"data_stored":767964,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2835924535},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13472,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1712,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-13122,"num_write_kb":-52500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-649,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.687857"},"pg_stats":[{"pgid":"1.f","version":"39'12904","reported_seq":17537,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-05T16:48:16.433370+0000","last_change":"2024-09-05T16:46:27.826222+0000","last_active":"2024-09-05T16:48:16.433370+0000","last_peered":"2024-09-05T16:48:16.433370+0000","last_clean":"2024-09-05T16:48:16.433370+0000","last_became_active":"2024-09-05T16:46:27.757928+0000","last_became_peered":"2024-09-05T16:46:27.757928+0000","last_unstale":"2024-09-05T16:48:16.433370+0000","last_undegraded":"2024-09-05T16:48:16.433370+0000","last_fullsized":"2024-09-05T16:48:16.433370+0000","mapping_epoch":40,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:32:35.713428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12904,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2262,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12552","reported_seq":17250,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-05T16:48:16.430268+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:48:16.430268+0000","last_peered":"2024-09-05T16:48:16.430268+0000","last_clean":"2024-09-05T16:48:16.430268+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:48:16.430268+0000","last_undegraded":"2024-09-05T16:48:16.430268+0000","last_fullsized":"2024-09-05T16:48:16.430268+0000","mapping_epoch":38,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1865,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12752","reported_seq":17405,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-05T16:48:16.432786+0000","last_change":"2024-09-05T16:46:51.992073+0000","last_active":"2024-09-05T16:48:16.432786+0000","last_peered":"2024-09-05T16:48:16.432786+0000","last_clean":"2024-09-05T16:48:16.432786+0000","last_became_active":"2024-09-05T16:46:51.667128+0000","last_became_peered":"2024-09-05T16:46:51.667128+0000","last_unstale":"2024-09-05T16:48:16.432786+0000","last_undegraded":"2024-09-05T16:48:16.432786+0000","last_fullsized":"2024-09-05T16:48:16.432786+0000","mapping_epoch":44,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12752","last_scrub_stamp":"2024-09-05T16:46:51.991995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:51.991995+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:04:54.120031+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12934","reported_seq":17710,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-05T16:48:16.429984+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:48:16.429984+0000","last_peered":"2024-09-05T16:48:16.429984+0000","last_clean":"2024-09-05T16:48:16.429984+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:48:16.429984+0000","last_undegraded":"2024-09-05T16:48:16.429984+0000","last_fullsized":"2024-09-05T16:48:16.429984+0000","mapping_epoch":34,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7234,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12593","reported_seq":17050,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-05T16:48:16.432562+0000","last_change":"2024-09-05T16:47:51.525026+0000","last_active":"2024-09-05T16:48:16.432562+0000","last_peered":"2024-09-05T16:48:16.432562+0000","last_clean":"2024-09-05T16:48:16.432562+0000","last_became_active":"2024-09-05T16:47:51.193016+0000","last_became_peered":"2024-09-05T16:47:51.193016+0000","last_unstale":"2024-09-05T16:48:16.432562+0000","last_undegraded":"2024-09-05T16:48:16.432562+0000","last_fullsized":"2024-09-05T16:48:16.432562+0000","mapping_epoch":54,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12593","last_scrub_stamp":"2024-09-05T16:47:51.524957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:51.524957+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:46:22.846859+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12772","reported_seq":17381,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-05T16:48:16.432461+0000","last_change":"2024-09-05T16:47:15.832530+0000","last_active":"2024-09-05T16:48:16.432461+0000","last_peered":"2024-09-05T16:48:16.432461+0000","last_clean":"2024-09-05T16:48:16.432461+0000","last_became_active":"2024-09-05T16:47:15.728759+0000","last_became_peered":"2024-09-05T16:47:15.728759+0000","last_unstale":"2024-09-05T16:48:16.432461+0000","last_undegraded":"2024-09-05T16:48:16.432461+0000","last_fullsized":"2024-09-05T16:48:16.432461+0000","mapping_epoch":48,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T02:39:11.515339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12772,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2250,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12660","reported_seq":17098,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-05T16:48:16.432380+0000","last_change":"2024-09-05T16:47:03.683348+0000","last_active":"2024-09-05T16:48:16.432380+0000","last_peered":"2024-09-05T16:48:16.432380+0000","last_clean":"2024-09-05T16:48:16.432380+0000","last_became_active":"2024-09-05T16:47:03.185143+0000","last_became_peered":"2024-09-05T16:47:03.185143+0000","last_unstale":"2024-09-05T16:48:16.432380+0000","last_undegraded":"2024-09-05T16:48:16.432380+0000","last_fullsized":"2024-09-05T16:48:16.432380+0000","mapping_epoch":46,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12660","last_scrub_stamp":"2024-09-05T16:47:03.683262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:03.683262+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:42:16.533299+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12777","reported_seq":12926,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-05T16:48:16.430157+0000","last_change":"2024-09-05T16:46:40.458315+0000","last_active":"2024-09-05T16:48:16.430157+0000","last_peered":"2024-09-05T16:48:16.430157+0000","last_clean":"2024-09-05T16:48:16.430157+0000","last_became_active":"2024-09-05T16:46:40.184288+0000","last_became_peered":"2024-09-05T16:46:40.184288+0000","last_unstale":"2024-09-05T16:48:16.430157+0000","last_undegraded":"2024-09-05T16:48:16.430157+0000","last_fullsized":"2024-09-05T16:48:16.430157+0000","mapping_epoch":42,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12777","last_scrub_stamp":"2024-09-05T16:46:40.458226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:40.458226+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:38:58.734210+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'12770","reported_seq":12917,"reported_epoch":59,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:48:16.432658+0000","last_change":"2024-09-05T16:34:56.974151+0000","last_active":"2024-09-05T16:48:16.432658+0000","last_peered":"2024-09-05T16:48:16.432658+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:48:16.432658+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12767,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2240}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12808","reported_seq":17488,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-05T16:48:16.432883+0000","last_change":"2024-09-05T16:48:04.288466+0000","last_active":"2024-09-05T16:48:16.432883+0000","last_peered":"2024-09-05T16:48:16.432883+0000","last_clean":"2024-09-05T16:48:16.432883+0000","last_became_active":"2024-09-05T16:48:03.239894+0000","last_became_peered":"2024-09-05T16:48:03.239894+0000","last_unstale":"2024-09-05T16:48:16.432883+0000","last_undegraded":"2024-09-05T16:48:16.432883+0000","last_fullsized":"2024-09-05T16:48:16.432883+0000","mapping_epoch":56,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12808","last_scrub_stamp":"2024-09-05T16:48:04.288388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:48:04.288388+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T21:11:32.044919+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13120","reported_seq":15952,"reported_epoch":59,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:48:23.209534+0000","last_change":"2024-09-05T16:48:14.899129+0000","last_active":"2024-09-05T16:48:23.209534+0000","last_peered":"2024-09-05T16:48:23.209534+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:48:23.209534+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13117,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1362,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-952}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12758","reported_seq":12903,"reported_epoch":59,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:48:16.432928+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:48:16.432928+0000","last_peered":"2024-09-05T16:48:16.432928+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:48:16.432928+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2237}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12808","reported_seq":17590,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-05T16:48:16.429984+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:48:16.429984+0000","last_peered":"2024-09-05T16:48:16.429984+0000","last_clean":"2024-09-05T16:48:16.429984+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:48:16.429984+0000","last_undegraded":"2024-09-05T16:48:16.429984+0000","last_fullsized":"2024-09-05T16:48:16.429984+0000","mapping_epoch":36,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4408,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13124","reported_seq":13261,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-05T16:48:16.582230+0000","last_change":"2024-09-05T16:48:16.582142+0000","last_active":"2024-09-05T16:48:16.582230+0000","last_peered":"2024-09-05T16:48:16.582230+0000","last_clean":"2024-09-05T16:48:16.582230+0000","last_became_active":"2024-09-05T16:48:16.435091+0000","last_became_peered":"2024-09-05T16:48:16.435091+0000","last_unstale":"2024-09-05T16:48:16.582230+0000","last_undegraded":"2024-09-05T16:48:16.582230+0000","last_fullsized":"2024-09-05T16:48:16.582230+0000","mapping_epoch":58,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'13124","last_scrub_stamp":"2024-09-05T16:48:16.582038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:48:16.582038+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T17:44:25.891565+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"39'12831","reported_seq":12984,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-05T16:48:16.430173+0000","last_change":"2024-09-05T16:47:40.451380+0000","last_active":"2024-09-05T16:48:16.430173+0000","last_peered":"2024-09-05T16:48:16.430173+0000","last_clean":"2024-09-05T16:48:16.430173+0000","last_became_active":"2024-09-05T16:47:40.189447+0000","last_became_peered":"2024-09-05T16:47:40.189447+0000","last_unstale":"2024-09-05T16:48:16.430173+0000","last_undegraded":"2024-09-05T16:48:16.430173+0000","last_fullsized":"2024-09-05T16:48:16.430173+0000","mapping_epoch":52,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12831","last_scrub_stamp":"2024-09-05T16:47:40.451295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:40.451295+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:46:20.946333+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12773","reported_seq":12926,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-05T16:48:16.430339+0000","last_change":"2024-09-05T16:47:27.437275+0000","last_active":"2024-09-05T16:48:16.430339+0000","last_peered":"2024-09-05T16:48:16.430339+0000","last_clean":"2024-09-05T16:48:16.430339+0000","last_became_active":"2024-09-05T16:47:27.204989+0000","last_became_peered":"2024-09-05T16:47:27.204989+0000","last_unstale":"2024-09-05T16:48:16.430339+0000","last_undegraded":"2024-09-05T16:48:16.430339+0000","last_fullsized":"2024-09-05T16:48:16.430339+0000","mapping_epoch":50,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12773","last_scrub_stamp":"2024-09-05T16:47:27.437187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:27.437187+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T16:53:55.961513+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":53888,"num_objects":8,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5441,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":77824,"num_write_kb":263452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5921,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084460,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":726332,"kb_used_data":484,"kb_used_omap":4,"kb_used_meta":725819,"kb_avail":93645508,"statfs":{"total":96636764160,"available":95893000192,"internally_reserved":0,"allocated":495616,"data_stored":187895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":743239379},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986337,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":618052,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":617531,"kb_avail":93753788,"statfs":{"total":96636764160,"available":96003878912,"internally_reserved":0,"allocated":512000,"data_stored":204279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":632352466},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673143,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713340,"kb_used_data":484,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":93658500,"statfs":{"total":96636764160,"available":95906304000,"internally_reserved":0,"allocated":495616,"data_stored":187895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":729938373},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182574,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713788,"kb_used_data":484,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658052,"statfs":{"total":96636764160,"available":95905845248,"internally_reserved":0,"allocated":495616,"data_stored":187895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":730394317},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":20480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:48:29.978 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:48:30.274 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:48:30.275 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:48:30.287 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":508,"stamp":"2024-09-05T16:48:29.228540+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":53888,"num_objects":8,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4377,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":77824,"num_write_kb":263452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6985,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2763180,"kb_used_data":1940,"kb_used_omap":14,"kb_used_meta":2761137,"kb_avail":374724180,"statfs":{"total":386547056640,"available":383717560320,"internally_reserved":0,"allocated":1986560,"data_stored":755676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2827404855},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1930,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1930,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.111801"},"pg_stats":[{"pgid":"1.f","version":"39'12904","reported_seq":17537,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-05T16:48:16.433370+0000","last_change":"2024-09-05T16:46:27.826222+0000","last_active":"2024-09-05T16:48:16.433370+0000","last_peered":"2024-09-05T16:48:16.433370+0000","last_clean":"2024-09-05T16:48:16.433370+0000","last_became_active":"2024-09-05T16:46:27.757928+0000","last_became_peered":"2024-09-05T16:46:27.757928+0000","last_unstale":"2024-09-05T16:48:16.433370+0000","last_undegraded":"2024-09-05T16:48:16.433370+0000","last_fullsized":"2024-09-05T16:48:16.433370+0000","mapping_epoch":40,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:32:35.713428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12904,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2262,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12552","reported_seq":17250,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-05T16:48:16.430268+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:48:16.430268+0000","last_peered":"2024-09-05T16:48:16.430268+0000","last_clean":"2024-09-05T16:48:16.430268+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:48:16.430268+0000","last_undegraded":"2024-09-05T16:48:16.430268+0000","last_fullsized":"2024-09-05T16:48:16.430268+0000","mapping_epoch":38,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1865,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12752","reported_seq":17405,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-05T16:48:16.432786+0000","last_change":"2024-09-05T16:46:51.992073+0000","last_active":"2024-09-05T16:48:16.432786+0000","last_peered":"2024-09-05T16:48:16.432786+0000","last_clean":"2024-09-05T16:48:16.432786+0000","last_became_active":"2024-09-05T16:46:51.667128+0000","last_became_peered":"2024-09-05T16:46:51.667128+0000","last_unstale":"2024-09-05T16:48:16.432786+0000","last_undegraded":"2024-09-05T16:48:16.432786+0000","last_fullsized":"2024-09-05T16:48:16.432786+0000","mapping_epoch":44,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12752","last_scrub_stamp":"2024-09-05T16:46:51.991995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:51.991995+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:04:54.120031+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12934","reported_seq":17710,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-05T16:48:16.429984+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:48:16.429984+0000","last_peered":"2024-09-05T16:48:16.429984+0000","last_clean":"2024-09-05T16:48:16.429984+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:48:16.429984+0000","last_undegraded":"2024-09-05T16:48:16.429984+0000","last_fullsized":"2024-09-05T16:48:16.429984+0000","mapping_epoch":34,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7234,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12593","reported_seq":17050,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-05T16:48:16.432562+0000","last_change":"2024-09-05T16:47:51.525026+0000","last_active":"2024-09-05T16:48:16.432562+0000","last_peered":"2024-09-05T16:48:16.432562+0000","last_clean":"2024-09-05T16:48:16.432562+0000","last_became_active":"2024-09-05T16:47:51.193016+0000","last_became_peered":"2024-09-05T16:47:51.193016+0000","last_unstale":"2024-09-05T16:48:16.432562+0000","last_undegraded":"2024-09-05T16:48:16.432562+0000","last_fullsized":"2024-09-05T16:48:16.432562+0000","mapping_epoch":54,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12593","last_scrub_stamp":"2024-09-05T16:47:51.524957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:51.524957+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:46:22.846859+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12772","reported_seq":17381,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-05T16:48:16.432461+0000","last_change":"2024-09-05T16:47:15.832530+0000","last_active":"2024-09-05T16:48:16.432461+0000","last_peered":"2024-09-05T16:48:16.432461+0000","last_clean":"2024-09-05T16:48:16.432461+0000","last_became_active":"2024-09-05T16:47:15.728759+0000","last_became_peered":"2024-09-05T16:47:15.728759+0000","last_unstale":"2024-09-05T16:48:16.432461+0000","last_undegraded":"2024-09-05T16:48:16.432461+0000","last_fullsized":"2024-09-05T16:48:16.432461+0000","mapping_epoch":48,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T02:39:11.515339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12772,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2250,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12660","reported_seq":17098,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-05T16:48:16.432380+0000","last_change":"2024-09-05T16:47:03.683348+0000","last_active":"2024-09-05T16:48:16.432380+0000","last_peered":"2024-09-05T16:48:16.432380+0000","last_clean":"2024-09-05T16:48:16.432380+0000","last_became_active":"2024-09-05T16:47:03.185143+0000","last_became_peered":"2024-09-05T16:47:03.185143+0000","last_unstale":"2024-09-05T16:48:16.432380+0000","last_undegraded":"2024-09-05T16:48:16.432380+0000","last_fullsized":"2024-09-05T16:48:16.432380+0000","mapping_epoch":46,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12660","last_scrub_stamp":"2024-09-05T16:47:03.683262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:03.683262+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:42:16.533299+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12777","reported_seq":12926,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-05T16:48:16.430157+0000","last_change":"2024-09-05T16:46:40.458315+0000","last_active":"2024-09-05T16:48:16.430157+0000","last_peered":"2024-09-05T16:48:16.430157+0000","last_clean":"2024-09-05T16:48:16.430157+0000","last_became_active":"2024-09-05T16:46:40.184288+0000","last_became_peered":"2024-09-05T16:46:40.184288+0000","last_unstale":"2024-09-05T16:48:16.430157+0000","last_undegraded":"2024-09-05T16:48:16.430157+0000","last_fullsized":"2024-09-05T16:48:16.430157+0000","mapping_epoch":42,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12777","last_scrub_stamp":"2024-09-05T16:46:40.458226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:40.458226+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:38:58.734210+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'12770","reported_seq":13136,"reported_epoch":59,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:48:28.213071+0000","last_change":"2024-09-05T16:48:27.426520+0000","last_active":"2024-09-05T16:48:28.213071+0000","last_peered":"2024-09-05T16:48:28.213071+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:48:28.213071+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12767,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":129,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2134}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12808","reported_seq":17489,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-05T16:48:16.432919+0000","last_change":"2024-09-05T16:48:04.288466+0000","last_active":"2024-09-05T16:48:16.432919+0000","last_peered":"2024-09-05T16:48:16.432919+0000","last_clean":"2024-09-05T16:48:16.432919+0000","last_became_active":"2024-09-05T16:48:03.239894+0000","last_became_peered":"2024-09-05T16:48:03.239894+0000","last_unstale":"2024-09-05T16:48:16.432919+0000","last_undegraded":"2024-09-05T16:48:16.432919+0000","last_fullsized":"2024-09-05T16:48:16.432919+0000","mapping_epoch":56,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12808","last_scrub_stamp":"2024-09-05T16:48:04.288388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:48:04.288388+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T21:11:32.044919+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13120","reported_seq":17863,"reported_epoch":59,"state":"active+undersized+remapped","last_fresh":"2024-09-05T16:48:27.425432+0000","last_change":"2024-09-05T16:48:27.425639+0000","last_active":"2024-09-05T16:48:27.425432+0000","last_peered":"2024-09-05T16:48:27.425432+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:48:27.425432+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13117,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2317,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","1(2)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12758","reported_seq":12903,"reported_epoch":59,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:48:16.432928+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:48:16.432928+0000","last_peered":"2024-09-05T16:48:16.432928+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:48:16.432928+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2237}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12808","reported_seq":17590,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-05T16:48:16.429984+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:48:16.429984+0000","last_peered":"2024-09-05T16:48:16.429984+0000","last_clean":"2024-09-05T16:48:16.429984+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:48:16.429984+0000","last_undegraded":"2024-09-05T16:48:16.429984+0000","last_fullsized":"2024-09-05T16:48:16.429984+0000","mapping_epoch":36,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4408,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13124","reported_seq":13261,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-05T16:48:16.582230+0000","last_change":"2024-09-05T16:48:16.582142+0000","last_active":"2024-09-05T16:48:16.582230+0000","last_peered":"2024-09-05T16:48:16.582230+0000","last_clean":"2024-09-05T16:48:16.582230+0000","last_became_active":"2024-09-05T16:48:16.435091+0000","last_became_peered":"2024-09-05T16:48:16.435091+0000","last_unstale":"2024-09-05T16:48:16.582230+0000","last_undegraded":"2024-09-05T16:48:16.582230+0000","last_fullsized":"2024-09-05T16:48:16.582230+0000","mapping_epoch":58,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'13124","last_scrub_stamp":"2024-09-05T16:48:16.582038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:48:16.582038+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T17:44:25.891565+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"39'12831","reported_seq":12984,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-05T16:48:16.430173+0000","last_change":"2024-09-05T16:47:40.451380+0000","last_active":"2024-09-05T16:48:16.430173+0000","last_peered":"2024-09-05T16:48:16.430173+0000","last_clean":"2024-09-05T16:48:16.430173+0000","last_became_active":"2024-09-05T16:47:40.189447+0000","last_became_peered":"2024-09-05T16:47:40.189447+0000","last_unstale":"2024-09-05T16:48:16.430173+0000","last_undegraded":"2024-09-05T16:48:16.430173+0000","last_fullsized":"2024-09-05T16:48:16.430173+0000","mapping_epoch":52,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12831","last_scrub_stamp":"2024-09-05T16:47:40.451295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:40.451295+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:46:20.946333+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12773","reported_seq":12926,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-05T16:48:16.430339+0000","last_change":"2024-09-05T16:47:27.437275+0000","last_active":"2024-09-05T16:48:16.430339+0000","last_peered":"2024-09-05T16:48:16.430339+0000","last_clean":"2024-09-05T16:48:16.430339+0000","last_became_active":"2024-09-05T16:47:27.204989+0000","last_became_peered":"2024-09-05T16:47:27.204989+0000","last_unstale":"2024-09-05T16:48:16.430339+0000","last_undegraded":"2024-09-05T16:48:16.430339+0000","last_fullsized":"2024-09-05T16:48:16.430339+0000","mapping_epoch":50,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12773","last_scrub_stamp":"2024-09-05T16:47:27.437187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:27.437187+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T16:53:55.961513+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":53888,"num_objects":8,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4377,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":77824,"num_write_kb":263452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6985,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":20480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084461,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":726332,"kb_used_data":484,"kb_used_omap":4,"kb_used_meta":725819,"kb_avail":93645508,"statfs":{"total":96636764160,"available":95893000192,"internally_reserved":0,"allocated":495616,"data_stored":187895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":743239379},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986338,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":609720,"kb_used_data":488,"kb_used_omap":4,"kb_used_meta":609211,"kb_avail":93762120,"statfs":{"total":96636764160,"available":96012410880,"internally_reserved":0,"allocated":499712,"data_stored":191991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":623832786},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673144,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713340,"kb_used_data":484,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":93658500,"statfs":{"total":96636764160,"available":95906304000,"internally_reserved":0,"allocated":495616,"data_stored":187895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":729938373},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182575,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713788,"kb_used_data":484,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658052,"statfs":{"total":96636764160,"available":95905845248,"internally_reserved":0,"allocated":495616,"data_stored":187895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":730394317},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:48:30.288 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:48:30.598 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:48:30.599 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:48:30.611 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":508,"stamp":"2024-09-05T16:48:29.228540+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":53888,"num_objects":8,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4377,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":77824,"num_write_kb":263452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6985,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2763180,"kb_used_data":1940,"kb_used_omap":14,"kb_used_meta":2761137,"kb_avail":374724180,"statfs":{"total":386547056640,"available":383717560320,"internally_reserved":0,"allocated":1986560,"data_stored":755676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2827404855},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1930,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1930,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.111801"},"pg_stats":[{"pgid":"1.f","version":"39'12904","reported_seq":17537,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-05T16:48:16.433370+0000","last_change":"2024-09-05T16:46:27.826222+0000","last_active":"2024-09-05T16:48:16.433370+0000","last_peered":"2024-09-05T16:48:16.433370+0000","last_clean":"2024-09-05T16:48:16.433370+0000","last_became_active":"2024-09-05T16:46:27.757928+0000","last_became_peered":"2024-09-05T16:46:27.757928+0000","last_unstale":"2024-09-05T16:48:16.433370+0000","last_undegraded":"2024-09-05T16:48:16.433370+0000","last_fullsized":"2024-09-05T16:48:16.433370+0000","mapping_epoch":40,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:32:35.713428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12904,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2262,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12552","reported_seq":17250,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-05T16:48:16.430268+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:48:16.430268+0000","last_peered":"2024-09-05T16:48:16.430268+0000","last_clean":"2024-09-05T16:48:16.430268+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:48:16.430268+0000","last_undegraded":"2024-09-05T16:48:16.430268+0000","last_fullsized":"2024-09-05T16:48:16.430268+0000","mapping_epoch":38,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1865,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12752","reported_seq":17405,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-05T16:48:16.432786+0000","last_change":"2024-09-05T16:46:51.992073+0000","last_active":"2024-09-05T16:48:16.432786+0000","last_peered":"2024-09-05T16:48:16.432786+0000","last_clean":"2024-09-05T16:48:16.432786+0000","last_became_active":"2024-09-05T16:46:51.667128+0000","last_became_peered":"2024-09-05T16:46:51.667128+0000","last_unstale":"2024-09-05T16:48:16.432786+0000","last_undegraded":"2024-09-05T16:48:16.432786+0000","last_fullsized":"2024-09-05T16:48:16.432786+0000","mapping_epoch":44,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12752","last_scrub_stamp":"2024-09-05T16:46:51.991995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:51.991995+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:04:54.120031+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12934","reported_seq":17710,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-05T16:48:16.429984+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:48:16.429984+0000","last_peered":"2024-09-05T16:48:16.429984+0000","last_clean":"2024-09-05T16:48:16.429984+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:48:16.429984+0000","last_undegraded":"2024-09-05T16:48:16.429984+0000","last_fullsized":"2024-09-05T16:48:16.429984+0000","mapping_epoch":34,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7234,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12593","reported_seq":17050,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-05T16:48:16.432562+0000","last_change":"2024-09-05T16:47:51.525026+0000","last_active":"2024-09-05T16:48:16.432562+0000","last_peered":"2024-09-05T16:48:16.432562+0000","last_clean":"2024-09-05T16:48:16.432562+0000","last_became_active":"2024-09-05T16:47:51.193016+0000","last_became_peered":"2024-09-05T16:47:51.193016+0000","last_unstale":"2024-09-05T16:48:16.432562+0000","last_undegraded":"2024-09-05T16:48:16.432562+0000","last_fullsized":"2024-09-05T16:48:16.432562+0000","mapping_epoch":54,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12593","last_scrub_stamp":"2024-09-05T16:47:51.524957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:51.524957+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:46:22.846859+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12772","reported_seq":17381,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-05T16:48:16.432461+0000","last_change":"2024-09-05T16:47:15.832530+0000","last_active":"2024-09-05T16:48:16.432461+0000","last_peered":"2024-09-05T16:48:16.432461+0000","last_clean":"2024-09-05T16:48:16.432461+0000","last_became_active":"2024-09-05T16:47:15.728759+0000","last_became_peered":"2024-09-05T16:47:15.728759+0000","last_unstale":"2024-09-05T16:48:16.432461+0000","last_undegraded":"2024-09-05T16:48:16.432461+0000","last_fullsized":"2024-09-05T16:48:16.432461+0000","mapping_epoch":48,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T02:39:11.515339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12772,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2250,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12660","reported_seq":17098,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-05T16:48:16.432380+0000","last_change":"2024-09-05T16:47:03.683348+0000","last_active":"2024-09-05T16:48:16.432380+0000","last_peered":"2024-09-05T16:48:16.432380+0000","last_clean":"2024-09-05T16:48:16.432380+0000","last_became_active":"2024-09-05T16:47:03.185143+0000","last_became_peered":"2024-09-05T16:47:03.185143+0000","last_unstale":"2024-09-05T16:48:16.432380+0000","last_undegraded":"2024-09-05T16:48:16.432380+0000","last_fullsized":"2024-09-05T16:48:16.432380+0000","mapping_epoch":46,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12660","last_scrub_stamp":"2024-09-05T16:47:03.683262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:03.683262+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:42:16.533299+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12777","reported_seq":12926,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-05T16:48:16.430157+0000","last_change":"2024-09-05T16:46:40.458315+0000","last_active":"2024-09-05T16:48:16.430157+0000","last_peered":"2024-09-05T16:48:16.430157+0000","last_clean":"2024-09-05T16:48:16.430157+0000","last_became_active":"2024-09-05T16:46:40.184288+0000","last_became_peered":"2024-09-05T16:46:40.184288+0000","last_unstale":"2024-09-05T16:48:16.430157+0000","last_undegraded":"2024-09-05T16:48:16.430157+0000","last_fullsized":"2024-09-05T16:48:16.430157+0000","mapping_epoch":42,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12777","last_scrub_stamp":"2024-09-05T16:46:40.458226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:40.458226+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:38:58.734210+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'12770","reported_seq":13136,"reported_epoch":59,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:48:28.213071+0000","last_change":"2024-09-05T16:48:27.426520+0000","last_active":"2024-09-05T16:48:28.213071+0000","last_peered":"2024-09-05T16:48:28.213071+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:48:28.213071+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12767,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":129,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2134}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12808","reported_seq":17489,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-05T16:48:16.432919+0000","last_change":"2024-09-05T16:48:04.288466+0000","last_active":"2024-09-05T16:48:16.432919+0000","last_peered":"2024-09-05T16:48:16.432919+0000","last_clean":"2024-09-05T16:48:16.432919+0000","last_became_active":"2024-09-05T16:48:03.239894+0000","last_became_peered":"2024-09-05T16:48:03.239894+0000","last_unstale":"2024-09-05T16:48:16.432919+0000","last_undegraded":"2024-09-05T16:48:16.432919+0000","last_fullsized":"2024-09-05T16:48:16.432919+0000","mapping_epoch":56,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12808","last_scrub_stamp":"2024-09-05T16:48:04.288388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:48:04.288388+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T21:11:32.044919+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13120","reported_seq":17863,"reported_epoch":59,"state":"active+undersized+remapped","last_fresh":"2024-09-05T16:48:27.425432+0000","last_change":"2024-09-05T16:48:27.425639+0000","last_active":"2024-09-05T16:48:27.425432+0000","last_peered":"2024-09-05T16:48:27.425432+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:48:27.425432+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13117,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2317,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","1(2)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12758","reported_seq":12903,"reported_epoch":59,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:48:16.432928+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:48:16.432928+0000","last_peered":"2024-09-05T16:48:16.432928+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:48:16.432928+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2237}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12808","reported_seq":17590,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-05T16:48:16.429984+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:48:16.429984+0000","last_peered":"2024-09-05T16:48:16.429984+0000","last_clean":"2024-09-05T16:48:16.429984+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:48:16.429984+0000","last_undegraded":"2024-09-05T16:48:16.429984+0000","last_fullsized":"2024-09-05T16:48:16.429984+0000","mapping_epoch":36,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4408,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13124","reported_seq":13261,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-05T16:48:16.582230+0000","last_change":"2024-09-05T16:48:16.582142+0000","last_active":"2024-09-05T16:48:16.582230+0000","last_peered":"2024-09-05T16:48:16.582230+0000","last_clean":"2024-09-05T16:48:16.582230+0000","last_became_active":"2024-09-05T16:48:16.435091+0000","last_became_peered":"2024-09-05T16:48:16.435091+0000","last_unstale":"2024-09-05T16:48:16.582230+0000","last_undegraded":"2024-09-05T16:48:16.582230+0000","last_fullsized":"2024-09-05T16:48:16.582230+0000","mapping_epoch":58,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'13124","last_scrub_stamp":"2024-09-05T16:48:16.582038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:48:16.582038+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T17:44:25.891565+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"39'12831","reported_seq":12984,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-05T16:48:16.430173+0000","last_change":"2024-09-05T16:47:40.451380+0000","last_active":"2024-09-05T16:48:16.430173+0000","last_peered":"2024-09-05T16:48:16.430173+0000","last_clean":"2024-09-05T16:48:16.430173+0000","last_became_active":"2024-09-05T16:47:40.189447+0000","last_became_peered":"2024-09-05T16:47:40.189447+0000","last_unstale":"2024-09-05T16:48:16.430173+0000","last_undegraded":"2024-09-05T16:48:16.430173+0000","last_fullsized":"2024-09-05T16:48:16.430173+0000","mapping_epoch":52,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12831","last_scrub_stamp":"2024-09-05T16:47:40.451295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:40.451295+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:46:20.946333+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12773","reported_seq":12926,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-05T16:48:16.430339+0000","last_change":"2024-09-05T16:47:27.437275+0000","last_active":"2024-09-05T16:48:16.430339+0000","last_peered":"2024-09-05T16:48:16.430339+0000","last_clean":"2024-09-05T16:48:16.430339+0000","last_became_active":"2024-09-05T16:47:27.204989+0000","last_became_peered":"2024-09-05T16:47:27.204989+0000","last_unstale":"2024-09-05T16:48:16.430339+0000","last_undegraded":"2024-09-05T16:48:16.430339+0000","last_fullsized":"2024-09-05T16:48:16.430339+0000","mapping_epoch":50,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12773","last_scrub_stamp":"2024-09-05T16:47:27.437187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:27.437187+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T16:53:55.961513+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":53888,"num_objects":8,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4377,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":77824,"num_write_kb":263452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6985,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":20480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084461,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":726332,"kb_used_data":484,"kb_used_omap":4,"kb_used_meta":725819,"kb_avail":93645508,"statfs":{"total":96636764160,"available":95893000192,"internally_reserved":0,"allocated":495616,"data_stored":187895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":743239379},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986338,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":609720,"kb_used_data":488,"kb_used_omap":4,"kb_used_meta":609211,"kb_avail":93762120,"statfs":{"total":96636764160,"available":96012410880,"internally_reserved":0,"allocated":499712,"data_stored":191991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":623832786},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673144,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713340,"kb_used_data":484,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":93658500,"statfs":{"total":96636764160,"available":95906304000,"internally_reserved":0,"allocated":495616,"data_stored":187895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":729938373},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182575,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713788,"kb_used_data":484,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658052,"statfs":{"total":96636764160,"available":95905845248,"internally_reserved":0,"allocated":495616,"data_stored":187895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":730394317},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:48:30.612 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:48:30.612 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "39'12770", 'reported_seq': 13136, 'reported_epoch': 59, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:48:28.213071+0000', 'last_change': '2024-09-05T16:48:27.426520+0000', 'last_active': '2024-09-05T16:48:28.213071+0000', 'last_peered': '2024-09-05T16:48:28.213071+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:48:28.213071+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "33'2700", 'ondisk_log_start': "33'2700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10070, 'log_dups_size': 0, 'ondisk_log_size': 10070, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26944, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 16, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2138, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12767, 'num_write_kb': 51081, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 129, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2134}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:48:30.612 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:48:30.613 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "39'13120", 'reported_seq': 17863, 'reported_epoch': 59, 'state': 'active+undersized+remapped', 'last_fresh': '2024-09-05T16:48:27.425432+0000', 'last_change': '2024-09-05T16:48:27.425639+0000', 'last_active': '2024-09-05T16:48:27.425432+0000', 'last_peered': '2024-09-05T16:48:27.425432+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:48:27.425432+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "33'3100", 'ondisk_log_start': "33'3100", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20208, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13117, 'num_write_kb': 52476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2317, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '1(2)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:48:30.613 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:48:30.613 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "39'12758", 'reported_seq': 12903, 'reported_epoch': 59, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:48:16.432928+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:48:16.432928+0000', 'last_peered': '2024-09-05T16:48:16.432928+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:48:16.432928+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "33'2700", 'ondisk_log_start': "33'2700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10058, 'log_dups_size': 0, 'ondisk_log_size': 10058, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6736, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12757, 'num_write_kb': 51031, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2237}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:48:30.613 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:48:30.987 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:48:30.987 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 4377/32 objects degraded (13678.125%), 2 pgs degraded, 3 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":914,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":61,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":2},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":13},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+undersized+remapped","count":1},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":8,"data_bytes":53888,"bytes_used":2829496320,"bytes_avail":383717560320,"bytes_total":386547056640,"degraded_objects":4377,"degraded_total":32,"degraded_ratio":136.78125,"recovering_objects_per_sec":192,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1930,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:48:31.000 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:48:31.000 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:48:31.314 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:48:31.314 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:48:31.327 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":509,"stamp":"2024-09-05T16:48:31.118358+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":53888,"num_objects":8,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4377,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":77824,"num_write_kb":263452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6985,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2763196,"kb_used_data":1956,"kb_used_omap":14,"kb_used_meta":2761137,"kb_avail":374724164,"statfs":{"total":386547056640,"available":383717543936,"internally_reserved":0,"allocated":2002944,"data_stored":761939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2827404855},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1930,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1930,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001316"},"pg_stats":[{"pgid":"1.f","version":"39'12904","reported_seq":17537,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-05T16:48:16.433370+0000","last_change":"2024-09-05T16:46:27.826222+0000","last_active":"2024-09-05T16:48:16.433370+0000","last_peered":"2024-09-05T16:48:16.433370+0000","last_clean":"2024-09-05T16:48:16.433370+0000","last_became_active":"2024-09-05T16:46:27.757928+0000","last_became_peered":"2024-09-05T16:46:27.757928+0000","last_unstale":"2024-09-05T16:48:16.433370+0000","last_undegraded":"2024-09-05T16:48:16.433370+0000","last_fullsized":"2024-09-05T16:48:16.433370+0000","mapping_epoch":40,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:32:35.713428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12904,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2262,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12552","reported_seq":17250,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-05T16:48:16.430268+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:48:16.430268+0000","last_peered":"2024-09-05T16:48:16.430268+0000","last_clean":"2024-09-05T16:48:16.430268+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:48:16.430268+0000","last_undegraded":"2024-09-05T16:48:16.430268+0000","last_fullsized":"2024-09-05T16:48:16.430268+0000","mapping_epoch":38,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1865,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12752","reported_seq":17409,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-05T16:48:29.233031+0000","last_change":"2024-09-05T16:46:51.992073+0000","last_active":"2024-09-05T16:48:29.233031+0000","last_peered":"2024-09-05T16:48:29.233031+0000","last_clean":"2024-09-05T16:48:29.233031+0000","last_became_active":"2024-09-05T16:46:51.667128+0000","last_became_peered":"2024-09-05T16:46:51.667128+0000","last_unstale":"2024-09-05T16:48:29.233031+0000","last_undegraded":"2024-09-05T16:48:29.233031+0000","last_fullsized":"2024-09-05T16:48:29.233031+0000","mapping_epoch":44,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12752","last_scrub_stamp":"2024-09-05T16:46:51.991995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:51.991995+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:04:54.120031+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12934","reported_seq":17710,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-05T16:48:16.429984+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:48:16.429984+0000","last_peered":"2024-09-05T16:48:16.429984+0000","last_clean":"2024-09-05T16:48:16.429984+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:48:16.429984+0000","last_undegraded":"2024-09-05T16:48:16.429984+0000","last_fullsized":"2024-09-05T16:48:16.429984+0000","mapping_epoch":34,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7234,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12593","reported_seq":17050,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-05T16:48:16.432562+0000","last_change":"2024-09-05T16:47:51.525026+0000","last_active":"2024-09-05T16:48:16.432562+0000","last_peered":"2024-09-05T16:48:16.432562+0000","last_clean":"2024-09-05T16:48:16.432562+0000","last_became_active":"2024-09-05T16:47:51.193016+0000","last_became_peered":"2024-09-05T16:47:51.193016+0000","last_unstale":"2024-09-05T16:48:16.432562+0000","last_undegraded":"2024-09-05T16:48:16.432562+0000","last_fullsized":"2024-09-05T16:48:16.432562+0000","mapping_epoch":54,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12593","last_scrub_stamp":"2024-09-05T16:47:51.524957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:51.524957+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:46:22.846859+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12772","reported_seq":17381,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-05T16:48:16.432461+0000","last_change":"2024-09-05T16:47:15.832530+0000","last_active":"2024-09-05T16:48:16.432461+0000","last_peered":"2024-09-05T16:48:16.432461+0000","last_clean":"2024-09-05T16:48:16.432461+0000","last_became_active":"2024-09-05T16:47:15.728759+0000","last_became_peered":"2024-09-05T16:47:15.728759+0000","last_unstale":"2024-09-05T16:48:16.432461+0000","last_undegraded":"2024-09-05T16:48:16.432461+0000","last_fullsized":"2024-09-05T16:48:16.432461+0000","mapping_epoch":48,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T02:39:11.515339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12772,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2250,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12660","reported_seq":17098,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-05T16:48:16.432380+0000","last_change":"2024-09-05T16:47:03.683348+0000","last_active":"2024-09-05T16:48:16.432380+0000","last_peered":"2024-09-05T16:48:16.432380+0000","last_clean":"2024-09-05T16:48:16.432380+0000","last_became_active":"2024-09-05T16:47:03.185143+0000","last_became_peered":"2024-09-05T16:47:03.185143+0000","last_unstale":"2024-09-05T16:48:16.432380+0000","last_undegraded":"2024-09-05T16:48:16.432380+0000","last_fullsized":"2024-09-05T16:48:16.432380+0000","mapping_epoch":46,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12660","last_scrub_stamp":"2024-09-05T16:47:03.683262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:03.683262+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:42:16.533299+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12777","reported_seq":12926,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-05T16:48:16.430157+0000","last_change":"2024-09-05T16:46:40.458315+0000","last_active":"2024-09-05T16:48:16.430157+0000","last_peered":"2024-09-05T16:48:16.430157+0000","last_clean":"2024-09-05T16:48:16.430157+0000","last_became_active":"2024-09-05T16:46:40.184288+0000","last_became_peered":"2024-09-05T16:46:40.184288+0000","last_unstale":"2024-09-05T16:48:16.430157+0000","last_undegraded":"2024-09-05T16:48:16.430157+0000","last_fullsized":"2024-09-05T16:48:16.430157+0000","mapping_epoch":42,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12777","last_scrub_stamp":"2024-09-05T16:46:40.458226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:40.458226+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:38:58.734210+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'12770","reported_seq":13136,"reported_epoch":59,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:48:28.213071+0000","last_change":"2024-09-05T16:48:27.426520+0000","last_active":"2024-09-05T16:48:28.213071+0000","last_peered":"2024-09-05T16:48:28.213071+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:48:28.213071+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12767,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":129,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2134}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12808","reported_seq":17493,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-05T16:48:29.233129+0000","last_change":"2024-09-05T16:48:04.288466+0000","last_active":"2024-09-05T16:48:29.233129+0000","last_peered":"2024-09-05T16:48:29.233129+0000","last_clean":"2024-09-05T16:48:29.233129+0000","last_became_active":"2024-09-05T16:48:03.239894+0000","last_became_peered":"2024-09-05T16:48:03.239894+0000","last_unstale":"2024-09-05T16:48:29.233129+0000","last_undegraded":"2024-09-05T16:48:29.233129+0000","last_fullsized":"2024-09-05T16:48:29.233129+0000","mapping_epoch":56,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12808","last_scrub_stamp":"2024-09-05T16:48:04.288388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:48:04.288388+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T21:11:32.044919+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13120","reported_seq":17863,"reported_epoch":59,"state":"active+undersized+remapped","last_fresh":"2024-09-05T16:48:27.425432+0000","last_change":"2024-09-05T16:48:27.425639+0000","last_active":"2024-09-05T16:48:27.425432+0000","last_peered":"2024-09-05T16:48:27.425432+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:48:27.425432+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13117,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2317,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","1(2)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12758","reported_seq":12903,"reported_epoch":59,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:48:16.432928+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:48:16.432928+0000","last_peered":"2024-09-05T16:48:16.432928+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:48:16.432928+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2237}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12808","reported_seq":17590,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-05T16:48:16.429984+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:48:16.429984+0000","last_peered":"2024-09-05T16:48:16.429984+0000","last_clean":"2024-09-05T16:48:16.429984+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:48:16.429984+0000","last_undegraded":"2024-09-05T16:48:16.429984+0000","last_fullsized":"2024-09-05T16:48:16.429984+0000","mapping_epoch":36,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4408,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13124","reported_seq":13261,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-05T16:48:16.582230+0000","last_change":"2024-09-05T16:48:16.582142+0000","last_active":"2024-09-05T16:48:16.582230+0000","last_peered":"2024-09-05T16:48:16.582230+0000","last_clean":"2024-09-05T16:48:16.582230+0000","last_became_active":"2024-09-05T16:48:16.435091+0000","last_became_peered":"2024-09-05T16:48:16.435091+0000","last_unstale":"2024-09-05T16:48:16.582230+0000","last_undegraded":"2024-09-05T16:48:16.582230+0000","last_fullsized":"2024-09-05T16:48:16.582230+0000","mapping_epoch":58,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'13124","last_scrub_stamp":"2024-09-05T16:48:16.582038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:48:16.582038+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T17:44:25.891565+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"39'12831","reported_seq":12984,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-05T16:48:16.430173+0000","last_change":"2024-09-05T16:47:40.451380+0000","last_active":"2024-09-05T16:48:16.430173+0000","last_peered":"2024-09-05T16:48:16.430173+0000","last_clean":"2024-09-05T16:48:16.430173+0000","last_became_active":"2024-09-05T16:47:40.189447+0000","last_became_peered":"2024-09-05T16:47:40.189447+0000","last_unstale":"2024-09-05T16:48:16.430173+0000","last_undegraded":"2024-09-05T16:48:16.430173+0000","last_fullsized":"2024-09-05T16:48:16.430173+0000","mapping_epoch":52,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12831","last_scrub_stamp":"2024-09-05T16:47:40.451295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:40.451295+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:46:20.946333+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12773","reported_seq":12926,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-05T16:48:16.430339+0000","last_change":"2024-09-05T16:47:27.437275+0000","last_active":"2024-09-05T16:48:16.430339+0000","last_peered":"2024-09-05T16:48:16.430339+0000","last_clean":"2024-09-05T16:48:16.430339+0000","last_became_active":"2024-09-05T16:47:27.204989+0000","last_became_peered":"2024-09-05T16:47:27.204989+0000","last_unstale":"2024-09-05T16:48:16.430339+0000","last_undegraded":"2024-09-05T16:48:16.430339+0000","last_fullsized":"2024-09-05T16:48:16.430339+0000","mapping_epoch":50,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12773","last_scrub_stamp":"2024-09-05T16:47:27.437187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:27.437187+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T16:53:55.961513+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":53888,"num_objects":8,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4377,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":77824,"num_write_kb":263452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6985,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":20480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084461,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":726332,"kb_used_data":484,"kb_used_omap":4,"kb_used_meta":725819,"kb_avail":93645508,"statfs":{"total":96636764160,"available":95893000192,"internally_reserved":0,"allocated":495616,"data_stored":187895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":743239379},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986338,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":609720,"kb_used_data":488,"kb_used_omap":4,"kb_used_meta":609211,"kb_avail":93762120,"statfs":{"total":96636764160,"available":96012410880,"internally_reserved":0,"allocated":499712,"data_stored":191991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":623832786},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673145,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713356,"kb_used_data":500,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":93658484,"statfs":{"total":96636764160,"available":95906287616,"internally_reserved":0,"allocated":512000,"data_stored":194158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":729938373},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182575,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713788,"kb_used_data":484,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658052,"statfs":{"total":96636764160,"available":95905845248,"internally_reserved":0,"allocated":495616,"data_stored":187895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":730394317},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:48:34.330 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:48:34.624 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:48:34.625 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:48:34.638 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":510,"stamp":"2024-09-05T16:48:33.118713+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":53888,"num_objects":8,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4377,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":77824,"num_write_kb":263452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6985,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2771544,"kb_used_data":1988,"kb_used_omap":14,"kb_used_meta":2769457,"kb_avail":374715816,"statfs":{"total":386547056640,"available":383708995584,"internally_reserved":0,"allocated":2035712,"data_stored":774465,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2835924535},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1064,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1064,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001365"},"pg_stats":[{"pgid":"1.f","version":"39'12904","reported_seq":17537,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-05T16:48:16.433370+0000","last_change":"2024-09-05T16:46:27.826222+0000","last_active":"2024-09-05T16:48:16.433370+0000","last_peered":"2024-09-05T16:48:16.433370+0000","last_clean":"2024-09-05T16:48:16.433370+0000","last_became_active":"2024-09-05T16:46:27.757928+0000","last_became_peered":"2024-09-05T16:46:27.757928+0000","last_unstale":"2024-09-05T16:48:16.433370+0000","last_undegraded":"2024-09-05T16:48:16.433370+0000","last_fullsized":"2024-09-05T16:48:16.433370+0000","mapping_epoch":40,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:32:35.713428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12904,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2262,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12552","reported_seq":17254,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-05T16:48:29.230646+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:48:29.230646+0000","last_peered":"2024-09-05T16:48:29.230646+0000","last_clean":"2024-09-05T16:48:29.230646+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:48:29.230646+0000","last_undegraded":"2024-09-05T16:48:29.230646+0000","last_fullsized":"2024-09-05T16:48:29.230646+0000","mapping_epoch":38,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1865,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12752","reported_seq":17409,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-05T16:48:29.233031+0000","last_change":"2024-09-05T16:46:51.992073+0000","last_active":"2024-09-05T16:48:29.233031+0000","last_peered":"2024-09-05T16:48:29.233031+0000","last_clean":"2024-09-05T16:48:29.233031+0000","last_became_active":"2024-09-05T16:46:51.667128+0000","last_became_peered":"2024-09-05T16:46:51.667128+0000","last_unstale":"2024-09-05T16:48:29.233031+0000","last_undegraded":"2024-09-05T16:48:29.233031+0000","last_fullsized":"2024-09-05T16:48:29.233031+0000","mapping_epoch":44,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12752","last_scrub_stamp":"2024-09-05T16:46:51.991995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:51.991995+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:04:54.120031+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12934","reported_seq":17714,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-05T16:48:29.230556+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:48:29.230556+0000","last_peered":"2024-09-05T16:48:29.230556+0000","last_clean":"2024-09-05T16:48:29.230556+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:48:29.230556+0000","last_undegraded":"2024-09-05T16:48:29.230556+0000","last_fullsized":"2024-09-05T16:48:29.230556+0000","mapping_epoch":34,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7234,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12593","reported_seq":17050,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-05T16:48:16.432562+0000","last_change":"2024-09-05T16:47:51.525026+0000","last_active":"2024-09-05T16:48:16.432562+0000","last_peered":"2024-09-05T16:48:16.432562+0000","last_clean":"2024-09-05T16:48:16.432562+0000","last_became_active":"2024-09-05T16:47:51.193016+0000","last_became_peered":"2024-09-05T16:47:51.193016+0000","last_unstale":"2024-09-05T16:48:16.432562+0000","last_undegraded":"2024-09-05T16:48:16.432562+0000","last_fullsized":"2024-09-05T16:48:16.432562+0000","mapping_epoch":54,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12593","last_scrub_stamp":"2024-09-05T16:47:51.524957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:51.524957+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:46:22.846859+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12772","reported_seq":17381,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-05T16:48:16.432461+0000","last_change":"2024-09-05T16:47:15.832530+0000","last_active":"2024-09-05T16:48:16.432461+0000","last_peered":"2024-09-05T16:48:16.432461+0000","last_clean":"2024-09-05T16:48:16.432461+0000","last_became_active":"2024-09-05T16:47:15.728759+0000","last_became_peered":"2024-09-05T16:47:15.728759+0000","last_unstale":"2024-09-05T16:48:16.432461+0000","last_undegraded":"2024-09-05T16:48:16.432461+0000","last_fullsized":"2024-09-05T16:48:16.432461+0000","mapping_epoch":48,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T02:39:11.515339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12772,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2250,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12660","reported_seq":17102,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-05T16:48:29.229181+0000","last_change":"2024-09-05T16:47:03.683348+0000","last_active":"2024-09-05T16:48:29.229181+0000","last_peered":"2024-09-05T16:48:29.229181+0000","last_clean":"2024-09-05T16:48:29.229181+0000","last_became_active":"2024-09-05T16:47:03.185143+0000","last_became_peered":"2024-09-05T16:47:03.185143+0000","last_unstale":"2024-09-05T16:48:29.229181+0000","last_undegraded":"2024-09-05T16:48:29.229181+0000","last_fullsized":"2024-09-05T16:48:29.229181+0000","mapping_epoch":46,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12660","last_scrub_stamp":"2024-09-05T16:47:03.683262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:03.683262+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:42:16.533299+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12777","reported_seq":12930,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-05T16:48:29.230218+0000","last_change":"2024-09-05T16:46:40.458315+0000","last_active":"2024-09-05T16:48:29.230218+0000","last_peered":"2024-09-05T16:48:29.230218+0000","last_clean":"2024-09-05T16:48:29.230218+0000","last_became_active":"2024-09-05T16:46:40.184288+0000","last_became_peered":"2024-09-05T16:46:40.184288+0000","last_unstale":"2024-09-05T16:48:29.230218+0000","last_undegraded":"2024-09-05T16:48:29.230218+0000","last_fullsized":"2024-09-05T16:48:29.230218+0000","mapping_epoch":42,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12777","last_scrub_stamp":"2024-09-05T16:46:40.458226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:40.458226+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:38:58.734210+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'12770","reported_seq":13136,"reported_epoch":59,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:48:28.213071+0000","last_change":"2024-09-05T16:48:27.426520+0000","last_active":"2024-09-05T16:48:28.213071+0000","last_peered":"2024-09-05T16:48:28.213071+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:48:28.213071+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12767,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":129,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2134}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12808","reported_seq":17493,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-05T16:48:29.233129+0000","last_change":"2024-09-05T16:48:04.288466+0000","last_active":"2024-09-05T16:48:29.233129+0000","last_peered":"2024-09-05T16:48:29.233129+0000","last_clean":"2024-09-05T16:48:29.233129+0000","last_became_active":"2024-09-05T16:48:03.239894+0000","last_became_peered":"2024-09-05T16:48:03.239894+0000","last_unstale":"2024-09-05T16:48:29.233129+0000","last_undegraded":"2024-09-05T16:48:29.233129+0000","last_fullsized":"2024-09-05T16:48:29.233129+0000","mapping_epoch":56,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12808","last_scrub_stamp":"2024-09-05T16:48:04.288388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:48:04.288388+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T21:11:32.044919+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13120","reported_seq":17863,"reported_epoch":59,"state":"active+undersized+remapped","last_fresh":"2024-09-05T16:48:27.425432+0000","last_change":"2024-09-05T16:48:27.425639+0000","last_active":"2024-09-05T16:48:27.425432+0000","last_peered":"2024-09-05T16:48:27.425432+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:48:27.425432+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13117,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2317,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","1(2)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12758","reported_seq":12903,"reported_epoch":59,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:48:16.432928+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:48:16.432928+0000","last_peered":"2024-09-05T16:48:16.432928+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:48:16.432928+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2237}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12808","reported_seq":17594,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-05T16:48:29.230528+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:48:29.230528+0000","last_peered":"2024-09-05T16:48:29.230528+0000","last_clean":"2024-09-05T16:48:29.230528+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:48:29.230528+0000","last_undegraded":"2024-09-05T16:48:29.230528+0000","last_fullsized":"2024-09-05T16:48:29.230528+0000","mapping_epoch":36,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4408,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13124","reported_seq":13265,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-05T16:48:29.230184+0000","last_change":"2024-09-05T16:48:16.582142+0000","last_active":"2024-09-05T16:48:29.230184+0000","last_peered":"2024-09-05T16:48:29.230184+0000","last_clean":"2024-09-05T16:48:29.230184+0000","last_became_active":"2024-09-05T16:48:16.435091+0000","last_became_peered":"2024-09-05T16:48:16.435091+0000","last_unstale":"2024-09-05T16:48:29.230184+0000","last_undegraded":"2024-09-05T16:48:29.230184+0000","last_fullsized":"2024-09-05T16:48:29.230184+0000","mapping_epoch":58,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'13124","last_scrub_stamp":"2024-09-05T16:48:16.582038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:48:16.582038+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T17:44:25.891565+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"39'12831","reported_seq":12988,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-05T16:48:29.230099+0000","last_change":"2024-09-05T16:47:40.451380+0000","last_active":"2024-09-05T16:48:29.230099+0000","last_peered":"2024-09-05T16:48:29.230099+0000","last_clean":"2024-09-05T16:48:29.230099+0000","last_became_active":"2024-09-05T16:47:40.189447+0000","last_became_peered":"2024-09-05T16:47:40.189447+0000","last_unstale":"2024-09-05T16:48:29.230099+0000","last_undegraded":"2024-09-05T16:48:29.230099+0000","last_fullsized":"2024-09-05T16:48:29.230099+0000","mapping_epoch":52,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12831","last_scrub_stamp":"2024-09-05T16:47:40.451295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:40.451295+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:46:20.946333+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12773","reported_seq":12930,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-05T16:48:29.230307+0000","last_change":"2024-09-05T16:47:27.437275+0000","last_active":"2024-09-05T16:48:29.230307+0000","last_peered":"2024-09-05T16:48:29.230307+0000","last_clean":"2024-09-05T16:48:29.230307+0000","last_became_active":"2024-09-05T16:47:27.204989+0000","last_became_peered":"2024-09-05T16:47:27.204989+0000","last_unstale":"2024-09-05T16:48:29.230307+0000","last_undegraded":"2024-09-05T16:48:29.230307+0000","last_fullsized":"2024-09-05T16:48:29.230307+0000","mapping_epoch":50,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12773","last_scrub_stamp":"2024-09-05T16:47:27.437187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:27.437187+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T16:53:55.961513+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":53888,"num_objects":8,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4377,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":77824,"num_write_kb":263452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6985,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":20480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084461,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":726332,"kb_used_data":484,"kb_used_omap":4,"kb_used_meta":725819,"kb_avail":93645508,"statfs":{"total":96636764160,"available":95893000192,"internally_reserved":0,"allocated":495616,"data_stored":187895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":743239379},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986339,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":618056,"kb_used_data":504,"kb_used_omap":4,"kb_used_meta":617531,"kb_avail":93753784,"statfs":{"total":96636764160,"available":96003874816,"internally_reserved":0,"allocated":516096,"data_stored":198254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":632352466},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673145,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713356,"kb_used_data":500,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":93658484,"statfs":{"total":96636764160,"available":95906287616,"internally_reserved":0,"allocated":512000,"data_stored":194158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":729938373},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182576,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713800,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658040,"statfs":{"total":96636764160,"available":95905832960,"internally_reserved":0,"allocated":512000,"data_stored":194158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":730394317},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:48:34.639 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:48:34.936 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:48:34.937 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:48:34.949 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":510,"stamp":"2024-09-05T16:48:33.118713+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":53888,"num_objects":8,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4377,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":77824,"num_write_kb":263452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6985,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2771544,"kb_used_data":1988,"kb_used_omap":14,"kb_used_meta":2769457,"kb_avail":374715816,"statfs":{"total":386547056640,"available":383708995584,"internally_reserved":0,"allocated":2035712,"data_stored":774465,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2835924535},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1064,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1064,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001365"},"pg_stats":[{"pgid":"1.f","version":"39'12904","reported_seq":17537,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-05T16:48:16.433370+0000","last_change":"2024-09-05T16:46:27.826222+0000","last_active":"2024-09-05T16:48:16.433370+0000","last_peered":"2024-09-05T16:48:16.433370+0000","last_clean":"2024-09-05T16:48:16.433370+0000","last_became_active":"2024-09-05T16:46:27.757928+0000","last_became_peered":"2024-09-05T16:46:27.757928+0000","last_unstale":"2024-09-05T16:48:16.433370+0000","last_undegraded":"2024-09-05T16:48:16.433370+0000","last_fullsized":"2024-09-05T16:48:16.433370+0000","mapping_epoch":40,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:32:35.713428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12904,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2262,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12552","reported_seq":17254,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-05T16:48:29.230646+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:48:29.230646+0000","last_peered":"2024-09-05T16:48:29.230646+0000","last_clean":"2024-09-05T16:48:29.230646+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:48:29.230646+0000","last_undegraded":"2024-09-05T16:48:29.230646+0000","last_fullsized":"2024-09-05T16:48:29.230646+0000","mapping_epoch":38,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1865,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12752","reported_seq":17409,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-05T16:48:29.233031+0000","last_change":"2024-09-05T16:46:51.992073+0000","last_active":"2024-09-05T16:48:29.233031+0000","last_peered":"2024-09-05T16:48:29.233031+0000","last_clean":"2024-09-05T16:48:29.233031+0000","last_became_active":"2024-09-05T16:46:51.667128+0000","last_became_peered":"2024-09-05T16:46:51.667128+0000","last_unstale":"2024-09-05T16:48:29.233031+0000","last_undegraded":"2024-09-05T16:48:29.233031+0000","last_fullsized":"2024-09-05T16:48:29.233031+0000","mapping_epoch":44,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12752","last_scrub_stamp":"2024-09-05T16:46:51.991995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:51.991995+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:04:54.120031+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12934","reported_seq":17714,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-05T16:48:29.230556+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:48:29.230556+0000","last_peered":"2024-09-05T16:48:29.230556+0000","last_clean":"2024-09-05T16:48:29.230556+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:48:29.230556+0000","last_undegraded":"2024-09-05T16:48:29.230556+0000","last_fullsized":"2024-09-05T16:48:29.230556+0000","mapping_epoch":34,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7234,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12593","reported_seq":17050,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-05T16:48:16.432562+0000","last_change":"2024-09-05T16:47:51.525026+0000","last_active":"2024-09-05T16:48:16.432562+0000","last_peered":"2024-09-05T16:48:16.432562+0000","last_clean":"2024-09-05T16:48:16.432562+0000","last_became_active":"2024-09-05T16:47:51.193016+0000","last_became_peered":"2024-09-05T16:47:51.193016+0000","last_unstale":"2024-09-05T16:48:16.432562+0000","last_undegraded":"2024-09-05T16:48:16.432562+0000","last_fullsized":"2024-09-05T16:48:16.432562+0000","mapping_epoch":54,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12593","last_scrub_stamp":"2024-09-05T16:47:51.524957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:51.524957+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:46:22.846859+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12772","reported_seq":17381,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-05T16:48:16.432461+0000","last_change":"2024-09-05T16:47:15.832530+0000","last_active":"2024-09-05T16:48:16.432461+0000","last_peered":"2024-09-05T16:48:16.432461+0000","last_clean":"2024-09-05T16:48:16.432461+0000","last_became_active":"2024-09-05T16:47:15.728759+0000","last_became_peered":"2024-09-05T16:47:15.728759+0000","last_unstale":"2024-09-05T16:48:16.432461+0000","last_undegraded":"2024-09-05T16:48:16.432461+0000","last_fullsized":"2024-09-05T16:48:16.432461+0000","mapping_epoch":48,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T02:39:11.515339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12772,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2250,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12660","reported_seq":17102,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-05T16:48:29.229181+0000","last_change":"2024-09-05T16:47:03.683348+0000","last_active":"2024-09-05T16:48:29.229181+0000","last_peered":"2024-09-05T16:48:29.229181+0000","last_clean":"2024-09-05T16:48:29.229181+0000","last_became_active":"2024-09-05T16:47:03.185143+0000","last_became_peered":"2024-09-05T16:47:03.185143+0000","last_unstale":"2024-09-05T16:48:29.229181+0000","last_undegraded":"2024-09-05T16:48:29.229181+0000","last_fullsized":"2024-09-05T16:48:29.229181+0000","mapping_epoch":46,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12660","last_scrub_stamp":"2024-09-05T16:47:03.683262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:03.683262+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:42:16.533299+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12777","reported_seq":12930,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-05T16:48:29.230218+0000","last_change":"2024-09-05T16:46:40.458315+0000","last_active":"2024-09-05T16:48:29.230218+0000","last_peered":"2024-09-05T16:48:29.230218+0000","last_clean":"2024-09-05T16:48:29.230218+0000","last_became_active":"2024-09-05T16:46:40.184288+0000","last_became_peered":"2024-09-05T16:46:40.184288+0000","last_unstale":"2024-09-05T16:48:29.230218+0000","last_undegraded":"2024-09-05T16:48:29.230218+0000","last_fullsized":"2024-09-05T16:48:29.230218+0000","mapping_epoch":42,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12777","last_scrub_stamp":"2024-09-05T16:46:40.458226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:40.458226+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:38:58.734210+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'12770","reported_seq":13136,"reported_epoch":59,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:48:28.213071+0000","last_change":"2024-09-05T16:48:27.426520+0000","last_active":"2024-09-05T16:48:28.213071+0000","last_peered":"2024-09-05T16:48:28.213071+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:48:28.213071+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12767,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":129,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2134}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12808","reported_seq":17493,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-05T16:48:29.233129+0000","last_change":"2024-09-05T16:48:04.288466+0000","last_active":"2024-09-05T16:48:29.233129+0000","last_peered":"2024-09-05T16:48:29.233129+0000","last_clean":"2024-09-05T16:48:29.233129+0000","last_became_active":"2024-09-05T16:48:03.239894+0000","last_became_peered":"2024-09-05T16:48:03.239894+0000","last_unstale":"2024-09-05T16:48:29.233129+0000","last_undegraded":"2024-09-05T16:48:29.233129+0000","last_fullsized":"2024-09-05T16:48:29.233129+0000","mapping_epoch":56,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12808","last_scrub_stamp":"2024-09-05T16:48:04.288388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:48:04.288388+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T21:11:32.044919+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13120","reported_seq":17863,"reported_epoch":59,"state":"active+undersized+remapped","last_fresh":"2024-09-05T16:48:27.425432+0000","last_change":"2024-09-05T16:48:27.425639+0000","last_active":"2024-09-05T16:48:27.425432+0000","last_peered":"2024-09-05T16:48:27.425432+0000","last_clean":"2024-09-05T16:33:53.605270+0000","last_became_active":"2024-09-05T16:34:56.956041+0000","last_became_peered":"2024-09-05T16:34:56.956041+0000","last_unstale":"2024-09-05T16:48:27.425432+0000","last_undegraded":"2024-09-05T16:34:56.625557+0000","last_fullsized":"2024-09-05T16:34:56.557093+0000","mapping_epoch":32,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13117,"num_write_kb":52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2317,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","1(2)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12758","reported_seq":12903,"reported_epoch":59,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:48:16.432928+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:48:16.432928+0000","last_peered":"2024-09-05T16:48:16.432928+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:48:16.432928+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2237}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12808","reported_seq":17594,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-05T16:48:29.230528+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:48:29.230528+0000","last_peered":"2024-09-05T16:48:29.230528+0000","last_clean":"2024-09-05T16:48:29.230528+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:48:29.230528+0000","last_undegraded":"2024-09-05T16:48:29.230528+0000","last_fullsized":"2024-09-05T16:48:29.230528+0000","mapping_epoch":36,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4408,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13124","reported_seq":13265,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-05T16:48:29.230184+0000","last_change":"2024-09-05T16:48:16.582142+0000","last_active":"2024-09-05T16:48:29.230184+0000","last_peered":"2024-09-05T16:48:29.230184+0000","last_clean":"2024-09-05T16:48:29.230184+0000","last_became_active":"2024-09-05T16:48:16.435091+0000","last_became_peered":"2024-09-05T16:48:16.435091+0000","last_unstale":"2024-09-05T16:48:29.230184+0000","last_undegraded":"2024-09-05T16:48:29.230184+0000","last_fullsized":"2024-09-05T16:48:29.230184+0000","mapping_epoch":58,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'13124","last_scrub_stamp":"2024-09-05T16:48:16.582038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:48:16.582038+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T17:44:25.891565+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"39'12831","reported_seq":12988,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-05T16:48:29.230099+0000","last_change":"2024-09-05T16:47:40.451380+0000","last_active":"2024-09-05T16:48:29.230099+0000","last_peered":"2024-09-05T16:48:29.230099+0000","last_clean":"2024-09-05T16:48:29.230099+0000","last_became_active":"2024-09-05T16:47:40.189447+0000","last_became_peered":"2024-09-05T16:47:40.189447+0000","last_unstale":"2024-09-05T16:48:29.230099+0000","last_undegraded":"2024-09-05T16:48:29.230099+0000","last_fullsized":"2024-09-05T16:48:29.230099+0000","mapping_epoch":52,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12831","last_scrub_stamp":"2024-09-05T16:47:40.451295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:40.451295+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:46:20.946333+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12773","reported_seq":12930,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-05T16:48:29.230307+0000","last_change":"2024-09-05T16:47:27.437275+0000","last_active":"2024-09-05T16:48:29.230307+0000","last_peered":"2024-09-05T16:48:29.230307+0000","last_clean":"2024-09-05T16:48:29.230307+0000","last_became_active":"2024-09-05T16:47:27.204989+0000","last_became_peered":"2024-09-05T16:47:27.204989+0000","last_unstale":"2024-09-05T16:48:29.230307+0000","last_undegraded":"2024-09-05T16:48:29.230307+0000","last_fullsized":"2024-09-05T16:48:29.230307+0000","mapping_epoch":50,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12773","last_scrub_stamp":"2024-09-05T16:47:27.437187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:27.437187+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T16:53:55.961513+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":53888,"num_objects":8,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4377,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":77824,"num_write_kb":263452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6985,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":20480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084461,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":726332,"kb_used_data":484,"kb_used_omap":4,"kb_used_meta":725819,"kb_avail":93645508,"statfs":{"total":96636764160,"available":95893000192,"internally_reserved":0,"allocated":495616,"data_stored":187895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":743239379},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986339,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":618056,"kb_used_data":504,"kb_used_omap":4,"kb_used_meta":617531,"kb_avail":93753784,"statfs":{"total":96636764160,"available":96003874816,"internally_reserved":0,"allocated":516096,"data_stored":198254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":632352466},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673145,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713356,"kb_used_data":500,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":93658484,"statfs":{"total":96636764160,"available":95906287616,"internally_reserved":0,"allocated":512000,"data_stored":194158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":729938373},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182576,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713800,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658040,"statfs":{"total":96636764160,"available":95905832960,"internally_reserved":0,"allocated":512000,"data_stored":194158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":730394317},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:48:34.950 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:48:34.950 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "39'12770", 'reported_seq': 13136, 'reported_epoch': 59, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:48:28.213071+0000', 'last_change': '2024-09-05T16:48:27.426520+0000', 'last_active': '2024-09-05T16:48:28.213071+0000', 'last_peered': '2024-09-05T16:48:28.213071+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:48:28.213071+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "33'2700", 'ondisk_log_start': "33'2700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10070, 'log_dups_size': 0, 'ondisk_log_size': 10070, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26944, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 16, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2138, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12767, 'num_write_kb': 51081, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 129, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2134}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:48:34.950 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-05T16:48:34.950 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "39'13120", 'reported_seq': 17863, 'reported_epoch': 59, 'state': 'active+undersized+remapped', 'last_fresh': '2024-09-05T16:48:27.425432+0000', 'last_change': '2024-09-05T16:48:27.425639+0000', 'last_active': '2024-09-05T16:48:27.425432+0000', 'last_peered': '2024-09-05T16:48:27.425432+0000', 'last_clean': '2024-09-05T16:33:53.605270+0000', 'last_became_active': '2024-09-05T16:34:56.956041+0000', 'last_became_peered': '2024-09-05T16:34:56.956041+0000', 'last_unstale': '2024-09-05T16:48:27.425432+0000', 'last_undegraded': '2024-09-05T16:34:56.625557+0000', 'last_fullsized': '2024-09-05T16:34:56.557093+0000', 'mapping_epoch': 32, 'log_start': "33'3100", 'ondisk_log_start': "33'3100", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20208, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13117, 'num_write_kb': 52476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2317, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '1(2)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:48:34.950 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:48:34.950 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "39'12758", 'reported_seq': 12903, 'reported_epoch': 59, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:48:16.432928+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:48:16.432928+0000', 'last_peered': '2024-09-05T16:48:16.432928+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:48:16.432928+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "33'2700", 'ondisk_log_start': "33'2700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10058, 'log_dups_size': 0, 'ondisk_log_size': 10058, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6736, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12757, 'num_write_kb': 51031, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2237}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:48:34.951 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:48:35.337 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:48:35.337 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 3548/20 objects degraded (17740.000%), 2 pgs degraded, 2 pgs undersized","count":4},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":918,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":61,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":2},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":5,"data_bytes":33680,"bytes_used":2838073344,"bytes_avail":383708983296,"bytes_total":386547056640,"degraded_objects":3548,"degraded_total":20,"degraded_ratio":177.40000000000001},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:48:35.350 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-09-05T16:48:35.350 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:48:35.652 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:48:35.652 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:48:35.666 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":511,"stamp":"2024-09-05T16:48:35.119101+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":33680,"num_objects":5,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3548,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":64707,"num_write_kb":210976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5497,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2771556,"kb_used_data":2004,"kb_used_omap":14,"kb_used_meta":2769457,"kb_avail":374715804,"statfs":{"total":386547056640,"available":383708983296,"internally_reserved":0,"allocated":2052096,"data_stored":780728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2835924535},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20208,"num_objects":-3,"num_object_clones":0,"num_object_copies":-12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1893,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-13117,"num_write_kb":-52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-424,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001460"},"pg_stats":[{"pgid":"1.f","version":"39'12904","reported_seq":17541,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-05T16:48:29.230833+0000","last_change":"2024-09-05T16:46:27.826222+0000","last_active":"2024-09-05T16:48:29.230833+0000","last_peered":"2024-09-05T16:48:29.230833+0000","last_clean":"2024-09-05T16:48:29.230833+0000","last_became_active":"2024-09-05T16:46:27.757928+0000","last_became_peered":"2024-09-05T16:46:27.757928+0000","last_unstale":"2024-09-05T16:48:29.230833+0000","last_undegraded":"2024-09-05T16:48:29.230833+0000","last_fullsized":"2024-09-05T16:48:29.230833+0000","mapping_epoch":40,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:32:35.713428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12904,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2262,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12552","reported_seq":17254,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-05T16:48:29.230646+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:48:29.230646+0000","last_peered":"2024-09-05T16:48:29.230646+0000","last_clean":"2024-09-05T16:48:29.230646+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:48:29.230646+0000","last_undegraded":"2024-09-05T16:48:29.230646+0000","last_fullsized":"2024-09-05T16:48:29.230646+0000","mapping_epoch":38,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1865,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12752","reported_seq":17409,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-05T16:48:29.233031+0000","last_change":"2024-09-05T16:46:51.992073+0000","last_active":"2024-09-05T16:48:29.233031+0000","last_peered":"2024-09-05T16:48:29.233031+0000","last_clean":"2024-09-05T16:48:29.233031+0000","last_became_active":"2024-09-05T16:46:51.667128+0000","last_became_peered":"2024-09-05T16:46:51.667128+0000","last_unstale":"2024-09-05T16:48:29.233031+0000","last_undegraded":"2024-09-05T16:48:29.233031+0000","last_fullsized":"2024-09-05T16:48:29.233031+0000","mapping_epoch":44,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12752","last_scrub_stamp":"2024-09-05T16:46:51.991995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:51.991995+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:04:54.120031+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12934","reported_seq":17714,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-05T16:48:29.230556+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:48:29.230556+0000","last_peered":"2024-09-05T16:48:29.230556+0000","last_clean":"2024-09-05T16:48:29.230556+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:48:29.230556+0000","last_undegraded":"2024-09-05T16:48:29.230556+0000","last_fullsized":"2024-09-05T16:48:29.230556+0000","mapping_epoch":34,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7234,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12593","reported_seq":17055,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-05T16:48:29.230261+0000","last_change":"2024-09-05T16:47:51.525026+0000","last_active":"2024-09-05T16:48:29.230261+0000","last_peered":"2024-09-05T16:48:29.230261+0000","last_clean":"2024-09-05T16:48:29.230261+0000","last_became_active":"2024-09-05T16:47:51.193016+0000","last_became_peered":"2024-09-05T16:47:51.193016+0000","last_unstale":"2024-09-05T16:48:29.230261+0000","last_undegraded":"2024-09-05T16:48:29.230261+0000","last_fullsized":"2024-09-05T16:48:29.230261+0000","mapping_epoch":54,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12593","last_scrub_stamp":"2024-09-05T16:47:51.524957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:51.524957+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:46:22.846859+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12772","reported_seq":17385,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-05T16:48:29.230812+0000","last_change":"2024-09-05T16:47:15.832530+0000","last_active":"2024-09-05T16:48:29.230812+0000","last_peered":"2024-09-05T16:48:29.230812+0000","last_clean":"2024-09-05T16:48:29.230812+0000","last_became_active":"2024-09-05T16:47:15.728759+0000","last_became_peered":"2024-09-05T16:47:15.728759+0000","last_unstale":"2024-09-05T16:48:29.230812+0000","last_undegraded":"2024-09-05T16:48:29.230812+0000","last_fullsized":"2024-09-05T16:48:29.230812+0000","mapping_epoch":48,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T02:39:11.515339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12772,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2250,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12660","reported_seq":17102,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-05T16:48:29.229181+0000","last_change":"2024-09-05T16:47:03.683348+0000","last_active":"2024-09-05T16:48:29.229181+0000","last_peered":"2024-09-05T16:48:29.229181+0000","last_clean":"2024-09-05T16:48:29.229181+0000","last_became_active":"2024-09-05T16:47:03.185143+0000","last_became_peered":"2024-09-05T16:47:03.185143+0000","last_unstale":"2024-09-05T16:48:29.229181+0000","last_undegraded":"2024-09-05T16:48:29.229181+0000","last_fullsized":"2024-09-05T16:48:29.229181+0000","mapping_epoch":46,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12660","last_scrub_stamp":"2024-09-05T16:47:03.683262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:03.683262+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:42:16.533299+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12777","reported_seq":12930,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-05T16:48:29.230218+0000","last_change":"2024-09-05T16:46:40.458315+0000","last_active":"2024-09-05T16:48:29.230218+0000","last_peered":"2024-09-05T16:48:29.230218+0000","last_clean":"2024-09-05T16:48:29.230218+0000","last_became_active":"2024-09-05T16:46:40.184288+0000","last_became_peered":"2024-09-05T16:46:40.184288+0000","last_unstale":"2024-09-05T16:48:29.230218+0000","last_undegraded":"2024-09-05T16:48:29.230218+0000","last_fullsized":"2024-09-05T16:48:29.230218+0000","mapping_epoch":42,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12777","last_scrub_stamp":"2024-09-05T16:46:40.458226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:40.458226+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:38:58.734210+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'12770","reported_seq":14798,"reported_epoch":61,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:48:33.211522+0000","last_change":"2024-09-05T16:48:27.426520+0000","last_active":"2024-09-05T16:48:33.211522+0000","last_peered":"2024-09-05T16:48:33.211522+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:48:33.211522+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12767,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":958,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1305}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12808","reported_seq":17493,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-05T16:48:29.233129+0000","last_change":"2024-09-05T16:48:04.288466+0000","last_active":"2024-09-05T16:48:29.233129+0000","last_peered":"2024-09-05T16:48:29.233129+0000","last_clean":"2024-09-05T16:48:29.233129+0000","last_became_active":"2024-09-05T16:48:03.239894+0000","last_became_peered":"2024-09-05T16:48:03.239894+0000","last_unstale":"2024-09-05T16:48:29.233129+0000","last_undegraded":"2024-09-05T16:48:29.233129+0000","last_fullsized":"2024-09-05T16:48:29.233129+0000","mapping_epoch":56,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12808","last_scrub_stamp":"2024-09-05T16:48:04.288388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:48:04.288388+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T21:11:32.044919+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13120","reported_seq":17883,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-05T16:48:29.529847+0000","last_change":"2024-09-05T16:48:29.529847+0000","last_active":"2024-09-05T16:48:29.529847+0000","last_peered":"2024-09-05T16:48:29.529847+0000","last_clean":"2024-09-05T16:48:29.529847+0000","last_became_active":"2024-09-05T16:48:29.235000+0000","last_became_peered":"2024-09-05T16:48:29.235000+0000","last_unstale":"2024-09-05T16:48:29.529847+0000","last_undegraded":"2024-09-05T16:48:29.529847+0000","last_fullsized":"2024-09-05T16:48:29.529847+0000","mapping_epoch":60,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'13120","last_scrub_stamp":"2024-09-05T16:48:29.529768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:48:29.529768+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T02:18:10.992722+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12758","reported_seq":12907,"reported_epoch":61,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:48:29.230950+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:48:29.230950+0000","last_peered":"2024-09-05T16:48:29.230950+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:48:29.230950+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2237}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12808","reported_seq":17594,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-05T16:48:29.230528+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:48:29.230528+0000","last_peered":"2024-09-05T16:48:29.230528+0000","last_clean":"2024-09-05T16:48:29.230528+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:48:29.230528+0000","last_undegraded":"2024-09-05T16:48:29.230528+0000","last_fullsized":"2024-09-05T16:48:29.230528+0000","mapping_epoch":36,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4408,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13124","reported_seq":13265,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-05T16:48:29.230184+0000","last_change":"2024-09-05T16:48:16.582142+0000","last_active":"2024-09-05T16:48:29.230184+0000","last_peered":"2024-09-05T16:48:29.230184+0000","last_clean":"2024-09-05T16:48:29.230184+0000","last_became_active":"2024-09-05T16:48:16.435091+0000","last_became_peered":"2024-09-05T16:48:16.435091+0000","last_unstale":"2024-09-05T16:48:29.230184+0000","last_undegraded":"2024-09-05T16:48:29.230184+0000","last_fullsized":"2024-09-05T16:48:29.230184+0000","mapping_epoch":58,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'13124","last_scrub_stamp":"2024-09-05T16:48:16.582038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:48:16.582038+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T17:44:25.891565+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"39'12831","reported_seq":12988,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-05T16:48:29.230099+0000","last_change":"2024-09-05T16:47:40.451380+0000","last_active":"2024-09-05T16:48:29.230099+0000","last_peered":"2024-09-05T16:48:29.230099+0000","last_clean":"2024-09-05T16:48:29.230099+0000","last_became_active":"2024-09-05T16:47:40.189447+0000","last_became_peered":"2024-09-05T16:47:40.189447+0000","last_unstale":"2024-09-05T16:48:29.230099+0000","last_undegraded":"2024-09-05T16:48:29.230099+0000","last_fullsized":"2024-09-05T16:48:29.230099+0000","mapping_epoch":52,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12831","last_scrub_stamp":"2024-09-05T16:47:40.451295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:40.451295+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:46:20.946333+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12773","reported_seq":12930,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-05T16:48:29.230307+0000","last_change":"2024-09-05T16:47:27.437275+0000","last_active":"2024-09-05T16:48:29.230307+0000","last_peered":"2024-09-05T16:48:29.230307+0000","last_clean":"2024-09-05T16:48:29.230307+0000","last_became_active":"2024-09-05T16:47:27.204989+0000","last_became_peered":"2024-09-05T16:47:27.204989+0000","last_unstale":"2024-09-05T16:48:29.230307+0000","last_undegraded":"2024-09-05T16:48:29.230307+0000","last_fullsized":"2024-09-05T16:48:29.230307+0000","mapping_epoch":50,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12773","last_scrub_stamp":"2024-09-05T16:47:27.437187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:27.437187+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T16:53:55.961513+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":33680,"num_objects":5,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3548,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":64707,"num_write_kb":210976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5497,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":20480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084462,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":726344,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":725819,"kb_avail":93645496,"statfs":{"total":96636764160,"available":95892987904,"internally_reserved":0,"allocated":512000,"data_stored":194158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":743239379},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986339,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":618056,"kb_used_data":504,"kb_used_omap":4,"kb_used_meta":617531,"kb_avail":93753784,"statfs":{"total":96636764160,"available":96003874816,"internally_reserved":0,"allocated":516096,"data_stored":198254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":632352466},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673145,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713356,"kb_used_data":500,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":93658484,"statfs":{"total":96636764160,"available":95906287616,"internally_reserved":0,"allocated":512000,"data_stored":194158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":729938373},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182576,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713800,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658040,"statfs":{"total":96636764160,"available":95905832960,"internally_reserved":0,"allocated":512000,"data_stored":194158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":730394317},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:48:38.667 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:48:38.975 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:48:38.975 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:48:38.987 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":512,"stamp":"2024-09-05T16:48:37.119404+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":33680,"num_objects":5,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3548,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":64707,"num_write_kb":210976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5497,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2771556,"kb_used_data":2004,"kb_used_omap":14,"kb_used_meta":2769457,"kb_avail":374715804,"statfs":{"total":386547056640,"available":383708983296,"internally_reserved":0,"allocated":2052096,"data_stored":780728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2835924535},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20208,"num_objects":-3,"num_object_clones":0,"num_object_copies":-12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-829,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-13117,"num_write_kb":-52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1488,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.893064"},"pg_stats":[{"pgid":"1.f","version":"39'12904","reported_seq":17541,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-05T16:48:29.230833+0000","last_change":"2024-09-05T16:46:27.826222+0000","last_active":"2024-09-05T16:48:29.230833+0000","last_peered":"2024-09-05T16:48:29.230833+0000","last_clean":"2024-09-05T16:48:29.230833+0000","last_became_active":"2024-09-05T16:46:27.757928+0000","last_became_peered":"2024-09-05T16:46:27.757928+0000","last_unstale":"2024-09-05T16:48:29.230833+0000","last_undegraded":"2024-09-05T16:48:29.230833+0000","last_fullsized":"2024-09-05T16:48:29.230833+0000","mapping_epoch":40,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:32:35.713428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12904,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2262,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12552","reported_seq":17254,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-05T16:48:29.230646+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:48:29.230646+0000","last_peered":"2024-09-05T16:48:29.230646+0000","last_clean":"2024-09-05T16:48:29.230646+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:48:29.230646+0000","last_undegraded":"2024-09-05T16:48:29.230646+0000","last_fullsized":"2024-09-05T16:48:29.230646+0000","mapping_epoch":38,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1865,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12752","reported_seq":17409,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-05T16:48:29.233031+0000","last_change":"2024-09-05T16:46:51.992073+0000","last_active":"2024-09-05T16:48:29.233031+0000","last_peered":"2024-09-05T16:48:29.233031+0000","last_clean":"2024-09-05T16:48:29.233031+0000","last_became_active":"2024-09-05T16:46:51.667128+0000","last_became_peered":"2024-09-05T16:46:51.667128+0000","last_unstale":"2024-09-05T16:48:29.233031+0000","last_undegraded":"2024-09-05T16:48:29.233031+0000","last_fullsized":"2024-09-05T16:48:29.233031+0000","mapping_epoch":44,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12752","last_scrub_stamp":"2024-09-05T16:46:51.991995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:51.991995+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:04:54.120031+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12934","reported_seq":17714,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-05T16:48:29.230556+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:48:29.230556+0000","last_peered":"2024-09-05T16:48:29.230556+0000","last_clean":"2024-09-05T16:48:29.230556+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:48:29.230556+0000","last_undegraded":"2024-09-05T16:48:29.230556+0000","last_fullsized":"2024-09-05T16:48:29.230556+0000","mapping_epoch":34,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7234,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12593","reported_seq":17055,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-05T16:48:29.230261+0000","last_change":"2024-09-05T16:47:51.525026+0000","last_active":"2024-09-05T16:48:29.230261+0000","last_peered":"2024-09-05T16:48:29.230261+0000","last_clean":"2024-09-05T16:48:29.230261+0000","last_became_active":"2024-09-05T16:47:51.193016+0000","last_became_peered":"2024-09-05T16:47:51.193016+0000","last_unstale":"2024-09-05T16:48:29.230261+0000","last_undegraded":"2024-09-05T16:48:29.230261+0000","last_fullsized":"2024-09-05T16:48:29.230261+0000","mapping_epoch":54,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12593","last_scrub_stamp":"2024-09-05T16:47:51.524957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:51.524957+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:46:22.846859+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12772","reported_seq":17385,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-05T16:48:29.230812+0000","last_change":"2024-09-05T16:47:15.832530+0000","last_active":"2024-09-05T16:48:29.230812+0000","last_peered":"2024-09-05T16:48:29.230812+0000","last_clean":"2024-09-05T16:48:29.230812+0000","last_became_active":"2024-09-05T16:47:15.728759+0000","last_became_peered":"2024-09-05T16:47:15.728759+0000","last_unstale":"2024-09-05T16:48:29.230812+0000","last_undegraded":"2024-09-05T16:48:29.230812+0000","last_fullsized":"2024-09-05T16:48:29.230812+0000","mapping_epoch":48,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T02:39:11.515339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12772,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2250,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12660","reported_seq":17102,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-05T16:48:29.229181+0000","last_change":"2024-09-05T16:47:03.683348+0000","last_active":"2024-09-05T16:48:29.229181+0000","last_peered":"2024-09-05T16:48:29.229181+0000","last_clean":"2024-09-05T16:48:29.229181+0000","last_became_active":"2024-09-05T16:47:03.185143+0000","last_became_peered":"2024-09-05T16:47:03.185143+0000","last_unstale":"2024-09-05T16:48:29.229181+0000","last_undegraded":"2024-09-05T16:48:29.229181+0000","last_fullsized":"2024-09-05T16:48:29.229181+0000","mapping_epoch":46,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12660","last_scrub_stamp":"2024-09-05T16:47:03.683262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:03.683262+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:42:16.533299+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12777","reported_seq":12930,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-05T16:48:29.230218+0000","last_change":"2024-09-05T16:46:40.458315+0000","last_active":"2024-09-05T16:48:29.230218+0000","last_peered":"2024-09-05T16:48:29.230218+0000","last_clean":"2024-09-05T16:48:29.230218+0000","last_became_active":"2024-09-05T16:46:40.184288+0000","last_became_peered":"2024-09-05T16:46:40.184288+0000","last_unstale":"2024-09-05T16:48:29.230218+0000","last_undegraded":"2024-09-05T16:48:29.230218+0000","last_fullsized":"2024-09-05T16:48:29.230218+0000","mapping_epoch":42,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12777","last_scrub_stamp":"2024-09-05T16:46:40.458226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:40.458226+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:38:58.734210+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'12770","reported_seq":14798,"reported_epoch":61,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:48:33.211522+0000","last_change":"2024-09-05T16:48:27.426520+0000","last_active":"2024-09-05T16:48:33.211522+0000","last_peered":"2024-09-05T16:48:33.211522+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:48:33.211522+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12767,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":958,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1305}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12808","reported_seq":17493,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-05T16:48:29.233129+0000","last_change":"2024-09-05T16:48:04.288466+0000","last_active":"2024-09-05T16:48:29.233129+0000","last_peered":"2024-09-05T16:48:29.233129+0000","last_clean":"2024-09-05T16:48:29.233129+0000","last_became_active":"2024-09-05T16:48:03.239894+0000","last_became_peered":"2024-09-05T16:48:03.239894+0000","last_unstale":"2024-09-05T16:48:29.233129+0000","last_undegraded":"2024-09-05T16:48:29.233129+0000","last_fullsized":"2024-09-05T16:48:29.233129+0000","mapping_epoch":56,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12808","last_scrub_stamp":"2024-09-05T16:48:04.288388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:48:04.288388+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T21:11:32.044919+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13120","reported_seq":17883,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-05T16:48:29.529847+0000","last_change":"2024-09-05T16:48:29.529847+0000","last_active":"2024-09-05T16:48:29.529847+0000","last_peered":"2024-09-05T16:48:29.529847+0000","last_clean":"2024-09-05T16:48:29.529847+0000","last_became_active":"2024-09-05T16:48:29.235000+0000","last_became_peered":"2024-09-05T16:48:29.235000+0000","last_unstale":"2024-09-05T16:48:29.529847+0000","last_undegraded":"2024-09-05T16:48:29.529847+0000","last_fullsized":"2024-09-05T16:48:29.529847+0000","mapping_epoch":60,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'13120","last_scrub_stamp":"2024-09-05T16:48:29.529768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:48:29.529768+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T02:18:10.992722+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12758","reported_seq":12907,"reported_epoch":61,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:48:29.230950+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:48:29.230950+0000","last_peered":"2024-09-05T16:48:29.230950+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:48:29.230950+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2237}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12808","reported_seq":17594,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-05T16:48:29.230528+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:48:29.230528+0000","last_peered":"2024-09-05T16:48:29.230528+0000","last_clean":"2024-09-05T16:48:29.230528+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:48:29.230528+0000","last_undegraded":"2024-09-05T16:48:29.230528+0000","last_fullsized":"2024-09-05T16:48:29.230528+0000","mapping_epoch":36,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4408,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13124","reported_seq":13265,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-05T16:48:29.230184+0000","last_change":"2024-09-05T16:48:16.582142+0000","last_active":"2024-09-05T16:48:29.230184+0000","last_peered":"2024-09-05T16:48:29.230184+0000","last_clean":"2024-09-05T16:48:29.230184+0000","last_became_active":"2024-09-05T16:48:16.435091+0000","last_became_peered":"2024-09-05T16:48:16.435091+0000","last_unstale":"2024-09-05T16:48:29.230184+0000","last_undegraded":"2024-09-05T16:48:29.230184+0000","last_fullsized":"2024-09-05T16:48:29.230184+0000","mapping_epoch":58,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'13124","last_scrub_stamp":"2024-09-05T16:48:16.582038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:48:16.582038+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T17:44:25.891565+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"39'12831","reported_seq":12988,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-05T16:48:29.230099+0000","last_change":"2024-09-05T16:47:40.451380+0000","last_active":"2024-09-05T16:48:29.230099+0000","last_peered":"2024-09-05T16:48:29.230099+0000","last_clean":"2024-09-05T16:48:29.230099+0000","last_became_active":"2024-09-05T16:47:40.189447+0000","last_became_peered":"2024-09-05T16:47:40.189447+0000","last_unstale":"2024-09-05T16:48:29.230099+0000","last_undegraded":"2024-09-05T16:48:29.230099+0000","last_fullsized":"2024-09-05T16:48:29.230099+0000","mapping_epoch":52,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12831","last_scrub_stamp":"2024-09-05T16:47:40.451295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:40.451295+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:46:20.946333+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12773","reported_seq":12930,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-05T16:48:29.230307+0000","last_change":"2024-09-05T16:47:27.437275+0000","last_active":"2024-09-05T16:48:29.230307+0000","last_peered":"2024-09-05T16:48:29.230307+0000","last_clean":"2024-09-05T16:48:29.230307+0000","last_became_active":"2024-09-05T16:47:27.204989+0000","last_became_peered":"2024-09-05T16:47:27.204989+0000","last_unstale":"2024-09-05T16:48:29.230307+0000","last_undegraded":"2024-09-05T16:48:29.230307+0000","last_fullsized":"2024-09-05T16:48:29.230307+0000","mapping_epoch":50,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12773","last_scrub_stamp":"2024-09-05T16:47:27.437187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:27.437187+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T16:53:55.961513+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":33680,"num_objects":5,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3548,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":64707,"num_write_kb":210976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5497,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":20480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084462,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":726344,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":725819,"kb_avail":93645496,"statfs":{"total":96636764160,"available":95892987904,"internally_reserved":0,"allocated":512000,"data_stored":194158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":743239379},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986339,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":618056,"kb_used_data":504,"kb_used_omap":4,"kb_used_meta":617531,"kb_avail":93753784,"statfs":{"total":96636764160,"available":96003874816,"internally_reserved":0,"allocated":516096,"data_stored":198254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":632352466},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673146,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713356,"kb_used_data":500,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":93658484,"statfs":{"total":96636764160,"available":95906287616,"internally_reserved":0,"allocated":512000,"data_stored":194158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":729938373},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182577,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713800,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658040,"statfs":{"total":96636764160,"available":95905832960,"internally_reserved":0,"allocated":512000,"data_stored":194158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":730394317},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:48:38.988 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:48:39.299 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:48:39.300 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:48:39.313 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":513,"stamp":"2024-09-05T16:48:39.119702+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":33680,"num_objects":5,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2506,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":64707,"num_write_kb":210976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6539,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2781284,"kb_used_data":2004,"kb_used_omap":14,"kb_used_meta":2779185,"kb_avail":374706076,"statfs":{"total":386547056640,"available":383699021824,"internally_reserved":0,"allocated":2052096,"data_stored":780728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2845886007},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20208,"num_objects":-3,"num_object_clones":0,"num_object_copies":-12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1871,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-13117,"num_write_kb":-52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-446,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001701"},"pg_stats":[{"pgid":"1.f","version":"39'12904","reported_seq":17541,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-05T16:48:29.230833+0000","last_change":"2024-09-05T16:46:27.826222+0000","last_active":"2024-09-05T16:48:29.230833+0000","last_peered":"2024-09-05T16:48:29.230833+0000","last_clean":"2024-09-05T16:48:29.230833+0000","last_became_active":"2024-09-05T16:46:27.757928+0000","last_became_peered":"2024-09-05T16:46:27.757928+0000","last_unstale":"2024-09-05T16:48:29.230833+0000","last_undegraded":"2024-09-05T16:48:29.230833+0000","last_fullsized":"2024-09-05T16:48:29.230833+0000","mapping_epoch":40,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:32:35.713428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12904,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2262,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12552","reported_seq":17254,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-05T16:48:29.230646+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:48:29.230646+0000","last_peered":"2024-09-05T16:48:29.230646+0000","last_clean":"2024-09-05T16:48:29.230646+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:48:29.230646+0000","last_undegraded":"2024-09-05T16:48:29.230646+0000","last_fullsized":"2024-09-05T16:48:29.230646+0000","mapping_epoch":38,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1865,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12752","reported_seq":17409,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-05T16:48:29.233031+0000","last_change":"2024-09-05T16:46:51.992073+0000","last_active":"2024-09-05T16:48:29.233031+0000","last_peered":"2024-09-05T16:48:29.233031+0000","last_clean":"2024-09-05T16:48:29.233031+0000","last_became_active":"2024-09-05T16:46:51.667128+0000","last_became_peered":"2024-09-05T16:46:51.667128+0000","last_unstale":"2024-09-05T16:48:29.233031+0000","last_undegraded":"2024-09-05T16:48:29.233031+0000","last_fullsized":"2024-09-05T16:48:29.233031+0000","mapping_epoch":44,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12752","last_scrub_stamp":"2024-09-05T16:46:51.991995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:51.991995+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:04:54.120031+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12934","reported_seq":17714,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-05T16:48:29.230556+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:48:29.230556+0000","last_peered":"2024-09-05T16:48:29.230556+0000","last_clean":"2024-09-05T16:48:29.230556+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:48:29.230556+0000","last_undegraded":"2024-09-05T16:48:29.230556+0000","last_fullsized":"2024-09-05T16:48:29.230556+0000","mapping_epoch":34,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7234,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12593","reported_seq":17055,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-05T16:48:29.230261+0000","last_change":"2024-09-05T16:47:51.525026+0000","last_active":"2024-09-05T16:48:29.230261+0000","last_peered":"2024-09-05T16:48:29.230261+0000","last_clean":"2024-09-05T16:48:29.230261+0000","last_became_active":"2024-09-05T16:47:51.193016+0000","last_became_peered":"2024-09-05T16:47:51.193016+0000","last_unstale":"2024-09-05T16:48:29.230261+0000","last_undegraded":"2024-09-05T16:48:29.230261+0000","last_fullsized":"2024-09-05T16:48:29.230261+0000","mapping_epoch":54,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12593","last_scrub_stamp":"2024-09-05T16:47:51.524957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:51.524957+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:46:22.846859+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12772","reported_seq":17385,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-05T16:48:29.230812+0000","last_change":"2024-09-05T16:47:15.832530+0000","last_active":"2024-09-05T16:48:29.230812+0000","last_peered":"2024-09-05T16:48:29.230812+0000","last_clean":"2024-09-05T16:48:29.230812+0000","last_became_active":"2024-09-05T16:47:15.728759+0000","last_became_peered":"2024-09-05T16:47:15.728759+0000","last_unstale":"2024-09-05T16:48:29.230812+0000","last_undegraded":"2024-09-05T16:48:29.230812+0000","last_fullsized":"2024-09-05T16:48:29.230812+0000","mapping_epoch":48,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T02:39:11.515339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12772,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2250,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12660","reported_seq":17102,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-05T16:48:29.229181+0000","last_change":"2024-09-05T16:47:03.683348+0000","last_active":"2024-09-05T16:48:29.229181+0000","last_peered":"2024-09-05T16:48:29.229181+0000","last_clean":"2024-09-05T16:48:29.229181+0000","last_became_active":"2024-09-05T16:47:03.185143+0000","last_became_peered":"2024-09-05T16:47:03.185143+0000","last_unstale":"2024-09-05T16:48:29.229181+0000","last_undegraded":"2024-09-05T16:48:29.229181+0000","last_fullsized":"2024-09-05T16:48:29.229181+0000","mapping_epoch":46,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12660","last_scrub_stamp":"2024-09-05T16:47:03.683262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:03.683262+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:42:16.533299+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12777","reported_seq":12930,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-05T16:48:29.230218+0000","last_change":"2024-09-05T16:46:40.458315+0000","last_active":"2024-09-05T16:48:29.230218+0000","last_peered":"2024-09-05T16:48:29.230218+0000","last_clean":"2024-09-05T16:48:29.230218+0000","last_became_active":"2024-09-05T16:46:40.184288+0000","last_became_peered":"2024-09-05T16:46:40.184288+0000","last_unstale":"2024-09-05T16:48:29.230218+0000","last_undegraded":"2024-09-05T16:48:29.230218+0000","last_fullsized":"2024-09-05T16:48:29.230218+0000","mapping_epoch":42,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12777","last_scrub_stamp":"2024-09-05T16:46:40.458226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:40.458226+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:38:58.734210+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'12770","reported_seq":16882,"reported_epoch":61,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:48:38.214378+0000","last_change":"2024-09-05T16:48:27.426520+0000","last_active":"2024-09-05T16:48:38.214378+0000","last_peered":"2024-09-05T16:48:38.214378+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:48:38.214378+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":267,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12767,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2000,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-263}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12808","reported_seq":17493,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-05T16:48:29.233129+0000","last_change":"2024-09-05T16:48:04.288466+0000","last_active":"2024-09-05T16:48:29.233129+0000","last_peered":"2024-09-05T16:48:29.233129+0000","last_clean":"2024-09-05T16:48:29.233129+0000","last_became_active":"2024-09-05T16:48:03.239894+0000","last_became_peered":"2024-09-05T16:48:03.239894+0000","last_unstale":"2024-09-05T16:48:29.233129+0000","last_undegraded":"2024-09-05T16:48:29.233129+0000","last_fullsized":"2024-09-05T16:48:29.233129+0000","mapping_epoch":56,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12808","last_scrub_stamp":"2024-09-05T16:48:04.288388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:48:04.288388+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T21:11:32.044919+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13120","reported_seq":17883,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-05T16:48:29.529847+0000","last_change":"2024-09-05T16:48:29.529847+0000","last_active":"2024-09-05T16:48:29.529847+0000","last_peered":"2024-09-05T16:48:29.529847+0000","last_clean":"2024-09-05T16:48:29.529847+0000","last_became_active":"2024-09-05T16:48:29.235000+0000","last_became_peered":"2024-09-05T16:48:29.235000+0000","last_unstale":"2024-09-05T16:48:29.529847+0000","last_undegraded":"2024-09-05T16:48:29.529847+0000","last_fullsized":"2024-09-05T16:48:29.529847+0000","mapping_epoch":60,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'13120","last_scrub_stamp":"2024-09-05T16:48:29.529768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:48:29.529768+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T02:18:10.992722+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12758","reported_seq":12907,"reported_epoch":61,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:48:29.230950+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:48:29.230950+0000","last_peered":"2024-09-05T16:48:29.230950+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:48:29.230950+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2237}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12808","reported_seq":17594,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-05T16:48:29.230528+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:48:29.230528+0000","last_peered":"2024-09-05T16:48:29.230528+0000","last_clean":"2024-09-05T16:48:29.230528+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:48:29.230528+0000","last_undegraded":"2024-09-05T16:48:29.230528+0000","last_fullsized":"2024-09-05T16:48:29.230528+0000","mapping_epoch":36,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4408,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13124","reported_seq":13265,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-05T16:48:29.230184+0000","last_change":"2024-09-05T16:48:16.582142+0000","last_active":"2024-09-05T16:48:29.230184+0000","last_peered":"2024-09-05T16:48:29.230184+0000","last_clean":"2024-09-05T16:48:29.230184+0000","last_became_active":"2024-09-05T16:48:16.435091+0000","last_became_peered":"2024-09-05T16:48:16.435091+0000","last_unstale":"2024-09-05T16:48:29.230184+0000","last_undegraded":"2024-09-05T16:48:29.230184+0000","last_fullsized":"2024-09-05T16:48:29.230184+0000","mapping_epoch":58,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'13124","last_scrub_stamp":"2024-09-05T16:48:16.582038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:48:16.582038+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T17:44:25.891565+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"39'12831","reported_seq":12988,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-05T16:48:29.230099+0000","last_change":"2024-09-05T16:47:40.451380+0000","last_active":"2024-09-05T16:48:29.230099+0000","last_peered":"2024-09-05T16:48:29.230099+0000","last_clean":"2024-09-05T16:48:29.230099+0000","last_became_active":"2024-09-05T16:47:40.189447+0000","last_became_peered":"2024-09-05T16:47:40.189447+0000","last_unstale":"2024-09-05T16:48:29.230099+0000","last_undegraded":"2024-09-05T16:48:29.230099+0000","last_fullsized":"2024-09-05T16:48:29.230099+0000","mapping_epoch":52,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12831","last_scrub_stamp":"2024-09-05T16:47:40.451295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:40.451295+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:46:20.946333+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12773","reported_seq":12930,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-05T16:48:29.230307+0000","last_change":"2024-09-05T16:47:27.437275+0000","last_active":"2024-09-05T16:48:29.230307+0000","last_peered":"2024-09-05T16:48:29.230307+0000","last_clean":"2024-09-05T16:48:29.230307+0000","last_became_active":"2024-09-05T16:47:27.204989+0000","last_became_peered":"2024-09-05T16:47:27.204989+0000","last_unstale":"2024-09-05T16:48:29.230307+0000","last_undegraded":"2024-09-05T16:48:29.230307+0000","last_fullsized":"2024-09-05T16:48:29.230307+0000","mapping_epoch":50,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12773","last_scrub_stamp":"2024-09-05T16:47:27.437187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:27.437187+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T16:53:55.961513+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":33680,"num_objects":5,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2506,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":64707,"num_write_kb":210976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6539,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":20480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084463,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":726344,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":725819,"kb_avail":93645496,"statfs":{"total":96636764160,"available":95892987904,"internally_reserved":0,"allocated":512000,"data_stored":194158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":743239379},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986340,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":627784,"kb_used_data":504,"kb_used_omap":4,"kb_used_meta":627259,"kb_avail":93744056,"statfs":{"total":96636764160,"available":95993913344,"internally_reserved":0,"allocated":516096,"data_stored":198254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":642313938},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673146,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713356,"kb_used_data":500,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":93658484,"statfs":{"total":96636764160,"available":95906287616,"internally_reserved":0,"allocated":512000,"data_stored":194158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":729938373},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182577,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713800,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658040,"statfs":{"total":96636764160,"available":95905832960,"internally_reserved":0,"allocated":512000,"data_stored":194158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":730394317},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:48:39.314 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:48:39.314 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "39'12770", 'reported_seq': 16882, 'reported_epoch': 61, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:48:38.214378+0000', 'last_change': '2024-09-05T16:48:27.426520+0000', 'last_active': '2024-09-05T16:48:38.214378+0000', 'last_peered': '2024-09-05T16:48:38.214378+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:48:38.214378+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "33'2700", 'ondisk_log_start': "33'2700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10070, 'log_dups_size': 0, 'ondisk_log_size': 10070, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26944, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 16, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 267, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12767, 'num_write_kb': 51081, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2000, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -263}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:48:39.314 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:48:39.314 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "39'12758", 'reported_seq': 12907, 'reported_epoch': 61, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:48:29.230950+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:48:29.230950+0000', 'last_peered': '2024-09-05T16:48:29.230950+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:48:29.230950+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "33'2700", 'ondisk_log_start': "33'2700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10058, 'log_dups_size': 0, 'ondisk_log_size': 10058, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6736, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12757, 'num_write_kb': 51031, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2237}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:48:39.314 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:48:39.749 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:48:39.749 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2506/20 objects degraded (12530.000%), 2 pgs degraded, 2 pgs undersized","count":4},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":922,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":61,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":2},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":5,"data_bytes":33680,"bytes_used":2848034816,"bytes_avail":383699021824,"bytes_total":386547056640,"degraded_objects":2506,"degraded_total":20,"degraded_ratio":125.3},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:48:39.762 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-09-05T16:48:39.763 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:48:40.086 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:48:40.087 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:48:40.101 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":513,"stamp":"2024-09-05T16:48:39.119702+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":33680,"num_objects":5,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2506,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":64707,"num_write_kb":210976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6539,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2781284,"kb_used_data":2004,"kb_used_omap":14,"kb_used_meta":2779185,"kb_avail":374706076,"statfs":{"total":386547056640,"available":383699021824,"internally_reserved":0,"allocated":2052096,"data_stored":780728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2845886007},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20208,"num_objects":-3,"num_object_clones":0,"num_object_copies":-12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1871,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-13117,"num_write_kb":-52476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-446,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001701"},"pg_stats":[{"pgid":"1.f","version":"39'12904","reported_seq":17541,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-05T16:48:29.230833+0000","last_change":"2024-09-05T16:46:27.826222+0000","last_active":"2024-09-05T16:48:29.230833+0000","last_peered":"2024-09-05T16:48:29.230833+0000","last_clean":"2024-09-05T16:48:29.230833+0000","last_became_active":"2024-09-05T16:46:27.757928+0000","last_became_peered":"2024-09-05T16:46:27.757928+0000","last_unstale":"2024-09-05T16:48:29.230833+0000","last_undegraded":"2024-09-05T16:48:29.230833+0000","last_fullsized":"2024-09-05T16:48:29.230833+0000","mapping_epoch":40,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:32:35.713428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12904,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2262,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12552","reported_seq":17254,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-05T16:48:29.230646+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:48:29.230646+0000","last_peered":"2024-09-05T16:48:29.230646+0000","last_clean":"2024-09-05T16:48:29.230646+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:48:29.230646+0000","last_undegraded":"2024-09-05T16:48:29.230646+0000","last_fullsized":"2024-09-05T16:48:29.230646+0000","mapping_epoch":38,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1865,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12752","reported_seq":17409,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-05T16:48:29.233031+0000","last_change":"2024-09-05T16:46:51.992073+0000","last_active":"2024-09-05T16:48:29.233031+0000","last_peered":"2024-09-05T16:48:29.233031+0000","last_clean":"2024-09-05T16:48:29.233031+0000","last_became_active":"2024-09-05T16:46:51.667128+0000","last_became_peered":"2024-09-05T16:46:51.667128+0000","last_unstale":"2024-09-05T16:48:29.233031+0000","last_undegraded":"2024-09-05T16:48:29.233031+0000","last_fullsized":"2024-09-05T16:48:29.233031+0000","mapping_epoch":44,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12752","last_scrub_stamp":"2024-09-05T16:46:51.991995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:51.991995+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:04:54.120031+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12934","reported_seq":17714,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-05T16:48:29.230556+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:48:29.230556+0000","last_peered":"2024-09-05T16:48:29.230556+0000","last_clean":"2024-09-05T16:48:29.230556+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:48:29.230556+0000","last_undegraded":"2024-09-05T16:48:29.230556+0000","last_fullsized":"2024-09-05T16:48:29.230556+0000","mapping_epoch":34,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7234,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12593","reported_seq":17055,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-05T16:48:29.230261+0000","last_change":"2024-09-05T16:47:51.525026+0000","last_active":"2024-09-05T16:48:29.230261+0000","last_peered":"2024-09-05T16:48:29.230261+0000","last_clean":"2024-09-05T16:48:29.230261+0000","last_became_active":"2024-09-05T16:47:51.193016+0000","last_became_peered":"2024-09-05T16:47:51.193016+0000","last_unstale":"2024-09-05T16:48:29.230261+0000","last_undegraded":"2024-09-05T16:48:29.230261+0000","last_fullsized":"2024-09-05T16:48:29.230261+0000","mapping_epoch":54,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12593","last_scrub_stamp":"2024-09-05T16:47:51.524957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:51.524957+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:46:22.846859+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12772","reported_seq":17385,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-05T16:48:29.230812+0000","last_change":"2024-09-05T16:47:15.832530+0000","last_active":"2024-09-05T16:48:29.230812+0000","last_peered":"2024-09-05T16:48:29.230812+0000","last_clean":"2024-09-05T16:48:29.230812+0000","last_became_active":"2024-09-05T16:47:15.728759+0000","last_became_peered":"2024-09-05T16:47:15.728759+0000","last_unstale":"2024-09-05T16:48:29.230812+0000","last_undegraded":"2024-09-05T16:48:29.230812+0000","last_fullsized":"2024-09-05T16:48:29.230812+0000","mapping_epoch":48,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T02:39:11.515339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12772,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2250,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12660","reported_seq":17102,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-05T16:48:29.229181+0000","last_change":"2024-09-05T16:47:03.683348+0000","last_active":"2024-09-05T16:48:29.229181+0000","last_peered":"2024-09-05T16:48:29.229181+0000","last_clean":"2024-09-05T16:48:29.229181+0000","last_became_active":"2024-09-05T16:47:03.185143+0000","last_became_peered":"2024-09-05T16:47:03.185143+0000","last_unstale":"2024-09-05T16:48:29.229181+0000","last_undegraded":"2024-09-05T16:48:29.229181+0000","last_fullsized":"2024-09-05T16:48:29.229181+0000","mapping_epoch":46,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12660","last_scrub_stamp":"2024-09-05T16:47:03.683262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:03.683262+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:42:16.533299+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12777","reported_seq":12930,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-05T16:48:29.230218+0000","last_change":"2024-09-05T16:46:40.458315+0000","last_active":"2024-09-05T16:48:29.230218+0000","last_peered":"2024-09-05T16:48:29.230218+0000","last_clean":"2024-09-05T16:48:29.230218+0000","last_became_active":"2024-09-05T16:46:40.184288+0000","last_became_peered":"2024-09-05T16:46:40.184288+0000","last_unstale":"2024-09-05T16:48:29.230218+0000","last_undegraded":"2024-09-05T16:48:29.230218+0000","last_fullsized":"2024-09-05T16:48:29.230218+0000","mapping_epoch":42,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12777","last_scrub_stamp":"2024-09-05T16:46:40.458226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:40.458226+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:38:58.734210+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'12770","reported_seq":16882,"reported_epoch":61,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:48:38.214378+0000","last_change":"2024-09-05T16:48:27.426520+0000","last_active":"2024-09-05T16:48:38.214378+0000","last_peered":"2024-09-05T16:48:38.214378+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:48:38.214378+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":267,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12767,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2000,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-263}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12808","reported_seq":17493,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-05T16:48:29.233129+0000","last_change":"2024-09-05T16:48:04.288466+0000","last_active":"2024-09-05T16:48:29.233129+0000","last_peered":"2024-09-05T16:48:29.233129+0000","last_clean":"2024-09-05T16:48:29.233129+0000","last_became_active":"2024-09-05T16:48:03.239894+0000","last_became_peered":"2024-09-05T16:48:03.239894+0000","last_unstale":"2024-09-05T16:48:29.233129+0000","last_undegraded":"2024-09-05T16:48:29.233129+0000","last_fullsized":"2024-09-05T16:48:29.233129+0000","mapping_epoch":56,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12808","last_scrub_stamp":"2024-09-05T16:48:04.288388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:48:04.288388+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T21:11:32.044919+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13120","reported_seq":17883,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-05T16:48:29.529847+0000","last_change":"2024-09-05T16:48:29.529847+0000","last_active":"2024-09-05T16:48:29.529847+0000","last_peered":"2024-09-05T16:48:29.529847+0000","last_clean":"2024-09-05T16:48:29.529847+0000","last_became_active":"2024-09-05T16:48:29.235000+0000","last_became_peered":"2024-09-05T16:48:29.235000+0000","last_unstale":"2024-09-05T16:48:29.529847+0000","last_undegraded":"2024-09-05T16:48:29.529847+0000","last_fullsized":"2024-09-05T16:48:29.529847+0000","mapping_epoch":60,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'13120","last_scrub_stamp":"2024-09-05T16:48:29.529768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:48:29.529768+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T02:18:10.992722+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12758","reported_seq":12907,"reported_epoch":61,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:48:29.230950+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:48:29.230950+0000","last_peered":"2024-09-05T16:48:29.230950+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:48:29.230950+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2237}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12808","reported_seq":17594,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-05T16:48:29.230528+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:48:29.230528+0000","last_peered":"2024-09-05T16:48:29.230528+0000","last_clean":"2024-09-05T16:48:29.230528+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:48:29.230528+0000","last_undegraded":"2024-09-05T16:48:29.230528+0000","last_fullsized":"2024-09-05T16:48:29.230528+0000","mapping_epoch":36,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4408,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13124","reported_seq":13265,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-05T16:48:29.230184+0000","last_change":"2024-09-05T16:48:16.582142+0000","last_active":"2024-09-05T16:48:29.230184+0000","last_peered":"2024-09-05T16:48:29.230184+0000","last_clean":"2024-09-05T16:48:29.230184+0000","last_became_active":"2024-09-05T16:48:16.435091+0000","last_became_peered":"2024-09-05T16:48:16.435091+0000","last_unstale":"2024-09-05T16:48:29.230184+0000","last_undegraded":"2024-09-05T16:48:29.230184+0000","last_fullsized":"2024-09-05T16:48:29.230184+0000","mapping_epoch":58,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'13124","last_scrub_stamp":"2024-09-05T16:48:16.582038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:48:16.582038+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T17:44:25.891565+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"39'12831","reported_seq":12988,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-05T16:48:29.230099+0000","last_change":"2024-09-05T16:47:40.451380+0000","last_active":"2024-09-05T16:48:29.230099+0000","last_peered":"2024-09-05T16:48:29.230099+0000","last_clean":"2024-09-05T16:48:29.230099+0000","last_became_active":"2024-09-05T16:47:40.189447+0000","last_became_peered":"2024-09-05T16:47:40.189447+0000","last_unstale":"2024-09-05T16:48:29.230099+0000","last_undegraded":"2024-09-05T16:48:29.230099+0000","last_fullsized":"2024-09-05T16:48:29.230099+0000","mapping_epoch":52,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12831","last_scrub_stamp":"2024-09-05T16:47:40.451295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:40.451295+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:46:20.946333+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12773","reported_seq":12930,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-05T16:48:29.230307+0000","last_change":"2024-09-05T16:47:27.437275+0000","last_active":"2024-09-05T16:48:29.230307+0000","last_peered":"2024-09-05T16:48:29.230307+0000","last_clean":"2024-09-05T16:48:29.230307+0000","last_became_active":"2024-09-05T16:47:27.204989+0000","last_became_peered":"2024-09-05T16:47:27.204989+0000","last_unstale":"2024-09-05T16:48:29.230307+0000","last_undegraded":"2024-09-05T16:48:29.230307+0000","last_fullsized":"2024-09-05T16:48:29.230307+0000","mapping_epoch":50,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12773","last_scrub_stamp":"2024-09-05T16:47:27.437187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:27.437187+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T16:53:55.961513+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":33680,"num_objects":5,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2506,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":64707,"num_write_kb":210976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6539,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":20480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084463,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":726344,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":725819,"kb_avail":93645496,"statfs":{"total":96636764160,"available":95892987904,"internally_reserved":0,"allocated":512000,"data_stored":194158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":743239379},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986340,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":627784,"kb_used_data":504,"kb_used_omap":4,"kb_used_meta":627259,"kb_avail":93744056,"statfs":{"total":96636764160,"available":95993913344,"internally_reserved":0,"allocated":516096,"data_stored":198254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":642313938},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673146,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713356,"kb_used_data":500,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":93658484,"statfs":{"total":96636764160,"available":95906287616,"internally_reserved":0,"allocated":512000,"data_stored":194158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":729938373},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182577,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713800,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658040,"statfs":{"total":96636764160,"available":95905832960,"internally_reserved":0,"allocated":512000,"data_stored":194158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":730394317},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:48:43.103 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:48:43.990 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:48:43.991 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:48:44.003 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":517,"stamp":"2024-09-05T16:48:43.120401+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":33680,"num_objects":5,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2506,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":64707,"num_write_kb":210976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6539,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2789628,"kb_used_data":2032,"kb_used_omap":14,"kb_used_meta":2787505,"kb_avail":374697732,"statfs":{"total":386547056640,"available":383690477568,"internally_reserved":0,"allocated":2080768,"data_stored":789031,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2854405687},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1042,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1042,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001300"},"pg_stats":[{"pgid":"1.f","version":"39'12904","reported_seq":17541,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-05T16:48:29.230833+0000","last_change":"2024-09-05T16:46:27.826222+0000","last_active":"2024-09-05T16:48:29.230833+0000","last_peered":"2024-09-05T16:48:29.230833+0000","last_clean":"2024-09-05T16:48:29.230833+0000","last_became_active":"2024-09-05T16:46:27.757928+0000","last_became_peered":"2024-09-05T16:46:27.757928+0000","last_unstale":"2024-09-05T16:48:29.230833+0000","last_undegraded":"2024-09-05T16:48:29.230833+0000","last_fullsized":"2024-09-05T16:48:29.230833+0000","mapping_epoch":40,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:32:35.713428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12904,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2262,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12552","reported_seq":17258,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-05T16:48:41.234615+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:48:41.234615+0000","last_peered":"2024-09-05T16:48:41.234615+0000","last_clean":"2024-09-05T16:48:41.234615+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:48:41.234615+0000","last_undegraded":"2024-09-05T16:48:41.234615+0000","last_fullsized":"2024-09-05T16:48:41.234615+0000","mapping_epoch":38,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1865,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12752","reported_seq":17411,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-05T16:48:40.225783+0000","last_change":"2024-09-05T16:46:51.992073+0000","last_active":"2024-09-05T16:48:40.225783+0000","last_peered":"2024-09-05T16:48:40.225783+0000","last_clean":"2024-09-05T16:48:40.225783+0000","last_became_active":"2024-09-05T16:46:51.667128+0000","last_became_peered":"2024-09-05T16:46:51.667128+0000","last_unstale":"2024-09-05T16:48:40.225783+0000","last_undegraded":"2024-09-05T16:48:40.225783+0000","last_fullsized":"2024-09-05T16:48:40.225783+0000","mapping_epoch":44,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12752","last_scrub_stamp":"2024-09-05T16:46:51.991995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:51.991995+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:04:54.120031+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12934","reported_seq":17718,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-05T16:48:41.234020+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:48:41.234020+0000","last_peered":"2024-09-05T16:48:41.234020+0000","last_clean":"2024-09-05T16:48:41.234020+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:48:41.234020+0000","last_undegraded":"2024-09-05T16:48:41.234020+0000","last_fullsized":"2024-09-05T16:48:41.234020+0000","mapping_epoch":34,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7234,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12593","reported_seq":17055,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-05T16:48:29.230261+0000","last_change":"2024-09-05T16:47:51.525026+0000","last_active":"2024-09-05T16:48:29.230261+0000","last_peered":"2024-09-05T16:48:29.230261+0000","last_clean":"2024-09-05T16:48:29.230261+0000","last_became_active":"2024-09-05T16:47:51.193016+0000","last_became_peered":"2024-09-05T16:47:51.193016+0000","last_unstale":"2024-09-05T16:48:29.230261+0000","last_undegraded":"2024-09-05T16:48:29.230261+0000","last_fullsized":"2024-09-05T16:48:29.230261+0000","mapping_epoch":54,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12593","last_scrub_stamp":"2024-09-05T16:47:51.524957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:51.524957+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:46:22.846859+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12772","reported_seq":17385,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-05T16:48:29.230812+0000","last_change":"2024-09-05T16:47:15.832530+0000","last_active":"2024-09-05T16:48:29.230812+0000","last_peered":"2024-09-05T16:48:29.230812+0000","last_clean":"2024-09-05T16:48:29.230812+0000","last_became_active":"2024-09-05T16:47:15.728759+0000","last_became_peered":"2024-09-05T16:47:15.728759+0000","last_unstale":"2024-09-05T16:48:29.230812+0000","last_undegraded":"2024-09-05T16:48:29.230812+0000","last_fullsized":"2024-09-05T16:48:29.230812+0000","mapping_epoch":48,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T02:39:11.515339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12772,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2250,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12660","reported_seq":17106,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-05T16:48:41.231170+0000","last_change":"2024-09-05T16:47:03.683348+0000","last_active":"2024-09-05T16:48:41.231170+0000","last_peered":"2024-09-05T16:48:41.231170+0000","last_clean":"2024-09-05T16:48:41.231170+0000","last_became_active":"2024-09-05T16:47:03.185143+0000","last_became_peered":"2024-09-05T16:47:03.185143+0000","last_unstale":"2024-09-05T16:48:41.231170+0000","last_undegraded":"2024-09-05T16:48:41.231170+0000","last_fullsized":"2024-09-05T16:48:41.231170+0000","mapping_epoch":46,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12660","last_scrub_stamp":"2024-09-05T16:47:03.683262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:03.683262+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:42:16.533299+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12777","reported_seq":12934,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-05T16:48:41.234021+0000","last_change":"2024-09-05T16:46:40.458315+0000","last_active":"2024-09-05T16:48:41.234021+0000","last_peered":"2024-09-05T16:48:41.234021+0000","last_clean":"2024-09-05T16:48:41.234021+0000","last_became_active":"2024-09-05T16:46:40.184288+0000","last_became_peered":"2024-09-05T16:46:40.184288+0000","last_unstale":"2024-09-05T16:48:41.234021+0000","last_undegraded":"2024-09-05T16:48:41.234021+0000","last_fullsized":"2024-09-05T16:48:41.234021+0000","mapping_epoch":42,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12777","last_scrub_stamp":"2024-09-05T16:46:40.458226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:40.458226+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:38:58.734210+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'12770","reported_seq":16882,"reported_epoch":61,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:48:38.214378+0000","last_change":"2024-09-05T16:48:27.426520+0000","last_active":"2024-09-05T16:48:38.214378+0000","last_peered":"2024-09-05T16:48:38.214378+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:48:38.214378+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":267,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12767,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2000,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-263}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12808","reported_seq":17495,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-05T16:48:40.225796+0000","last_change":"2024-09-05T16:48:04.288466+0000","last_active":"2024-09-05T16:48:40.225796+0000","last_peered":"2024-09-05T16:48:40.225796+0000","last_clean":"2024-09-05T16:48:40.225796+0000","last_became_active":"2024-09-05T16:48:03.239894+0000","last_became_peered":"2024-09-05T16:48:03.239894+0000","last_unstale":"2024-09-05T16:48:40.225796+0000","last_undegraded":"2024-09-05T16:48:40.225796+0000","last_fullsized":"2024-09-05T16:48:40.225796+0000","mapping_epoch":56,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12808","last_scrub_stamp":"2024-09-05T16:48:04.288388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:48:04.288388+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T21:11:32.044919+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13120","reported_seq":17883,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-05T16:48:29.529847+0000","last_change":"2024-09-05T16:48:29.529847+0000","last_active":"2024-09-05T16:48:29.529847+0000","last_peered":"2024-09-05T16:48:29.529847+0000","last_clean":"2024-09-05T16:48:29.529847+0000","last_became_active":"2024-09-05T16:48:29.235000+0000","last_became_peered":"2024-09-05T16:48:29.235000+0000","last_unstale":"2024-09-05T16:48:29.529847+0000","last_undegraded":"2024-09-05T16:48:29.529847+0000","last_fullsized":"2024-09-05T16:48:29.529847+0000","mapping_epoch":60,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'13120","last_scrub_stamp":"2024-09-05T16:48:29.529768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:48:29.529768+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T02:18:10.992722+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12758","reported_seq":12907,"reported_epoch":61,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:48:29.230950+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:48:29.230950+0000","last_peered":"2024-09-05T16:48:29.230950+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:48:29.230950+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2237}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12808","reported_seq":17598,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-05T16:48:41.234065+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:48:41.234065+0000","last_peered":"2024-09-05T16:48:41.234065+0000","last_clean":"2024-09-05T16:48:41.234065+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:48:41.234065+0000","last_undegraded":"2024-09-05T16:48:41.234065+0000","last_fullsized":"2024-09-05T16:48:41.234065+0000","mapping_epoch":36,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4408,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13124","reported_seq":13269,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-05T16:48:41.234257+0000","last_change":"2024-09-05T16:48:16.582142+0000","last_active":"2024-09-05T16:48:41.234257+0000","last_peered":"2024-09-05T16:48:41.234257+0000","last_clean":"2024-09-05T16:48:41.234257+0000","last_became_active":"2024-09-05T16:48:16.435091+0000","last_became_peered":"2024-09-05T16:48:16.435091+0000","last_unstale":"2024-09-05T16:48:41.234257+0000","last_undegraded":"2024-09-05T16:48:41.234257+0000","last_fullsized":"2024-09-05T16:48:41.234257+0000","mapping_epoch":58,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'13124","last_scrub_stamp":"2024-09-05T16:48:16.582038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:48:16.582038+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T17:44:25.891565+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"39'12831","reported_seq":12992,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-05T16:48:41.234342+0000","last_change":"2024-09-05T16:47:40.451380+0000","last_active":"2024-09-05T16:48:41.234342+0000","last_peered":"2024-09-05T16:48:41.234342+0000","last_clean":"2024-09-05T16:48:41.234342+0000","last_became_active":"2024-09-05T16:47:40.189447+0000","last_became_peered":"2024-09-05T16:47:40.189447+0000","last_unstale":"2024-09-05T16:48:41.234342+0000","last_undegraded":"2024-09-05T16:48:41.234342+0000","last_fullsized":"2024-09-05T16:48:41.234342+0000","mapping_epoch":52,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12831","last_scrub_stamp":"2024-09-05T16:47:40.451295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:40.451295+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:46:20.946333+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12773","reported_seq":12934,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-05T16:48:41.234360+0000","last_change":"2024-09-05T16:47:27.437275+0000","last_active":"2024-09-05T16:48:41.234360+0000","last_peered":"2024-09-05T16:48:41.234360+0000","last_clean":"2024-09-05T16:48:41.234360+0000","last_became_active":"2024-09-05T16:47:27.204989+0000","last_became_peered":"2024-09-05T16:47:27.204989+0000","last_unstale":"2024-09-05T16:48:41.234360+0000","last_undegraded":"2024-09-05T16:48:41.234360+0000","last_fullsized":"2024-09-05T16:48:41.234360+0000","mapping_epoch":50,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12773","last_scrub_stamp":"2024-09-05T16:47:27.437187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:27.437187+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T16:53:55.961513+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":33680,"num_objects":5,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2506,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":64707,"num_write_kb":210976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6539,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084463,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":726344,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":725819,"kb_avail":93645496,"statfs":{"total":96636764160,"available":95892987904,"internally_reserved":0,"allocated":512000,"data_stored":194158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":743239379},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986341,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":636116,"kb_used_data":516,"kb_used_omap":4,"kb_used_meta":635579,"kb_avail":93735724,"statfs":{"total":96636764160,"available":95985381376,"internally_reserved":0,"allocated":528384,"data_stored":200347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":650833618},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673147,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713360,"kb_used_data":508,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":93658480,"statfs":{"total":96636764160,"available":95906283520,"internally_reserved":0,"allocated":520192,"data_stored":197263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":729938373},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182578,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713808,"kb_used_data":508,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658032,"statfs":{"total":96636764160,"available":95905824768,"internally_reserved":0,"allocated":520192,"data_stored":197263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":730394317},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:48:44.004 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:48:44.314 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:48:44.315 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:48:44.329 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":517,"stamp":"2024-09-05T16:48:43.120401+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":33680,"num_objects":5,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2506,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":64707,"num_write_kb":210976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6539,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2789628,"kb_used_data":2032,"kb_used_omap":14,"kb_used_meta":2787505,"kb_avail":374697732,"statfs":{"total":386547056640,"available":383690477568,"internally_reserved":0,"allocated":2080768,"data_stored":789031,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2854405687},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1042,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1042,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001300"},"pg_stats":[{"pgid":"1.f","version":"39'12904","reported_seq":17541,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-05T16:48:29.230833+0000","last_change":"2024-09-05T16:46:27.826222+0000","last_active":"2024-09-05T16:48:29.230833+0000","last_peered":"2024-09-05T16:48:29.230833+0000","last_clean":"2024-09-05T16:48:29.230833+0000","last_became_active":"2024-09-05T16:46:27.757928+0000","last_became_peered":"2024-09-05T16:46:27.757928+0000","last_unstale":"2024-09-05T16:48:29.230833+0000","last_undegraded":"2024-09-05T16:48:29.230833+0000","last_fullsized":"2024-09-05T16:48:29.230833+0000","mapping_epoch":40,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:32:35.713428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12904,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2262,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12552","reported_seq":17258,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-05T16:48:41.234615+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:48:41.234615+0000","last_peered":"2024-09-05T16:48:41.234615+0000","last_clean":"2024-09-05T16:48:41.234615+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:48:41.234615+0000","last_undegraded":"2024-09-05T16:48:41.234615+0000","last_fullsized":"2024-09-05T16:48:41.234615+0000","mapping_epoch":38,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1865,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12752","reported_seq":17411,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-05T16:48:40.225783+0000","last_change":"2024-09-05T16:46:51.992073+0000","last_active":"2024-09-05T16:48:40.225783+0000","last_peered":"2024-09-05T16:48:40.225783+0000","last_clean":"2024-09-05T16:48:40.225783+0000","last_became_active":"2024-09-05T16:46:51.667128+0000","last_became_peered":"2024-09-05T16:46:51.667128+0000","last_unstale":"2024-09-05T16:48:40.225783+0000","last_undegraded":"2024-09-05T16:48:40.225783+0000","last_fullsized":"2024-09-05T16:48:40.225783+0000","mapping_epoch":44,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12752","last_scrub_stamp":"2024-09-05T16:46:51.991995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:51.991995+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:04:54.120031+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12934","reported_seq":17718,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-05T16:48:41.234020+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:48:41.234020+0000","last_peered":"2024-09-05T16:48:41.234020+0000","last_clean":"2024-09-05T16:48:41.234020+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:48:41.234020+0000","last_undegraded":"2024-09-05T16:48:41.234020+0000","last_fullsized":"2024-09-05T16:48:41.234020+0000","mapping_epoch":34,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7234,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12593","reported_seq":17055,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-05T16:48:29.230261+0000","last_change":"2024-09-05T16:47:51.525026+0000","last_active":"2024-09-05T16:48:29.230261+0000","last_peered":"2024-09-05T16:48:29.230261+0000","last_clean":"2024-09-05T16:48:29.230261+0000","last_became_active":"2024-09-05T16:47:51.193016+0000","last_became_peered":"2024-09-05T16:47:51.193016+0000","last_unstale":"2024-09-05T16:48:29.230261+0000","last_undegraded":"2024-09-05T16:48:29.230261+0000","last_fullsized":"2024-09-05T16:48:29.230261+0000","mapping_epoch":54,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12593","last_scrub_stamp":"2024-09-05T16:47:51.524957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:51.524957+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:46:22.846859+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12772","reported_seq":17385,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-05T16:48:29.230812+0000","last_change":"2024-09-05T16:47:15.832530+0000","last_active":"2024-09-05T16:48:29.230812+0000","last_peered":"2024-09-05T16:48:29.230812+0000","last_clean":"2024-09-05T16:48:29.230812+0000","last_became_active":"2024-09-05T16:47:15.728759+0000","last_became_peered":"2024-09-05T16:47:15.728759+0000","last_unstale":"2024-09-05T16:48:29.230812+0000","last_undegraded":"2024-09-05T16:48:29.230812+0000","last_fullsized":"2024-09-05T16:48:29.230812+0000","mapping_epoch":48,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T02:39:11.515339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12772,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2250,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12660","reported_seq":17106,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-05T16:48:41.231170+0000","last_change":"2024-09-05T16:47:03.683348+0000","last_active":"2024-09-05T16:48:41.231170+0000","last_peered":"2024-09-05T16:48:41.231170+0000","last_clean":"2024-09-05T16:48:41.231170+0000","last_became_active":"2024-09-05T16:47:03.185143+0000","last_became_peered":"2024-09-05T16:47:03.185143+0000","last_unstale":"2024-09-05T16:48:41.231170+0000","last_undegraded":"2024-09-05T16:48:41.231170+0000","last_fullsized":"2024-09-05T16:48:41.231170+0000","mapping_epoch":46,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12660","last_scrub_stamp":"2024-09-05T16:47:03.683262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:03.683262+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:42:16.533299+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12777","reported_seq":12934,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-05T16:48:41.234021+0000","last_change":"2024-09-05T16:46:40.458315+0000","last_active":"2024-09-05T16:48:41.234021+0000","last_peered":"2024-09-05T16:48:41.234021+0000","last_clean":"2024-09-05T16:48:41.234021+0000","last_became_active":"2024-09-05T16:46:40.184288+0000","last_became_peered":"2024-09-05T16:46:40.184288+0000","last_unstale":"2024-09-05T16:48:41.234021+0000","last_undegraded":"2024-09-05T16:48:41.234021+0000","last_fullsized":"2024-09-05T16:48:41.234021+0000","mapping_epoch":42,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12777","last_scrub_stamp":"2024-09-05T16:46:40.458226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:40.458226+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:38:58.734210+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'12770","reported_seq":16882,"reported_epoch":61,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:48:38.214378+0000","last_change":"2024-09-05T16:48:27.426520+0000","last_active":"2024-09-05T16:48:38.214378+0000","last_peered":"2024-09-05T16:48:38.214378+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:48:38.214378+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":267,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12767,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2000,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-263}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12808","reported_seq":17495,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-05T16:48:40.225796+0000","last_change":"2024-09-05T16:48:04.288466+0000","last_active":"2024-09-05T16:48:40.225796+0000","last_peered":"2024-09-05T16:48:40.225796+0000","last_clean":"2024-09-05T16:48:40.225796+0000","last_became_active":"2024-09-05T16:48:03.239894+0000","last_became_peered":"2024-09-05T16:48:03.239894+0000","last_unstale":"2024-09-05T16:48:40.225796+0000","last_undegraded":"2024-09-05T16:48:40.225796+0000","last_fullsized":"2024-09-05T16:48:40.225796+0000","mapping_epoch":56,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12808","last_scrub_stamp":"2024-09-05T16:48:04.288388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:48:04.288388+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T21:11:32.044919+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13120","reported_seq":17883,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-05T16:48:29.529847+0000","last_change":"2024-09-05T16:48:29.529847+0000","last_active":"2024-09-05T16:48:29.529847+0000","last_peered":"2024-09-05T16:48:29.529847+0000","last_clean":"2024-09-05T16:48:29.529847+0000","last_became_active":"2024-09-05T16:48:29.235000+0000","last_became_peered":"2024-09-05T16:48:29.235000+0000","last_unstale":"2024-09-05T16:48:29.529847+0000","last_undegraded":"2024-09-05T16:48:29.529847+0000","last_fullsized":"2024-09-05T16:48:29.529847+0000","mapping_epoch":60,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'13120","last_scrub_stamp":"2024-09-05T16:48:29.529768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:48:29.529768+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T02:18:10.992722+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12758","reported_seq":12907,"reported_epoch":61,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:48:29.230950+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:48:29.230950+0000","last_peered":"2024-09-05T16:48:29.230950+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:48:29.230950+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2237}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12808","reported_seq":17598,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-05T16:48:41.234065+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:48:41.234065+0000","last_peered":"2024-09-05T16:48:41.234065+0000","last_clean":"2024-09-05T16:48:41.234065+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:48:41.234065+0000","last_undegraded":"2024-09-05T16:48:41.234065+0000","last_fullsized":"2024-09-05T16:48:41.234065+0000","mapping_epoch":36,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4408,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13124","reported_seq":13269,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-05T16:48:41.234257+0000","last_change":"2024-09-05T16:48:16.582142+0000","last_active":"2024-09-05T16:48:41.234257+0000","last_peered":"2024-09-05T16:48:41.234257+0000","last_clean":"2024-09-05T16:48:41.234257+0000","last_became_active":"2024-09-05T16:48:16.435091+0000","last_became_peered":"2024-09-05T16:48:16.435091+0000","last_unstale":"2024-09-05T16:48:41.234257+0000","last_undegraded":"2024-09-05T16:48:41.234257+0000","last_fullsized":"2024-09-05T16:48:41.234257+0000","mapping_epoch":58,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'13124","last_scrub_stamp":"2024-09-05T16:48:16.582038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:48:16.582038+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T17:44:25.891565+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"39'12831","reported_seq":12992,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-05T16:48:41.234342+0000","last_change":"2024-09-05T16:47:40.451380+0000","last_active":"2024-09-05T16:48:41.234342+0000","last_peered":"2024-09-05T16:48:41.234342+0000","last_clean":"2024-09-05T16:48:41.234342+0000","last_became_active":"2024-09-05T16:47:40.189447+0000","last_became_peered":"2024-09-05T16:47:40.189447+0000","last_unstale":"2024-09-05T16:48:41.234342+0000","last_undegraded":"2024-09-05T16:48:41.234342+0000","last_fullsized":"2024-09-05T16:48:41.234342+0000","mapping_epoch":52,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12831","last_scrub_stamp":"2024-09-05T16:47:40.451295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:40.451295+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:46:20.946333+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12773","reported_seq":12934,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-05T16:48:41.234360+0000","last_change":"2024-09-05T16:47:27.437275+0000","last_active":"2024-09-05T16:48:41.234360+0000","last_peered":"2024-09-05T16:48:41.234360+0000","last_clean":"2024-09-05T16:48:41.234360+0000","last_became_active":"2024-09-05T16:47:27.204989+0000","last_became_peered":"2024-09-05T16:47:27.204989+0000","last_unstale":"2024-09-05T16:48:41.234360+0000","last_undegraded":"2024-09-05T16:48:41.234360+0000","last_fullsized":"2024-09-05T16:48:41.234360+0000","mapping_epoch":50,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12773","last_scrub_stamp":"2024-09-05T16:47:27.437187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:27.437187+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T16:53:55.961513+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":33680,"num_objects":5,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2506,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":64707,"num_write_kb":210976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6539,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084463,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":726344,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":725819,"kb_avail":93645496,"statfs":{"total":96636764160,"available":95892987904,"internally_reserved":0,"allocated":512000,"data_stored":194158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":743239379},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986341,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":636116,"kb_used_data":516,"kb_used_omap":4,"kb_used_meta":635579,"kb_avail":93735724,"statfs":{"total":96636764160,"available":95985381376,"internally_reserved":0,"allocated":528384,"data_stored":200347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":650833618},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673147,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713360,"kb_used_data":508,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":93658480,"statfs":{"total":96636764160,"available":95906283520,"internally_reserved":0,"allocated":520192,"data_stored":197263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":729938373},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182578,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713808,"kb_used_data":508,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658032,"statfs":{"total":96636764160,"available":95905824768,"internally_reserved":0,"allocated":520192,"data_stored":197263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":730394317},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:48:44.330 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-05T16:48:44.330 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "39'12770", 'reported_seq': 16882, 'reported_epoch': 61, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:48:38.214378+0000', 'last_change': '2024-09-05T16:48:27.426520+0000', 'last_active': '2024-09-05T16:48:38.214378+0000', 'last_peered': '2024-09-05T16:48:38.214378+0000', 'last_clean': '2024-09-05T16:33:53.354779+0000', 'last_became_active': '2024-09-05T16:34:56.962613+0000', 'last_became_peered': '2024-09-05T16:34:56.962613+0000', 'last_unstale': '2024-09-05T16:48:38.214378+0000', 'last_undegraded': '2024-09-05T16:34:56.632414+0000', 'last_fullsized': '2024-09-05T16:34:56.557335+0000', 'mapping_epoch': 32, 'log_start': "33'2700", 'ondisk_log_start': "33'2700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10070, 'log_dups_size': 0, 'ondisk_log_size': 10070, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26944, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 16, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 267, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12767, 'num_write_kb': 51081, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2000, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -263}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:48:44.330 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:48:44.330 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "39'12758", 'reported_seq': 12907, 'reported_epoch': 61, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:48:29.230950+0000', 'last_change': '2024-09-05T16:34:56.985136+0000', 'last_active': '2024-09-05T16:48:29.230950+0000', 'last_peered': '2024-09-05T16:48:29.230950+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:48:29.230950+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "33'2700", 'ondisk_log_start': "33'2700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10058, 'log_dups_size': 0, 'ondisk_log_size': 10058, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6736, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12757, 'num_write_kb': 51031, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2237}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:48:44.331 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:48:44.741 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:48:44.742 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2506/20 objects degraded (12530.000%), 2 pgs degraded, 2 pgs undersized","count":4},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":927,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":63,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":5,"data_bytes":33680,"bytes_used":2856579072,"bytes_avail":383690477568,"bytes_total":386547056640,"degraded_objects":2506,"degraded_total":20,"degraded_ratio":125.3,"recovering_objects_per_sec":130,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1042,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:48:44.757 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-05T16:48:44.757 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:48:45.069 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:48:45.070 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:48:45.082 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":517,"stamp":"2024-09-05T16:48:43.120401+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":33680,"num_objects":5,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2506,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":64707,"num_write_kb":210976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6539,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2789628,"kb_used_data":2032,"kb_used_omap":14,"kb_used_meta":2787505,"kb_avail":374697732,"statfs":{"total":386547056640,"available":383690477568,"internally_reserved":0,"allocated":2080768,"data_stored":789031,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2854405687},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1042,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1042,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001300"},"pg_stats":[{"pgid":"1.f","version":"39'12904","reported_seq":17541,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-05T16:48:29.230833+0000","last_change":"2024-09-05T16:46:27.826222+0000","last_active":"2024-09-05T16:48:29.230833+0000","last_peered":"2024-09-05T16:48:29.230833+0000","last_clean":"2024-09-05T16:48:29.230833+0000","last_became_active":"2024-09-05T16:46:27.757928+0000","last_became_peered":"2024-09-05T16:46:27.757928+0000","last_unstale":"2024-09-05T16:48:29.230833+0000","last_undegraded":"2024-09-05T16:48:29.230833+0000","last_fullsized":"2024-09-05T16:48:29.230833+0000","mapping_epoch":40,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:32:35.713428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12904,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2262,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12552","reported_seq":17258,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-05T16:48:41.234615+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:48:41.234615+0000","last_peered":"2024-09-05T16:48:41.234615+0000","last_clean":"2024-09-05T16:48:41.234615+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:48:41.234615+0000","last_undegraded":"2024-09-05T16:48:41.234615+0000","last_fullsized":"2024-09-05T16:48:41.234615+0000","mapping_epoch":38,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1865,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12752","reported_seq":17411,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-05T16:48:40.225783+0000","last_change":"2024-09-05T16:46:51.992073+0000","last_active":"2024-09-05T16:48:40.225783+0000","last_peered":"2024-09-05T16:48:40.225783+0000","last_clean":"2024-09-05T16:48:40.225783+0000","last_became_active":"2024-09-05T16:46:51.667128+0000","last_became_peered":"2024-09-05T16:46:51.667128+0000","last_unstale":"2024-09-05T16:48:40.225783+0000","last_undegraded":"2024-09-05T16:48:40.225783+0000","last_fullsized":"2024-09-05T16:48:40.225783+0000","mapping_epoch":44,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12752","last_scrub_stamp":"2024-09-05T16:46:51.991995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:51.991995+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:04:54.120031+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12934","reported_seq":17718,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-05T16:48:41.234020+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:48:41.234020+0000","last_peered":"2024-09-05T16:48:41.234020+0000","last_clean":"2024-09-05T16:48:41.234020+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:48:41.234020+0000","last_undegraded":"2024-09-05T16:48:41.234020+0000","last_fullsized":"2024-09-05T16:48:41.234020+0000","mapping_epoch":34,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7234,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12593","reported_seq":17055,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-05T16:48:29.230261+0000","last_change":"2024-09-05T16:47:51.525026+0000","last_active":"2024-09-05T16:48:29.230261+0000","last_peered":"2024-09-05T16:48:29.230261+0000","last_clean":"2024-09-05T16:48:29.230261+0000","last_became_active":"2024-09-05T16:47:51.193016+0000","last_became_peered":"2024-09-05T16:47:51.193016+0000","last_unstale":"2024-09-05T16:48:29.230261+0000","last_undegraded":"2024-09-05T16:48:29.230261+0000","last_fullsized":"2024-09-05T16:48:29.230261+0000","mapping_epoch":54,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12593","last_scrub_stamp":"2024-09-05T16:47:51.524957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:51.524957+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:46:22.846859+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12772","reported_seq":17385,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-05T16:48:29.230812+0000","last_change":"2024-09-05T16:47:15.832530+0000","last_active":"2024-09-05T16:48:29.230812+0000","last_peered":"2024-09-05T16:48:29.230812+0000","last_clean":"2024-09-05T16:48:29.230812+0000","last_became_active":"2024-09-05T16:47:15.728759+0000","last_became_peered":"2024-09-05T16:47:15.728759+0000","last_unstale":"2024-09-05T16:48:29.230812+0000","last_undegraded":"2024-09-05T16:48:29.230812+0000","last_fullsized":"2024-09-05T16:48:29.230812+0000","mapping_epoch":48,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T02:39:11.515339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12772,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2250,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12660","reported_seq":17106,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-05T16:48:41.231170+0000","last_change":"2024-09-05T16:47:03.683348+0000","last_active":"2024-09-05T16:48:41.231170+0000","last_peered":"2024-09-05T16:48:41.231170+0000","last_clean":"2024-09-05T16:48:41.231170+0000","last_became_active":"2024-09-05T16:47:03.185143+0000","last_became_peered":"2024-09-05T16:47:03.185143+0000","last_unstale":"2024-09-05T16:48:41.231170+0000","last_undegraded":"2024-09-05T16:48:41.231170+0000","last_fullsized":"2024-09-05T16:48:41.231170+0000","mapping_epoch":46,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12660","last_scrub_stamp":"2024-09-05T16:47:03.683262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:03.683262+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:42:16.533299+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12777","reported_seq":12934,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-05T16:48:41.234021+0000","last_change":"2024-09-05T16:46:40.458315+0000","last_active":"2024-09-05T16:48:41.234021+0000","last_peered":"2024-09-05T16:48:41.234021+0000","last_clean":"2024-09-05T16:48:41.234021+0000","last_became_active":"2024-09-05T16:46:40.184288+0000","last_became_peered":"2024-09-05T16:46:40.184288+0000","last_unstale":"2024-09-05T16:48:41.234021+0000","last_undegraded":"2024-09-05T16:48:41.234021+0000","last_fullsized":"2024-09-05T16:48:41.234021+0000","mapping_epoch":42,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12777","last_scrub_stamp":"2024-09-05T16:46:40.458226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:40.458226+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:38:58.734210+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'12770","reported_seq":16882,"reported_epoch":61,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:48:38.214378+0000","last_change":"2024-09-05T16:48:27.426520+0000","last_active":"2024-09-05T16:48:38.214378+0000","last_peered":"2024-09-05T16:48:38.214378+0000","last_clean":"2024-09-05T16:33:53.354779+0000","last_became_active":"2024-09-05T16:34:56.962613+0000","last_became_peered":"2024-09-05T16:34:56.962613+0000","last_unstale":"2024-09-05T16:48:38.214378+0000","last_undegraded":"2024-09-05T16:34:56.632414+0000","last_fullsized":"2024-09-05T16:34:56.557335+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":267,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12767,"num_write_kb":51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2000,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-263}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12808","reported_seq":17495,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-05T16:48:40.225796+0000","last_change":"2024-09-05T16:48:04.288466+0000","last_active":"2024-09-05T16:48:40.225796+0000","last_peered":"2024-09-05T16:48:40.225796+0000","last_clean":"2024-09-05T16:48:40.225796+0000","last_became_active":"2024-09-05T16:48:03.239894+0000","last_became_peered":"2024-09-05T16:48:03.239894+0000","last_unstale":"2024-09-05T16:48:40.225796+0000","last_undegraded":"2024-09-05T16:48:40.225796+0000","last_fullsized":"2024-09-05T16:48:40.225796+0000","mapping_epoch":56,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12808","last_scrub_stamp":"2024-09-05T16:48:04.288388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:48:04.288388+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T21:11:32.044919+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13120","reported_seq":17883,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-05T16:48:29.529847+0000","last_change":"2024-09-05T16:48:29.529847+0000","last_active":"2024-09-05T16:48:29.529847+0000","last_peered":"2024-09-05T16:48:29.529847+0000","last_clean":"2024-09-05T16:48:29.529847+0000","last_became_active":"2024-09-05T16:48:29.235000+0000","last_became_peered":"2024-09-05T16:48:29.235000+0000","last_unstale":"2024-09-05T16:48:29.529847+0000","last_undegraded":"2024-09-05T16:48:29.529847+0000","last_fullsized":"2024-09-05T16:48:29.529847+0000","mapping_epoch":60,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'13120","last_scrub_stamp":"2024-09-05T16:48:29.529768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:48:29.529768+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T02:18:10.992722+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12758","reported_seq":12907,"reported_epoch":61,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-05T16:48:29.230950+0000","last_change":"2024-09-05T16:34:56.985136+0000","last_active":"2024-09-05T16:48:29.230950+0000","last_peered":"2024-09-05T16:48:29.230950+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:48:29.230950+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2237}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12808","reported_seq":17598,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-05T16:48:41.234065+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:48:41.234065+0000","last_peered":"2024-09-05T16:48:41.234065+0000","last_clean":"2024-09-05T16:48:41.234065+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:48:41.234065+0000","last_undegraded":"2024-09-05T16:48:41.234065+0000","last_fullsized":"2024-09-05T16:48:41.234065+0000","mapping_epoch":36,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4408,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13124","reported_seq":13269,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-05T16:48:41.234257+0000","last_change":"2024-09-05T16:48:16.582142+0000","last_active":"2024-09-05T16:48:41.234257+0000","last_peered":"2024-09-05T16:48:41.234257+0000","last_clean":"2024-09-05T16:48:41.234257+0000","last_became_active":"2024-09-05T16:48:16.435091+0000","last_became_peered":"2024-09-05T16:48:16.435091+0000","last_unstale":"2024-09-05T16:48:41.234257+0000","last_undegraded":"2024-09-05T16:48:41.234257+0000","last_fullsized":"2024-09-05T16:48:41.234257+0000","mapping_epoch":58,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'13124","last_scrub_stamp":"2024-09-05T16:48:16.582038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:48:16.582038+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T17:44:25.891565+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"39'12831","reported_seq":12992,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-05T16:48:41.234342+0000","last_change":"2024-09-05T16:47:40.451380+0000","last_active":"2024-09-05T16:48:41.234342+0000","last_peered":"2024-09-05T16:48:41.234342+0000","last_clean":"2024-09-05T16:48:41.234342+0000","last_became_active":"2024-09-05T16:47:40.189447+0000","last_became_peered":"2024-09-05T16:47:40.189447+0000","last_unstale":"2024-09-05T16:48:41.234342+0000","last_undegraded":"2024-09-05T16:48:41.234342+0000","last_fullsized":"2024-09-05T16:48:41.234342+0000","mapping_epoch":52,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12831","last_scrub_stamp":"2024-09-05T16:47:40.451295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:40.451295+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:46:20.946333+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12773","reported_seq":12934,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-05T16:48:41.234360+0000","last_change":"2024-09-05T16:47:27.437275+0000","last_active":"2024-09-05T16:48:41.234360+0000","last_peered":"2024-09-05T16:48:41.234360+0000","last_clean":"2024-09-05T16:48:41.234360+0000","last_became_active":"2024-09-05T16:47:27.204989+0000","last_became_peered":"2024-09-05T16:47:27.204989+0000","last_unstale":"2024-09-05T16:48:41.234360+0000","last_undegraded":"2024-09-05T16:48:41.234360+0000","last_fullsized":"2024-09-05T16:48:41.234360+0000","mapping_epoch":50,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12773","last_scrub_stamp":"2024-09-05T16:47:27.437187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:27.437187+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T16:53:55.961513+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":33680,"num_objects":5,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2506,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":64707,"num_write_kb":210976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6539,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084463,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":726344,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":725819,"kb_avail":93645496,"statfs":{"total":96636764160,"available":95892987904,"internally_reserved":0,"allocated":512000,"data_stored":194158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":743239379},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986341,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":636116,"kb_used_data":516,"kb_used_omap":4,"kb_used_meta":635579,"kb_avail":93735724,"statfs":{"total":96636764160,"available":95985381376,"internally_reserved":0,"allocated":528384,"data_stored":200347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":650833618},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673147,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713360,"kb_used_data":508,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":93658480,"statfs":{"total":96636764160,"available":95906283520,"internally_reserved":0,"allocated":520192,"data_stored":197263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":729938373},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182578,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713808,"kb_used_data":508,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658032,"statfs":{"total":96636764160,"available":95905824768,"internally_reserved":0,"allocated":520192,"data_stored":197263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":730394317},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:48:48.084 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:48:48.409 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:48:48.410 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:48:48.422 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":519,"stamp":"2024-09-05T16:48:47.121035+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1624,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":51940,"num_write_kb":159895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5154,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2789664,"kb_used_data":2064,"kb_used_omap":14,"kb_used_meta":2787505,"kb_avail":374697696,"statfs":{"total":386547056640,"available":383690440704,"internally_reserved":0,"allocated":2113536,"data_stored":801388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2854405687},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20208,"num_objects":-3,"num_object_clones":0,"num_object_copies":-12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-882,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12767,"num_write_kb":-51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1385,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001333"},"pg_stats":[{"pgid":"1.f","version":"39'12904","reported_seq":17545,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-05T16:48:41.233263+0000","last_change":"2024-09-05T16:46:27.826222+0000","last_active":"2024-09-05T16:48:41.233263+0000","last_peered":"2024-09-05T16:48:41.233263+0000","last_clean":"2024-09-05T16:48:41.233263+0000","last_became_active":"2024-09-05T16:46:27.757928+0000","last_became_peered":"2024-09-05T16:46:27.757928+0000","last_unstale":"2024-09-05T16:48:41.233263+0000","last_undegraded":"2024-09-05T16:48:41.233263+0000","last_fullsized":"2024-09-05T16:48:41.233263+0000","mapping_epoch":40,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:32:35.713428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12904,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2262,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12552","reported_seq":17258,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-05T16:48:41.234615+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:48:41.234615+0000","last_peered":"2024-09-05T16:48:41.234615+0000","last_clean":"2024-09-05T16:48:41.234615+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:48:41.234615+0000","last_undegraded":"2024-09-05T16:48:41.234615+0000","last_fullsized":"2024-09-05T16:48:41.234615+0000","mapping_epoch":38,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1865,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12752","reported_seq":17413,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-05T16:48:41.235308+0000","last_change":"2024-09-05T16:46:51.992073+0000","last_active":"2024-09-05T16:48:41.235308+0000","last_peered":"2024-09-05T16:48:41.235308+0000","last_clean":"2024-09-05T16:48:41.235308+0000","last_became_active":"2024-09-05T16:46:51.667128+0000","last_became_peered":"2024-09-05T16:46:51.667128+0000","last_unstale":"2024-09-05T16:48:41.235308+0000","last_undegraded":"2024-09-05T16:48:41.235308+0000","last_fullsized":"2024-09-05T16:48:41.235308+0000","mapping_epoch":44,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12752","last_scrub_stamp":"2024-09-05T16:46:51.991995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:51.991995+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:04:54.120031+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12934","reported_seq":17718,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-05T16:48:41.234020+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:48:41.234020+0000","last_peered":"2024-09-05T16:48:41.234020+0000","last_clean":"2024-09-05T16:48:41.234020+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:48:41.234020+0000","last_undegraded":"2024-09-05T16:48:41.234020+0000","last_fullsized":"2024-09-05T16:48:41.234020+0000","mapping_epoch":34,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7234,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12593","reported_seq":17059,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-05T16:48:41.232825+0000","last_change":"2024-09-05T16:47:51.525026+0000","last_active":"2024-09-05T16:48:41.232825+0000","last_peered":"2024-09-05T16:48:41.232825+0000","last_clean":"2024-09-05T16:48:41.232825+0000","last_became_active":"2024-09-05T16:47:51.193016+0000","last_became_peered":"2024-09-05T16:47:51.193016+0000","last_unstale":"2024-09-05T16:48:41.232825+0000","last_undegraded":"2024-09-05T16:48:41.232825+0000","last_fullsized":"2024-09-05T16:48:41.232825+0000","mapping_epoch":54,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12593","last_scrub_stamp":"2024-09-05T16:47:51.524957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:51.524957+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:46:22.846859+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12772","reported_seq":17389,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-05T16:48:41.233316+0000","last_change":"2024-09-05T16:47:15.832530+0000","last_active":"2024-09-05T16:48:41.233316+0000","last_peered":"2024-09-05T16:48:41.233316+0000","last_clean":"2024-09-05T16:48:41.233316+0000","last_became_active":"2024-09-05T16:47:15.728759+0000","last_became_peered":"2024-09-05T16:47:15.728759+0000","last_unstale":"2024-09-05T16:48:41.233316+0000","last_undegraded":"2024-09-05T16:48:41.233316+0000","last_fullsized":"2024-09-05T16:48:41.233316+0000","mapping_epoch":48,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T02:39:11.515339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12772,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2250,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12660","reported_seq":17107,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-05T16:48:41.231222+0000","last_change":"2024-09-05T16:47:03.683348+0000","last_active":"2024-09-05T16:48:41.231222+0000","last_peered":"2024-09-05T16:48:41.231222+0000","last_clean":"2024-09-05T16:48:41.231222+0000","last_became_active":"2024-09-05T16:47:03.185143+0000","last_became_peered":"2024-09-05T16:47:03.185143+0000","last_unstale":"2024-09-05T16:48:41.231222+0000","last_undegraded":"2024-09-05T16:48:41.231222+0000","last_fullsized":"2024-09-05T16:48:41.231222+0000","mapping_epoch":46,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12660","last_scrub_stamp":"2024-09-05T16:47:03.683262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:03.683262+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:42:16.533299+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12777","reported_seq":12934,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-05T16:48:41.234021+0000","last_change":"2024-09-05T16:46:40.458315+0000","last_active":"2024-09-05T16:48:41.234021+0000","last_peered":"2024-09-05T16:48:41.234021+0000","last_clean":"2024-09-05T16:48:41.234021+0000","last_became_active":"2024-09-05T16:46:40.184288+0000","last_became_peered":"2024-09-05T16:46:40.184288+0000","last_unstale":"2024-09-05T16:48:41.234021+0000","last_undegraded":"2024-09-05T16:48:41.234021+0000","last_fullsized":"2024-09-05T16:48:41.234021+0000","mapping_epoch":42,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12777","last_scrub_stamp":"2024-09-05T16:46:40.458226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:40.458226+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:38:58.734210+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'12770","reported_seq":17437,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-05T16:48:41.629203+0000","last_change":"2024-09-05T16:48:41.629203+0000","last_active":"2024-09-05T16:48:41.629203+0000","last_peered":"2024-09-05T16:48:41.629203+0000","last_clean":"2024-09-05T16:48:41.629203+0000","last_became_active":"2024-09-05T16:48:41.236479+0000","last_became_peered":"2024-09-05T16:48:41.236479+0000","last_unstale":"2024-09-05T16:48:41.629203+0000","last_undegraded":"2024-09-05T16:48:41.629203+0000","last_fullsized":"2024-09-05T16:48:41.629203+0000","mapping_epoch":62,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12770","last_scrub_stamp":"2024-09-05T16:48:41.629128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:48:41.629128+0000","objects_scrubbed":1,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T02:22:35.076855+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12808","reported_seq":17497,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-05T16:48:41.235475+0000","last_change":"2024-09-05T16:48:04.288466+0000","last_active":"2024-09-05T16:48:41.235475+0000","last_peered":"2024-09-05T16:48:41.235475+0000","last_clean":"2024-09-05T16:48:41.235475+0000","last_became_active":"2024-09-05T16:48:03.239894+0000","last_became_peered":"2024-09-05T16:48:03.239894+0000","last_unstale":"2024-09-05T16:48:41.235475+0000","last_undegraded":"2024-09-05T16:48:41.235475+0000","last_fullsized":"2024-09-05T16:48:41.235475+0000","mapping_epoch":56,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12808","last_scrub_stamp":"2024-09-05T16:48:04.288388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:48:04.288388+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T21:11:32.044919+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13120","reported_seq":17887,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-05T16:48:41.233200+0000","last_change":"2024-09-05T16:48:29.529847+0000","last_active":"2024-09-05T16:48:41.233200+0000","last_peered":"2024-09-05T16:48:41.233200+0000","last_clean":"2024-09-05T16:48:41.233200+0000","last_became_active":"2024-09-05T16:48:29.235000+0000","last_became_peered":"2024-09-05T16:48:29.235000+0000","last_unstale":"2024-09-05T16:48:41.233200+0000","last_undegraded":"2024-09-05T16:48:41.233200+0000","last_fullsized":"2024-09-05T16:48:41.233200+0000","mapping_epoch":60,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'13120","last_scrub_stamp":"2024-09-05T16:48:29.529768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:48:29.529768+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T02:18:10.992722+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12758","reported_seq":14142,"reported_epoch":63,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:48:43.215284+0000","last_change":"2024-09-05T16:48:39.285711+0000","last_active":"2024-09-05T16:48:43.215284+0000","last_peered":"2024-09-05T16:48:43.215284+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:48:43.215284+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1624,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":642,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1623}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12808","reported_seq":17598,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-05T16:48:41.234065+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:48:41.234065+0000","last_peered":"2024-09-05T16:48:41.234065+0000","last_clean":"2024-09-05T16:48:41.234065+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:48:41.234065+0000","last_undegraded":"2024-09-05T16:48:41.234065+0000","last_fullsized":"2024-09-05T16:48:41.234065+0000","mapping_epoch":36,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4408,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13124","reported_seq":13269,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-05T16:48:41.234257+0000","last_change":"2024-09-05T16:48:16.582142+0000","last_active":"2024-09-05T16:48:41.234257+0000","last_peered":"2024-09-05T16:48:41.234257+0000","last_clean":"2024-09-05T16:48:41.234257+0000","last_became_active":"2024-09-05T16:48:16.435091+0000","last_became_peered":"2024-09-05T16:48:16.435091+0000","last_unstale":"2024-09-05T16:48:41.234257+0000","last_undegraded":"2024-09-05T16:48:41.234257+0000","last_fullsized":"2024-09-05T16:48:41.234257+0000","mapping_epoch":58,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'13124","last_scrub_stamp":"2024-09-05T16:48:16.582038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:48:16.582038+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T17:44:25.891565+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"39'12831","reported_seq":12992,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-05T16:48:41.234342+0000","last_change":"2024-09-05T16:47:40.451380+0000","last_active":"2024-09-05T16:48:41.234342+0000","last_peered":"2024-09-05T16:48:41.234342+0000","last_clean":"2024-09-05T16:48:41.234342+0000","last_became_active":"2024-09-05T16:47:40.189447+0000","last_became_peered":"2024-09-05T16:47:40.189447+0000","last_unstale":"2024-09-05T16:48:41.234342+0000","last_undegraded":"2024-09-05T16:48:41.234342+0000","last_fullsized":"2024-09-05T16:48:41.234342+0000","mapping_epoch":52,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12831","last_scrub_stamp":"2024-09-05T16:47:40.451295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:40.451295+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:46:20.946333+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12773","reported_seq":12934,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-05T16:48:41.234360+0000","last_change":"2024-09-05T16:47:27.437275+0000","last_active":"2024-09-05T16:48:41.234360+0000","last_peered":"2024-09-05T16:48:41.234360+0000","last_clean":"2024-09-05T16:48:41.234360+0000","last_became_active":"2024-09-05T16:47:27.204989+0000","last_became_peered":"2024-09-05T16:47:27.204989+0000","last_unstale":"2024-09-05T16:48:41.234360+0000","last_undegraded":"2024-09-05T16:48:41.234360+0000","last_fullsized":"2024-09-05T16:48:41.234360+0000","mapping_epoch":50,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12773","last_scrub_stamp":"2024-09-05T16:47:27.437187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:27.437187+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T16:53:55.961513+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1624,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":51940,"num_write_kb":159895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5154,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084464,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":726360,"kb_used_data":516,"kb_used_omap":4,"kb_used_meta":725819,"kb_avail":93645480,"statfs":{"total":96636764160,"available":95892971520,"internally_reserved":0,"allocated":528384,"data_stored":200347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":743239379},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986341,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":636116,"kb_used_data":516,"kb_used_omap":4,"kb_used_meta":635579,"kb_avail":93735724,"statfs":{"total":96636764160,"available":95985381376,"internally_reserved":0,"allocated":528384,"data_stored":200347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":650833618},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673148,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713372,"kb_used_data":516,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":93658468,"statfs":{"total":96636764160,"available":95906271232,"internally_reserved":0,"allocated":528384,"data_stored":200347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":729938373},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182579,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713816,"kb_used_data":516,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658024,"statfs":{"total":96636764160,"available":95905816576,"internally_reserved":0,"allocated":528384,"data_stored":200347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":730394317},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:48:48.424 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:48:48.726 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:48:48.727 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:48:48.738 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":519,"stamp":"2024-09-05T16:48:47.121035+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1624,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":51940,"num_write_kb":159895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5154,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2789664,"kb_used_data":2064,"kb_used_omap":14,"kb_used_meta":2787505,"kb_avail":374697696,"statfs":{"total":386547056640,"available":383690440704,"internally_reserved":0,"allocated":2113536,"data_stored":801388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2854405687},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20208,"num_objects":-3,"num_object_clones":0,"num_object_copies":-12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-882,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12767,"num_write_kb":-51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1385,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001333"},"pg_stats":[{"pgid":"1.f","version":"39'12904","reported_seq":17545,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-05T16:48:41.233263+0000","last_change":"2024-09-05T16:46:27.826222+0000","last_active":"2024-09-05T16:48:41.233263+0000","last_peered":"2024-09-05T16:48:41.233263+0000","last_clean":"2024-09-05T16:48:41.233263+0000","last_became_active":"2024-09-05T16:46:27.757928+0000","last_became_peered":"2024-09-05T16:46:27.757928+0000","last_unstale":"2024-09-05T16:48:41.233263+0000","last_undegraded":"2024-09-05T16:48:41.233263+0000","last_fullsized":"2024-09-05T16:48:41.233263+0000","mapping_epoch":40,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:32:35.713428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12904,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2262,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12552","reported_seq":17258,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-05T16:48:41.234615+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:48:41.234615+0000","last_peered":"2024-09-05T16:48:41.234615+0000","last_clean":"2024-09-05T16:48:41.234615+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:48:41.234615+0000","last_undegraded":"2024-09-05T16:48:41.234615+0000","last_fullsized":"2024-09-05T16:48:41.234615+0000","mapping_epoch":38,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1865,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12752","reported_seq":17413,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-05T16:48:41.235308+0000","last_change":"2024-09-05T16:46:51.992073+0000","last_active":"2024-09-05T16:48:41.235308+0000","last_peered":"2024-09-05T16:48:41.235308+0000","last_clean":"2024-09-05T16:48:41.235308+0000","last_became_active":"2024-09-05T16:46:51.667128+0000","last_became_peered":"2024-09-05T16:46:51.667128+0000","last_unstale":"2024-09-05T16:48:41.235308+0000","last_undegraded":"2024-09-05T16:48:41.235308+0000","last_fullsized":"2024-09-05T16:48:41.235308+0000","mapping_epoch":44,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12752","last_scrub_stamp":"2024-09-05T16:46:51.991995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:51.991995+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:04:54.120031+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12934","reported_seq":17718,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-05T16:48:41.234020+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:48:41.234020+0000","last_peered":"2024-09-05T16:48:41.234020+0000","last_clean":"2024-09-05T16:48:41.234020+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:48:41.234020+0000","last_undegraded":"2024-09-05T16:48:41.234020+0000","last_fullsized":"2024-09-05T16:48:41.234020+0000","mapping_epoch":34,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7234,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12593","reported_seq":17059,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-05T16:48:41.232825+0000","last_change":"2024-09-05T16:47:51.525026+0000","last_active":"2024-09-05T16:48:41.232825+0000","last_peered":"2024-09-05T16:48:41.232825+0000","last_clean":"2024-09-05T16:48:41.232825+0000","last_became_active":"2024-09-05T16:47:51.193016+0000","last_became_peered":"2024-09-05T16:47:51.193016+0000","last_unstale":"2024-09-05T16:48:41.232825+0000","last_undegraded":"2024-09-05T16:48:41.232825+0000","last_fullsized":"2024-09-05T16:48:41.232825+0000","mapping_epoch":54,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12593","last_scrub_stamp":"2024-09-05T16:47:51.524957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:51.524957+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:46:22.846859+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12772","reported_seq":17389,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-05T16:48:41.233316+0000","last_change":"2024-09-05T16:47:15.832530+0000","last_active":"2024-09-05T16:48:41.233316+0000","last_peered":"2024-09-05T16:48:41.233316+0000","last_clean":"2024-09-05T16:48:41.233316+0000","last_became_active":"2024-09-05T16:47:15.728759+0000","last_became_peered":"2024-09-05T16:47:15.728759+0000","last_unstale":"2024-09-05T16:48:41.233316+0000","last_undegraded":"2024-09-05T16:48:41.233316+0000","last_fullsized":"2024-09-05T16:48:41.233316+0000","mapping_epoch":48,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T02:39:11.515339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12772,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2250,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12660","reported_seq":17107,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-05T16:48:41.231222+0000","last_change":"2024-09-05T16:47:03.683348+0000","last_active":"2024-09-05T16:48:41.231222+0000","last_peered":"2024-09-05T16:48:41.231222+0000","last_clean":"2024-09-05T16:48:41.231222+0000","last_became_active":"2024-09-05T16:47:03.185143+0000","last_became_peered":"2024-09-05T16:47:03.185143+0000","last_unstale":"2024-09-05T16:48:41.231222+0000","last_undegraded":"2024-09-05T16:48:41.231222+0000","last_fullsized":"2024-09-05T16:48:41.231222+0000","mapping_epoch":46,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12660","last_scrub_stamp":"2024-09-05T16:47:03.683262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:03.683262+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:42:16.533299+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12777","reported_seq":12934,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-05T16:48:41.234021+0000","last_change":"2024-09-05T16:46:40.458315+0000","last_active":"2024-09-05T16:48:41.234021+0000","last_peered":"2024-09-05T16:48:41.234021+0000","last_clean":"2024-09-05T16:48:41.234021+0000","last_became_active":"2024-09-05T16:46:40.184288+0000","last_became_peered":"2024-09-05T16:46:40.184288+0000","last_unstale":"2024-09-05T16:48:41.234021+0000","last_undegraded":"2024-09-05T16:48:41.234021+0000","last_fullsized":"2024-09-05T16:48:41.234021+0000","mapping_epoch":42,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12777","last_scrub_stamp":"2024-09-05T16:46:40.458226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:40.458226+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:38:58.734210+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'12770","reported_seq":17437,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-05T16:48:41.629203+0000","last_change":"2024-09-05T16:48:41.629203+0000","last_active":"2024-09-05T16:48:41.629203+0000","last_peered":"2024-09-05T16:48:41.629203+0000","last_clean":"2024-09-05T16:48:41.629203+0000","last_became_active":"2024-09-05T16:48:41.236479+0000","last_became_peered":"2024-09-05T16:48:41.236479+0000","last_unstale":"2024-09-05T16:48:41.629203+0000","last_undegraded":"2024-09-05T16:48:41.629203+0000","last_fullsized":"2024-09-05T16:48:41.629203+0000","mapping_epoch":62,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12770","last_scrub_stamp":"2024-09-05T16:48:41.629128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:48:41.629128+0000","objects_scrubbed":1,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T02:22:35.076855+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12808","reported_seq":17497,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-05T16:48:41.235475+0000","last_change":"2024-09-05T16:48:04.288466+0000","last_active":"2024-09-05T16:48:41.235475+0000","last_peered":"2024-09-05T16:48:41.235475+0000","last_clean":"2024-09-05T16:48:41.235475+0000","last_became_active":"2024-09-05T16:48:03.239894+0000","last_became_peered":"2024-09-05T16:48:03.239894+0000","last_unstale":"2024-09-05T16:48:41.235475+0000","last_undegraded":"2024-09-05T16:48:41.235475+0000","last_fullsized":"2024-09-05T16:48:41.235475+0000","mapping_epoch":56,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12808","last_scrub_stamp":"2024-09-05T16:48:04.288388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:48:04.288388+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T21:11:32.044919+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13120","reported_seq":17887,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-05T16:48:41.233200+0000","last_change":"2024-09-05T16:48:29.529847+0000","last_active":"2024-09-05T16:48:41.233200+0000","last_peered":"2024-09-05T16:48:41.233200+0000","last_clean":"2024-09-05T16:48:41.233200+0000","last_became_active":"2024-09-05T16:48:29.235000+0000","last_became_peered":"2024-09-05T16:48:29.235000+0000","last_unstale":"2024-09-05T16:48:41.233200+0000","last_undegraded":"2024-09-05T16:48:41.233200+0000","last_fullsized":"2024-09-05T16:48:41.233200+0000","mapping_epoch":60,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'13120","last_scrub_stamp":"2024-09-05T16:48:29.529768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:48:29.529768+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T02:18:10.992722+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12758","reported_seq":14142,"reported_epoch":63,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:48:43.215284+0000","last_change":"2024-09-05T16:48:39.285711+0000","last_active":"2024-09-05T16:48:43.215284+0000","last_peered":"2024-09-05T16:48:43.215284+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:48:43.215284+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1624,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":642,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1623}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12808","reported_seq":17598,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-05T16:48:41.234065+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:48:41.234065+0000","last_peered":"2024-09-05T16:48:41.234065+0000","last_clean":"2024-09-05T16:48:41.234065+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:48:41.234065+0000","last_undegraded":"2024-09-05T16:48:41.234065+0000","last_fullsized":"2024-09-05T16:48:41.234065+0000","mapping_epoch":36,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4408,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13124","reported_seq":13269,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-05T16:48:41.234257+0000","last_change":"2024-09-05T16:48:16.582142+0000","last_active":"2024-09-05T16:48:41.234257+0000","last_peered":"2024-09-05T16:48:41.234257+0000","last_clean":"2024-09-05T16:48:41.234257+0000","last_became_active":"2024-09-05T16:48:16.435091+0000","last_became_peered":"2024-09-05T16:48:16.435091+0000","last_unstale":"2024-09-05T16:48:41.234257+0000","last_undegraded":"2024-09-05T16:48:41.234257+0000","last_fullsized":"2024-09-05T16:48:41.234257+0000","mapping_epoch":58,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'13124","last_scrub_stamp":"2024-09-05T16:48:16.582038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:48:16.582038+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T17:44:25.891565+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"39'12831","reported_seq":12992,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-05T16:48:41.234342+0000","last_change":"2024-09-05T16:47:40.451380+0000","last_active":"2024-09-05T16:48:41.234342+0000","last_peered":"2024-09-05T16:48:41.234342+0000","last_clean":"2024-09-05T16:48:41.234342+0000","last_became_active":"2024-09-05T16:47:40.189447+0000","last_became_peered":"2024-09-05T16:47:40.189447+0000","last_unstale":"2024-09-05T16:48:41.234342+0000","last_undegraded":"2024-09-05T16:48:41.234342+0000","last_fullsized":"2024-09-05T16:48:41.234342+0000","mapping_epoch":52,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12831","last_scrub_stamp":"2024-09-05T16:47:40.451295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:40.451295+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:46:20.946333+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12773","reported_seq":12934,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-05T16:48:41.234360+0000","last_change":"2024-09-05T16:47:27.437275+0000","last_active":"2024-09-05T16:48:41.234360+0000","last_peered":"2024-09-05T16:48:41.234360+0000","last_clean":"2024-09-05T16:48:41.234360+0000","last_became_active":"2024-09-05T16:47:27.204989+0000","last_became_peered":"2024-09-05T16:47:27.204989+0000","last_unstale":"2024-09-05T16:48:41.234360+0000","last_undegraded":"2024-09-05T16:48:41.234360+0000","last_fullsized":"2024-09-05T16:48:41.234360+0000","mapping_epoch":50,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12773","last_scrub_stamp":"2024-09-05T16:47:27.437187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:27.437187+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T16:53:55.961513+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1624,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":51940,"num_write_kb":159895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5154,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084464,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":726360,"kb_used_data":516,"kb_used_omap":4,"kb_used_meta":725819,"kb_avail":93645480,"statfs":{"total":96636764160,"available":95892971520,"internally_reserved":0,"allocated":528384,"data_stored":200347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":743239379},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986341,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":636116,"kb_used_data":516,"kb_used_omap":4,"kb_used_meta":635579,"kb_avail":93735724,"statfs":{"total":96636764160,"available":95985381376,"internally_reserved":0,"allocated":528384,"data_stored":200347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":650833618},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673148,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713372,"kb_used_data":516,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":93658468,"statfs":{"total":96636764160,"available":95906271232,"internally_reserved":0,"allocated":528384,"data_stored":200347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":729938373},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182579,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713816,"kb_used_data":516,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658024,"statfs":{"total":96636764160,"available":95905816576,"internally_reserved":0,"allocated":528384,"data_stored":200347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":730394317},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:48:48.739 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:48:48.740 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "39'12758", 'reported_seq': 14142, 'reported_epoch': 63, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:48:43.215284+0000', 'last_change': '2024-09-05T16:48:39.285711+0000', 'last_active': '2024-09-05T16:48:43.215284+0000', 'last_peered': '2024-09-05T16:48:43.215284+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:48:43.215284+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "33'2700", 'ondisk_log_start': "33'2700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10058, 'log_dups_size': 0, 'ondisk_log_size': 10058, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6736, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1624, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12757, 'num_write_kb': 51031, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 642, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1623}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:48:48.740 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:48:49.135 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:48:49.135 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 1624/8 objects degraded (20300.000%), 1 pg degraded, 1 pg undersized","count":2},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":932,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":63,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":2,"data_bytes":13472,"bytes_used":2856615936,"bytes_avail":383690440704,"bytes_total":386547056640,"degraded_objects":1624,"degraded_total":8,"degraded_ratio":203},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:48:49.149 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-09-05T16:48:49.149 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:48:49.466 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:48:49.466 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:48:49.481 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":520,"stamp":"2024-09-05T16:48:49.121333+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":682,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":51940,"num_write_kb":159895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6096,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2777184,"kb_used_data":2064,"kb_used_omap":14,"kb_used_meta":2775025,"kb_avail":374710176,"statfs":{"total":386547056640,"available":383703220224,"internally_reserved":0,"allocated":2113536,"data_stored":801388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2841626167},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20208,"num_objects":-3,"num_object_clones":0,"num_object_copies":-12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1824,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12767,"num_write_kb":-51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-443,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.898439"},"pg_stats":[{"pgid":"1.f","version":"39'12904","reported_seq":17545,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-05T16:48:41.233263+0000","last_change":"2024-09-05T16:46:27.826222+0000","last_active":"2024-09-05T16:48:41.233263+0000","last_peered":"2024-09-05T16:48:41.233263+0000","last_clean":"2024-09-05T16:48:41.233263+0000","last_became_active":"2024-09-05T16:46:27.757928+0000","last_became_peered":"2024-09-05T16:46:27.757928+0000","last_unstale":"2024-09-05T16:48:41.233263+0000","last_undegraded":"2024-09-05T16:48:41.233263+0000","last_fullsized":"2024-09-05T16:48:41.233263+0000","mapping_epoch":40,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:32:35.713428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12904,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2262,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12552","reported_seq":17258,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-05T16:48:41.234615+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:48:41.234615+0000","last_peered":"2024-09-05T16:48:41.234615+0000","last_clean":"2024-09-05T16:48:41.234615+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:48:41.234615+0000","last_undegraded":"2024-09-05T16:48:41.234615+0000","last_fullsized":"2024-09-05T16:48:41.234615+0000","mapping_epoch":38,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1865,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12752","reported_seq":17413,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-05T16:48:41.235308+0000","last_change":"2024-09-05T16:46:51.992073+0000","last_active":"2024-09-05T16:48:41.235308+0000","last_peered":"2024-09-05T16:48:41.235308+0000","last_clean":"2024-09-05T16:48:41.235308+0000","last_became_active":"2024-09-05T16:46:51.667128+0000","last_became_peered":"2024-09-05T16:46:51.667128+0000","last_unstale":"2024-09-05T16:48:41.235308+0000","last_undegraded":"2024-09-05T16:48:41.235308+0000","last_fullsized":"2024-09-05T16:48:41.235308+0000","mapping_epoch":44,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12752","last_scrub_stamp":"2024-09-05T16:46:51.991995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:51.991995+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:04:54.120031+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12934","reported_seq":17718,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-05T16:48:41.234020+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:48:41.234020+0000","last_peered":"2024-09-05T16:48:41.234020+0000","last_clean":"2024-09-05T16:48:41.234020+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:48:41.234020+0000","last_undegraded":"2024-09-05T16:48:41.234020+0000","last_fullsized":"2024-09-05T16:48:41.234020+0000","mapping_epoch":34,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7234,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12593","reported_seq":17059,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-05T16:48:41.232825+0000","last_change":"2024-09-05T16:47:51.525026+0000","last_active":"2024-09-05T16:48:41.232825+0000","last_peered":"2024-09-05T16:48:41.232825+0000","last_clean":"2024-09-05T16:48:41.232825+0000","last_became_active":"2024-09-05T16:47:51.193016+0000","last_became_peered":"2024-09-05T16:47:51.193016+0000","last_unstale":"2024-09-05T16:48:41.232825+0000","last_undegraded":"2024-09-05T16:48:41.232825+0000","last_fullsized":"2024-09-05T16:48:41.232825+0000","mapping_epoch":54,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12593","last_scrub_stamp":"2024-09-05T16:47:51.524957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:51.524957+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:46:22.846859+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12772","reported_seq":17389,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-05T16:48:41.233316+0000","last_change":"2024-09-05T16:47:15.832530+0000","last_active":"2024-09-05T16:48:41.233316+0000","last_peered":"2024-09-05T16:48:41.233316+0000","last_clean":"2024-09-05T16:48:41.233316+0000","last_became_active":"2024-09-05T16:47:15.728759+0000","last_became_peered":"2024-09-05T16:47:15.728759+0000","last_unstale":"2024-09-05T16:48:41.233316+0000","last_undegraded":"2024-09-05T16:48:41.233316+0000","last_fullsized":"2024-09-05T16:48:41.233316+0000","mapping_epoch":48,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T02:39:11.515339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12772,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2250,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12660","reported_seq":17107,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-05T16:48:41.231222+0000","last_change":"2024-09-05T16:47:03.683348+0000","last_active":"2024-09-05T16:48:41.231222+0000","last_peered":"2024-09-05T16:48:41.231222+0000","last_clean":"2024-09-05T16:48:41.231222+0000","last_became_active":"2024-09-05T16:47:03.185143+0000","last_became_peered":"2024-09-05T16:47:03.185143+0000","last_unstale":"2024-09-05T16:48:41.231222+0000","last_undegraded":"2024-09-05T16:48:41.231222+0000","last_fullsized":"2024-09-05T16:48:41.231222+0000","mapping_epoch":46,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12660","last_scrub_stamp":"2024-09-05T16:47:03.683262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:03.683262+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:42:16.533299+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12777","reported_seq":12935,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-05T16:48:41.234074+0000","last_change":"2024-09-05T16:46:40.458315+0000","last_active":"2024-09-05T16:48:41.234074+0000","last_peered":"2024-09-05T16:48:41.234074+0000","last_clean":"2024-09-05T16:48:41.234074+0000","last_became_active":"2024-09-05T16:46:40.184288+0000","last_became_peered":"2024-09-05T16:46:40.184288+0000","last_unstale":"2024-09-05T16:48:41.234074+0000","last_undegraded":"2024-09-05T16:48:41.234074+0000","last_fullsized":"2024-09-05T16:48:41.234074+0000","mapping_epoch":42,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12777","last_scrub_stamp":"2024-09-05T16:46:40.458226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:40.458226+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:38:58.734210+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'12770","reported_seq":17437,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-05T16:48:41.629203+0000","last_change":"2024-09-05T16:48:41.629203+0000","last_active":"2024-09-05T16:48:41.629203+0000","last_peered":"2024-09-05T16:48:41.629203+0000","last_clean":"2024-09-05T16:48:41.629203+0000","last_became_active":"2024-09-05T16:48:41.236479+0000","last_became_peered":"2024-09-05T16:48:41.236479+0000","last_unstale":"2024-09-05T16:48:41.629203+0000","last_undegraded":"2024-09-05T16:48:41.629203+0000","last_fullsized":"2024-09-05T16:48:41.629203+0000","mapping_epoch":62,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12770","last_scrub_stamp":"2024-09-05T16:48:41.629128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:48:41.629128+0000","objects_scrubbed":1,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T02:22:35.076855+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12808","reported_seq":17497,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-05T16:48:41.235475+0000","last_change":"2024-09-05T16:48:04.288466+0000","last_active":"2024-09-05T16:48:41.235475+0000","last_peered":"2024-09-05T16:48:41.235475+0000","last_clean":"2024-09-05T16:48:41.235475+0000","last_became_active":"2024-09-05T16:48:03.239894+0000","last_became_peered":"2024-09-05T16:48:03.239894+0000","last_unstale":"2024-09-05T16:48:41.235475+0000","last_undegraded":"2024-09-05T16:48:41.235475+0000","last_fullsized":"2024-09-05T16:48:41.235475+0000","mapping_epoch":56,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12808","last_scrub_stamp":"2024-09-05T16:48:04.288388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:48:04.288388+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T21:11:32.044919+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13120","reported_seq":17887,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-05T16:48:41.233200+0000","last_change":"2024-09-05T16:48:29.529847+0000","last_active":"2024-09-05T16:48:41.233200+0000","last_peered":"2024-09-05T16:48:41.233200+0000","last_clean":"2024-09-05T16:48:41.233200+0000","last_became_active":"2024-09-05T16:48:29.235000+0000","last_became_peered":"2024-09-05T16:48:29.235000+0000","last_unstale":"2024-09-05T16:48:41.233200+0000","last_undegraded":"2024-09-05T16:48:41.233200+0000","last_fullsized":"2024-09-05T16:48:41.233200+0000","mapping_epoch":60,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'13120","last_scrub_stamp":"2024-09-05T16:48:29.529768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:48:29.529768+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T02:18:10.992722+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12758","reported_seq":16026,"reported_epoch":63,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:48:48.215255+0000","last_change":"2024-09-05T16:48:39.285711+0000","last_active":"2024-09-05T16:48:48.215255+0000","last_peered":"2024-09-05T16:48:48.215255+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:48:48.215255+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":682,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1584,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-681}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12808","reported_seq":17598,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-05T16:48:41.234065+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:48:41.234065+0000","last_peered":"2024-09-05T16:48:41.234065+0000","last_clean":"2024-09-05T16:48:41.234065+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:48:41.234065+0000","last_undegraded":"2024-09-05T16:48:41.234065+0000","last_fullsized":"2024-09-05T16:48:41.234065+0000","mapping_epoch":36,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4408,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13124","reported_seq":13269,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-05T16:48:41.234257+0000","last_change":"2024-09-05T16:48:16.582142+0000","last_active":"2024-09-05T16:48:41.234257+0000","last_peered":"2024-09-05T16:48:41.234257+0000","last_clean":"2024-09-05T16:48:41.234257+0000","last_became_active":"2024-09-05T16:48:16.435091+0000","last_became_peered":"2024-09-05T16:48:16.435091+0000","last_unstale":"2024-09-05T16:48:41.234257+0000","last_undegraded":"2024-09-05T16:48:41.234257+0000","last_fullsized":"2024-09-05T16:48:41.234257+0000","mapping_epoch":58,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'13124","last_scrub_stamp":"2024-09-05T16:48:16.582038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:48:16.582038+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T17:44:25.891565+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"39'12831","reported_seq":12992,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-05T16:48:41.234342+0000","last_change":"2024-09-05T16:47:40.451380+0000","last_active":"2024-09-05T16:48:41.234342+0000","last_peered":"2024-09-05T16:48:41.234342+0000","last_clean":"2024-09-05T16:48:41.234342+0000","last_became_active":"2024-09-05T16:47:40.189447+0000","last_became_peered":"2024-09-05T16:47:40.189447+0000","last_unstale":"2024-09-05T16:48:41.234342+0000","last_undegraded":"2024-09-05T16:48:41.234342+0000","last_fullsized":"2024-09-05T16:48:41.234342+0000","mapping_epoch":52,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12831","last_scrub_stamp":"2024-09-05T16:47:40.451295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:40.451295+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:46:20.946333+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12773","reported_seq":12934,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-05T16:48:41.234360+0000","last_change":"2024-09-05T16:47:27.437275+0000","last_active":"2024-09-05T16:48:41.234360+0000","last_peered":"2024-09-05T16:48:41.234360+0000","last_clean":"2024-09-05T16:48:41.234360+0000","last_became_active":"2024-09-05T16:47:27.204989+0000","last_became_peered":"2024-09-05T16:47:27.204989+0000","last_unstale":"2024-09-05T16:48:41.234360+0000","last_undegraded":"2024-09-05T16:48:41.234360+0000","last_fullsized":"2024-09-05T16:48:41.234360+0000","mapping_epoch":50,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12773","last_scrub_stamp":"2024-09-05T16:47:27.437187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:27.437187+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T16:53:55.961513+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":682,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":51940,"num_write_kb":159895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6096,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084465,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":726360,"kb_used_data":516,"kb_used_omap":4,"kb_used_meta":725819,"kb_avail":93645480,"statfs":{"total":96636764160,"available":95892971520,"internally_reserved":0,"allocated":528384,"data_stored":200347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":743239379},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986342,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":623636,"kb_used_data":516,"kb_used_omap":4,"kb_used_meta":623099,"kb_avail":93748204,"statfs":{"total":96636764160,"available":95998160896,"internally_reserved":0,"allocated":528384,"data_stored":200347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":638054098},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673148,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713372,"kb_used_data":516,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":93658468,"statfs":{"total":96636764160,"available":95906271232,"internally_reserved":0,"allocated":528384,"data_stored":200347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":729938373},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182579,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713816,"kb_used_data":516,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658024,"statfs":{"total":96636764160,"available":95905816576,"internally_reserved":0,"allocated":528384,"data_stored":200347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":730394317},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:48:52.483 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:48:52.788 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:48:52.788 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:48:52.800 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":523,"stamp":"2024-09-05T16:48:52.670485+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":682,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":51940,"num_write_kb":159895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6096,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2777184,"kb_used_data":2064,"kb_used_omap":14,"kb_used_meta":2775025,"kb_avail":374710176,"statfs":{"total":386547056640,"available":383703220224,"internally_reserved":0,"allocated":2113536,"data_stored":801388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2841626167},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20208,"num_objects":-3,"num_object_clones":0,"num_object_copies":-12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1824,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12767,"num_write_kb":-51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-443,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.550084"},"pg_stats":[{"pgid":"1.f","version":"39'12904","reported_seq":17545,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-05T16:48:41.233263+0000","last_change":"2024-09-05T16:46:27.826222+0000","last_active":"2024-09-05T16:48:41.233263+0000","last_peered":"2024-09-05T16:48:41.233263+0000","last_clean":"2024-09-05T16:48:41.233263+0000","last_became_active":"2024-09-05T16:46:27.757928+0000","last_became_peered":"2024-09-05T16:46:27.757928+0000","last_unstale":"2024-09-05T16:48:41.233263+0000","last_undegraded":"2024-09-05T16:48:41.233263+0000","last_fullsized":"2024-09-05T16:48:41.233263+0000","mapping_epoch":40,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:32:35.713428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12904,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2262,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12552","reported_seq":17258,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-05T16:48:41.234615+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:48:41.234615+0000","last_peered":"2024-09-05T16:48:41.234615+0000","last_clean":"2024-09-05T16:48:41.234615+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:48:41.234615+0000","last_undegraded":"2024-09-05T16:48:41.234615+0000","last_fullsized":"2024-09-05T16:48:41.234615+0000","mapping_epoch":38,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1865,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12752","reported_seq":17413,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-05T16:48:41.235308+0000","last_change":"2024-09-05T16:46:51.992073+0000","last_active":"2024-09-05T16:48:41.235308+0000","last_peered":"2024-09-05T16:48:41.235308+0000","last_clean":"2024-09-05T16:48:41.235308+0000","last_became_active":"2024-09-05T16:46:51.667128+0000","last_became_peered":"2024-09-05T16:46:51.667128+0000","last_unstale":"2024-09-05T16:48:41.235308+0000","last_undegraded":"2024-09-05T16:48:41.235308+0000","last_fullsized":"2024-09-05T16:48:41.235308+0000","mapping_epoch":44,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12752","last_scrub_stamp":"2024-09-05T16:46:51.991995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:51.991995+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:04:54.120031+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12934","reported_seq":17718,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-05T16:48:41.234020+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:48:41.234020+0000","last_peered":"2024-09-05T16:48:41.234020+0000","last_clean":"2024-09-05T16:48:41.234020+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:48:41.234020+0000","last_undegraded":"2024-09-05T16:48:41.234020+0000","last_fullsized":"2024-09-05T16:48:41.234020+0000","mapping_epoch":34,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7234,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12593","reported_seq":17059,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-05T16:48:41.232825+0000","last_change":"2024-09-05T16:47:51.525026+0000","last_active":"2024-09-05T16:48:41.232825+0000","last_peered":"2024-09-05T16:48:41.232825+0000","last_clean":"2024-09-05T16:48:41.232825+0000","last_became_active":"2024-09-05T16:47:51.193016+0000","last_became_peered":"2024-09-05T16:47:51.193016+0000","last_unstale":"2024-09-05T16:48:41.232825+0000","last_undegraded":"2024-09-05T16:48:41.232825+0000","last_fullsized":"2024-09-05T16:48:41.232825+0000","mapping_epoch":54,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12593","last_scrub_stamp":"2024-09-05T16:47:51.524957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:51.524957+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:46:22.846859+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12772","reported_seq":17389,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-05T16:48:41.233316+0000","last_change":"2024-09-05T16:47:15.832530+0000","last_active":"2024-09-05T16:48:41.233316+0000","last_peered":"2024-09-05T16:48:41.233316+0000","last_clean":"2024-09-05T16:48:41.233316+0000","last_became_active":"2024-09-05T16:47:15.728759+0000","last_became_peered":"2024-09-05T16:47:15.728759+0000","last_unstale":"2024-09-05T16:48:41.233316+0000","last_undegraded":"2024-09-05T16:48:41.233316+0000","last_fullsized":"2024-09-05T16:48:41.233316+0000","mapping_epoch":48,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T02:39:11.515339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12772,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2250,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12660","reported_seq":17107,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-05T16:48:41.231222+0000","last_change":"2024-09-05T16:47:03.683348+0000","last_active":"2024-09-05T16:48:41.231222+0000","last_peered":"2024-09-05T16:48:41.231222+0000","last_clean":"2024-09-05T16:48:41.231222+0000","last_became_active":"2024-09-05T16:47:03.185143+0000","last_became_peered":"2024-09-05T16:47:03.185143+0000","last_unstale":"2024-09-05T16:48:41.231222+0000","last_undegraded":"2024-09-05T16:48:41.231222+0000","last_fullsized":"2024-09-05T16:48:41.231222+0000","mapping_epoch":46,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12660","last_scrub_stamp":"2024-09-05T16:47:03.683262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:03.683262+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:42:16.533299+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12777","reported_seq":12935,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-05T16:48:41.234074+0000","last_change":"2024-09-05T16:46:40.458315+0000","last_active":"2024-09-05T16:48:41.234074+0000","last_peered":"2024-09-05T16:48:41.234074+0000","last_clean":"2024-09-05T16:48:41.234074+0000","last_became_active":"2024-09-05T16:46:40.184288+0000","last_became_peered":"2024-09-05T16:46:40.184288+0000","last_unstale":"2024-09-05T16:48:41.234074+0000","last_undegraded":"2024-09-05T16:48:41.234074+0000","last_fullsized":"2024-09-05T16:48:41.234074+0000","mapping_epoch":42,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12777","last_scrub_stamp":"2024-09-05T16:46:40.458226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:40.458226+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:38:58.734210+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'12770","reported_seq":17437,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-05T16:48:41.629203+0000","last_change":"2024-09-05T16:48:41.629203+0000","last_active":"2024-09-05T16:48:41.629203+0000","last_peered":"2024-09-05T16:48:41.629203+0000","last_clean":"2024-09-05T16:48:41.629203+0000","last_became_active":"2024-09-05T16:48:41.236479+0000","last_became_peered":"2024-09-05T16:48:41.236479+0000","last_unstale":"2024-09-05T16:48:41.629203+0000","last_undegraded":"2024-09-05T16:48:41.629203+0000","last_fullsized":"2024-09-05T16:48:41.629203+0000","mapping_epoch":62,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12770","last_scrub_stamp":"2024-09-05T16:48:41.629128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:48:41.629128+0000","objects_scrubbed":1,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T02:22:35.076855+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12808","reported_seq":17497,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-05T16:48:41.235475+0000","last_change":"2024-09-05T16:48:04.288466+0000","last_active":"2024-09-05T16:48:41.235475+0000","last_peered":"2024-09-05T16:48:41.235475+0000","last_clean":"2024-09-05T16:48:41.235475+0000","last_became_active":"2024-09-05T16:48:03.239894+0000","last_became_peered":"2024-09-05T16:48:03.239894+0000","last_unstale":"2024-09-05T16:48:41.235475+0000","last_undegraded":"2024-09-05T16:48:41.235475+0000","last_fullsized":"2024-09-05T16:48:41.235475+0000","mapping_epoch":56,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12808","last_scrub_stamp":"2024-09-05T16:48:04.288388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:48:04.288388+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T21:11:32.044919+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13120","reported_seq":17887,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-05T16:48:41.233200+0000","last_change":"2024-09-05T16:48:29.529847+0000","last_active":"2024-09-05T16:48:41.233200+0000","last_peered":"2024-09-05T16:48:41.233200+0000","last_clean":"2024-09-05T16:48:41.233200+0000","last_became_active":"2024-09-05T16:48:29.235000+0000","last_became_peered":"2024-09-05T16:48:29.235000+0000","last_unstale":"2024-09-05T16:48:41.233200+0000","last_undegraded":"2024-09-05T16:48:41.233200+0000","last_fullsized":"2024-09-05T16:48:41.233200+0000","mapping_epoch":60,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'13120","last_scrub_stamp":"2024-09-05T16:48:29.529768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:48:29.529768+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T02:18:10.992722+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12758","reported_seq":16026,"reported_epoch":63,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:48:48.215255+0000","last_change":"2024-09-05T16:48:39.285711+0000","last_active":"2024-09-05T16:48:48.215255+0000","last_peered":"2024-09-05T16:48:48.215255+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:48:48.215255+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":682,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1584,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-681}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12808","reported_seq":17598,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-05T16:48:41.234065+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:48:41.234065+0000","last_peered":"2024-09-05T16:48:41.234065+0000","last_clean":"2024-09-05T16:48:41.234065+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:48:41.234065+0000","last_undegraded":"2024-09-05T16:48:41.234065+0000","last_fullsized":"2024-09-05T16:48:41.234065+0000","mapping_epoch":36,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4408,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13124","reported_seq":13269,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-05T16:48:41.234257+0000","last_change":"2024-09-05T16:48:16.582142+0000","last_active":"2024-09-05T16:48:41.234257+0000","last_peered":"2024-09-05T16:48:41.234257+0000","last_clean":"2024-09-05T16:48:41.234257+0000","last_became_active":"2024-09-05T16:48:16.435091+0000","last_became_peered":"2024-09-05T16:48:16.435091+0000","last_unstale":"2024-09-05T16:48:41.234257+0000","last_undegraded":"2024-09-05T16:48:41.234257+0000","last_fullsized":"2024-09-05T16:48:41.234257+0000","mapping_epoch":58,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'13124","last_scrub_stamp":"2024-09-05T16:48:16.582038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:48:16.582038+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T17:44:25.891565+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"39'12831","reported_seq":12992,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-05T16:48:41.234342+0000","last_change":"2024-09-05T16:47:40.451380+0000","last_active":"2024-09-05T16:48:41.234342+0000","last_peered":"2024-09-05T16:48:41.234342+0000","last_clean":"2024-09-05T16:48:41.234342+0000","last_became_active":"2024-09-05T16:47:40.189447+0000","last_became_peered":"2024-09-05T16:47:40.189447+0000","last_unstale":"2024-09-05T16:48:41.234342+0000","last_undegraded":"2024-09-05T16:48:41.234342+0000","last_fullsized":"2024-09-05T16:48:41.234342+0000","mapping_epoch":52,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12831","last_scrub_stamp":"2024-09-05T16:47:40.451295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:40.451295+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:46:20.946333+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12773","reported_seq":12934,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-05T16:48:41.234360+0000","last_change":"2024-09-05T16:47:27.437275+0000","last_active":"2024-09-05T16:48:41.234360+0000","last_peered":"2024-09-05T16:48:41.234360+0000","last_clean":"2024-09-05T16:48:41.234360+0000","last_became_active":"2024-09-05T16:47:27.204989+0000","last_became_peered":"2024-09-05T16:47:27.204989+0000","last_unstale":"2024-09-05T16:48:41.234360+0000","last_undegraded":"2024-09-05T16:48:41.234360+0000","last_fullsized":"2024-09-05T16:48:41.234360+0000","mapping_epoch":50,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12773","last_scrub_stamp":"2024-09-05T16:47:27.437187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:27.437187+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T16:53:55.961513+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":682,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":51940,"num_write_kb":159895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6096,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084465,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":726360,"kb_used_data":516,"kb_used_omap":4,"kb_used_meta":725819,"kb_avail":93645480,"statfs":{"total":96636764160,"available":95892971520,"internally_reserved":0,"allocated":528384,"data_stored":200347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":743239379},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986342,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":623636,"kb_used_data":516,"kb_used_omap":4,"kb_used_meta":623099,"kb_avail":93748204,"statfs":{"total":96636764160,"available":95998160896,"internally_reserved":0,"allocated":528384,"data_stored":200347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":638054098},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673149,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713372,"kb_used_data":516,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":93658468,"statfs":{"total":96636764160,"available":95906271232,"internally_reserved":0,"allocated":528384,"data_stored":200347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":729938373},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182580,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713816,"kb_used_data":516,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658024,"statfs":{"total":96636764160,"available":95905816576,"internally_reserved":0,"allocated":528384,"data_stored":200347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":730394317},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:48:52.801 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:48:53.081 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:48:53.082 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:48:53.093 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":523,"stamp":"2024-09-05T16:48:52.670485+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":682,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":51940,"num_write_kb":159895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6096,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2777184,"kb_used_data":2064,"kb_used_omap":14,"kb_used_meta":2775025,"kb_avail":374710176,"statfs":{"total":386547056640,"available":383703220224,"internally_reserved":0,"allocated":2113536,"data_stored":801388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2841626167},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20208,"num_objects":-3,"num_object_clones":0,"num_object_copies":-12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1824,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12767,"num_write_kb":-51081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-443,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.550084"},"pg_stats":[{"pgid":"1.f","version":"39'12904","reported_seq":17545,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-05T16:48:41.233263+0000","last_change":"2024-09-05T16:46:27.826222+0000","last_active":"2024-09-05T16:48:41.233263+0000","last_peered":"2024-09-05T16:48:41.233263+0000","last_clean":"2024-09-05T16:48:41.233263+0000","last_became_active":"2024-09-05T16:46:27.757928+0000","last_became_peered":"2024-09-05T16:46:27.757928+0000","last_unstale":"2024-09-05T16:48:41.233263+0000","last_undegraded":"2024-09-05T16:48:41.233263+0000","last_fullsized":"2024-09-05T16:48:41.233263+0000","mapping_epoch":40,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:32:35.713428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12904,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2262,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12552","reported_seq":17258,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-05T16:48:41.234615+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:48:41.234615+0000","last_peered":"2024-09-05T16:48:41.234615+0000","last_clean":"2024-09-05T16:48:41.234615+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:48:41.234615+0000","last_undegraded":"2024-09-05T16:48:41.234615+0000","last_fullsized":"2024-09-05T16:48:41.234615+0000","mapping_epoch":38,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1865,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12752","reported_seq":17413,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-05T16:48:41.235308+0000","last_change":"2024-09-05T16:46:51.992073+0000","last_active":"2024-09-05T16:48:41.235308+0000","last_peered":"2024-09-05T16:48:41.235308+0000","last_clean":"2024-09-05T16:48:41.235308+0000","last_became_active":"2024-09-05T16:46:51.667128+0000","last_became_peered":"2024-09-05T16:46:51.667128+0000","last_unstale":"2024-09-05T16:48:41.235308+0000","last_undegraded":"2024-09-05T16:48:41.235308+0000","last_fullsized":"2024-09-05T16:48:41.235308+0000","mapping_epoch":44,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12752","last_scrub_stamp":"2024-09-05T16:46:51.991995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:51.991995+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:04:54.120031+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12934","reported_seq":17718,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-05T16:48:41.234020+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:48:41.234020+0000","last_peered":"2024-09-05T16:48:41.234020+0000","last_clean":"2024-09-05T16:48:41.234020+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:48:41.234020+0000","last_undegraded":"2024-09-05T16:48:41.234020+0000","last_fullsized":"2024-09-05T16:48:41.234020+0000","mapping_epoch":34,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7234,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12593","reported_seq":17059,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-05T16:48:41.232825+0000","last_change":"2024-09-05T16:47:51.525026+0000","last_active":"2024-09-05T16:48:41.232825+0000","last_peered":"2024-09-05T16:48:41.232825+0000","last_clean":"2024-09-05T16:48:41.232825+0000","last_became_active":"2024-09-05T16:47:51.193016+0000","last_became_peered":"2024-09-05T16:47:51.193016+0000","last_unstale":"2024-09-05T16:48:41.232825+0000","last_undegraded":"2024-09-05T16:48:41.232825+0000","last_fullsized":"2024-09-05T16:48:41.232825+0000","mapping_epoch":54,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12593","last_scrub_stamp":"2024-09-05T16:47:51.524957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:51.524957+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:46:22.846859+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12772","reported_seq":17389,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-05T16:48:41.233316+0000","last_change":"2024-09-05T16:47:15.832530+0000","last_active":"2024-09-05T16:48:41.233316+0000","last_peered":"2024-09-05T16:48:41.233316+0000","last_clean":"2024-09-05T16:48:41.233316+0000","last_became_active":"2024-09-05T16:47:15.728759+0000","last_became_peered":"2024-09-05T16:47:15.728759+0000","last_unstale":"2024-09-05T16:48:41.233316+0000","last_undegraded":"2024-09-05T16:48:41.233316+0000","last_fullsized":"2024-09-05T16:48:41.233316+0000","mapping_epoch":48,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T02:39:11.515339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12772,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2250,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12660","reported_seq":17107,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-05T16:48:41.231222+0000","last_change":"2024-09-05T16:47:03.683348+0000","last_active":"2024-09-05T16:48:41.231222+0000","last_peered":"2024-09-05T16:48:41.231222+0000","last_clean":"2024-09-05T16:48:41.231222+0000","last_became_active":"2024-09-05T16:47:03.185143+0000","last_became_peered":"2024-09-05T16:47:03.185143+0000","last_unstale":"2024-09-05T16:48:41.231222+0000","last_undegraded":"2024-09-05T16:48:41.231222+0000","last_fullsized":"2024-09-05T16:48:41.231222+0000","mapping_epoch":46,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12660","last_scrub_stamp":"2024-09-05T16:47:03.683262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:03.683262+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:42:16.533299+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12777","reported_seq":12935,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-05T16:48:41.234074+0000","last_change":"2024-09-05T16:46:40.458315+0000","last_active":"2024-09-05T16:48:41.234074+0000","last_peered":"2024-09-05T16:48:41.234074+0000","last_clean":"2024-09-05T16:48:41.234074+0000","last_became_active":"2024-09-05T16:46:40.184288+0000","last_became_peered":"2024-09-05T16:46:40.184288+0000","last_unstale":"2024-09-05T16:48:41.234074+0000","last_undegraded":"2024-09-05T16:48:41.234074+0000","last_fullsized":"2024-09-05T16:48:41.234074+0000","mapping_epoch":42,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12777","last_scrub_stamp":"2024-09-05T16:46:40.458226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:40.458226+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:38:58.734210+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'12770","reported_seq":17437,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-05T16:48:41.629203+0000","last_change":"2024-09-05T16:48:41.629203+0000","last_active":"2024-09-05T16:48:41.629203+0000","last_peered":"2024-09-05T16:48:41.629203+0000","last_clean":"2024-09-05T16:48:41.629203+0000","last_became_active":"2024-09-05T16:48:41.236479+0000","last_became_peered":"2024-09-05T16:48:41.236479+0000","last_unstale":"2024-09-05T16:48:41.629203+0000","last_undegraded":"2024-09-05T16:48:41.629203+0000","last_fullsized":"2024-09-05T16:48:41.629203+0000","mapping_epoch":62,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12770","last_scrub_stamp":"2024-09-05T16:48:41.629128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:48:41.629128+0000","objects_scrubbed":1,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T02:22:35.076855+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12808","reported_seq":17497,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-05T16:48:41.235475+0000","last_change":"2024-09-05T16:48:04.288466+0000","last_active":"2024-09-05T16:48:41.235475+0000","last_peered":"2024-09-05T16:48:41.235475+0000","last_clean":"2024-09-05T16:48:41.235475+0000","last_became_active":"2024-09-05T16:48:03.239894+0000","last_became_peered":"2024-09-05T16:48:03.239894+0000","last_unstale":"2024-09-05T16:48:41.235475+0000","last_undegraded":"2024-09-05T16:48:41.235475+0000","last_fullsized":"2024-09-05T16:48:41.235475+0000","mapping_epoch":56,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12808","last_scrub_stamp":"2024-09-05T16:48:04.288388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:48:04.288388+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T21:11:32.044919+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13120","reported_seq":17887,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-05T16:48:41.233200+0000","last_change":"2024-09-05T16:48:29.529847+0000","last_active":"2024-09-05T16:48:41.233200+0000","last_peered":"2024-09-05T16:48:41.233200+0000","last_clean":"2024-09-05T16:48:41.233200+0000","last_became_active":"2024-09-05T16:48:29.235000+0000","last_became_peered":"2024-09-05T16:48:29.235000+0000","last_unstale":"2024-09-05T16:48:41.233200+0000","last_undegraded":"2024-09-05T16:48:41.233200+0000","last_fullsized":"2024-09-05T16:48:41.233200+0000","mapping_epoch":60,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'13120","last_scrub_stamp":"2024-09-05T16:48:29.529768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:48:29.529768+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T02:18:10.992722+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12758","reported_seq":16026,"reported_epoch":63,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-05T16:48:48.215255+0000","last_change":"2024-09-05T16:48:39.285711+0000","last_active":"2024-09-05T16:48:48.215255+0000","last_peered":"2024-09-05T16:48:48.215255+0000","last_clean":"2024-09-05T16:33:45.082120+0000","last_became_active":"2024-09-05T16:34:56.977336+0000","last_became_peered":"2024-09-05T16:34:56.977336+0000","last_unstale":"2024-09-05T16:48:48.215255+0000","last_undegraded":"2024-09-05T16:34:56.623966+0000","last_fullsized":"2024-09-05T16:34:56.558662+0000","mapping_epoch":32,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":682,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1584,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-681}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"39'12808","reported_seq":17598,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-05T16:48:41.234065+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:48:41.234065+0000","last_peered":"2024-09-05T16:48:41.234065+0000","last_clean":"2024-09-05T16:48:41.234065+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:48:41.234065+0000","last_undegraded":"2024-09-05T16:48:41.234065+0000","last_fullsized":"2024-09-05T16:48:41.234065+0000","mapping_epoch":36,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4408,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13124","reported_seq":13269,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-05T16:48:41.234257+0000","last_change":"2024-09-05T16:48:16.582142+0000","last_active":"2024-09-05T16:48:41.234257+0000","last_peered":"2024-09-05T16:48:41.234257+0000","last_clean":"2024-09-05T16:48:41.234257+0000","last_became_active":"2024-09-05T16:48:16.435091+0000","last_became_peered":"2024-09-05T16:48:16.435091+0000","last_unstale":"2024-09-05T16:48:41.234257+0000","last_undegraded":"2024-09-05T16:48:41.234257+0000","last_fullsized":"2024-09-05T16:48:41.234257+0000","mapping_epoch":58,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'13124","last_scrub_stamp":"2024-09-05T16:48:16.582038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:48:16.582038+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T17:44:25.891565+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"39'12831","reported_seq":12992,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-05T16:48:41.234342+0000","last_change":"2024-09-05T16:47:40.451380+0000","last_active":"2024-09-05T16:48:41.234342+0000","last_peered":"2024-09-05T16:48:41.234342+0000","last_clean":"2024-09-05T16:48:41.234342+0000","last_became_active":"2024-09-05T16:47:40.189447+0000","last_became_peered":"2024-09-05T16:47:40.189447+0000","last_unstale":"2024-09-05T16:48:41.234342+0000","last_undegraded":"2024-09-05T16:48:41.234342+0000","last_fullsized":"2024-09-05T16:48:41.234342+0000","mapping_epoch":52,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12831","last_scrub_stamp":"2024-09-05T16:47:40.451295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:40.451295+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:46:20.946333+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12773","reported_seq":12934,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-05T16:48:41.234360+0000","last_change":"2024-09-05T16:47:27.437275+0000","last_active":"2024-09-05T16:48:41.234360+0000","last_peered":"2024-09-05T16:48:41.234360+0000","last_clean":"2024-09-05T16:48:41.234360+0000","last_became_active":"2024-09-05T16:47:27.204989+0000","last_became_peered":"2024-09-05T16:47:27.204989+0000","last_unstale":"2024-09-05T16:48:41.234360+0000","last_undegraded":"2024-09-05T16:48:41.234360+0000","last_fullsized":"2024-09-05T16:48:41.234360+0000","mapping_epoch":50,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12773","last_scrub_stamp":"2024-09-05T16:47:27.437187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:27.437187+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T16:53:55.961513+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":682,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":51940,"num_write_kb":159895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6096,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084465,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":726360,"kb_used_data":516,"kb_used_omap":4,"kb_used_meta":725819,"kb_avail":93645480,"statfs":{"total":96636764160,"available":95892971520,"internally_reserved":0,"allocated":528384,"data_stored":200347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":743239379},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986342,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":623636,"kb_used_data":516,"kb_used_omap":4,"kb_used_meta":623099,"kb_avail":93748204,"statfs":{"total":96636764160,"available":95998160896,"internally_reserved":0,"allocated":528384,"data_stored":200347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":638054098},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673149,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713372,"kb_used_data":516,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":93658468,"statfs":{"total":96636764160,"available":95906271232,"internally_reserved":0,"allocated":528384,"data_stored":200347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":729938373},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182580,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713816,"kb_used_data":516,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658024,"statfs":{"total":96636764160,"available":95905816576,"internally_reserved":0,"allocated":528384,"data_stored":200347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":730394317},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:48:53.094 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-05T16:48:53.094 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "39'12758", 'reported_seq': 16026, 'reported_epoch': 63, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-05T16:48:48.215255+0000', 'last_change': '2024-09-05T16:48:39.285711+0000', 'last_active': '2024-09-05T16:48:48.215255+0000', 'last_peered': '2024-09-05T16:48:48.215255+0000', 'last_clean': '2024-09-05T16:33:45.082120+0000', 'last_became_active': '2024-09-05T16:34:56.977336+0000', 'last_became_peered': '2024-09-05T16:34:56.977336+0000', 'last_unstale': '2024-09-05T16:48:48.215255+0000', 'last_undegraded': '2024-09-05T16:34:56.623966+0000', 'last_fullsized': '2024-09-05T16:34:56.558662+0000', 'mapping_epoch': 32, 'log_start': "33'2700", 'ondisk_log_start': "33'2700", 'created': 15, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'last_clean_scrub_stamp': '2024-09-05T16:33:41.856813+0000', 'objects_scrubbed': 0, 'log_size': 10058, 'log_dups_size': 0, 'ondisk_log_size': 10058, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6736, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 682, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12757, 'num_write_kb': 51031, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1584, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -681}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-05T16:48:53.095 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-05T16:48:53.451 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:48:53.451 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 682/8 objects degraded (8525.000%), 1 pg degraded, 1 pg undersized","count":2},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":936,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":65,"num_osds":4,"num_up_osds":4,"osd_up_since":1725554094,"num_in_osds":4,"osd_in_since":1725554001,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":2,"data_bytes":13472,"bytes_used":2843836416,"bytes_avail":383703220224,"bytes_total":386547056640,"degraded_objects":682,"degraded_total":8,"degraded_ratio":85.25},"fsmap":{"epoch":1,"btime":"2024-09-05T16:33:15:768647+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-05T16:35:54.815205+0000","services":{}},"progress_events":{}} 2024-09-05T16:48:53.464 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-09-05T16:48:53.465 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:48:53.747 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:48:53.748 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:48:53.759 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":524,"stamp":"2024-09-05T16:48:53.122270+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":51940,"num_write_kb":159895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4539,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2785516,"kb_used_data":2072,"kb_used_omap":14,"kb_used_meta":2783345,"kb_avail":374701844,"statfs":{"total":386547056640,"available":383694688256,"internally_reserved":0,"allocated":2121728,"data_stored":804456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2850145847},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1624,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-615,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001560"},"pg_stats":[{"pgid":"1.f","version":"39'12904","reported_seq":17545,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-05T16:48:41.233263+0000","last_change":"2024-09-05T16:46:27.826222+0000","last_active":"2024-09-05T16:48:41.233263+0000","last_peered":"2024-09-05T16:48:41.233263+0000","last_clean":"2024-09-05T16:48:41.233263+0000","last_became_active":"2024-09-05T16:46:27.757928+0000","last_became_peered":"2024-09-05T16:46:27.757928+0000","last_unstale":"2024-09-05T16:48:41.233263+0000","last_undegraded":"2024-09-05T16:48:41.233263+0000","last_fullsized":"2024-09-05T16:48:41.233263+0000","mapping_epoch":40,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:32:35.713428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12904,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2262,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12552","reported_seq":17262,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:48:52.673735+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:48:52.673735+0000","last_peered":"2024-09-05T16:48:52.673735+0000","last_clean":"2024-09-05T16:48:52.673735+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:48:52.673735+0000","last_undegraded":"2024-09-05T16:48:52.673735+0000","last_fullsized":"2024-09-05T16:48:52.673735+0000","mapping_epoch":38,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1865,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12752","reported_seq":17413,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-05T16:48:41.235308+0000","last_change":"2024-09-05T16:46:51.992073+0000","last_active":"2024-09-05T16:48:41.235308+0000","last_peered":"2024-09-05T16:48:41.235308+0000","last_clean":"2024-09-05T16:48:41.235308+0000","last_became_active":"2024-09-05T16:46:51.667128+0000","last_became_peered":"2024-09-05T16:46:51.667128+0000","last_unstale":"2024-09-05T16:48:41.235308+0000","last_undegraded":"2024-09-05T16:48:41.235308+0000","last_fullsized":"2024-09-05T16:48:41.235308+0000","mapping_epoch":44,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12752","last_scrub_stamp":"2024-09-05T16:46:51.991995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:51.991995+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:04:54.120031+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12934","reported_seq":17722,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:48:52.673535+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:48:52.673535+0000","last_peered":"2024-09-05T16:48:52.673535+0000","last_clean":"2024-09-05T16:48:52.673535+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:48:52.673535+0000","last_undegraded":"2024-09-05T16:48:52.673535+0000","last_fullsized":"2024-09-05T16:48:52.673535+0000","mapping_epoch":34,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7234,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12593","reported_seq":17059,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-05T16:48:41.232825+0000","last_change":"2024-09-05T16:47:51.525026+0000","last_active":"2024-09-05T16:48:41.232825+0000","last_peered":"2024-09-05T16:48:41.232825+0000","last_clean":"2024-09-05T16:48:41.232825+0000","last_became_active":"2024-09-05T16:47:51.193016+0000","last_became_peered":"2024-09-05T16:47:51.193016+0000","last_unstale":"2024-09-05T16:48:41.232825+0000","last_undegraded":"2024-09-05T16:48:41.232825+0000","last_fullsized":"2024-09-05T16:48:41.232825+0000","mapping_epoch":54,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12593","last_scrub_stamp":"2024-09-05T16:47:51.524957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:51.524957+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:46:22.846859+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12772","reported_seq":17389,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-05T16:48:41.233316+0000","last_change":"2024-09-05T16:47:15.832530+0000","last_active":"2024-09-05T16:48:41.233316+0000","last_peered":"2024-09-05T16:48:41.233316+0000","last_clean":"2024-09-05T16:48:41.233316+0000","last_became_active":"2024-09-05T16:47:15.728759+0000","last_became_peered":"2024-09-05T16:47:15.728759+0000","last_unstale":"2024-09-05T16:48:41.233316+0000","last_undegraded":"2024-09-05T16:48:41.233316+0000","last_fullsized":"2024-09-05T16:48:41.233316+0000","mapping_epoch":48,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T02:39:11.515339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12772,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2250,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12660","reported_seq":17107,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-05T16:48:41.231222+0000","last_change":"2024-09-05T16:47:03.683348+0000","last_active":"2024-09-05T16:48:41.231222+0000","last_peered":"2024-09-05T16:48:41.231222+0000","last_clean":"2024-09-05T16:48:41.231222+0000","last_became_active":"2024-09-05T16:47:03.185143+0000","last_became_peered":"2024-09-05T16:47:03.185143+0000","last_unstale":"2024-09-05T16:48:41.231222+0000","last_undegraded":"2024-09-05T16:48:41.231222+0000","last_fullsized":"2024-09-05T16:48:41.231222+0000","mapping_epoch":46,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12660","last_scrub_stamp":"2024-09-05T16:47:03.683262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:03.683262+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:42:16.533299+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12777","reported_seq":12939,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:48:52.673772+0000","last_change":"2024-09-05T16:46:40.458315+0000","last_active":"2024-09-05T16:48:52.673772+0000","last_peered":"2024-09-05T16:48:52.673772+0000","last_clean":"2024-09-05T16:48:52.673772+0000","last_became_active":"2024-09-05T16:46:40.184288+0000","last_became_peered":"2024-09-05T16:46:40.184288+0000","last_unstale":"2024-09-05T16:48:52.673772+0000","last_undegraded":"2024-09-05T16:48:52.673772+0000","last_fullsized":"2024-09-05T16:48:52.673772+0000","mapping_epoch":42,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12777","last_scrub_stamp":"2024-09-05T16:46:40.458226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:40.458226+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:38:58.734210+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'12770","reported_seq":17437,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-05T16:48:41.629203+0000","last_change":"2024-09-05T16:48:41.629203+0000","last_active":"2024-09-05T16:48:41.629203+0000","last_peered":"2024-09-05T16:48:41.629203+0000","last_clean":"2024-09-05T16:48:41.629203+0000","last_became_active":"2024-09-05T16:48:41.236479+0000","last_became_peered":"2024-09-05T16:48:41.236479+0000","last_unstale":"2024-09-05T16:48:41.629203+0000","last_undegraded":"2024-09-05T16:48:41.629203+0000","last_fullsized":"2024-09-05T16:48:41.629203+0000","mapping_epoch":62,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12770","last_scrub_stamp":"2024-09-05T16:48:41.629128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:48:41.629128+0000","objects_scrubbed":1,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T02:22:35.076855+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12808","reported_seq":17497,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-05T16:48:41.235475+0000","last_change":"2024-09-05T16:48:04.288466+0000","last_active":"2024-09-05T16:48:41.235475+0000","last_peered":"2024-09-05T16:48:41.235475+0000","last_clean":"2024-09-05T16:48:41.235475+0000","last_became_active":"2024-09-05T16:48:03.239894+0000","last_became_peered":"2024-09-05T16:48:03.239894+0000","last_unstale":"2024-09-05T16:48:41.235475+0000","last_undegraded":"2024-09-05T16:48:41.235475+0000","last_fullsized":"2024-09-05T16:48:41.235475+0000","mapping_epoch":56,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12808","last_scrub_stamp":"2024-09-05T16:48:04.288388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:48:04.288388+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T21:11:32.044919+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13120","reported_seq":17887,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-05T16:48:41.233200+0000","last_change":"2024-09-05T16:48:29.529847+0000","last_active":"2024-09-05T16:48:41.233200+0000","last_peered":"2024-09-05T16:48:41.233200+0000","last_clean":"2024-09-05T16:48:41.233200+0000","last_became_active":"2024-09-05T16:48:29.235000+0000","last_became_peered":"2024-09-05T16:48:29.235000+0000","last_unstale":"2024-09-05T16:48:41.233200+0000","last_undegraded":"2024-09-05T16:48:41.233200+0000","last_fullsized":"2024-09-05T16:48:41.233200+0000","mapping_epoch":60,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'13120","last_scrub_stamp":"2024-09-05T16:48:29.529768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:48:29.529768+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T02:18:10.992722+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12758","reported_seq":12876,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:48:52.754550+0000","last_change":"2024-09-05T16:48:52.754442+0000","last_active":"2024-09-05T16:48:52.754550+0000","last_peered":"2024-09-05T16:48:52.754550+0000","last_clean":"2024-09-05T16:48:52.754550+0000","last_became_active":"2024-09-05T16:48:52.676533+0000","last_became_peered":"2024-09-05T16:48:52.676533+0000","last_unstale":"2024-09-05T16:48:52.754550+0000","last_undegraded":"2024-09-05T16:48:52.754550+0000","last_fullsized":"2024-09-05T16:48:52.754550+0000","mapping_epoch":64,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"39'12808","reported_seq":17602,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:48:52.673654+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:48:52.673654+0000","last_peered":"2024-09-05T16:48:52.673654+0000","last_clean":"2024-09-05T16:48:52.673654+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:48:52.673654+0000","last_undegraded":"2024-09-05T16:48:52.673654+0000","last_fullsized":"2024-09-05T16:48:52.673654+0000","mapping_epoch":36,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4408,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13124","reported_seq":13273,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:48:52.673627+0000","last_change":"2024-09-05T16:48:16.582142+0000","last_active":"2024-09-05T16:48:52.673627+0000","last_peered":"2024-09-05T16:48:52.673627+0000","last_clean":"2024-09-05T16:48:52.673627+0000","last_became_active":"2024-09-05T16:48:16.435091+0000","last_became_peered":"2024-09-05T16:48:16.435091+0000","last_unstale":"2024-09-05T16:48:52.673627+0000","last_undegraded":"2024-09-05T16:48:52.673627+0000","last_fullsized":"2024-09-05T16:48:52.673627+0000","mapping_epoch":58,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'13124","last_scrub_stamp":"2024-09-05T16:48:16.582038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:48:16.582038+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T17:44:25.891565+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"39'12831","reported_seq":12996,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:48:52.673625+0000","last_change":"2024-09-05T16:47:40.451380+0000","last_active":"2024-09-05T16:48:52.673625+0000","last_peered":"2024-09-05T16:48:52.673625+0000","last_clean":"2024-09-05T16:48:52.673625+0000","last_became_active":"2024-09-05T16:47:40.189447+0000","last_became_peered":"2024-09-05T16:47:40.189447+0000","last_unstale":"2024-09-05T16:48:52.673625+0000","last_undegraded":"2024-09-05T16:48:52.673625+0000","last_fullsized":"2024-09-05T16:48:52.673625+0000","mapping_epoch":52,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12831","last_scrub_stamp":"2024-09-05T16:47:40.451295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:40.451295+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:46:20.946333+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12773","reported_seq":12938,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:48:52.673802+0000","last_change":"2024-09-05T16:47:27.437275+0000","last_active":"2024-09-05T16:48:52.673802+0000","last_peered":"2024-09-05T16:48:52.673802+0000","last_clean":"2024-09-05T16:48:52.673802+0000","last_became_active":"2024-09-05T16:47:27.204989+0000","last_became_peered":"2024-09-05T16:47:27.204989+0000","last_unstale":"2024-09-05T16:48:52.673802+0000","last_undegraded":"2024-09-05T16:48:52.673802+0000","last_fullsized":"2024-09-05T16:48:52.673802+0000","mapping_epoch":50,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12773","last_scrub_stamp":"2024-09-05T16:47:27.437187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:27.437187+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T16:53:55.961513+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":51940,"num_write_kb":159895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4539,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084465,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":726360,"kb_used_data":516,"kb_used_omap":4,"kb_used_meta":725819,"kb_avail":93645480,"statfs":{"total":96636764160,"available":95892971520,"internally_reserved":0,"allocated":528384,"data_stored":200347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":743239379},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986343,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":631968,"kb_used_data":524,"kb_used_omap":4,"kb_used_meta":631419,"kb_avail":93739872,"statfs":{"total":96636764160,"available":95989628928,"internally_reserved":0,"allocated":536576,"data_stored":203415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":646573778},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673149,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713372,"kb_used_data":516,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":93658468,"statfs":{"total":96636764160,"available":95906271232,"internally_reserved":0,"allocated":528384,"data_stored":200347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":729938373},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182580,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713816,"kb_used_data":516,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658024,"statfs":{"total":96636764160,"available":95905816576,"internally_reserved":0,"allocated":528384,"data_stored":200347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":730394317},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:48:56.762 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:48:57.056 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:48:57.057 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:48:57.069 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":525,"stamp":"2024-09-05T16:48:55.122662+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":51940,"num_write_kb":159895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4539,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2785524,"kb_used_data":2080,"kb_used_omap":14,"kb_used_meta":2783345,"kb_avail":374701836,"statfs":{"total":386547056640,"available":383694680064,"internally_reserved":0,"allocated":2129920,"data_stored":807524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2850145847},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1624,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-615,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001627"},"pg_stats":[{"pgid":"1.f","version":"39'12904","reported_seq":17549,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:48:52.672782+0000","last_change":"2024-09-05T16:46:27.826222+0000","last_active":"2024-09-05T16:48:52.672782+0000","last_peered":"2024-09-05T16:48:52.672782+0000","last_clean":"2024-09-05T16:48:52.672782+0000","last_became_active":"2024-09-05T16:46:27.757928+0000","last_became_peered":"2024-09-05T16:46:27.757928+0000","last_unstale":"2024-09-05T16:48:52.672782+0000","last_undegraded":"2024-09-05T16:48:52.672782+0000","last_fullsized":"2024-09-05T16:48:52.672782+0000","mapping_epoch":40,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:32:35.713428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12904,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2262,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12552","reported_seq":17262,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:48:52.673735+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:48:52.673735+0000","last_peered":"2024-09-05T16:48:52.673735+0000","last_clean":"2024-09-05T16:48:52.673735+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:48:52.673735+0000","last_undegraded":"2024-09-05T16:48:52.673735+0000","last_fullsized":"2024-09-05T16:48:52.673735+0000","mapping_epoch":38,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1865,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12752","reported_seq":17413,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-05T16:48:41.235308+0000","last_change":"2024-09-05T16:46:51.992073+0000","last_active":"2024-09-05T16:48:41.235308+0000","last_peered":"2024-09-05T16:48:41.235308+0000","last_clean":"2024-09-05T16:48:41.235308+0000","last_became_active":"2024-09-05T16:46:51.667128+0000","last_became_peered":"2024-09-05T16:46:51.667128+0000","last_unstale":"2024-09-05T16:48:41.235308+0000","last_undegraded":"2024-09-05T16:48:41.235308+0000","last_fullsized":"2024-09-05T16:48:41.235308+0000","mapping_epoch":44,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12752","last_scrub_stamp":"2024-09-05T16:46:51.991995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:51.991995+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:04:54.120031+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12934","reported_seq":17722,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:48:52.673535+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:48:52.673535+0000","last_peered":"2024-09-05T16:48:52.673535+0000","last_clean":"2024-09-05T16:48:52.673535+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:48:52.673535+0000","last_undegraded":"2024-09-05T16:48:52.673535+0000","last_fullsized":"2024-09-05T16:48:52.673535+0000","mapping_epoch":34,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7234,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12593","reported_seq":17063,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:48:52.672736+0000","last_change":"2024-09-05T16:47:51.525026+0000","last_active":"2024-09-05T16:48:52.672736+0000","last_peered":"2024-09-05T16:48:52.672736+0000","last_clean":"2024-09-05T16:48:52.672736+0000","last_became_active":"2024-09-05T16:47:51.193016+0000","last_became_peered":"2024-09-05T16:47:51.193016+0000","last_unstale":"2024-09-05T16:48:52.672736+0000","last_undegraded":"2024-09-05T16:48:52.672736+0000","last_fullsized":"2024-09-05T16:48:52.672736+0000","mapping_epoch":54,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12593","last_scrub_stamp":"2024-09-05T16:47:51.524957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:51.524957+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:46:22.846859+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12772","reported_seq":17393,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:48:52.673036+0000","last_change":"2024-09-05T16:47:15.832530+0000","last_active":"2024-09-05T16:48:52.673036+0000","last_peered":"2024-09-05T16:48:52.673036+0000","last_clean":"2024-09-05T16:48:52.673036+0000","last_became_active":"2024-09-05T16:47:15.728759+0000","last_became_peered":"2024-09-05T16:47:15.728759+0000","last_unstale":"2024-09-05T16:48:52.673036+0000","last_undegraded":"2024-09-05T16:48:52.673036+0000","last_fullsized":"2024-09-05T16:48:52.673036+0000","mapping_epoch":48,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T02:39:11.515339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12772,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2250,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12660","reported_seq":17107,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-05T16:48:41.231222+0000","last_change":"2024-09-05T16:47:03.683348+0000","last_active":"2024-09-05T16:48:41.231222+0000","last_peered":"2024-09-05T16:48:41.231222+0000","last_clean":"2024-09-05T16:48:41.231222+0000","last_became_active":"2024-09-05T16:47:03.185143+0000","last_became_peered":"2024-09-05T16:47:03.185143+0000","last_unstale":"2024-09-05T16:48:41.231222+0000","last_undegraded":"2024-09-05T16:48:41.231222+0000","last_fullsized":"2024-09-05T16:48:41.231222+0000","mapping_epoch":46,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12660","last_scrub_stamp":"2024-09-05T16:47:03.683262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:03.683262+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:42:16.533299+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12777","reported_seq":12939,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:48:52.673772+0000","last_change":"2024-09-05T16:46:40.458315+0000","last_active":"2024-09-05T16:48:52.673772+0000","last_peered":"2024-09-05T16:48:52.673772+0000","last_clean":"2024-09-05T16:48:52.673772+0000","last_became_active":"2024-09-05T16:46:40.184288+0000","last_became_peered":"2024-09-05T16:46:40.184288+0000","last_unstale":"2024-09-05T16:48:52.673772+0000","last_undegraded":"2024-09-05T16:48:52.673772+0000","last_fullsized":"2024-09-05T16:48:52.673772+0000","mapping_epoch":42,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12777","last_scrub_stamp":"2024-09-05T16:46:40.458226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:40.458226+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:38:58.734210+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'12770","reported_seq":17441,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:48:52.672936+0000","last_change":"2024-09-05T16:48:41.629203+0000","last_active":"2024-09-05T16:48:52.672936+0000","last_peered":"2024-09-05T16:48:52.672936+0000","last_clean":"2024-09-05T16:48:52.672936+0000","last_became_active":"2024-09-05T16:48:41.236479+0000","last_became_peered":"2024-09-05T16:48:41.236479+0000","last_unstale":"2024-09-05T16:48:52.672936+0000","last_undegraded":"2024-09-05T16:48:52.672936+0000","last_fullsized":"2024-09-05T16:48:52.672936+0000","mapping_epoch":62,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12770","last_scrub_stamp":"2024-09-05T16:48:41.629128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:48:41.629128+0000","objects_scrubbed":1,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T02:22:35.076855+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12808","reported_seq":17497,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-05T16:48:41.235475+0000","last_change":"2024-09-05T16:48:04.288466+0000","last_active":"2024-09-05T16:48:41.235475+0000","last_peered":"2024-09-05T16:48:41.235475+0000","last_clean":"2024-09-05T16:48:41.235475+0000","last_became_active":"2024-09-05T16:48:03.239894+0000","last_became_peered":"2024-09-05T16:48:03.239894+0000","last_unstale":"2024-09-05T16:48:41.235475+0000","last_undegraded":"2024-09-05T16:48:41.235475+0000","last_fullsized":"2024-09-05T16:48:41.235475+0000","mapping_epoch":56,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12808","last_scrub_stamp":"2024-09-05T16:48:04.288388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:48:04.288388+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T21:11:32.044919+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13120","reported_seq":17891,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:48:52.672814+0000","last_change":"2024-09-05T16:48:29.529847+0000","last_active":"2024-09-05T16:48:52.672814+0000","last_peered":"2024-09-05T16:48:52.672814+0000","last_clean":"2024-09-05T16:48:52.672814+0000","last_became_active":"2024-09-05T16:48:29.235000+0000","last_became_peered":"2024-09-05T16:48:29.235000+0000","last_unstale":"2024-09-05T16:48:52.672814+0000","last_undegraded":"2024-09-05T16:48:52.672814+0000","last_fullsized":"2024-09-05T16:48:52.672814+0000","mapping_epoch":60,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'13120","last_scrub_stamp":"2024-09-05T16:48:29.529768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:48:29.529768+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T02:18:10.992722+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12758","reported_seq":12876,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:48:52.754550+0000","last_change":"2024-09-05T16:48:52.754442+0000","last_active":"2024-09-05T16:48:52.754550+0000","last_peered":"2024-09-05T16:48:52.754550+0000","last_clean":"2024-09-05T16:48:52.754550+0000","last_became_active":"2024-09-05T16:48:52.676533+0000","last_became_peered":"2024-09-05T16:48:52.676533+0000","last_unstale":"2024-09-05T16:48:52.754550+0000","last_undegraded":"2024-09-05T16:48:52.754550+0000","last_fullsized":"2024-09-05T16:48:52.754550+0000","mapping_epoch":64,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"39'12808","reported_seq":17602,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:48:52.673654+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:48:52.673654+0000","last_peered":"2024-09-05T16:48:52.673654+0000","last_clean":"2024-09-05T16:48:52.673654+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:48:52.673654+0000","last_undegraded":"2024-09-05T16:48:52.673654+0000","last_fullsized":"2024-09-05T16:48:52.673654+0000","mapping_epoch":36,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4408,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13124","reported_seq":13273,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:48:52.673627+0000","last_change":"2024-09-05T16:48:16.582142+0000","last_active":"2024-09-05T16:48:52.673627+0000","last_peered":"2024-09-05T16:48:52.673627+0000","last_clean":"2024-09-05T16:48:52.673627+0000","last_became_active":"2024-09-05T16:48:16.435091+0000","last_became_peered":"2024-09-05T16:48:16.435091+0000","last_unstale":"2024-09-05T16:48:52.673627+0000","last_undegraded":"2024-09-05T16:48:52.673627+0000","last_fullsized":"2024-09-05T16:48:52.673627+0000","mapping_epoch":58,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'13124","last_scrub_stamp":"2024-09-05T16:48:16.582038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:48:16.582038+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T17:44:25.891565+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"39'12831","reported_seq":12996,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:48:52.673625+0000","last_change":"2024-09-05T16:47:40.451380+0000","last_active":"2024-09-05T16:48:52.673625+0000","last_peered":"2024-09-05T16:48:52.673625+0000","last_clean":"2024-09-05T16:48:52.673625+0000","last_became_active":"2024-09-05T16:47:40.189447+0000","last_became_peered":"2024-09-05T16:47:40.189447+0000","last_unstale":"2024-09-05T16:48:52.673625+0000","last_undegraded":"2024-09-05T16:48:52.673625+0000","last_fullsized":"2024-09-05T16:48:52.673625+0000","mapping_epoch":52,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12831","last_scrub_stamp":"2024-09-05T16:47:40.451295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:40.451295+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:46:20.946333+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12773","reported_seq":12938,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:48:52.673802+0000","last_change":"2024-09-05T16:47:27.437275+0000","last_active":"2024-09-05T16:48:52.673802+0000","last_peered":"2024-09-05T16:48:52.673802+0000","last_clean":"2024-09-05T16:48:52.673802+0000","last_became_active":"2024-09-05T16:47:27.204989+0000","last_became_peered":"2024-09-05T16:47:27.204989+0000","last_unstale":"2024-09-05T16:48:52.673802+0000","last_undegraded":"2024-09-05T16:48:52.673802+0000","last_fullsized":"2024-09-05T16:48:52.673802+0000","mapping_epoch":50,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12773","last_scrub_stamp":"2024-09-05T16:47:27.437187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:27.437187+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T16:53:55.961513+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":51940,"num_write_kb":159895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4539,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084466,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":726368,"kb_used_data":524,"kb_used_omap":4,"kb_used_meta":725819,"kb_avail":93645472,"statfs":{"total":96636764160,"available":95892963328,"internally_reserved":0,"allocated":536576,"data_stored":203415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":743239379},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986343,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":631968,"kb_used_data":524,"kb_used_omap":4,"kb_used_meta":631419,"kb_avail":93739872,"statfs":{"total":96636764160,"available":95989628928,"internally_reserved":0,"allocated":536576,"data_stored":203415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":646573778},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673149,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713372,"kb_used_data":516,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":93658468,"statfs":{"total":96636764160,"available":95906271232,"internally_reserved":0,"allocated":528384,"data_stored":200347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":729938373},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182580,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713816,"kb_used_data":516,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658024,"statfs":{"total":96636764160,"available":95905816576,"internally_reserved":0,"allocated":528384,"data_stored":200347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":730394317},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:48:57.070 INFO:tasks.ec_lost_unfound.ceph_manager:clean! 2024-09-05T16:48:57.070 INFO:tasks.ec_lost_unfound.ceph_manager:waiting for clean 2024-09-05T16:48:57.070 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:48:57.354 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:48:57.354 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:48:57.366 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":526,"stamp":"2024-09-05T16:48:57.122967+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":51940,"num_write_kb":159895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4539,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2785560,"kb_used_data":2112,"kb_used_omap":14,"kb_used_meta":2783345,"kb_avail":374701800,"statfs":{"total":386547056640,"available":383694643200,"internally_reserved":0,"allocated":2162688,"data_stored":819754,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2850145847},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-682,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1557,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001633"},"pg_stats":[{"pgid":"1.f","version":"39'12904","reported_seq":17549,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:48:52.672782+0000","last_change":"2024-09-05T16:46:27.826222+0000","last_active":"2024-09-05T16:48:52.672782+0000","last_peered":"2024-09-05T16:48:52.672782+0000","last_clean":"2024-09-05T16:48:52.672782+0000","last_became_active":"2024-09-05T16:46:27.757928+0000","last_became_peered":"2024-09-05T16:46:27.757928+0000","last_unstale":"2024-09-05T16:48:52.672782+0000","last_undegraded":"2024-09-05T16:48:52.672782+0000","last_fullsized":"2024-09-05T16:48:52.672782+0000","mapping_epoch":40,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:32:35.713428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12904,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2262,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12552","reported_seq":17262,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:48:52.673735+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:48:52.673735+0000","last_peered":"2024-09-05T16:48:52.673735+0000","last_clean":"2024-09-05T16:48:52.673735+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:48:52.673735+0000","last_undegraded":"2024-09-05T16:48:52.673735+0000","last_fullsized":"2024-09-05T16:48:52.673735+0000","mapping_epoch":38,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1865,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12752","reported_seq":17417,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:48:52.675313+0000","last_change":"2024-09-05T16:46:51.992073+0000","last_active":"2024-09-05T16:48:52.675313+0000","last_peered":"2024-09-05T16:48:52.675313+0000","last_clean":"2024-09-05T16:48:52.675313+0000","last_became_active":"2024-09-05T16:46:51.667128+0000","last_became_peered":"2024-09-05T16:46:51.667128+0000","last_unstale":"2024-09-05T16:48:52.675313+0000","last_undegraded":"2024-09-05T16:48:52.675313+0000","last_fullsized":"2024-09-05T16:48:52.675313+0000","mapping_epoch":44,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12752","last_scrub_stamp":"2024-09-05T16:46:51.991995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:51.991995+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:04:54.120031+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12934","reported_seq":17722,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:48:52.673535+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:48:52.673535+0000","last_peered":"2024-09-05T16:48:52.673535+0000","last_clean":"2024-09-05T16:48:52.673535+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:48:52.673535+0000","last_undegraded":"2024-09-05T16:48:52.673535+0000","last_fullsized":"2024-09-05T16:48:52.673535+0000","mapping_epoch":34,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7234,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12593","reported_seq":17063,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:48:52.672736+0000","last_change":"2024-09-05T16:47:51.525026+0000","last_active":"2024-09-05T16:48:52.672736+0000","last_peered":"2024-09-05T16:48:52.672736+0000","last_clean":"2024-09-05T16:48:52.672736+0000","last_became_active":"2024-09-05T16:47:51.193016+0000","last_became_peered":"2024-09-05T16:47:51.193016+0000","last_unstale":"2024-09-05T16:48:52.672736+0000","last_undegraded":"2024-09-05T16:48:52.672736+0000","last_fullsized":"2024-09-05T16:48:52.672736+0000","mapping_epoch":54,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12593","last_scrub_stamp":"2024-09-05T16:47:51.524957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:51.524957+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:46:22.846859+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12772","reported_seq":17393,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:48:52.673036+0000","last_change":"2024-09-05T16:47:15.832530+0000","last_active":"2024-09-05T16:48:52.673036+0000","last_peered":"2024-09-05T16:48:52.673036+0000","last_clean":"2024-09-05T16:48:52.673036+0000","last_became_active":"2024-09-05T16:47:15.728759+0000","last_became_peered":"2024-09-05T16:47:15.728759+0000","last_unstale":"2024-09-05T16:48:52.673036+0000","last_undegraded":"2024-09-05T16:48:52.673036+0000","last_fullsized":"2024-09-05T16:48:52.673036+0000","mapping_epoch":48,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T02:39:11.515339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12772,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2250,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12660","reported_seq":17111,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:48:52.671899+0000","last_change":"2024-09-05T16:47:03.683348+0000","last_active":"2024-09-05T16:48:52.671899+0000","last_peered":"2024-09-05T16:48:52.671899+0000","last_clean":"2024-09-05T16:48:52.671899+0000","last_became_active":"2024-09-05T16:47:03.185143+0000","last_became_peered":"2024-09-05T16:47:03.185143+0000","last_unstale":"2024-09-05T16:48:52.671899+0000","last_undegraded":"2024-09-05T16:48:52.671899+0000","last_fullsized":"2024-09-05T16:48:52.671899+0000","mapping_epoch":46,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12660","last_scrub_stamp":"2024-09-05T16:47:03.683262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:03.683262+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:42:16.533299+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12777","reported_seq":12939,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:48:52.673772+0000","last_change":"2024-09-05T16:46:40.458315+0000","last_active":"2024-09-05T16:48:52.673772+0000","last_peered":"2024-09-05T16:48:52.673772+0000","last_clean":"2024-09-05T16:48:52.673772+0000","last_became_active":"2024-09-05T16:46:40.184288+0000","last_became_peered":"2024-09-05T16:46:40.184288+0000","last_unstale":"2024-09-05T16:48:52.673772+0000","last_undegraded":"2024-09-05T16:48:52.673772+0000","last_fullsized":"2024-09-05T16:48:52.673772+0000","mapping_epoch":42,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12777","last_scrub_stamp":"2024-09-05T16:46:40.458226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:40.458226+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:38:58.734210+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'12770","reported_seq":17441,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:48:52.672936+0000","last_change":"2024-09-05T16:48:41.629203+0000","last_active":"2024-09-05T16:48:52.672936+0000","last_peered":"2024-09-05T16:48:52.672936+0000","last_clean":"2024-09-05T16:48:52.672936+0000","last_became_active":"2024-09-05T16:48:41.236479+0000","last_became_peered":"2024-09-05T16:48:41.236479+0000","last_unstale":"2024-09-05T16:48:52.672936+0000","last_undegraded":"2024-09-05T16:48:52.672936+0000","last_fullsized":"2024-09-05T16:48:52.672936+0000","mapping_epoch":62,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12770","last_scrub_stamp":"2024-09-05T16:48:41.629128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:48:41.629128+0000","objects_scrubbed":1,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T02:22:35.076855+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12808","reported_seq":17501,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:48:52.675410+0000","last_change":"2024-09-05T16:48:04.288466+0000","last_active":"2024-09-05T16:48:52.675410+0000","last_peered":"2024-09-05T16:48:52.675410+0000","last_clean":"2024-09-05T16:48:52.675410+0000","last_became_active":"2024-09-05T16:48:03.239894+0000","last_became_peered":"2024-09-05T16:48:03.239894+0000","last_unstale":"2024-09-05T16:48:52.675410+0000","last_undegraded":"2024-09-05T16:48:52.675410+0000","last_fullsized":"2024-09-05T16:48:52.675410+0000","mapping_epoch":56,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12808","last_scrub_stamp":"2024-09-05T16:48:04.288388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:48:04.288388+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T21:11:32.044919+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13120","reported_seq":17891,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:48:52.672814+0000","last_change":"2024-09-05T16:48:29.529847+0000","last_active":"2024-09-05T16:48:52.672814+0000","last_peered":"2024-09-05T16:48:52.672814+0000","last_clean":"2024-09-05T16:48:52.672814+0000","last_became_active":"2024-09-05T16:48:29.235000+0000","last_became_peered":"2024-09-05T16:48:29.235000+0000","last_unstale":"2024-09-05T16:48:52.672814+0000","last_undegraded":"2024-09-05T16:48:52.672814+0000","last_fullsized":"2024-09-05T16:48:52.672814+0000","mapping_epoch":60,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'13120","last_scrub_stamp":"2024-09-05T16:48:29.529768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:48:29.529768+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T02:18:10.992722+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12758","reported_seq":12876,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:48:52.754550+0000","last_change":"2024-09-05T16:48:52.754442+0000","last_active":"2024-09-05T16:48:52.754550+0000","last_peered":"2024-09-05T16:48:52.754550+0000","last_clean":"2024-09-05T16:48:52.754550+0000","last_became_active":"2024-09-05T16:48:52.676533+0000","last_became_peered":"2024-09-05T16:48:52.676533+0000","last_unstale":"2024-09-05T16:48:52.754550+0000","last_undegraded":"2024-09-05T16:48:52.754550+0000","last_fullsized":"2024-09-05T16:48:52.754550+0000","mapping_epoch":64,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"39'12808","reported_seq":17602,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:48:52.673654+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:48:52.673654+0000","last_peered":"2024-09-05T16:48:52.673654+0000","last_clean":"2024-09-05T16:48:52.673654+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:48:52.673654+0000","last_undegraded":"2024-09-05T16:48:52.673654+0000","last_fullsized":"2024-09-05T16:48:52.673654+0000","mapping_epoch":36,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4408,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13124","reported_seq":13273,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:48:52.673627+0000","last_change":"2024-09-05T16:48:16.582142+0000","last_active":"2024-09-05T16:48:52.673627+0000","last_peered":"2024-09-05T16:48:52.673627+0000","last_clean":"2024-09-05T16:48:52.673627+0000","last_became_active":"2024-09-05T16:48:16.435091+0000","last_became_peered":"2024-09-05T16:48:16.435091+0000","last_unstale":"2024-09-05T16:48:52.673627+0000","last_undegraded":"2024-09-05T16:48:52.673627+0000","last_fullsized":"2024-09-05T16:48:52.673627+0000","mapping_epoch":58,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'13124","last_scrub_stamp":"2024-09-05T16:48:16.582038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:48:16.582038+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T17:44:25.891565+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"39'12831","reported_seq":12996,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:48:52.673625+0000","last_change":"2024-09-05T16:47:40.451380+0000","last_active":"2024-09-05T16:48:52.673625+0000","last_peered":"2024-09-05T16:48:52.673625+0000","last_clean":"2024-09-05T16:48:52.673625+0000","last_became_active":"2024-09-05T16:47:40.189447+0000","last_became_peered":"2024-09-05T16:47:40.189447+0000","last_unstale":"2024-09-05T16:48:52.673625+0000","last_undegraded":"2024-09-05T16:48:52.673625+0000","last_fullsized":"2024-09-05T16:48:52.673625+0000","mapping_epoch":52,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12831","last_scrub_stamp":"2024-09-05T16:47:40.451295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:40.451295+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:46:20.946333+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12773","reported_seq":12938,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:48:52.673802+0000","last_change":"2024-09-05T16:47:27.437275+0000","last_active":"2024-09-05T16:48:52.673802+0000","last_peered":"2024-09-05T16:48:52.673802+0000","last_clean":"2024-09-05T16:48:52.673802+0000","last_became_active":"2024-09-05T16:47:27.204989+0000","last_became_peered":"2024-09-05T16:47:27.204989+0000","last_unstale":"2024-09-05T16:48:52.673802+0000","last_undegraded":"2024-09-05T16:48:52.673802+0000","last_fullsized":"2024-09-05T16:48:52.673802+0000","mapping_epoch":50,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12773","last_scrub_stamp":"2024-09-05T16:47:27.437187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:27.437187+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T16:53:55.961513+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":51940,"num_write_kb":159895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4539,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084466,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":726368,"kb_used_data":524,"kb_used_omap":4,"kb_used_meta":725819,"kb_avail":93645472,"statfs":{"total":96636764160,"available":95892963328,"internally_reserved":0,"allocated":536576,"data_stored":203415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":743239379},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986343,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":631968,"kb_used_data":524,"kb_used_omap":4,"kb_used_meta":631419,"kb_avail":93739872,"statfs":{"total":96636764160,"available":95989628928,"internally_reserved":0,"allocated":536576,"data_stored":203415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":646573778},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673150,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713388,"kb_used_data":532,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":93658452,"statfs":{"total":96636764160,"available":95906254848,"internally_reserved":0,"allocated":544768,"data_stored":206462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":729938373},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182581,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713836,"kb_used_data":532,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658004,"statfs":{"total":96636764160,"available":95905796096,"internally_reserved":0,"allocated":544768,"data_stored":206462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":730394317},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:48:57.367 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:48:57.647 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:48:57.647 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:48:57.659 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":526,"stamp":"2024-09-05T16:48:57.122967+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":51940,"num_write_kb":159895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4539,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2785560,"kb_used_data":2112,"kb_used_omap":14,"kb_used_meta":2783345,"kb_avail":374701800,"statfs":{"total":386547056640,"available":383694643200,"internally_reserved":0,"allocated":2162688,"data_stored":819754,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2850145847},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-682,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1557,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001633"},"pg_stats":[{"pgid":"1.f","version":"39'12904","reported_seq":17549,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:48:52.672782+0000","last_change":"2024-09-05T16:46:27.826222+0000","last_active":"2024-09-05T16:48:52.672782+0000","last_peered":"2024-09-05T16:48:52.672782+0000","last_clean":"2024-09-05T16:48:52.672782+0000","last_became_active":"2024-09-05T16:46:27.757928+0000","last_became_peered":"2024-09-05T16:46:27.757928+0000","last_unstale":"2024-09-05T16:48:52.672782+0000","last_undegraded":"2024-09-05T16:48:52.672782+0000","last_fullsized":"2024-09-05T16:48:52.672782+0000","mapping_epoch":40,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:32:35.713428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12904,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2262,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12552","reported_seq":17262,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:48:52.673735+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:48:52.673735+0000","last_peered":"2024-09-05T16:48:52.673735+0000","last_clean":"2024-09-05T16:48:52.673735+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:48:52.673735+0000","last_undegraded":"2024-09-05T16:48:52.673735+0000","last_fullsized":"2024-09-05T16:48:52.673735+0000","mapping_epoch":38,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1865,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12752","reported_seq":17417,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:48:52.675313+0000","last_change":"2024-09-05T16:46:51.992073+0000","last_active":"2024-09-05T16:48:52.675313+0000","last_peered":"2024-09-05T16:48:52.675313+0000","last_clean":"2024-09-05T16:48:52.675313+0000","last_became_active":"2024-09-05T16:46:51.667128+0000","last_became_peered":"2024-09-05T16:46:51.667128+0000","last_unstale":"2024-09-05T16:48:52.675313+0000","last_undegraded":"2024-09-05T16:48:52.675313+0000","last_fullsized":"2024-09-05T16:48:52.675313+0000","mapping_epoch":44,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12752","last_scrub_stamp":"2024-09-05T16:46:51.991995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:51.991995+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:04:54.120031+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12934","reported_seq":17722,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:48:52.673535+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:48:52.673535+0000","last_peered":"2024-09-05T16:48:52.673535+0000","last_clean":"2024-09-05T16:48:52.673535+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:48:52.673535+0000","last_undegraded":"2024-09-05T16:48:52.673535+0000","last_fullsized":"2024-09-05T16:48:52.673535+0000","mapping_epoch":34,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7234,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12593","reported_seq":17063,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:48:52.672736+0000","last_change":"2024-09-05T16:47:51.525026+0000","last_active":"2024-09-05T16:48:52.672736+0000","last_peered":"2024-09-05T16:48:52.672736+0000","last_clean":"2024-09-05T16:48:52.672736+0000","last_became_active":"2024-09-05T16:47:51.193016+0000","last_became_peered":"2024-09-05T16:47:51.193016+0000","last_unstale":"2024-09-05T16:48:52.672736+0000","last_undegraded":"2024-09-05T16:48:52.672736+0000","last_fullsized":"2024-09-05T16:48:52.672736+0000","mapping_epoch":54,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12593","last_scrub_stamp":"2024-09-05T16:47:51.524957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:51.524957+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:46:22.846859+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12772","reported_seq":17393,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:48:52.673036+0000","last_change":"2024-09-05T16:47:15.832530+0000","last_active":"2024-09-05T16:48:52.673036+0000","last_peered":"2024-09-05T16:48:52.673036+0000","last_clean":"2024-09-05T16:48:52.673036+0000","last_became_active":"2024-09-05T16:47:15.728759+0000","last_became_peered":"2024-09-05T16:47:15.728759+0000","last_unstale":"2024-09-05T16:48:52.673036+0000","last_undegraded":"2024-09-05T16:48:52.673036+0000","last_fullsized":"2024-09-05T16:48:52.673036+0000","mapping_epoch":48,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T02:39:11.515339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12772,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2250,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12660","reported_seq":17111,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:48:52.671899+0000","last_change":"2024-09-05T16:47:03.683348+0000","last_active":"2024-09-05T16:48:52.671899+0000","last_peered":"2024-09-05T16:48:52.671899+0000","last_clean":"2024-09-05T16:48:52.671899+0000","last_became_active":"2024-09-05T16:47:03.185143+0000","last_became_peered":"2024-09-05T16:47:03.185143+0000","last_unstale":"2024-09-05T16:48:52.671899+0000","last_undegraded":"2024-09-05T16:48:52.671899+0000","last_fullsized":"2024-09-05T16:48:52.671899+0000","mapping_epoch":46,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12660","last_scrub_stamp":"2024-09-05T16:47:03.683262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:03.683262+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:42:16.533299+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12777","reported_seq":12939,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:48:52.673772+0000","last_change":"2024-09-05T16:46:40.458315+0000","last_active":"2024-09-05T16:48:52.673772+0000","last_peered":"2024-09-05T16:48:52.673772+0000","last_clean":"2024-09-05T16:48:52.673772+0000","last_became_active":"2024-09-05T16:46:40.184288+0000","last_became_peered":"2024-09-05T16:46:40.184288+0000","last_unstale":"2024-09-05T16:48:52.673772+0000","last_undegraded":"2024-09-05T16:48:52.673772+0000","last_fullsized":"2024-09-05T16:48:52.673772+0000","mapping_epoch":42,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12777","last_scrub_stamp":"2024-09-05T16:46:40.458226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:40.458226+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:38:58.734210+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'12770","reported_seq":17441,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:48:52.672936+0000","last_change":"2024-09-05T16:48:41.629203+0000","last_active":"2024-09-05T16:48:52.672936+0000","last_peered":"2024-09-05T16:48:52.672936+0000","last_clean":"2024-09-05T16:48:52.672936+0000","last_became_active":"2024-09-05T16:48:41.236479+0000","last_became_peered":"2024-09-05T16:48:41.236479+0000","last_unstale":"2024-09-05T16:48:52.672936+0000","last_undegraded":"2024-09-05T16:48:52.672936+0000","last_fullsized":"2024-09-05T16:48:52.672936+0000","mapping_epoch":62,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12770","last_scrub_stamp":"2024-09-05T16:48:41.629128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:48:41.629128+0000","objects_scrubbed":1,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T02:22:35.076855+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12808","reported_seq":17501,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:48:52.675410+0000","last_change":"2024-09-05T16:48:04.288466+0000","last_active":"2024-09-05T16:48:52.675410+0000","last_peered":"2024-09-05T16:48:52.675410+0000","last_clean":"2024-09-05T16:48:52.675410+0000","last_became_active":"2024-09-05T16:48:03.239894+0000","last_became_peered":"2024-09-05T16:48:03.239894+0000","last_unstale":"2024-09-05T16:48:52.675410+0000","last_undegraded":"2024-09-05T16:48:52.675410+0000","last_fullsized":"2024-09-05T16:48:52.675410+0000","mapping_epoch":56,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12808","last_scrub_stamp":"2024-09-05T16:48:04.288388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:48:04.288388+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T21:11:32.044919+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13120","reported_seq":17891,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:48:52.672814+0000","last_change":"2024-09-05T16:48:29.529847+0000","last_active":"2024-09-05T16:48:52.672814+0000","last_peered":"2024-09-05T16:48:52.672814+0000","last_clean":"2024-09-05T16:48:52.672814+0000","last_became_active":"2024-09-05T16:48:29.235000+0000","last_became_peered":"2024-09-05T16:48:29.235000+0000","last_unstale":"2024-09-05T16:48:52.672814+0000","last_undegraded":"2024-09-05T16:48:52.672814+0000","last_fullsized":"2024-09-05T16:48:52.672814+0000","mapping_epoch":60,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'13120","last_scrub_stamp":"2024-09-05T16:48:29.529768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:48:29.529768+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T02:18:10.992722+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12758","reported_seq":12876,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:48:52.754550+0000","last_change":"2024-09-05T16:48:52.754442+0000","last_active":"2024-09-05T16:48:52.754550+0000","last_peered":"2024-09-05T16:48:52.754550+0000","last_clean":"2024-09-05T16:48:52.754550+0000","last_became_active":"2024-09-05T16:48:52.676533+0000","last_became_peered":"2024-09-05T16:48:52.676533+0000","last_unstale":"2024-09-05T16:48:52.754550+0000","last_undegraded":"2024-09-05T16:48:52.754550+0000","last_fullsized":"2024-09-05T16:48:52.754550+0000","mapping_epoch":64,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"39'12808","reported_seq":17602,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:48:52.673654+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:48:52.673654+0000","last_peered":"2024-09-05T16:48:52.673654+0000","last_clean":"2024-09-05T16:48:52.673654+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:48:52.673654+0000","last_undegraded":"2024-09-05T16:48:52.673654+0000","last_fullsized":"2024-09-05T16:48:52.673654+0000","mapping_epoch":36,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4408,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13124","reported_seq":13273,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:48:52.673627+0000","last_change":"2024-09-05T16:48:16.582142+0000","last_active":"2024-09-05T16:48:52.673627+0000","last_peered":"2024-09-05T16:48:52.673627+0000","last_clean":"2024-09-05T16:48:52.673627+0000","last_became_active":"2024-09-05T16:48:16.435091+0000","last_became_peered":"2024-09-05T16:48:16.435091+0000","last_unstale":"2024-09-05T16:48:52.673627+0000","last_undegraded":"2024-09-05T16:48:52.673627+0000","last_fullsized":"2024-09-05T16:48:52.673627+0000","mapping_epoch":58,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'13124","last_scrub_stamp":"2024-09-05T16:48:16.582038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:48:16.582038+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T17:44:25.891565+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"39'12831","reported_seq":12996,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:48:52.673625+0000","last_change":"2024-09-05T16:47:40.451380+0000","last_active":"2024-09-05T16:48:52.673625+0000","last_peered":"2024-09-05T16:48:52.673625+0000","last_clean":"2024-09-05T16:48:52.673625+0000","last_became_active":"2024-09-05T16:47:40.189447+0000","last_became_peered":"2024-09-05T16:47:40.189447+0000","last_unstale":"2024-09-05T16:48:52.673625+0000","last_undegraded":"2024-09-05T16:48:52.673625+0000","last_fullsized":"2024-09-05T16:48:52.673625+0000","mapping_epoch":52,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12831","last_scrub_stamp":"2024-09-05T16:47:40.451295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:40.451295+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:46:20.946333+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12773","reported_seq":12938,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:48:52.673802+0000","last_change":"2024-09-05T16:47:27.437275+0000","last_active":"2024-09-05T16:48:52.673802+0000","last_peered":"2024-09-05T16:48:52.673802+0000","last_clean":"2024-09-05T16:48:52.673802+0000","last_became_active":"2024-09-05T16:47:27.204989+0000","last_became_peered":"2024-09-05T16:47:27.204989+0000","last_unstale":"2024-09-05T16:48:52.673802+0000","last_undegraded":"2024-09-05T16:48:52.673802+0000","last_fullsized":"2024-09-05T16:48:52.673802+0000","mapping_epoch":50,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12773","last_scrub_stamp":"2024-09-05T16:47:27.437187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:27.437187+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T16:53:55.961513+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":51940,"num_write_kb":159895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4539,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084466,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":726368,"kb_used_data":524,"kb_used_omap":4,"kb_used_meta":725819,"kb_avail":93645472,"statfs":{"total":96636764160,"available":95892963328,"internally_reserved":0,"allocated":536576,"data_stored":203415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":743239379},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986343,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":631968,"kb_used_data":524,"kb_used_omap":4,"kb_used_meta":631419,"kb_avail":93739872,"statfs":{"total":96636764160,"available":95989628928,"internally_reserved":0,"allocated":536576,"data_stored":203415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":646573778},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673150,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713388,"kb_used_data":532,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":93658452,"statfs":{"total":96636764160,"available":95906254848,"internally_reserved":0,"allocated":544768,"data_stored":206462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":729938373},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182581,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713836,"kb_used_data":532,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658004,"statfs":{"total":96636764160,"available":95905796096,"internally_reserved":0,"allocated":544768,"data_stored":206462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":730394317},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:48:57.660 INFO:tasks.ec_lost_unfound.ceph_manager:clean! 2024-09-05T16:48:57.661 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-05T16:48:57.670 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-05T16:48:57.670 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:48:57.952 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:48:57.952 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:48:57.964 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":526,"stamp":"2024-09-05T16:48:57.122967+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":51940,"num_write_kb":159895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4539,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2785560,"kb_used_data":2112,"kb_used_omap":14,"kb_used_meta":2783345,"kb_avail":374701800,"statfs":{"total":386547056640,"available":383694643200,"internally_reserved":0,"allocated":2162688,"data_stored":819754,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2850145847},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-682,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1557,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001633"},"pg_stats":[{"pgid":"1.f","version":"39'12904","reported_seq":17549,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:48:52.672782+0000","last_change":"2024-09-05T16:46:27.826222+0000","last_active":"2024-09-05T16:48:52.672782+0000","last_peered":"2024-09-05T16:48:52.672782+0000","last_clean":"2024-09-05T16:48:52.672782+0000","last_became_active":"2024-09-05T16:46:27.757928+0000","last_became_peered":"2024-09-05T16:46:27.757928+0000","last_unstale":"2024-09-05T16:48:52.672782+0000","last_undegraded":"2024-09-05T16:48:52.672782+0000","last_fullsized":"2024-09-05T16:48:52.672782+0000","mapping_epoch":40,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:32:35.713428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12904,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2262,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12552","reported_seq":17262,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:48:52.673735+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:48:52.673735+0000","last_peered":"2024-09-05T16:48:52.673735+0000","last_clean":"2024-09-05T16:48:52.673735+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:48:52.673735+0000","last_undegraded":"2024-09-05T16:48:52.673735+0000","last_fullsized":"2024-09-05T16:48:52.673735+0000","mapping_epoch":38,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1865,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12752","reported_seq":17417,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:48:52.675313+0000","last_change":"2024-09-05T16:46:51.992073+0000","last_active":"2024-09-05T16:48:52.675313+0000","last_peered":"2024-09-05T16:48:52.675313+0000","last_clean":"2024-09-05T16:48:52.675313+0000","last_became_active":"2024-09-05T16:46:51.667128+0000","last_became_peered":"2024-09-05T16:46:51.667128+0000","last_unstale":"2024-09-05T16:48:52.675313+0000","last_undegraded":"2024-09-05T16:48:52.675313+0000","last_fullsized":"2024-09-05T16:48:52.675313+0000","mapping_epoch":44,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12752","last_scrub_stamp":"2024-09-05T16:46:51.991995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:51.991995+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:04:54.120031+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12934","reported_seq":17722,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:48:52.673535+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:48:52.673535+0000","last_peered":"2024-09-05T16:48:52.673535+0000","last_clean":"2024-09-05T16:48:52.673535+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:48:52.673535+0000","last_undegraded":"2024-09-05T16:48:52.673535+0000","last_fullsized":"2024-09-05T16:48:52.673535+0000","mapping_epoch":34,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7234,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12593","reported_seq":17063,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:48:52.672736+0000","last_change":"2024-09-05T16:47:51.525026+0000","last_active":"2024-09-05T16:48:52.672736+0000","last_peered":"2024-09-05T16:48:52.672736+0000","last_clean":"2024-09-05T16:48:52.672736+0000","last_became_active":"2024-09-05T16:47:51.193016+0000","last_became_peered":"2024-09-05T16:47:51.193016+0000","last_unstale":"2024-09-05T16:48:52.672736+0000","last_undegraded":"2024-09-05T16:48:52.672736+0000","last_fullsized":"2024-09-05T16:48:52.672736+0000","mapping_epoch":54,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12593","last_scrub_stamp":"2024-09-05T16:47:51.524957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:51.524957+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:46:22.846859+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12772","reported_seq":17393,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:48:52.673036+0000","last_change":"2024-09-05T16:47:15.832530+0000","last_active":"2024-09-05T16:48:52.673036+0000","last_peered":"2024-09-05T16:48:52.673036+0000","last_clean":"2024-09-05T16:48:52.673036+0000","last_became_active":"2024-09-05T16:47:15.728759+0000","last_became_peered":"2024-09-05T16:47:15.728759+0000","last_unstale":"2024-09-05T16:48:52.673036+0000","last_undegraded":"2024-09-05T16:48:52.673036+0000","last_fullsized":"2024-09-05T16:48:52.673036+0000","mapping_epoch":48,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T02:39:11.515339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12772,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2250,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12660","reported_seq":17111,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:48:52.671899+0000","last_change":"2024-09-05T16:47:03.683348+0000","last_active":"2024-09-05T16:48:52.671899+0000","last_peered":"2024-09-05T16:48:52.671899+0000","last_clean":"2024-09-05T16:48:52.671899+0000","last_became_active":"2024-09-05T16:47:03.185143+0000","last_became_peered":"2024-09-05T16:47:03.185143+0000","last_unstale":"2024-09-05T16:48:52.671899+0000","last_undegraded":"2024-09-05T16:48:52.671899+0000","last_fullsized":"2024-09-05T16:48:52.671899+0000","mapping_epoch":46,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12660","last_scrub_stamp":"2024-09-05T16:47:03.683262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:03.683262+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:42:16.533299+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12777","reported_seq":12939,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:48:52.673772+0000","last_change":"2024-09-05T16:46:40.458315+0000","last_active":"2024-09-05T16:48:52.673772+0000","last_peered":"2024-09-05T16:48:52.673772+0000","last_clean":"2024-09-05T16:48:52.673772+0000","last_became_active":"2024-09-05T16:46:40.184288+0000","last_became_peered":"2024-09-05T16:46:40.184288+0000","last_unstale":"2024-09-05T16:48:52.673772+0000","last_undegraded":"2024-09-05T16:48:52.673772+0000","last_fullsized":"2024-09-05T16:48:52.673772+0000","mapping_epoch":42,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12777","last_scrub_stamp":"2024-09-05T16:46:40.458226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:40.458226+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:38:58.734210+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'12770","reported_seq":17441,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:48:52.672936+0000","last_change":"2024-09-05T16:48:41.629203+0000","last_active":"2024-09-05T16:48:52.672936+0000","last_peered":"2024-09-05T16:48:52.672936+0000","last_clean":"2024-09-05T16:48:52.672936+0000","last_became_active":"2024-09-05T16:48:41.236479+0000","last_became_peered":"2024-09-05T16:48:41.236479+0000","last_unstale":"2024-09-05T16:48:52.672936+0000","last_undegraded":"2024-09-05T16:48:52.672936+0000","last_fullsized":"2024-09-05T16:48:52.672936+0000","mapping_epoch":62,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12770","last_scrub_stamp":"2024-09-05T16:48:41.629128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:48:41.629128+0000","objects_scrubbed":1,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T02:22:35.076855+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12808","reported_seq":17501,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:48:52.675410+0000","last_change":"2024-09-05T16:48:04.288466+0000","last_active":"2024-09-05T16:48:52.675410+0000","last_peered":"2024-09-05T16:48:52.675410+0000","last_clean":"2024-09-05T16:48:52.675410+0000","last_became_active":"2024-09-05T16:48:03.239894+0000","last_became_peered":"2024-09-05T16:48:03.239894+0000","last_unstale":"2024-09-05T16:48:52.675410+0000","last_undegraded":"2024-09-05T16:48:52.675410+0000","last_fullsized":"2024-09-05T16:48:52.675410+0000","mapping_epoch":56,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12808","last_scrub_stamp":"2024-09-05T16:48:04.288388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:48:04.288388+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T21:11:32.044919+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13120","reported_seq":17891,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:48:52.672814+0000","last_change":"2024-09-05T16:48:29.529847+0000","last_active":"2024-09-05T16:48:52.672814+0000","last_peered":"2024-09-05T16:48:52.672814+0000","last_clean":"2024-09-05T16:48:52.672814+0000","last_became_active":"2024-09-05T16:48:29.235000+0000","last_became_peered":"2024-09-05T16:48:29.235000+0000","last_unstale":"2024-09-05T16:48:52.672814+0000","last_undegraded":"2024-09-05T16:48:52.672814+0000","last_fullsized":"2024-09-05T16:48:52.672814+0000","mapping_epoch":60,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'13120","last_scrub_stamp":"2024-09-05T16:48:29.529768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:48:29.529768+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T02:18:10.992722+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12758","reported_seq":12876,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:48:52.754550+0000","last_change":"2024-09-05T16:48:52.754442+0000","last_active":"2024-09-05T16:48:52.754550+0000","last_peered":"2024-09-05T16:48:52.754550+0000","last_clean":"2024-09-05T16:48:52.754550+0000","last_became_active":"2024-09-05T16:48:52.676533+0000","last_became_peered":"2024-09-05T16:48:52.676533+0000","last_unstale":"2024-09-05T16:48:52.754550+0000","last_undegraded":"2024-09-05T16:48:52.754550+0000","last_fullsized":"2024-09-05T16:48:52.754550+0000","mapping_epoch":64,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"39'12808","reported_seq":17602,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:48:52.673654+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:48:52.673654+0000","last_peered":"2024-09-05T16:48:52.673654+0000","last_clean":"2024-09-05T16:48:52.673654+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:48:52.673654+0000","last_undegraded":"2024-09-05T16:48:52.673654+0000","last_fullsized":"2024-09-05T16:48:52.673654+0000","mapping_epoch":36,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4408,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13124","reported_seq":13273,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:48:52.673627+0000","last_change":"2024-09-05T16:48:16.582142+0000","last_active":"2024-09-05T16:48:52.673627+0000","last_peered":"2024-09-05T16:48:52.673627+0000","last_clean":"2024-09-05T16:48:52.673627+0000","last_became_active":"2024-09-05T16:48:16.435091+0000","last_became_peered":"2024-09-05T16:48:16.435091+0000","last_unstale":"2024-09-05T16:48:52.673627+0000","last_undegraded":"2024-09-05T16:48:52.673627+0000","last_fullsized":"2024-09-05T16:48:52.673627+0000","mapping_epoch":58,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'13124","last_scrub_stamp":"2024-09-05T16:48:16.582038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:48:16.582038+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T17:44:25.891565+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"39'12831","reported_seq":12996,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:48:52.673625+0000","last_change":"2024-09-05T16:47:40.451380+0000","last_active":"2024-09-05T16:48:52.673625+0000","last_peered":"2024-09-05T16:48:52.673625+0000","last_clean":"2024-09-05T16:48:52.673625+0000","last_became_active":"2024-09-05T16:47:40.189447+0000","last_became_peered":"2024-09-05T16:47:40.189447+0000","last_unstale":"2024-09-05T16:48:52.673625+0000","last_undegraded":"2024-09-05T16:48:52.673625+0000","last_fullsized":"2024-09-05T16:48:52.673625+0000","mapping_epoch":52,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12831","last_scrub_stamp":"2024-09-05T16:47:40.451295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:40.451295+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:46:20.946333+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12773","reported_seq":12938,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:48:52.673802+0000","last_change":"2024-09-05T16:47:27.437275+0000","last_active":"2024-09-05T16:48:52.673802+0000","last_peered":"2024-09-05T16:48:52.673802+0000","last_clean":"2024-09-05T16:48:52.673802+0000","last_became_active":"2024-09-05T16:47:27.204989+0000","last_became_peered":"2024-09-05T16:47:27.204989+0000","last_unstale":"2024-09-05T16:48:52.673802+0000","last_undegraded":"2024-09-05T16:48:52.673802+0000","last_fullsized":"2024-09-05T16:48:52.673802+0000","mapping_epoch":50,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12773","last_scrub_stamp":"2024-09-05T16:47:27.437187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:27.437187+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T16:53:55.961513+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":51940,"num_write_kb":159895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4539,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084466,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":726368,"kb_used_data":524,"kb_used_omap":4,"kb_used_meta":725819,"kb_avail":93645472,"statfs":{"total":96636764160,"available":95892963328,"internally_reserved":0,"allocated":536576,"data_stored":203415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":743239379},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986343,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":631968,"kb_used_data":524,"kb_used_omap":4,"kb_used_meta":631419,"kb_avail":93739872,"statfs":{"total":96636764160,"available":95989628928,"internally_reserved":0,"allocated":536576,"data_stored":203415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":646573778},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673150,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713388,"kb_used_data":532,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":93658452,"statfs":{"total":96636764160,"available":95906254848,"internally_reserved":0,"allocated":544768,"data_stored":206462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":729938373},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182581,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713836,"kb_used_data":532,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658004,"statfs":{"total":96636764160,"available":95905796096,"internally_reserved":0,"allocated":544768,"data_stored":206462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":730394317},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:48:57.965 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:48:58.246 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:48:58.247 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:48:58.259 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":526,"stamp":"2024-09-05T16:48:57.122967+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":51940,"num_write_kb":159895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4539,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2785560,"kb_used_data":2112,"kb_used_omap":14,"kb_used_meta":2783345,"kb_avail":374701800,"statfs":{"total":386547056640,"available":383694643200,"internally_reserved":0,"allocated":2162688,"data_stored":819754,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2850145847},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-682,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1557,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001633"},"pg_stats":[{"pgid":"1.f","version":"39'12904","reported_seq":17549,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:48:52.672782+0000","last_change":"2024-09-05T16:46:27.826222+0000","last_active":"2024-09-05T16:48:52.672782+0000","last_peered":"2024-09-05T16:48:52.672782+0000","last_clean":"2024-09-05T16:48:52.672782+0000","last_became_active":"2024-09-05T16:46:27.757928+0000","last_became_peered":"2024-09-05T16:46:27.757928+0000","last_unstale":"2024-09-05T16:48:52.672782+0000","last_undegraded":"2024-09-05T16:48:52.672782+0000","last_fullsized":"2024-09-05T16:48:52.672782+0000","mapping_epoch":40,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:32:35.713428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12904,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2262,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12552","reported_seq":17262,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:48:52.673735+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:48:52.673735+0000","last_peered":"2024-09-05T16:48:52.673735+0000","last_clean":"2024-09-05T16:48:52.673735+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:48:52.673735+0000","last_undegraded":"2024-09-05T16:48:52.673735+0000","last_fullsized":"2024-09-05T16:48:52.673735+0000","mapping_epoch":38,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1865,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12752","reported_seq":17417,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:48:52.675313+0000","last_change":"2024-09-05T16:46:51.992073+0000","last_active":"2024-09-05T16:48:52.675313+0000","last_peered":"2024-09-05T16:48:52.675313+0000","last_clean":"2024-09-05T16:48:52.675313+0000","last_became_active":"2024-09-05T16:46:51.667128+0000","last_became_peered":"2024-09-05T16:46:51.667128+0000","last_unstale":"2024-09-05T16:48:52.675313+0000","last_undegraded":"2024-09-05T16:48:52.675313+0000","last_fullsized":"2024-09-05T16:48:52.675313+0000","mapping_epoch":44,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12752","last_scrub_stamp":"2024-09-05T16:46:51.991995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:51.991995+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:04:54.120031+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12934","reported_seq":17722,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:48:52.673535+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:48:52.673535+0000","last_peered":"2024-09-05T16:48:52.673535+0000","last_clean":"2024-09-05T16:48:52.673535+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:48:52.673535+0000","last_undegraded":"2024-09-05T16:48:52.673535+0000","last_fullsized":"2024-09-05T16:48:52.673535+0000","mapping_epoch":34,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7234,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12593","reported_seq":17063,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:48:52.672736+0000","last_change":"2024-09-05T16:47:51.525026+0000","last_active":"2024-09-05T16:48:52.672736+0000","last_peered":"2024-09-05T16:48:52.672736+0000","last_clean":"2024-09-05T16:48:52.672736+0000","last_became_active":"2024-09-05T16:47:51.193016+0000","last_became_peered":"2024-09-05T16:47:51.193016+0000","last_unstale":"2024-09-05T16:48:52.672736+0000","last_undegraded":"2024-09-05T16:48:52.672736+0000","last_fullsized":"2024-09-05T16:48:52.672736+0000","mapping_epoch":54,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12593","last_scrub_stamp":"2024-09-05T16:47:51.524957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:51.524957+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:46:22.846859+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12772","reported_seq":17393,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:48:52.673036+0000","last_change":"2024-09-05T16:47:15.832530+0000","last_active":"2024-09-05T16:48:52.673036+0000","last_peered":"2024-09-05T16:48:52.673036+0000","last_clean":"2024-09-05T16:48:52.673036+0000","last_became_active":"2024-09-05T16:47:15.728759+0000","last_became_peered":"2024-09-05T16:47:15.728759+0000","last_unstale":"2024-09-05T16:48:52.673036+0000","last_undegraded":"2024-09-05T16:48:52.673036+0000","last_fullsized":"2024-09-05T16:48:52.673036+0000","mapping_epoch":48,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T02:39:11.515339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12772,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2250,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12660","reported_seq":17111,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:48:52.671899+0000","last_change":"2024-09-05T16:47:03.683348+0000","last_active":"2024-09-05T16:48:52.671899+0000","last_peered":"2024-09-05T16:48:52.671899+0000","last_clean":"2024-09-05T16:48:52.671899+0000","last_became_active":"2024-09-05T16:47:03.185143+0000","last_became_peered":"2024-09-05T16:47:03.185143+0000","last_unstale":"2024-09-05T16:48:52.671899+0000","last_undegraded":"2024-09-05T16:48:52.671899+0000","last_fullsized":"2024-09-05T16:48:52.671899+0000","mapping_epoch":46,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12660","last_scrub_stamp":"2024-09-05T16:47:03.683262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:03.683262+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:42:16.533299+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12777","reported_seq":12939,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:48:52.673772+0000","last_change":"2024-09-05T16:46:40.458315+0000","last_active":"2024-09-05T16:48:52.673772+0000","last_peered":"2024-09-05T16:48:52.673772+0000","last_clean":"2024-09-05T16:48:52.673772+0000","last_became_active":"2024-09-05T16:46:40.184288+0000","last_became_peered":"2024-09-05T16:46:40.184288+0000","last_unstale":"2024-09-05T16:48:52.673772+0000","last_undegraded":"2024-09-05T16:48:52.673772+0000","last_fullsized":"2024-09-05T16:48:52.673772+0000","mapping_epoch":42,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12777","last_scrub_stamp":"2024-09-05T16:46:40.458226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:40.458226+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:38:58.734210+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'12770","reported_seq":17441,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:48:52.672936+0000","last_change":"2024-09-05T16:48:41.629203+0000","last_active":"2024-09-05T16:48:52.672936+0000","last_peered":"2024-09-05T16:48:52.672936+0000","last_clean":"2024-09-05T16:48:52.672936+0000","last_became_active":"2024-09-05T16:48:41.236479+0000","last_became_peered":"2024-09-05T16:48:41.236479+0000","last_unstale":"2024-09-05T16:48:52.672936+0000","last_undegraded":"2024-09-05T16:48:52.672936+0000","last_fullsized":"2024-09-05T16:48:52.672936+0000","mapping_epoch":62,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12770","last_scrub_stamp":"2024-09-05T16:48:41.629128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:48:41.629128+0000","objects_scrubbed":1,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T02:22:35.076855+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12808","reported_seq":17501,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:48:52.675410+0000","last_change":"2024-09-05T16:48:04.288466+0000","last_active":"2024-09-05T16:48:52.675410+0000","last_peered":"2024-09-05T16:48:52.675410+0000","last_clean":"2024-09-05T16:48:52.675410+0000","last_became_active":"2024-09-05T16:48:03.239894+0000","last_became_peered":"2024-09-05T16:48:03.239894+0000","last_unstale":"2024-09-05T16:48:52.675410+0000","last_undegraded":"2024-09-05T16:48:52.675410+0000","last_fullsized":"2024-09-05T16:48:52.675410+0000","mapping_epoch":56,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12808","last_scrub_stamp":"2024-09-05T16:48:04.288388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:48:04.288388+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T21:11:32.044919+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13120","reported_seq":17891,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:48:52.672814+0000","last_change":"2024-09-05T16:48:29.529847+0000","last_active":"2024-09-05T16:48:52.672814+0000","last_peered":"2024-09-05T16:48:52.672814+0000","last_clean":"2024-09-05T16:48:52.672814+0000","last_became_active":"2024-09-05T16:48:29.235000+0000","last_became_peered":"2024-09-05T16:48:29.235000+0000","last_unstale":"2024-09-05T16:48:52.672814+0000","last_undegraded":"2024-09-05T16:48:52.672814+0000","last_fullsized":"2024-09-05T16:48:52.672814+0000","mapping_epoch":60,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'13120","last_scrub_stamp":"2024-09-05T16:48:29.529768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:48:29.529768+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T02:18:10.992722+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12758","reported_seq":12876,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:48:52.754550+0000","last_change":"2024-09-05T16:48:52.754442+0000","last_active":"2024-09-05T16:48:52.754550+0000","last_peered":"2024-09-05T16:48:52.754550+0000","last_clean":"2024-09-05T16:48:52.754550+0000","last_became_active":"2024-09-05T16:48:52.676533+0000","last_became_peered":"2024-09-05T16:48:52.676533+0000","last_unstale":"2024-09-05T16:48:52.754550+0000","last_undegraded":"2024-09-05T16:48:52.754550+0000","last_fullsized":"2024-09-05T16:48:52.754550+0000","mapping_epoch":64,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"39'12808","reported_seq":17602,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:48:52.673654+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:48:52.673654+0000","last_peered":"2024-09-05T16:48:52.673654+0000","last_clean":"2024-09-05T16:48:52.673654+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:48:52.673654+0000","last_undegraded":"2024-09-05T16:48:52.673654+0000","last_fullsized":"2024-09-05T16:48:52.673654+0000","mapping_epoch":36,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4408,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13124","reported_seq":13273,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:48:52.673627+0000","last_change":"2024-09-05T16:48:16.582142+0000","last_active":"2024-09-05T16:48:52.673627+0000","last_peered":"2024-09-05T16:48:52.673627+0000","last_clean":"2024-09-05T16:48:52.673627+0000","last_became_active":"2024-09-05T16:48:16.435091+0000","last_became_peered":"2024-09-05T16:48:16.435091+0000","last_unstale":"2024-09-05T16:48:52.673627+0000","last_undegraded":"2024-09-05T16:48:52.673627+0000","last_fullsized":"2024-09-05T16:48:52.673627+0000","mapping_epoch":58,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'13124","last_scrub_stamp":"2024-09-05T16:48:16.582038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:48:16.582038+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T17:44:25.891565+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"39'12831","reported_seq":12996,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:48:52.673625+0000","last_change":"2024-09-05T16:47:40.451380+0000","last_active":"2024-09-05T16:48:52.673625+0000","last_peered":"2024-09-05T16:48:52.673625+0000","last_clean":"2024-09-05T16:48:52.673625+0000","last_became_active":"2024-09-05T16:47:40.189447+0000","last_became_peered":"2024-09-05T16:47:40.189447+0000","last_unstale":"2024-09-05T16:48:52.673625+0000","last_undegraded":"2024-09-05T16:48:52.673625+0000","last_fullsized":"2024-09-05T16:48:52.673625+0000","mapping_epoch":52,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12831","last_scrub_stamp":"2024-09-05T16:47:40.451295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:40.451295+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:46:20.946333+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12773","reported_seq":12938,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:48:52.673802+0000","last_change":"2024-09-05T16:47:27.437275+0000","last_active":"2024-09-05T16:48:52.673802+0000","last_peered":"2024-09-05T16:48:52.673802+0000","last_clean":"2024-09-05T16:48:52.673802+0000","last_became_active":"2024-09-05T16:47:27.204989+0000","last_became_peered":"2024-09-05T16:47:27.204989+0000","last_unstale":"2024-09-05T16:48:52.673802+0000","last_undegraded":"2024-09-05T16:48:52.673802+0000","last_fullsized":"2024-09-05T16:48:52.673802+0000","mapping_epoch":50,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12773","last_scrub_stamp":"2024-09-05T16:47:27.437187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:27.437187+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T16:53:55.961513+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":51940,"num_write_kb":159895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4539,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084466,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":726368,"kb_used_data":524,"kb_used_omap":4,"kb_used_meta":725819,"kb_avail":93645472,"statfs":{"total":96636764160,"available":95892963328,"internally_reserved":0,"allocated":536576,"data_stored":203415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":743239379},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986343,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":631968,"kb_used_data":524,"kb_used_omap":4,"kb_used_meta":631419,"kb_avail":93739872,"statfs":{"total":96636764160,"available":95989628928,"internally_reserved":0,"allocated":536576,"data_stored":203415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":646573778},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673150,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713388,"kb_used_data":532,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":93658452,"statfs":{"total":96636764160,"available":95906254848,"internally_reserved":0,"allocated":544768,"data_stored":206462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":729938373},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182581,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713836,"kb_used_data":532,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658004,"statfs":{"total":96636764160,"available":95905796096,"internally_reserved":0,"allocated":544768,"data_stored":206462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":730394317},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:48:58.260 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-05T16:48:58.260 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:48:58.540 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:48:58.541 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:48:58.552 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":526,"stamp":"2024-09-05T16:48:57.122967+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":51940,"num_write_kb":159895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4539,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2785560,"kb_used_data":2112,"kb_used_omap":14,"kb_used_meta":2783345,"kb_avail":374701800,"statfs":{"total":386547056640,"available":383694643200,"internally_reserved":0,"allocated":2162688,"data_stored":819754,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2850145847},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-682,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1557,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001633"},"pg_stats":[{"pgid":"1.f","version":"39'12904","reported_seq":17549,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:48:52.672782+0000","last_change":"2024-09-05T16:46:27.826222+0000","last_active":"2024-09-05T16:48:52.672782+0000","last_peered":"2024-09-05T16:48:52.672782+0000","last_clean":"2024-09-05T16:48:52.672782+0000","last_became_active":"2024-09-05T16:46:27.757928+0000","last_became_peered":"2024-09-05T16:46:27.757928+0000","last_unstale":"2024-09-05T16:48:52.672782+0000","last_undegraded":"2024-09-05T16:48:52.672782+0000","last_fullsized":"2024-09-05T16:48:52.672782+0000","mapping_epoch":40,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:32:35.713428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12904,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2262,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12552","reported_seq":17262,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:48:52.673735+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:48:52.673735+0000","last_peered":"2024-09-05T16:48:52.673735+0000","last_clean":"2024-09-05T16:48:52.673735+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:48:52.673735+0000","last_undegraded":"2024-09-05T16:48:52.673735+0000","last_fullsized":"2024-09-05T16:48:52.673735+0000","mapping_epoch":38,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1865,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12752","reported_seq":17417,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:48:52.675313+0000","last_change":"2024-09-05T16:46:51.992073+0000","last_active":"2024-09-05T16:48:52.675313+0000","last_peered":"2024-09-05T16:48:52.675313+0000","last_clean":"2024-09-05T16:48:52.675313+0000","last_became_active":"2024-09-05T16:46:51.667128+0000","last_became_peered":"2024-09-05T16:46:51.667128+0000","last_unstale":"2024-09-05T16:48:52.675313+0000","last_undegraded":"2024-09-05T16:48:52.675313+0000","last_fullsized":"2024-09-05T16:48:52.675313+0000","mapping_epoch":44,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12752","last_scrub_stamp":"2024-09-05T16:46:51.991995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:51.991995+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:04:54.120031+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12934","reported_seq":17722,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:48:52.673535+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:48:52.673535+0000","last_peered":"2024-09-05T16:48:52.673535+0000","last_clean":"2024-09-05T16:48:52.673535+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:48:52.673535+0000","last_undegraded":"2024-09-05T16:48:52.673535+0000","last_fullsized":"2024-09-05T16:48:52.673535+0000","mapping_epoch":34,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7234,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12593","reported_seq":17063,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:48:52.672736+0000","last_change":"2024-09-05T16:47:51.525026+0000","last_active":"2024-09-05T16:48:52.672736+0000","last_peered":"2024-09-05T16:48:52.672736+0000","last_clean":"2024-09-05T16:48:52.672736+0000","last_became_active":"2024-09-05T16:47:51.193016+0000","last_became_peered":"2024-09-05T16:47:51.193016+0000","last_unstale":"2024-09-05T16:48:52.672736+0000","last_undegraded":"2024-09-05T16:48:52.672736+0000","last_fullsized":"2024-09-05T16:48:52.672736+0000","mapping_epoch":54,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12593","last_scrub_stamp":"2024-09-05T16:47:51.524957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:51.524957+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:46:22.846859+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12772","reported_seq":17393,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:48:52.673036+0000","last_change":"2024-09-05T16:47:15.832530+0000","last_active":"2024-09-05T16:48:52.673036+0000","last_peered":"2024-09-05T16:48:52.673036+0000","last_clean":"2024-09-05T16:48:52.673036+0000","last_became_active":"2024-09-05T16:47:15.728759+0000","last_became_peered":"2024-09-05T16:47:15.728759+0000","last_unstale":"2024-09-05T16:48:52.673036+0000","last_undegraded":"2024-09-05T16:48:52.673036+0000","last_fullsized":"2024-09-05T16:48:52.673036+0000","mapping_epoch":48,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T02:39:11.515339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12772,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2250,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12660","reported_seq":17111,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:48:52.671899+0000","last_change":"2024-09-05T16:47:03.683348+0000","last_active":"2024-09-05T16:48:52.671899+0000","last_peered":"2024-09-05T16:48:52.671899+0000","last_clean":"2024-09-05T16:48:52.671899+0000","last_became_active":"2024-09-05T16:47:03.185143+0000","last_became_peered":"2024-09-05T16:47:03.185143+0000","last_unstale":"2024-09-05T16:48:52.671899+0000","last_undegraded":"2024-09-05T16:48:52.671899+0000","last_fullsized":"2024-09-05T16:48:52.671899+0000","mapping_epoch":46,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12660","last_scrub_stamp":"2024-09-05T16:47:03.683262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:03.683262+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:42:16.533299+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12777","reported_seq":12939,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:48:52.673772+0000","last_change":"2024-09-05T16:46:40.458315+0000","last_active":"2024-09-05T16:48:52.673772+0000","last_peered":"2024-09-05T16:48:52.673772+0000","last_clean":"2024-09-05T16:48:52.673772+0000","last_became_active":"2024-09-05T16:46:40.184288+0000","last_became_peered":"2024-09-05T16:46:40.184288+0000","last_unstale":"2024-09-05T16:48:52.673772+0000","last_undegraded":"2024-09-05T16:48:52.673772+0000","last_fullsized":"2024-09-05T16:48:52.673772+0000","mapping_epoch":42,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12777","last_scrub_stamp":"2024-09-05T16:46:40.458226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:40.458226+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:38:58.734210+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'12770","reported_seq":17441,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:48:52.672936+0000","last_change":"2024-09-05T16:48:41.629203+0000","last_active":"2024-09-05T16:48:52.672936+0000","last_peered":"2024-09-05T16:48:52.672936+0000","last_clean":"2024-09-05T16:48:52.672936+0000","last_became_active":"2024-09-05T16:48:41.236479+0000","last_became_peered":"2024-09-05T16:48:41.236479+0000","last_unstale":"2024-09-05T16:48:52.672936+0000","last_undegraded":"2024-09-05T16:48:52.672936+0000","last_fullsized":"2024-09-05T16:48:52.672936+0000","mapping_epoch":62,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12770","last_scrub_stamp":"2024-09-05T16:48:41.629128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:48:41.629128+0000","objects_scrubbed":1,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T02:22:35.076855+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12808","reported_seq":17501,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:48:52.675410+0000","last_change":"2024-09-05T16:48:04.288466+0000","last_active":"2024-09-05T16:48:52.675410+0000","last_peered":"2024-09-05T16:48:52.675410+0000","last_clean":"2024-09-05T16:48:52.675410+0000","last_became_active":"2024-09-05T16:48:03.239894+0000","last_became_peered":"2024-09-05T16:48:03.239894+0000","last_unstale":"2024-09-05T16:48:52.675410+0000","last_undegraded":"2024-09-05T16:48:52.675410+0000","last_fullsized":"2024-09-05T16:48:52.675410+0000","mapping_epoch":56,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12808","last_scrub_stamp":"2024-09-05T16:48:04.288388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:48:04.288388+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T21:11:32.044919+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13120","reported_seq":17891,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:48:52.672814+0000","last_change":"2024-09-05T16:48:29.529847+0000","last_active":"2024-09-05T16:48:52.672814+0000","last_peered":"2024-09-05T16:48:52.672814+0000","last_clean":"2024-09-05T16:48:52.672814+0000","last_became_active":"2024-09-05T16:48:29.235000+0000","last_became_peered":"2024-09-05T16:48:29.235000+0000","last_unstale":"2024-09-05T16:48:52.672814+0000","last_undegraded":"2024-09-05T16:48:52.672814+0000","last_fullsized":"2024-09-05T16:48:52.672814+0000","mapping_epoch":60,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'13120","last_scrub_stamp":"2024-09-05T16:48:29.529768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:48:29.529768+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T02:18:10.992722+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12758","reported_seq":12876,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:48:52.754550+0000","last_change":"2024-09-05T16:48:52.754442+0000","last_active":"2024-09-05T16:48:52.754550+0000","last_peered":"2024-09-05T16:48:52.754550+0000","last_clean":"2024-09-05T16:48:52.754550+0000","last_became_active":"2024-09-05T16:48:52.676533+0000","last_became_peered":"2024-09-05T16:48:52.676533+0000","last_unstale":"2024-09-05T16:48:52.754550+0000","last_undegraded":"2024-09-05T16:48:52.754550+0000","last_fullsized":"2024-09-05T16:48:52.754550+0000","mapping_epoch":64,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"39'12808","reported_seq":17602,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:48:52.673654+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:48:52.673654+0000","last_peered":"2024-09-05T16:48:52.673654+0000","last_clean":"2024-09-05T16:48:52.673654+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:48:52.673654+0000","last_undegraded":"2024-09-05T16:48:52.673654+0000","last_fullsized":"2024-09-05T16:48:52.673654+0000","mapping_epoch":36,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4408,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13124","reported_seq":13273,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:48:52.673627+0000","last_change":"2024-09-05T16:48:16.582142+0000","last_active":"2024-09-05T16:48:52.673627+0000","last_peered":"2024-09-05T16:48:52.673627+0000","last_clean":"2024-09-05T16:48:52.673627+0000","last_became_active":"2024-09-05T16:48:16.435091+0000","last_became_peered":"2024-09-05T16:48:16.435091+0000","last_unstale":"2024-09-05T16:48:52.673627+0000","last_undegraded":"2024-09-05T16:48:52.673627+0000","last_fullsized":"2024-09-05T16:48:52.673627+0000","mapping_epoch":58,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'13124","last_scrub_stamp":"2024-09-05T16:48:16.582038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:48:16.582038+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T17:44:25.891565+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"39'12831","reported_seq":12996,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:48:52.673625+0000","last_change":"2024-09-05T16:47:40.451380+0000","last_active":"2024-09-05T16:48:52.673625+0000","last_peered":"2024-09-05T16:48:52.673625+0000","last_clean":"2024-09-05T16:48:52.673625+0000","last_became_active":"2024-09-05T16:47:40.189447+0000","last_became_peered":"2024-09-05T16:47:40.189447+0000","last_unstale":"2024-09-05T16:48:52.673625+0000","last_undegraded":"2024-09-05T16:48:52.673625+0000","last_fullsized":"2024-09-05T16:48:52.673625+0000","mapping_epoch":52,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12831","last_scrub_stamp":"2024-09-05T16:47:40.451295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:40.451295+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:46:20.946333+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12773","reported_seq":12938,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:48:52.673802+0000","last_change":"2024-09-05T16:47:27.437275+0000","last_active":"2024-09-05T16:48:52.673802+0000","last_peered":"2024-09-05T16:48:52.673802+0000","last_clean":"2024-09-05T16:48:52.673802+0000","last_became_active":"2024-09-05T16:47:27.204989+0000","last_became_peered":"2024-09-05T16:47:27.204989+0000","last_unstale":"2024-09-05T16:48:52.673802+0000","last_undegraded":"2024-09-05T16:48:52.673802+0000","last_fullsized":"2024-09-05T16:48:52.673802+0000","mapping_epoch":50,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12773","last_scrub_stamp":"2024-09-05T16:47:27.437187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:27.437187+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T16:53:55.961513+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":51940,"num_write_kb":159895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4539,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084466,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":726368,"kb_used_data":524,"kb_used_omap":4,"kb_used_meta":725819,"kb_avail":93645472,"statfs":{"total":96636764160,"available":95892963328,"internally_reserved":0,"allocated":536576,"data_stored":203415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":743239379},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986343,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":631968,"kb_used_data":524,"kb_used_omap":4,"kb_used_meta":631419,"kb_avail":93739872,"statfs":{"total":96636764160,"available":95989628928,"internally_reserved":0,"allocated":536576,"data_stored":203415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":646573778},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673150,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713388,"kb_used_data":532,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":93658452,"statfs":{"total":96636764160,"available":95906254848,"internally_reserved":0,"allocated":544768,"data_stored":206462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":729938373},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182581,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713836,"kb_used_data":532,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658004,"statfs":{"total":96636764160,"available":95905796096,"internally_reserved":0,"allocated":544768,"data_stored":206462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":730394317},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:48:58.553 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-05T16:48:58.834 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:48:58.834 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":65,"fsid":"f1aad5cb-b9b1-40bd-87b3-d80e7f482bdc","created":"2024-09-05T16:33:15.770967+0000","modified":"2024-09-05T16:48:52.668277+0000","last_up_change":"2024-09-05T16:34:54.893123+0000","last_in_change":"2024-09-05T16:33:21.020289+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":9,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"unique_pool_0","create_time":"2024-09-05T16:33:40.865276+0000","flags":1,"flags_names":"hashpspool","type":3,"size":4,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"lost_unfound","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"f76bac77-d755-450f-9fbf-fb53a6033c24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":18,"up_from":25,"up_thru":49,"down_at":19,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6804","nonce":67704509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6805","nonce":67704509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6807","nonce":67704509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6806","nonce":67704509}]},"public_addr":"172.21.15.87:6804/67704509","cluster_addr":"172.21.15.87:6805/67704509","heartbeat_back_addr":"172.21.15.87:6807/67704509","heartbeat_front_addr":"172.21.15.87:6806/67704509","state":["exists","up"]},{"osd":1,"uuid":"941684ea-0764-4182-94cc-136349df4f23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":11,"last_clean_end":21,"up_from":31,"up_thru":64,"down_at":22,"lost_at":22,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6812","nonce":3937864697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6813","nonce":3937864697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6815","nonce":3937864697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6814","nonce":3937864697}]},"public_addr":"172.21.15.87:6812/3937864697","cluster_addr":"172.21.15.87:6813/3937864697","heartbeat_back_addr":"172.21.15.87:6815/3937864697","heartbeat_front_addr":"172.21.15.87:6814/3937864697","state":["exists","up"]},{"osd":2,"uuid":"17135934-ecb7-445a-b1dc-27f55da9f8ca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6800","nonce":4234108552}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6801","nonce":4234108552}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6803","nonce":4234108552}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6802","nonce":4234108552}]},"public_addr":"172.21.15.87:6800/4234108552","cluster_addr":"172.21.15.87:6801/4234108552","heartbeat_back_addr":"172.21.15.87:6803/4234108552","heartbeat_front_addr":"172.21.15.87:6802/4234108552","state":["exists","up"]},{"osd":3,"uuid":"098f9cbd-3da5-4ee7-8439-610b8cae774e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":11,"last_clean_end":19,"up_from":28,"up_thru":63,"down_at":20,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6808","nonce":1182532508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6809","nonce":1182532508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6811","nonce":1182532508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6810","nonce":1182532508}]},"public_addr":"172.21.15.87:6808/1182532508","cluster_addr":"172.21.15.87:6809/1182532508","heartbeat_back_addr":"172.21.15.87:6811/1182532508","heartbeat_front_addr":"172.21.15.87:6810/1182532508","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-05T16:33:27.513154+0000","dead_epoch":19},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-05T16:33:27.800689+0000","dead_epoch":22},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-05T16:33:28.127983+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-05T16:33:28.046083+0000","dead_epoch":20}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"lost_unfound":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-05T16:48:59.848 INFO:tasks.ceph:Scrubbing osd.0 2024-09-05T16:48:59.848 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-05T16:48:59.982 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-09-05T16:48:59.982 INFO:teuthology.orchestra.run.smithi087.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-05T16:48:59.982 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-09-05T16:48:59.993 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-09-05T16:49:00.277 INFO:teuthology.orchestra.run.smithi087.stderr:instructed osd(s) 0 to deep-scrub 2024-09-05T16:49:00.291 INFO:tasks.ceph:Scrubbing osd.1 2024-09-05T16:49:00.291 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-05T16:49:00.412 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-09-05T16:49:00.412 INFO:teuthology.orchestra.run.smithi087.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-05T16:49:00.412 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-09-05T16:49:00.422 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-09-05T16:49:00.708 INFO:teuthology.orchestra.run.smithi087.stderr:instructed osd(s) 1 to deep-scrub 2024-09-05T16:49:00.721 INFO:tasks.ceph:Scrubbing osd.2 2024-09-05T16:49:00.721 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-05T16:49:00.842 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-09-05T16:49:00.843 INFO:teuthology.orchestra.run.smithi087.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-05T16:49:00.843 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-09-05T16:49:00.853 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-09-05T16:49:01.136 INFO:teuthology.orchestra.run.smithi087.stderr:instructed osd(s) 2 to deep-scrub 2024-09-05T16:49:01.150 INFO:tasks.ceph:Scrubbing osd.3 2024-09-05T16:49:01.150 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-09-05T16:49:01.271 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-09-05T16:49:01.271 INFO:teuthology.orchestra.run.smithi087.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-05T16:49:01.271 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-09-05T16:49:01.281 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-09-05T16:49:01.567 INFO:teuthology.orchestra.run.smithi087.stderr:instructed osd(s) 3 to deep-scrub 2024-09-05T16:49:01.580 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:49:01.861 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:49:01.861 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:49:01.873 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":528,"stamp":"2024-09-05T16:49:01.123563+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":39183,"num_write_kb":108864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4512,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2785576,"kb_used_data":2128,"kb_used_omap":14,"kb_used_meta":2783345,"kb_avail":374701784,"statfs":{"total":386547056640,"available":383694626816,"internally_reserved":0,"allocated":2179072,"data_stored":825848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2850145847},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-6736,"num_objects":-1,"num_object_clones":0,"num_object_copies":-4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-682,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12757,"num_write_kb":-51031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1584,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.452517"},"pg_stats":[{"pgid":"1.f","version":"39'12904","reported_seq":17549,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:48:52.672782+0000","last_change":"2024-09-05T16:46:27.826222+0000","last_active":"2024-09-05T16:48:52.672782+0000","last_peered":"2024-09-05T16:48:52.672782+0000","last_clean":"2024-09-05T16:48:52.672782+0000","last_became_active":"2024-09-05T16:46:27.757928+0000","last_became_peered":"2024-09-05T16:46:27.757928+0000","last_unstale":"2024-09-05T16:48:52.672782+0000","last_undegraded":"2024-09-05T16:48:52.672782+0000","last_fullsized":"2024-09-05T16:48:52.672782+0000","mapping_epoch":40,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:32:35.713428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12904,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2262,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12552","reported_seq":17262,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:48:52.673735+0000","last_change":"2024-09-05T16:43:49.998224+0000","last_active":"2024-09-05T16:48:52.673735+0000","last_peered":"2024-09-05T16:48:52.673735+0000","last_clean":"2024-09-05T16:48:52.673735+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:48:52.673735+0000","last_undegraded":"2024-09-05T16:48:52.673735+0000","last_fullsized":"2024-09-05T16:48:52.673735+0000","mapping_epoch":38,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'10715","last_scrub_stamp":"2024-09-05T16:43:49.997880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:43:49.997880+0000","objects_scrubbed":1865,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:33:57.271054+0000","scrub_duration":4156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1865,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12752","reported_seq":17417,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:48:52.675313+0000","last_change":"2024-09-05T16:46:51.992073+0000","last_active":"2024-09-05T16:48:52.675313+0000","last_peered":"2024-09-05T16:48:52.675313+0000","last_clean":"2024-09-05T16:48:52.675313+0000","last_became_active":"2024-09-05T16:46:51.667128+0000","last_became_peered":"2024-09-05T16:46:51.667128+0000","last_unstale":"2024-09-05T16:48:52.675313+0000","last_undegraded":"2024-09-05T16:48:52.675313+0000","last_fullsized":"2024-09-05T16:48:52.675313+0000","mapping_epoch":44,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12752","last_scrub_stamp":"2024-09-05T16:46:51.991995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:51.991995+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:04:54.120031+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12934","reported_seq":17722,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:48:52.673535+0000","last_change":"2024-09-05T16:37:39.395263+0000","last_active":"2024-09-05T16:48:52.673535+0000","last_peered":"2024-09-05T16:48:52.673535+0000","last_clean":"2024-09-05T16:48:52.673535+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:48:52.673535+0000","last_undegraded":"2024-09-05T16:48:52.673535+0000","last_fullsized":"2024-09-05T16:48:52.673535+0000","mapping_epoch":34,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5761","last_scrub_stamp":"2024-09-05T16:37:39.391141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:37:39.391141+0000","objects_scrubbed":5694,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:25:39.218271+0000","scrub_duration":6487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7234,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12593","reported_seq":17063,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:48:52.672736+0000","last_change":"2024-09-05T16:47:51.525026+0000","last_active":"2024-09-05T16:48:52.672736+0000","last_peered":"2024-09-05T16:48:52.672736+0000","last_clean":"2024-09-05T16:48:52.672736+0000","last_became_active":"2024-09-05T16:47:51.193016+0000","last_became_peered":"2024-09-05T16:47:51.193016+0000","last_unstale":"2024-09-05T16:48:52.672736+0000","last_undegraded":"2024-09-05T16:48:52.672736+0000","last_fullsized":"2024-09-05T16:48:52.672736+0000","mapping_epoch":54,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12593","last_scrub_stamp":"2024-09-05T16:47:51.524957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:51.524957+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:46:22.846859+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12772","reported_seq":17393,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:48:52.673036+0000","last_change":"2024-09-05T16:47:15.832530+0000","last_active":"2024-09-05T16:48:52.673036+0000","last_peered":"2024-09-05T16:48:52.673036+0000","last_clean":"2024-09-05T16:48:52.673036+0000","last_became_active":"2024-09-05T16:47:15.728759+0000","last_became_peered":"2024-09-05T16:47:15.728759+0000","last_unstale":"2024-09-05T16:48:52.673036+0000","last_undegraded":"2024-09-05T16:48:52.673036+0000","last_fullsized":"2024-09-05T16:48:52.673036+0000","mapping_epoch":48,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:33:41.856813+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T02:39:11.515339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12772,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2250,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12660","reported_seq":17111,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:48:52.671899+0000","last_change":"2024-09-05T16:47:03.683348+0000","last_active":"2024-09-05T16:48:52.671899+0000","last_peered":"2024-09-05T16:48:52.671899+0000","last_clean":"2024-09-05T16:48:52.671899+0000","last_became_active":"2024-09-05T16:47:03.185143+0000","last_became_peered":"2024-09-05T16:47:03.185143+0000","last_unstale":"2024-09-05T16:48:52.671899+0000","last_undegraded":"2024-09-05T16:48:52.671899+0000","last_fullsized":"2024-09-05T16:48:52.671899+0000","mapping_epoch":46,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12660","last_scrub_stamp":"2024-09-05T16:47:03.683262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:03.683262+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:42:16.533299+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12777","reported_seq":12939,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:48:52.673772+0000","last_change":"2024-09-05T16:46:40.458315+0000","last_active":"2024-09-05T16:48:52.673772+0000","last_peered":"2024-09-05T16:48:52.673772+0000","last_clean":"2024-09-05T16:48:52.673772+0000","last_became_active":"2024-09-05T16:46:40.184288+0000","last_became_peered":"2024-09-05T16:46:40.184288+0000","last_unstale":"2024-09-05T16:48:52.673772+0000","last_undegraded":"2024-09-05T16:48:52.673772+0000","last_fullsized":"2024-09-05T16:48:52.673772+0000","mapping_epoch":42,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12777","last_scrub_stamp":"2024-09-05T16:46:40.458226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:46:40.458226+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:38:58.734210+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'12770","reported_seq":17442,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:48:52.672985+0000","last_change":"2024-09-05T16:48:41.629203+0000","last_active":"2024-09-05T16:48:52.672985+0000","last_peered":"2024-09-05T16:48:52.672985+0000","last_clean":"2024-09-05T16:48:52.672985+0000","last_became_active":"2024-09-05T16:48:41.236479+0000","last_became_peered":"2024-09-05T16:48:41.236479+0000","last_unstale":"2024-09-05T16:48:52.672985+0000","last_undegraded":"2024-09-05T16:48:52.672985+0000","last_fullsized":"2024-09-05T16:48:52.672985+0000","mapping_epoch":62,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12770","last_scrub_stamp":"2024-09-05T16:48:41.629128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:48:41.629128+0000","objects_scrubbed":1,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T02:22:35.076855+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12808","reported_seq":17501,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:48:52.675410+0000","last_change":"2024-09-05T16:48:04.288466+0000","last_active":"2024-09-05T16:48:52.675410+0000","last_peered":"2024-09-05T16:48:52.675410+0000","last_clean":"2024-09-05T16:48:52.675410+0000","last_became_active":"2024-09-05T16:48:03.239894+0000","last_became_peered":"2024-09-05T16:48:03.239894+0000","last_unstale":"2024-09-05T16:48:52.675410+0000","last_undegraded":"2024-09-05T16:48:52.675410+0000","last_fullsized":"2024-09-05T16:48:52.675410+0000","mapping_epoch":56,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12808","last_scrub_stamp":"2024-09-05T16:48:04.288388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:48:04.288388+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T21:11:32.044919+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13120","reported_seq":17891,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:48:52.672814+0000","last_change":"2024-09-05T16:48:29.529847+0000","last_active":"2024-09-05T16:48:52.672814+0000","last_peered":"2024-09-05T16:48:52.672814+0000","last_clean":"2024-09-05T16:48:52.672814+0000","last_became_active":"2024-09-05T16:48:29.235000+0000","last_became_peered":"2024-09-05T16:48:29.235000+0000","last_unstale":"2024-09-05T16:48:52.672814+0000","last_undegraded":"2024-09-05T16:48:52.672814+0000","last_fullsized":"2024-09-05T16:48:52.672814+0000","mapping_epoch":60,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'13120","last_scrub_stamp":"2024-09-05T16:48:29.529768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:48:29.529768+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T02:18:10.992722+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12758","reported_seq":12885,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:48:53.323297+0000","last_change":"2024-09-05T16:48:53.323297+0000","last_active":"2024-09-05T16:48:53.323297+0000","last_peered":"2024-09-05T16:48:53.323297+0000","last_clean":"2024-09-05T16:48:53.323297+0000","last_became_active":"2024-09-05T16:48:52.676533+0000","last_became_peered":"2024-09-05T16:48:52.676533+0000","last_unstale":"2024-09-05T16:48:53.323297+0000","last_undegraded":"2024-09-05T16:48:53.323297+0000","last_fullsized":"2024-09-05T16:48:53.323297+0000","mapping_epoch":64,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12758","last_scrub_stamp":"2024-09-05T16:48:53.323216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:48:53.323216+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T21:11:56.460145+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"39'12808","reported_seq":17602,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:48:52.673654+0000","last_change":"2024-09-05T16:40:35.704279+0000","last_active":"2024-09-05T16:48:52.673654+0000","last_peered":"2024-09-05T16:48:52.673654+0000","last_clean":"2024-09-05T16:48:52.673654+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:48:52.673654+0000","last_undegraded":"2024-09-05T16:48:52.673654+0000","last_fullsized":"2024-09-05T16:48:52.673654+0000","mapping_epoch":36,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'8436","last_scrub_stamp":"2024-09-05T16:40:35.703959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:40:35.703959+0000","objects_scrubbed":4408,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:17:05.035264+0000","scrub_duration":5504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4408,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13124","reported_seq":13273,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:48:52.673627+0000","last_change":"2024-09-05T16:48:16.582142+0000","last_active":"2024-09-05T16:48:52.673627+0000","last_peered":"2024-09-05T16:48:52.673627+0000","last_clean":"2024-09-05T16:48:52.673627+0000","last_became_active":"2024-09-05T16:48:16.435091+0000","last_became_peered":"2024-09-05T16:48:16.435091+0000","last_unstale":"2024-09-05T16:48:52.673627+0000","last_undegraded":"2024-09-05T16:48:52.673627+0000","last_fullsized":"2024-09-05T16:48:52.673627+0000","mapping_epoch":58,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'13124","last_scrub_stamp":"2024-09-05T16:48:16.582038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:48:16.582038+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T17:44:25.891565+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"39'12831","reported_seq":12996,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:48:52.673625+0000","last_change":"2024-09-05T16:47:40.451380+0000","last_active":"2024-09-05T16:48:52.673625+0000","last_peered":"2024-09-05T16:48:52.673625+0000","last_clean":"2024-09-05T16:48:52.673625+0000","last_became_active":"2024-09-05T16:47:40.189447+0000","last_became_peered":"2024-09-05T16:47:40.189447+0000","last_unstale":"2024-09-05T16:48:52.673625+0000","last_undegraded":"2024-09-05T16:48:52.673625+0000","last_fullsized":"2024-09-05T16:48:52.673625+0000","mapping_epoch":52,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12831","last_scrub_stamp":"2024-09-05T16:47:40.451295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:40.451295+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:46:20.946333+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12773","reported_seq":12938,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:48:52.673802+0000","last_change":"2024-09-05T16:47:27.437275+0000","last_active":"2024-09-05T16:48:52.673802+0000","last_peered":"2024-09-05T16:48:52.673802+0000","last_clean":"2024-09-05T16:48:52.673802+0000","last_became_active":"2024-09-05T16:47:27.204989+0000","last_became_peered":"2024-09-05T16:47:27.204989+0000","last_unstale":"2024-09-05T16:48:52.673802+0000","last_undegraded":"2024-09-05T16:48:52.673802+0000","last_fullsized":"2024-09-05T16:48:52.673802+0000","mapping_epoch":50,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12773","last_scrub_stamp":"2024-09-05T16:47:27.437187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-05T16:33:41.856813+0000","last_clean_scrub_stamp":"2024-09-05T16:47:27.437187+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T16:53:55.961513+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":39183,"num_write_kb":108864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4512,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084467,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":726380,"kb_used_data":532,"kb_used_omap":4,"kb_used_meta":725819,"kb_avail":93645460,"statfs":{"total":96636764160,"available":95892951040,"internally_reserved":0,"allocated":544768,"data_stored":206462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":743239379},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986344,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":631972,"kb_used_data":532,"kb_used_omap":4,"kb_used_meta":631419,"kb_avail":93739868,"statfs":{"total":96636764160,"available":95989624832,"internally_reserved":0,"allocated":544768,"data_stored":206462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":646573778},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673151,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713388,"kb_used_data":532,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":93658452,"statfs":{"total":96636764160,"available":95906254848,"internally_reserved":0,"allocated":544768,"data_stored":206462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":729938373},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182581,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713836,"kb_used_data":532,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658004,"statfs":{"total":96636764160,"available":95905796096,"internally_reserved":0,"allocated":544768,"data_stored":206462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":730394317},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:49:01.877 INFO:tasks.ceph:pgid 1.f last_scrub_stamp 2024-09-05T16:33:41.856813+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=5, tm_hour=16, tm_min=33, tm_sec=41, tm_wday=3, tm_yday=249, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=5, tm_hour=16, tm_min=48, tm_sec=58, tm_wday=3, tm_yday=249, tm_isdst=0) 2024-09-05T16:49:01.877 INFO:tasks.ceph:pgid 1.e last_scrub_stamp 2024-09-05T16:43:49.997880+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=5, tm_hour=16, tm_min=43, tm_sec=49, tm_wday=3, tm_yday=249, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=5, tm_hour=16, tm_min=48, tm_sec=58, tm_wday=3, tm_yday=249, tm_isdst=0) 2024-09-05T16:49:01.878 INFO:tasks.ceph:pgid 1.d last_scrub_stamp 2024-09-05T16:46:51.991995+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=5, tm_hour=16, tm_min=46, tm_sec=51, tm_wday=3, tm_yday=249, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=5, tm_hour=16, tm_min=48, tm_sec=58, tm_wday=3, tm_yday=249, tm_isdst=0) 2024-09-05T16:49:01.878 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-05T16:37:39.391141+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=5, tm_hour=16, tm_min=37, tm_sec=39, tm_wday=3, tm_yday=249, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=5, tm_hour=16, tm_min=48, tm_sec=58, tm_wday=3, tm_yday=249, tm_isdst=0) 2024-09-05T16:49:01.878 INFO:tasks.ceph:pgid 1.1 last_scrub_stamp 2024-09-05T16:47:51.524957+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=5, tm_hour=16, tm_min=47, tm_sec=51, tm_wday=3, tm_yday=249, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=5, tm_hour=16, tm_min=48, tm_sec=58, tm_wday=3, tm_yday=249, tm_isdst=0) 2024-09-05T16:49:01.878 INFO:tasks.ceph:pgid 1.c last_scrub_stamp 2024-09-05T16:33:41.856813+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=5, tm_hour=16, tm_min=33, tm_sec=41, tm_wday=3, tm_yday=249, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=5, tm_hour=16, tm_min=48, tm_sec=58, tm_wday=3, tm_yday=249, tm_isdst=0) 2024-09-05T16:49:01.878 INFO:tasks.ceph:pgid 1.2 last_scrub_stamp 2024-09-05T16:47:03.683262+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=5, tm_hour=16, tm_min=47, tm_sec=3, tm_wday=3, tm_yday=249, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=5, tm_hour=16, tm_min=48, tm_sec=58, tm_wday=3, tm_yday=249, tm_isdst=0) 2024-09-05T16:49:01.878 INFO:tasks.ceph:pgid 1.3 last_scrub_stamp 2024-09-05T16:46:40.458226+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=5, tm_hour=16, tm_min=46, tm_sec=40, tm_wday=3, tm_yday=249, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=5, tm_hour=16, tm_min=48, tm_sec=58, tm_wday=3, tm_yday=249, tm_isdst=0) 2024-09-05T16:49:01.879 INFO:tasks.ceph:pgid 1.4 last_scrub_stamp 2024-09-05T16:48:41.629128+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=5, tm_hour=16, tm_min=48, tm_sec=41, tm_wday=3, tm_yday=249, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=5, tm_hour=16, tm_min=48, tm_sec=58, tm_wday=3, tm_yday=249, tm_isdst=0) 2024-09-05T16:49:01.879 INFO:tasks.ceph:pgid 1.5 last_scrub_stamp 2024-09-05T16:48:04.288388+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=5, tm_hour=16, tm_min=48, tm_sec=4, tm_wday=3, tm_yday=249, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=5, tm_hour=16, tm_min=48, tm_sec=58, tm_wday=3, tm_yday=249, tm_isdst=0) 2024-09-05T16:49:01.879 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2024-09-05T16:48:29.529768+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=5, tm_hour=16, tm_min=48, tm_sec=29, tm_wday=3, tm_yday=249, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=5, tm_hour=16, tm_min=48, tm_sec=58, tm_wday=3, tm_yday=249, tm_isdst=0) 2024-09-05T16:49:01.879 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2024-09-05T16:48:53.323216+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=5, tm_hour=16, tm_min=48, tm_sec=53, tm_wday=3, tm_yday=249, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=5, tm_hour=16, tm_min=48, tm_sec=58, tm_wday=3, tm_yday=249, tm_isdst=0) 2024-09-05T16:49:01.879 INFO:tasks.ceph:pgid 1.8 last_scrub_stamp 2024-09-05T16:40:35.703959+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=5, tm_hour=16, tm_min=40, tm_sec=35, tm_wday=3, tm_yday=249, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=5, tm_hour=16, tm_min=48, tm_sec=58, tm_wday=3, tm_yday=249, tm_isdst=0) 2024-09-05T16:49:01.879 INFO:tasks.ceph:pgid 1.9 last_scrub_stamp 2024-09-05T16:48:16.582038+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=5, tm_hour=16, tm_min=48, tm_sec=16, tm_wday=3, tm_yday=249, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=5, tm_hour=16, tm_min=48, tm_sec=58, tm_wday=3, tm_yday=249, tm_isdst=0) 2024-09-05T16:49:01.879 INFO:tasks.ceph:pgid 1.a last_scrub_stamp 2024-09-05T16:47:40.451295+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=5, tm_hour=16, tm_min=47, tm_sec=40, tm_wday=3, tm_yday=249, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=5, tm_hour=16, tm_min=48, tm_sec=58, tm_wday=3, tm_yday=249, tm_isdst=0) 2024-09-05T16:49:01.880 INFO:tasks.ceph:pgid 1.b last_scrub_stamp 2024-09-05T16:47:27.437187+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=5, tm_hour=16, tm_min=47, tm_sec=27, tm_wday=3, tm_yday=249, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=5, tm_hour=16, tm_min=48, tm_sec=58, tm_wday=3, tm_yday=249, tm_isdst=0) 2024-09-05T16:49:01.880 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-05T16:49:21.881 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-05T16:49:22.175 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-05T16:49:22.176 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-09-05T16:49:22.188 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":538,"stamp":"2024-09-05T16:49:21.127775+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":39183,"num_write_kb":108864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4512,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2785552,"kb_used_data":2128,"kb_used_omap":14,"kb_used_meta":2783345,"kb_avail":374701808,"statfs":{"total":386547056640,"available":383694651392,"internally_reserved":0,"allocated":2179072,"data_stored":825848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14793,"internal_metadata":2850145847},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002654"},"pg_stats":[{"pgid":"1.f","version":"39'12904","reported_seq":17557,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:49:05.627178+0000","last_change":"2024-09-05T16:49:05.627178+0000","last_active":"2024-09-05T16:49:05.627178+0000","last_peered":"2024-09-05T16:49:05.627178+0000","last_clean":"2024-09-05T16:49:05.627178+0000","last_became_active":"2024-09-05T16:46:27.757928+0000","last_became_peered":"2024-09-05T16:46:27.757928+0000","last_unstale":"2024-09-05T16:49:05.627178+0000","last_undegraded":"2024-09-05T16:49:05.627178+0000","last_fullsized":"2024-09-05T16:49:05.627178+0000","mapping_epoch":40,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12904","last_scrub_stamp":"2024-09-05T16:49:05.627072+0000","last_deep_scrub":"39'12904","last_deep_scrub_stamp":"2024-09-05T16:49:05.627072+0000","last_clean_scrub_stamp":"2024-09-05T16:49:05.627072+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T23:15:21.404213+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12904,"num_write_kb":51616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2262,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'12552","reported_seq":17270,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:49:07.406518+0000","last_change":"2024-09-05T16:49:07.406518+0000","last_active":"2024-09-05T16:49:07.406518+0000","last_peered":"2024-09-05T16:49:07.406518+0000","last_clean":"2024-09-05T16:49:07.406518+0000","last_became_active":"2024-09-05T16:43:44.806229+0000","last_became_peered":"2024-09-05T16:43:44.806229+0000","last_unstale":"2024-09-05T16:49:07.406518+0000","last_undegraded":"2024-09-05T16:49:07.406518+0000","last_fullsized":"2024-09-05T16:49:07.406518+0000","mapping_epoch":38,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12552","last_scrub_stamp":"2024-09-05T16:49:07.406395+0000","last_deep_scrub":"39'12552","last_deep_scrub_stamp":"2024-09-05T16:49:07.406395+0000","last_clean_scrub_stamp":"2024-09-05T16:49:07.406395+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T01:15:13.024284+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1865,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"39'12752","reported_seq":17425,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:49:02.160884+0000","last_change":"2024-09-05T16:49:02.160884+0000","last_active":"2024-09-05T16:49:02.160884+0000","last_peered":"2024-09-05T16:49:02.160884+0000","last_clean":"2024-09-05T16:49:02.160884+0000","last_became_active":"2024-09-05T16:46:51.667128+0000","last_became_peered":"2024-09-05T16:46:51.667128+0000","last_unstale":"2024-09-05T16:49:02.160884+0000","last_undegraded":"2024-09-05T16:49:02.160884+0000","last_fullsized":"2024-09-05T16:49:02.160884+0000","mapping_epoch":44,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12752","last_scrub_stamp":"2024-09-05T16:49:02.160799+0000","last_deep_scrub":"39'12752","last_deep_scrub_stamp":"2024-09-05T16:49:02.160799+0000","last_clean_scrub_stamp":"2024-09-05T16:49:02.160799+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T18:49:19.316245+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"39'12934","reported_seq":17730,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:49:01.359017+0000","last_change":"2024-09-05T16:49:01.359017+0000","last_active":"2024-09-05T16:49:01.359017+0000","last_peered":"2024-09-05T16:49:01.359017+0000","last_clean":"2024-09-05T16:49:01.359017+0000","last_became_active":"2024-09-05T16:37:31.984892+0000","last_became_peered":"2024-09-05T16:37:31.984892+0000","last_unstale":"2024-09-05T16:49:01.359017+0000","last_undegraded":"2024-09-05T16:49:01.359017+0000","last_fullsized":"2024-09-05T16:49:01.359017+0000","mapping_epoch":34,"log_start":"33'2900","ondisk_log_start":"33'2900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12934","last_scrub_stamp":"2024-09-05T16:49:01.358931+0000","last_deep_scrub":"39'12934","last_deep_scrub_stamp":"2024-09-05T16:49:01.358931+0000","last_clean_scrub_stamp":"2024-09-05T16:49:01.358931+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T19:56:04.713670+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7234,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"39'12593","reported_seq":17071,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:49:03.655519+0000","last_change":"2024-09-05T16:49:03.655519+0000","last_active":"2024-09-05T16:49:03.655519+0000","last_peered":"2024-09-05T16:49:03.655519+0000","last_clean":"2024-09-05T16:49:03.655519+0000","last_became_active":"2024-09-05T16:47:51.193016+0000","last_became_peered":"2024-09-05T16:47:51.193016+0000","last_unstale":"2024-09-05T16:49:03.655519+0000","last_undegraded":"2024-09-05T16:49:03.655519+0000","last_fullsized":"2024-09-05T16:49:03.655519+0000","mapping_epoch":54,"log_start":"33'2500","ondisk_log_start":"33'2500","created":15,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12593","last_scrub_stamp":"2024-09-05T16:49:03.655403+0000","last_deep_scrub":"39'12593","last_deep_scrub_stamp":"2024-09-05T16:49:03.655403+0000","last_clean_scrub_stamp":"2024-09-05T16:49:03.655403+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T01:44:22.261008+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"39'12772","reported_seq":17401,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:49:04.637492+0000","last_change":"2024-09-05T16:49:04.637492+0000","last_active":"2024-09-05T16:49:04.637492+0000","last_peered":"2024-09-05T16:49:04.637492+0000","last_clean":"2024-09-05T16:49:04.637492+0000","last_became_active":"2024-09-05T16:47:15.728759+0000","last_became_peered":"2024-09-05T16:47:15.728759+0000","last_unstale":"2024-09-05T16:49:04.637492+0000","last_undegraded":"2024-09-05T16:49:04.637492+0000","last_fullsized":"2024-09-05T16:49:04.637492+0000","mapping_epoch":48,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12772","last_scrub_stamp":"2024-09-05T16:49:04.637370+0000","last_deep_scrub":"39'12772","last_deep_scrub_stamp":"2024-09-05T16:49:04.637370+0000","last_clean_scrub_stamp":"2024-09-05T16:49:04.637370+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:05:42.007806+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12772,"num_write_kb":51088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2250,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"39'12660","reported_seq":17119,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:49:00.310755+0000","last_change":"2024-09-05T16:49:00.310755+0000","last_active":"2024-09-05T16:49:00.310755+0000","last_peered":"2024-09-05T16:49:00.310755+0000","last_clean":"2024-09-05T16:49:00.310755+0000","last_became_active":"2024-09-05T16:47:03.185143+0000","last_became_peered":"2024-09-05T16:47:03.185143+0000","last_unstale":"2024-09-05T16:49:00.310755+0000","last_undegraded":"2024-09-05T16:49:00.310755+0000","last_fullsized":"2024-09-05T16:49:00.310755+0000","mapping_epoch":46,"log_start":"33'2600","ondisk_log_start":"33'2600","created":15,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12660","last_scrub_stamp":"2024-09-05T16:49:00.310619+0000","last_deep_scrub":"39'12660","last_deep_scrub_stamp":"2024-09-05T16:49:00.310619+0000","last_clean_scrub_stamp":"2024-09-05T16:49:00.310619+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T21:26:03.126398+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'12777","reported_seq":12947,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:49:03.431674+0000","last_change":"2024-09-05T16:49:03.431674+0000","last_active":"2024-09-05T16:49:03.431674+0000","last_peered":"2024-09-05T16:49:03.431674+0000","last_clean":"2024-09-05T16:49:03.431674+0000","last_became_active":"2024-09-05T16:46:40.184288+0000","last_became_peered":"2024-09-05T16:46:40.184288+0000","last_unstale":"2024-09-05T16:49:03.431674+0000","last_undegraded":"2024-09-05T16:49:03.431674+0000","last_fullsized":"2024-09-05T16:49:03.431674+0000","mapping_epoch":42,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12777","last_scrub_stamp":"2024-09-05T16:49:03.431569+0000","last_deep_scrub":"39'12777","last_deep_scrub_stamp":"2024-09-05T16:49:03.431569+0000","last_clean_scrub_stamp":"2024-09-05T16:49:03.431569+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T17:32:02.503841+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"39'12770","reported_seq":17450,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:49:02.635601+0000","last_change":"2024-09-05T16:49:02.635601+0000","last_active":"2024-09-05T16:49:02.635601+0000","last_peered":"2024-09-05T16:49:02.635601+0000","last_clean":"2024-09-05T16:49:02.635601+0000","last_became_active":"2024-09-05T16:48:41.236479+0000","last_became_peered":"2024-09-05T16:48:41.236479+0000","last_unstale":"2024-09-05T16:49:02.635601+0000","last_undegraded":"2024-09-05T16:49:02.635601+0000","last_fullsized":"2024-09-05T16:49:02.635601+0000","mapping_epoch":62,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12770","last_scrub_stamp":"2024-09-05T16:49:02.635509+0000","last_deep_scrub":"39'12770","last_deep_scrub_stamp":"2024-09-05T16:49:02.635509+0000","last_clean_scrub_stamp":"2024-09-05T16:49:02.635509+0000","objects_scrubbed":1,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T01:23:46.216993+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"39'12808","reported_seq":17509,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:49:01.170251+0000","last_change":"2024-09-05T16:49:01.170251+0000","last_active":"2024-09-05T16:49:01.170251+0000","last_peered":"2024-09-05T16:49:01.170251+0000","last_clean":"2024-09-05T16:49:01.170251+0000","last_became_active":"2024-09-05T16:48:03.239894+0000","last_became_peered":"2024-09-05T16:48:03.239894+0000","last_unstale":"2024-09-05T16:49:01.170251+0000","last_undegraded":"2024-09-05T16:49:01.170251+0000","last_fullsized":"2024-09-05T16:49:01.170251+0000","mapping_epoch":56,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12808","last_scrub_stamp":"2024-09-05T16:49:01.170166+0000","last_deep_scrub":"39'12808","last_deep_scrub_stamp":"2024-09-05T16:49:01.170166+0000","last_clean_scrub_stamp":"2024-09-05T16:49:01.170166+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T21:12:22.180485+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"39'13120","reported_seq":17900,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:49:01.663944+0000","last_change":"2024-09-05T16:49:01.663890+0000","last_active":"2024-09-05T16:49:01.663944+0000","last_peered":"2024-09-05T16:49:01.663944+0000","last_clean":"2024-09-05T16:49:01.663944+0000","last_became_active":"2024-09-05T16:48:29.235000+0000","last_became_peered":"2024-09-05T16:48:29.235000+0000","last_unstale":"2024-09-05T16:49:01.663944+0000","last_undegraded":"2024-09-05T16:49:01.663944+0000","last_fullsized":"2024-09-05T16:49:01.663944+0000","mapping_epoch":60,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'13120","last_scrub_stamp":"2024-09-05T16:49:01.663804+0000","last_deep_scrub":"39'13120","last_deep_scrub_stamp":"2024-09-05T16:49:01.663804+0000","last_clean_scrub_stamp":"2024-09-05T16:49:01.663804+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:40:30.832893+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"39'12758","reported_seq":12893,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:49:02.381147+0000","last_change":"2024-09-05T16:49:02.381147+0000","last_active":"2024-09-05T16:49:02.381147+0000","last_peered":"2024-09-05T16:49:02.381147+0000","last_clean":"2024-09-05T16:49:02.381147+0000","last_became_active":"2024-09-05T16:48:52.676533+0000","last_became_peered":"2024-09-05T16:48:52.676533+0000","last_unstale":"2024-09-05T16:49:02.381147+0000","last_undegraded":"2024-09-05T16:49:02.381147+0000","last_fullsized":"2024-09-05T16:49:02.381147+0000","mapping_epoch":64,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12758","last_scrub_stamp":"2024-09-05T16:49:02.381061+0000","last_deep_scrub":"39'12758","last_deep_scrub_stamp":"2024-09-05T16:49:02.381061+0000","last_clean_scrub_stamp":"2024-09-05T16:49:02.381061+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T19:05:47.067542+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"39'12808","reported_seq":17611,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:49:04.410092+0000","last_change":"2024-09-05T16:49:04.410026+0000","last_active":"2024-09-05T16:49:04.410092+0000","last_peered":"2024-09-05T16:49:04.410092+0000","last_clean":"2024-09-05T16:49:04.410092+0000","last_became_active":"2024-09-05T16:40:29.954942+0000","last_became_peered":"2024-09-05T16:40:29.954942+0000","last_unstale":"2024-09-05T16:49:04.410092+0000","last_undegraded":"2024-09-05T16:49:04.410092+0000","last_fullsized":"2024-09-05T16:49:04.410092+0000","mapping_epoch":36,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12808","last_scrub_stamp":"2024-09-05T16:49:04.409926+0000","last_deep_scrub":"39'12808","last_deep_scrub_stamp":"2024-09-05T16:49:04.409926+0000","last_clean_scrub_stamp":"2024-09-05T16:49:04.409926+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T17:08:59.106564+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4408,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'13124","reported_seq":13281,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:49:05.425382+0000","last_change":"2024-09-05T16:49:05.425382+0000","last_active":"2024-09-05T16:49:05.425382+0000","last_peered":"2024-09-05T16:49:05.425382+0000","last_clean":"2024-09-05T16:49:05.425382+0000","last_became_active":"2024-09-05T16:48:16.435091+0000","last_became_peered":"2024-09-05T16:48:16.435091+0000","last_unstale":"2024-09-05T16:49:05.425382+0000","last_undegraded":"2024-09-05T16:49:05.425382+0000","last_fullsized":"2024-09-05T16:49:05.425382+0000","mapping_epoch":58,"log_start":"33'3100","ondisk_log_start":"33'3100","created":15,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'13124","last_scrub_stamp":"2024-09-05T16:49:05.425248+0000","last_deep_scrub":"39'13124","last_deep_scrub_stamp":"2024-09-05T16:49:05.425248+0000","last_clean_scrub_stamp":"2024-09-05T16:49:05.425248+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T22:14:53.776820+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"39'12831","reported_seq":13004,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:49:08.410988+0000","last_change":"2024-09-05T16:49:08.410988+0000","last_active":"2024-09-05T16:49:08.410988+0000","last_peered":"2024-09-05T16:49:08.410988+0000","last_clean":"2024-09-05T16:49:08.410988+0000","last_became_active":"2024-09-05T16:47:40.189447+0000","last_became_peered":"2024-09-05T16:47:40.189447+0000","last_unstale":"2024-09-05T16:49:08.410988+0000","last_undegraded":"2024-09-05T16:49:08.410988+0000","last_fullsized":"2024-09-05T16:49:08.410988+0000","mapping_epoch":52,"log_start":"33'2800","ondisk_log_start":"33'2800","created":15,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12831","last_scrub_stamp":"2024-09-05T16:49:08.410876+0000","last_deep_scrub":"39'12831","last_deep_scrub_stamp":"2024-09-05T16:49:08.410876+0000","last_clean_scrub_stamp":"2024-09-05T16:49:08.410876+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T03:13:20.233704+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'12773","reported_seq":12946,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-05T16:49:06.431272+0000","last_change":"2024-09-05T16:49:06.431272+0000","last_active":"2024-09-05T16:49:06.431272+0000","last_peered":"2024-09-05T16:49:06.431272+0000","last_clean":"2024-09-05T16:49:06.431272+0000","last_became_active":"2024-09-05T16:47:27.204989+0000","last_became_peered":"2024-09-05T16:47:27.204989+0000","last_unstale":"2024-09-05T16:49:06.431272+0000","last_undegraded":"2024-09-05T16:49:06.431272+0000","last_fullsized":"2024-09-05T16:49:06.431272+0000","mapping_epoch":50,"log_start":"33'2700","ondisk_log_start":"33'2700","created":15,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12773","last_scrub_stamp":"2024-09-05T16:49:06.431111+0000","last_deep_scrub":"39'12773","last_deep_scrub_stamp":"2024-09-05T16:49:06.431111+0000","last_clean_scrub_stamp":"2024-09-05T16:49:06.431111+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T20:39:03.616953+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":39183,"num_write_kb":108864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4512,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":28,"seq":120259084471,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":726372,"kb_used_data":532,"kb_used_omap":4,"kb_used_meta":725819,"kb_avail":93645468,"statfs":{"total":96636764160,"available":95892959232,"internally_reserved":0,"allocated":544768,"data_stored":206462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":743239379},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986348,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":631972,"kb_used_data":532,"kb_used_omap":4,"kb_used_meta":631419,"kb_avail":93739868,"statfs":{"total":96636764160,"available":95989624832,"internally_reserved":0,"allocated":544768,"data_stored":206462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":646573778},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673155,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713380,"kb_used_data":532,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":93658460,"statfs":{"total":96636764160,"available":95906263040,"internally_reserved":0,"allocated":544768,"data_stored":206462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":729938373},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":25,"seq":107374182585,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713828,"kb_used_data":532,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658012,"statfs":{"total":96636764160,"available":95905804288,"internally_reserved":0,"allocated":544768,"data_stored":206462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4403,"internal_metadata":730394317},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-05T16:49:22.189 DEBUG:teuthology.orchestra.run.smithi087:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-09-05T16:49:22.488 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-05T16:49:22.488 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-05T16:49:22.489 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-09-05T16:49:22.489 INFO:teuthology.orchestra.run:waiting for 300 2024-09-05T16:49:27.076 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2024-09-05T16:49:28.590 INFO:tasks.ceph.osd.0:Stopped 2024-09-05T16:49:28.591 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-05T16:49:28.591 INFO:teuthology.orchestra.run:waiting for 300 2024-09-05T16:49:28.677 INFO:tasks.ceph.osd.1:Stopped 2024-09-05T16:49:28.677 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-09-05T16:49:28.677 INFO:teuthology.orchestra.run:waiting for 300 2024-09-05T16:49:32.580 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2024-09-05T16:49:32.581 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 has been restored 2024-09-05T16:49:34.778 INFO:tasks.ceph.osd.2:Stopped 2024-09-05T16:49:34.779 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-09-05T16:49:34.779 INFO:teuthology.orchestra.run:waiting for 300 2024-09-05T16:49:37.984 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~0s 2024-09-05T16:49:37.985 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2024-09-05T16:49:40.885 INFO:tasks.ceph.osd.3:Stopped 2024-09-05T16:49:40.885 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-05T16:49:40.885 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-09-05T16:49:40.886 INFO:teuthology.orchestra.run:waiting for 300 2024-09-05T16:49:40.917 INFO:tasks.ceph.mgr.x:Stopped 2024-09-05T16:49:40.917 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-05T16:49:40.917 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-05T16:49:40.917 INFO:teuthology.orchestra.run:waiting for 300 2024-09-05T16:49:40.971 INFO:tasks.ceph.mon.a:Stopped 2024-09-05T16:49:40.971 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-05T16:49:40.971 INFO:teuthology.orchestra.run:waiting for 300 2024-09-05T16:49:41.025 INFO:tasks.ceph.mon.b:Stopped 2024-09-05T16:49:41.025 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-05T16:49:41.026 INFO:teuthology.orchestra.run:waiting for 300 2024-09-05T16:49:41.079 INFO:tasks.ceph.mon.c:Stopped 2024-09-05T16:49:41.079 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-05T16:49:41.079 DEBUG:teuthology.orchestra.run.smithi087:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v 'objects unfound and apparently lost' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v '\(MON_DOWN\)' | head -n 1 2024-09-05T16:49:41.180 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi087.front.sepia.ceph.com 2024-09-05T16:49:41.181 DEBUG:teuthology.orchestra.run.smithi087:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-09-05T16:49:41.758 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi087.front.sepia.ceph.com 2024-09-05T16:49:41.759 DEBUG:teuthology.orchestra.run.smithi087:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-09-05T16:49:41.882 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi087.front.sepia.ceph.com 2024-09-05T16:49:41.882 DEBUG:teuthology.orchestra.run.smithi087:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-09-05T16:49:41.959 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi087.front.sepia.ceph.com 2024-09-05T16:49:41.960 DEBUG:teuthology.orchestra.run.smithi087:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-09-05T16:49:42.059 INFO:tasks.ceph:Archiving mon data... 2024-09-05T16:49:42.061 DEBUG:teuthology.misc:Transferring archived files from smithi087:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/skanta-2024-09-05_12:54:40-rados-wip-bharath4-testing-2024-09-02-0957-distro-default-smithi/7890779/data/mon.a.tgz 2024-09-05T16:49:42.061 DEBUG:teuthology.orchestra.run.smithi087:> mktemp 2024-09-05T16:49:42.092 INFO:teuthology.orchestra.run.smithi087.stdout:/tmp/tmp.bdEOvKiDEl 2024-09-05T16:49:42.092 DEBUG:teuthology.orchestra.run.smithi087:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.bdEOvKiDEl 2024-09-05T16:49:42.223 DEBUG:teuthology.orchestra.run.smithi087:> sudo chmod 0666 /tmp/tmp.bdEOvKiDEl 2024-09-05T16:49:42.336 DEBUG:teuthology.orchestra.remote:smithi087:/tmp/tmp.bdEOvKiDEl is 297KB 2024-09-05T16:49:42.431 DEBUG:teuthology.orchestra.run.smithi087:> rm -fr /tmp/tmp.bdEOvKiDEl 2024-09-05T16:49:42.446 DEBUG:teuthology.misc:Transferring archived files from smithi087:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/skanta-2024-09-05_12:54:40-rados-wip-bharath4-testing-2024-09-02-0957-distro-default-smithi/7890779/data/mon.b.tgz 2024-09-05T16:49:42.447 DEBUG:teuthology.orchestra.run.smithi087:> mktemp 2024-09-05T16:49:42.500 INFO:teuthology.orchestra.run.smithi087.stdout:/tmp/tmp.urF3M4mUJ1 2024-09-05T16:49:42.501 DEBUG:teuthology.orchestra.run.smithi087:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.urF3M4mUJ1 2024-09-05T16:49:42.617 DEBUG:teuthology.orchestra.run.smithi087:> sudo chmod 0666 /tmp/tmp.urF3M4mUJ1 2024-09-05T16:49:42.701 DEBUG:teuthology.orchestra.remote:smithi087:/tmp/tmp.urF3M4mUJ1 is 300KB 2024-09-05T16:49:42.795 DEBUG:teuthology.orchestra.run.smithi087:> rm -fr /tmp/tmp.urF3M4mUJ1 2024-09-05T16:49:42.810 DEBUG:teuthology.misc:Transferring archived files from smithi087:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/skanta-2024-09-05_12:54:40-rados-wip-bharath4-testing-2024-09-02-0957-distro-default-smithi/7890779/data/mon.c.tgz 2024-09-05T16:49:42.810 DEBUG:teuthology.orchestra.run.smithi087:> mktemp 2024-09-05T16:49:42.864 INFO:teuthology.orchestra.run.smithi087.stdout:/tmp/tmp.ojNGductEb 2024-09-05T16:49:42.864 DEBUG:teuthology.orchestra.run.smithi087:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.ojNGductEb 2024-09-05T16:49:42.981 DEBUG:teuthology.orchestra.run.smithi087:> sudo chmod 0666 /tmp/tmp.ojNGductEb 2024-09-05T16:49:42.985 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 has been restored 2024-09-05T16:49:43.064 DEBUG:teuthology.orchestra.remote:smithi087:/tmp/tmp.ojNGductEb is 300KB 2024-09-05T16:49:43.134 DEBUG:teuthology.orchestra.run.smithi087:> rm -fr /tmp/tmp.ojNGductEb 2024-09-05T16:49:43.149 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-05T16:49:43.149 DEBUG:teuthology.orchestra.run.smithi087:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-05T16:49:43.288 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-05T16:49:43.288 INFO:tasks.ceph:Archiving crash dumps... 2024-09-05T16:49:43.290 DEBUG:teuthology.misc:Transferring archived files from smithi087:/var/lib/ceph/crash to /home/teuthworker/archive/skanta-2024-09-05_12:54:40-rados-wip-bharath4-testing-2024-09-02-0957-distro-default-smithi/7890779/remote/smithi087/crash 2024-09-05T16:49:43.291 DEBUG:teuthology.orchestra.run.smithi087:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-05T16:49:43.353 INFO:tasks.ceph:Compressing logs... 2024-09-05T16:49:43.353 DEBUG:teuthology.orchestra.run.smithi087:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-05T16:49:43.715 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-09-05T16:49:43.715 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47620.log 2024-09-05T16:49:43.716 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60579.log 2024-09-05T16:49:43.716 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.47620.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65035.log 2024-09-05T16:49:43.716 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.60579.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58142.log 2024-09-05T16:49:43.716 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60579.log.gz 2024-09-05T16:49:43.716 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.65035.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69994.log 2024-09-05T16:49:43.716 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65035.log.gz 2024-09-05T16:49:43.716 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.58142.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75162.log 2024-09-05T16:49:43.716 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.69994.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66422.log 2024-09-05T16:49:43.716 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69994.log.gz 2024-09-05T16:49:43.716 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.75162.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63606.log 2024-09-05T16:49:43.716 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.75162.log.gz 2024-09-05T16:49:43.717 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.66422.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61839.log 2024-09-05T16:49:43.717 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66422.log.gz 2024-09-05T16:49:43.717 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.63606.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66296.log 2024-09-05T16:49:43.717 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63606.log.gz 2024-09-05T16:49:43.717 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.61839.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67809.log 2024-09-05T16:49:43.717 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61839.log.gz 2024-09-05T16:49:43.717 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.66296.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74280.log 2024-09-05T16:49:43.717 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66296.log.gz 2024-09-05T16:49:43.717 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.67809.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45568.log 2024-09-05T16:49:43.717 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67809.log.gz 2024-09-05T16:49:43.717 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.74280.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74280.log.gz 2024-09-05T16:49:43.717 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41599.log 2024-09-05T16:49:43.717 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.45568.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41025.log 2024-09-05T16:49:43.718 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45568.log.gz 2024-09-05T16:49:43.718 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.41599.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55493.log 2024-09-05T16:49:43.718 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.41025.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57006.log 2024-09-05T16:49:43.718 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.55493.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50441.log 2024-09-05T16:49:43.718 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.57006.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64236.log 2024-09-05T16:49:43.718 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.50441.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58983.log 2024-09-05T16:49:43.718 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.64236.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40558.log 2024-09-05T16:49:43.718 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64236.log.gz 2024-09-05T16:49:43.718 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.58983.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42261.log 2024-09-05T16:49:43.718 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.40558.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65245.log 2024-09-05T16:49:43.718 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.42261.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58689.log 2024-09-05T16:49:43.718 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.65245.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72390.log 2024-09-05T16:49:43.718 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65245.log.gz 2024-09-05T16:49:43.718 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.58689.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51281.log 2024-09-05T16:49:43.719 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.72390.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60453.log 2024-09-05T16:49:43.719 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72390.log.gz 2024-09-05T16:49:43.719 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.51281.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75036.log 2024-09-05T16:49:43.719 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.60453.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67221.log 2024-09-05T16:49:43.719 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60453.log.gz 2024-09-05T16:49:43.719 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.75036.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53717.log 2024-09-05T16:49:43.719 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.75036.log.gz 2024-09-05T16:49:43.719 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.67221.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71172.log 2024-09-05T16:49:43.719 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67221.log.gz 2024-09-05T16:49:43.719 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.53717.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74868.log 2024-09-05T16:49:43.719 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.71172.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48175.log 2024-09-05T16:49:43.719 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71172.log.gz 2024-09-05T16:49:43.719 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.74868.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64110.log 2024-09-05T16:49:43.719 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74868.log.gz 2024-09-05T16:49:43.720 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.48175.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50147.log 2024-09-05T16:49:43.720 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.64110.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60621.log 2024-09-05T16:49:43.720 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64110.log.gz 2024-09-05T16:49:43.720 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.50147.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39357.log 2024-09-05T16:49:43.720 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.60621.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54851.log 2024-09-05T16:49:43.720 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60621.log.gz 2024-09-05T16:49:43.720 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.39357.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72054.log 2024-09-05T16:49:43.720 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40558.log.gz 2024-09-05T16:49:43.720 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41025.log.gz 2024-09-05T16:49:43.720 INFO:teuthology.orchestra.run.smithi087.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39357.log.gz 2024-09-05T16:49:43.720 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41599.log.gz 2024-09-05T16:49:43.720 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42261.log.gz 2024-09-05T16:49:43.720 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70246.log/var/log/ceph/ceph-client.admin.54851.log: 2024-09-05T16:49:43.721 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47620.log.gz 2024-09-05T16:49:43.721 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51281.log.gz 2024-09-05T16:49:43.721 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50147.log.gz 2024-09-05T16:49:43.721 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50441.log.gz 2024-09-05T16:49:43.721 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53717.log.gz -- replaced with /var/log/ceph/ceph-client.admin.58142.log.gz 2024-09-05T16:49:43.721 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-09-05T16:49:43.721 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54851.log.gz 2024-09-05T16:49:43.721 INFO:teuthology.orchestra.run.smithi087.stderr: -- replaced with /var/log/ceph/ceph-client.admin.55493.log.gz 2024-09-05T16:49:43.721 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48175.log.gz 2024-09-05T16:49:43.721 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57006.log.gz 2024-09-05T16:49:43.721 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.72054.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58689.log.gz 2024-09-05T16:49:43.721 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58983.log.gz 2024-09-05T16:49:43.721 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56795.log 2024-09-05T16:49:43.721 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72054.log.gz 2024-09-05T16:49:43.722 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.70246.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42076.log 2024-09-05T16:49:43.722 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70246.log.gz 2024-09-05T16:49:43.722 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.56795.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41683.log 2024-09-05T16:49:43.722 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56795.log.gz 2024-09-05T16:49:43.722 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.42076.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67977.log 2024-09-05T16:49:43.722 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42076.log.gz 2024-09-05T16:49:43.722 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.41683.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58941.log 2024-09-05T16:49:43.722 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41683.log.gz 2024-09-05T16:49:43.722 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.67977.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47509.log 2024-09-05T16:49:43.722 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67977.log.gz 2024-09-05T16:49:43.722 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.58941.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40803.log 2024-09-05T16:49:43.722 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58941.log.gz 2024-09-05T16:49:43.722 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.47509.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41358.log 2024-09-05T16:49:43.723 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47509.log.gz 2024-09-05T16:49:43.723 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.40803.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73692.log 2024-09-05T16:49:43.723 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40803.log.gz 2024-09-05T16:49:43.723 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.41358.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66212.log 2024-09-05T16:49:43.723 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41358.log.gz 2024-09-05T16:49:43.723 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.73692.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69237.log 2024-09-05T16:49:43.723 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73692.log.gz 2024-09-05T16:49:43.723 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.66212.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68061.log 2024-09-05T16:49:43.723 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66212.log.gz 2024-09-05T16:49:43.723 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.69237.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75498.log 2024-09-05T16:49:43.723 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69237.log.gz 2024-09-05T16:49:43.723 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.68061.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50945.log 2024-09-05T16:49:43.724 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68061.log.gz 2024-09-05T16:49:43.724 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.75498.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58015.log 2024-09-05T16:49:43.724 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.75498.log.gz 2024-09-05T16:49:43.724 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.50945.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52079.log 2024-09-05T16:49:43.724 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50945.log.gz 2024-09-05T16:49:43.724 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.58015.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75792.log 2024-09-05T16:49:43.724 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58015.log.gz 2024-09-05T16:49:43.724 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67431.log 2024-09-05T16:49:43.724 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.52079.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52079.log.gz 2024-09-05T16:49:43.724 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.75792.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51365.log 2024-09-05T16:49:43.724 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.75792.log.gz 2024-09-05T16:49:43.724 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.67431.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58815.log 2024-09-05T16:49:43.724 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67431.log.gz 2024-09-05T16:49:43.725 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.51365.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70120.log 2024-09-05T16:49:43.725 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51365.log.gz 2024-09-05T16:49:43.725 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.58815.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58815.log.gz 2024-09-05T16:49:43.725 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43684.log 2024-09-05T16:49:43.725 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.70120.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73818.log 2024-09-05T16:49:43.725 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70120.log.gz 2024-09-05T16:49:43.725 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.43684.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39762.log 2024-09-05T16:49:43.725 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43684.log.gz 2024-09-05T16:49:43.725 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.73818.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47031.log 2024-09-05T16:49:43.725 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73818.log.gz 2024-09-05T16:49:43.725 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.39762.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45229.log 2024-09-05T16:49:43.725 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39762.log.gz 2024-09-05T16:49:43.725 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.47031.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61713.log 2024-09-05T16:49:43.726 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47031.log.gz 2024-09-05T16:49:43.726 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.45229.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68817.log 2024-09-05T16:49:43.726 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45229.log.gz 2024-09-05T16:49:43.726 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.61713.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.75288.log 2024-09-05T16:49:43.726 INFO:teuthology.orchestra.run.smithi087.stderr: -- replaced with /var/log/ceph/ceph-client.admin.61713.log.gz 2024-09-05T16:49:43.726 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.68817.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60369.log 2024-09-05T16:49:43.726 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68817.log.gz 2024-09-05T16:49:43.726 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.75288.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64026.log 2024-09-05T16:49:43.726 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.75288.log.gz 2024-09-05T16:49:43.726 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.60369.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72138.log 2024-09-05T16:49:43.726 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60369.log.gz 2024-09-05T16:49:43.726 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.64026.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45148.log 2024-09-05T16:49:43.726 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64026.log.gz 2024-09-05T16:49:43.726 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.72138.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52709.log 2024-09-05T16:49:43.727 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72138.log.gz 2024-09-05T16:49:43.727 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.45148.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62346.log 2024-09-05T16:49:43.727 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45148.log.gz 2024-09-05T16:49:43.727 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.52709.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52373.log 2024-09-05T16:49:43.727 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52709.log.gz 2024-09-05T16:49:43.727 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.62346.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50609.log 2024-09-05T16:49:43.727 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62346.log.gz 2024-09-05T16:49:43.727 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.52373.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68439.log 2024-09-05T16:49:43.727 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52373.log.gz 2024-09-05T16:49:43.727 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.50609.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50609.log.gz 2024-09-05T16:49:43.727 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53213.log 2024-09-05T16:49:43.727 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.68439.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74532.log 2024-09-05T16:49:43.727 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68439.log.gz 2024-09-05T16:49:43.727 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.53213.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53087.log 2024-09-05T16:49:43.728 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53213.log.gz 2024-09-05T16:49:43.728 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.74532.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57552.log 2024-09-05T16:49:43.728 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74532.log.gz 2024-09-05T16:49:43.728 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.53087.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65456.log 2024-09-05T16:49:43.728 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53087.log.gz 2024-09-05T16:49:43.728 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.57552.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64152.log 2024-09-05T16:49:43.728 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57552.log.gz 2024-09-05T16:49:43.728 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.65456.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70582.log 2024-09-05T16:49:43.728 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65456.log.gz 2024-09-05T16:49:43.728 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.64152.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42039.log 2024-09-05T16:49:43.728 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64152.log.gz 2024-09-05T16:49:43.728 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.70582.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46110.log 2024-09-05T16:49:43.728 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70582.log.gz 2024-09-05T16:49:43.729 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.42039.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55871.log 2024-09-05T16:49:43.729 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42039.log.gz 2024-09-05T16:49:43.729 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66758.log 2024-09-05T16:49:43.729 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.46110.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46110.log.gz 2024-09-05T16:49:43.729 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.55871.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55871.log.gzgzip 2024-09-05T16:49:43.729 INFO:teuthology.orchestra.run.smithi087.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.53381.log 2024-09-05T16:49:43.729 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.66758.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64531.log 2024-09-05T16:49:43.729 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66758.log.gz 2024-09-05T16:49:43.729 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.53381.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50399.log 2024-09-05T16:49:43.729 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53381.log.gz 2024-09-05T16:49:43.729 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.64531.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39974.log 2024-09-05T16:49:43.730 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64531.log.gz 2024-09-05T16:49:43.730 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.50399.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65792.log 2024-09-05T16:49:43.730 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50399.log.gz 2024-09-05T16:49:43.730 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.39974.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61587.log 2024-09-05T16:49:43.730 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39974.log.gz 2024-09-05T16:49:43.730 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.65792.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57048.log 2024-09-05T16:49:43.730 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65792.log.gz 2024-09-05T16:49:43.730 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.61587.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68859.log 2024-09-05T16:49:43.730 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61587.log.gz 2024-09-05T16:49:43.730 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.57048.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57216.log 2024-09-05T16:49:43.730 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57048.log.gz 2024-09-05T16:49:43.730 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.68859.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68859.log.gz 2024-09-05T16:49:43.731 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73776.log 2024-09-05T16:49:43.731 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.57216.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64363.log 2024-09-05T16:49:43.731 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57216.log.gz 2024-09-05T16:49:43.731 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.73776.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62682.log 2024-09-05T16:49:43.731 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73776.log.gz 2024-09-05T16:49:43.731 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.64363.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55199.log 2024-09-05T16:49:43.731 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64363.log.gz 2024-09-05T16:49:43.731 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.62682.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53885.log 2024-09-05T16:49:43.731 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62682.log.gz 2024-09-05T16:49:43.731 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50357.log 2024-09-05T16:49:43.731 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.55199.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55199.log.gz 2024-09-05T16:49:43.731 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.53885.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-09-05T16:49:43.732 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53885.log.gz 2024-09-05T16:49:43.732 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.50357.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65330.log 2024-09-05T16:49:43.732 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50357.log.gz 2024-09-05T16:49:43.732 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55577.log 2024-09-05T16:49:43.732 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.65330.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64951.log 2024-09-05T16:49:43.732 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65330.log.gz 2024-09-05T16:49:43.732 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52037.log 2024-09-05T16:49:43.732 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.55577.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55577.log.gz 2024-09-05T16:49:43.732 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.64951.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45316.log 2024-09-05T16:49:43.732 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64951.log.gz 2024-09-05T16:49:43.732 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.52037.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47842.log 2024-09-05T16:49:43.732 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52037.log.gz 2024-09-05T16:49:43.733 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65918.log 2024-09-05T16:49:43.733 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.45316.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45316.log.gz 2024-09-05T16:49:43.733 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.47842.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75120.log 2024-09-05T16:49:43.733 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47842.log.gz 2024-09-05T16:49:43.733 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.65918.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69489.log 2024-09-05T16:49:43.733 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65918.log.gz 2024-09-05T16:49:43.733 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68985.log 2024-09-05T16:49:43.733 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.75120.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.75120.log.gz 2024-09-05T16:49:43.733 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.69489.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45648.log 2024-09-05T16:49:43.733 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69489.log.gz 2024-09-05T16:49:43.733 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.68985.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75330.log 2024-09-05T16:49:43.733 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68985.log.gz 2024-09-05T16:49:43.734 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70624.log 2024-09-05T16:49:43.734 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.45648.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45648.log.gz 2024-09-05T16:49:43.734 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.75330.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40581.log 2024-09-05T16:49:43.734 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.75330.log.gz 2024-09-05T16:49:43.734 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58563.log 2024-09-05T16:49:43.734 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.70624.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70624.log.gz 2024-09-05T16:49:43.734 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54725.log 2024-09-05T16:49:43.734 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.40581.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40581.log.gz 2024-09-05T16:49:43.734 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.58563.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61968.log 2024-09-05T16:49:43.734 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58563.log.gz 2024-09-05T16:49:43.734 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.54725.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45462.log 2024-09-05T16:49:43.734 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54725.log.gz 2024-09-05T16:49:43.734 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56123.log 2024-09-05T16:49:43.735 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.61968.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61968.log.gz 2024-09-05T16:49:43.735 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.45462.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49811.log 2024-09-05T16:49:43.735 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45462.log.gz 2024-09-05T16:49:43.735 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.56123.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59571.log 2024-09-05T16:49:43.735 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56123.log.gz 2024-09-05T16:49:43.735 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61209.log 2024-09-05T16:49:43.735 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.49811.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49811.log.gz 2024-09-05T16:49:43.735 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.59571.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47435.log 2024-09-05T16:49:43.735 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59571.log.gz 2024-09-05T16:49:43.735 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.61209.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47953.log 2024-09-05T16:49:43.735 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61209.log.gz 2024-09-05T16:49:43.735 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.47435.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75834.log 2024-09-05T16:49:43.735 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47435.log.gz 2024-09-05T16:49:43.736 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.47953.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71087.log 2024-09-05T16:49:43.736 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47953.log.gz 2024-09-05T16:49:43.736 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.75834.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.75834.log.gz -5 2024-09-05T16:49:43.736 INFO:teuthology.orchestra.run.smithi087.stderr: --verbose -- /var/log/ceph/ceph-client.admin.74700.log 2024-09-05T16:49:43.736 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.71087.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.68901.log 2024-09-05T16:49:43.736 INFO:teuthology.orchestra.run.smithi087.stderr: -- replaced with /var/log/ceph/ceph-client.admin.71087.log.gz 2024-09-05T16:49:43.736 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.74700.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46572.log 2024-09-05T16:49:43.736 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74700.log.gz 2024-09-05T16:49:43.736 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.68901.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42150.log 2024-09-05T16:49:43.736 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68901.log.gz 2024-09-05T16:49:43.736 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.46572.log: gzip 0.0% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.46572.log.gz -- 2024-09-05T16:49:43.736 INFO:teuthology.orchestra.run.smithi087.stderr: /var/log/ceph/ceph-client.admin.57090.log 2024-09-05T16:49:43.736 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.42150.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74658.log 2024-09-05T16:49:43.737 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42150.log.gz 2024-09-05T16:49:43.737 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.57090.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52163.log 2024-09-05T16:49:43.737 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57090.log.gz 2024-09-05T16:49:43.737 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.74658.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74742.log 2024-09-05T16:49:43.737 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74658.log.gz 2024-09-05T16:49:43.737 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.52163.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65708.log 2024-09-05T16:49:43.737 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52163.log.gz 2024-09-05T16:49:43.737 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52835.log 2024-09-05T16:49:43.737 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.74742.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74742.log.gz 2024-09-05T16:49:43.737 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.65708.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39720.log 2024-09-05T16:49:43.737 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65708.log.gz 2024-09-05T16:49:43.737 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.52835.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.52835.log.gz 2024-09-05T16:49:43.738 INFO:teuthology.orchestra.run.smithi087.stderr: --verbose -- /var/log/ceph/ceph-client.admin.58731.log 2024-09-05T16:49:43.738 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.39720.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41210.log 2024-09-05T16:49:43.738 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39720.log.gz 2024-09-05T16:49:43.738 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.58731.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72264.log 2024-09-05T16:49:43.738 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58731.log.gz 2024-09-05T16:49:43.738 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.41210.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57846.log 2024-09-05T16:49:43.738 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41210.log.gz 2024-09-05T16:49:43.738 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.72264.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72264.log.gz 2024-09-05T16:49:43.738 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70498.log 2024-09-05T16:49:43.738 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.57846.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73566.log 2024-09-05T16:49:43.738 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57846.log.gz 2024-09-05T16:49:43.738 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.70498.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50861.log 2024-09-05T16:49:43.738 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70498.log.gz 2024-09-05T16:49:43.738 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.73566.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59991.log 2024-09-05T16:49:43.739 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73566.log.gz 2024-09-05T16:49:43.739 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.50861.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50861.log.gz 2024-09-05T16:49:43.739 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75708.log 2024-09-05T16:49:43.739 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.59991.log: -- /var/log/ceph/ceph-client.admin.53255.log 2024-09-05T16:49:43.739 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59991.log.gz 2024-09-05T16:49:43.739 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.75708.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39487.log 2024-09-05T16:49:43.739 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.75708.log.gz 2024-09-05T16:49:43.739 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.53255.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55283.log 2024-09-05T16:49:43.739 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53255.log.gz 2024-09-05T16:49:43.739 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.39487.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76042.log 2024-09-05T16:49:43.739 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39487.log.gz 2024-09-05T16:49:43.739 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.55283.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56417.log 2024-09-05T16:49:43.739 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55283.log.gz 2024-09-05T16:49:43.740 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.76042.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57972.log 2024-09-05T16:49:43.740 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.76042.log.gz 2024-09-05T16:49:43.740 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.56417.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53969.log 2024-09-05T16:49:43.740 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56417.log.gz 2024-09-05T16:49:43.740 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.57972.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56459.log 2024-09-05T16:49:43.740 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57972.log.gz 2024-09-05T16:49:43.740 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.53969.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65624.log 2024-09-05T16:49:43.740 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53969.log.gz 2024-09-05T16:49:43.740 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.56459.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49895.log 2024-09-05T16:49:43.740 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56459.log.gz 2024-09-05T16:49:43.740 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.65624.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68355.log 2024-09-05T16:49:43.740 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65624.log.gz 2024-09-05T16:49:43.740 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.49895.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50063.log 2024-09-05T16:49:43.741 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49895.log.gz 2024-09-05T16:49:43.741 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.68355.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50105.log 2024-09-05T16:49:43.741 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68355.log.gz 2024-09-05T16:49:43.741 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.50063.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63732.log 2024-09-05T16:49:43.741 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50063.log.gz 2024-09-05T16:49:43.741 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.50105.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56081.log 2024-09-05T16:49:43.741 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50105.log.gz 2024-09-05T16:49:43.741 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.63732.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75960.log 2024-09-05T16:49:43.741 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63732.log.gz 2024-09-05T16:49:43.741 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.56081.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52961.log 2024-09-05T16:49:43.741 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56081.log.gz 2024-09-05T16:49:43.741 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.75960.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59697.log 2024-09-05T16:49:43.741 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.75960.log.gz 2024-09-05T16:49:43.742 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.52961.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53549.log 2024-09-05T16:49:43.742 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52961.log.gz 2024-09-05T16:49:43.742 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.59697.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65161.log 2024-09-05T16:49:43.742 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59697.log.gz 2024-09-05T16:49:43.742 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.53549.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54641.log 2024-09-05T16:49:43.742 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53549.log.gz 2024-09-05T16:49:43.742 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.65161.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71802.log 2024-09-05T16:49:43.742 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65161.log.gz 2024-09-05T16:49:43.742 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.54641.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56375.log 2024-09-05T16:49:43.742 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54641.log.gz 2024-09-05T16:49:43.742 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.71802.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53675.log 2024-09-05T16:49:43.742 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71802.log.gz 2024-09-05T16:49:43.742 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.56375.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49601.log 2024-09-05T16:49:43.743 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56375.log.gz 2024-09-05T16:49:43.743 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.53675.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47157.log 2024-09-05T16:49:43.743 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53675.log.gz 2024-09-05T16:49:43.743 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.49601.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59445.log 2024-09-05T16:49:43.743 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49601.log.gz 2024-09-05T16:49:43.743 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.47157.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59529.log 2024-09-05T16:49:43.743 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47157.log.gz 2024-09-05T16:49:43.743 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.59445.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61377.log 2024-09-05T16:49:43.743 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59445.log.gz 2024-09-05T16:49:43.743 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.59529.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52793.log 2024-09-05T16:49:43.743 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59529.log.gz 2024-09-05T16:49:43.743 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.61377.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47805.log 2024-09-05T16:49:43.743 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61377.log.gz 2024-09-05T16:49:43.743 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.52793.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59739.log 2024-09-05T16:49:43.744 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52793.log.gz 2024-09-05T16:49:43.744 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.47805.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65372.log 2024-09-05T16:49:43.744 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47805.log.gz 2024-09-05T16:49:43.744 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48323.log 2024-09-05T16:49:43.744 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.59739.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59739.log.gz 2024-09-05T16:49:43.744 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.65372.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49727.log 2024-09-05T16:49:43.744 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65372.log.gz 2024-09-05T16:49:43.744 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.48323.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60873.log 2024-09-05T16:49:43.744 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48323.log.gz 2024-09-05T16:49:43.744 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.49727.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53129.log 2024-09-05T16:49:43.744 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49727.log.gz 2024-09-05T16:49:43.744 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.60873.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41928.log 2024-09-05T16:49:43.744 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60873.log.gz 2024-09-05T16:49:43.745 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.53129.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63564.log 2024-09-05T16:49:43.745 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53129.log.gz 2024-09-05T16:49:43.745 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.41928.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74196.log 2024-09-05T16:49:43.745 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41928.log.gz 2024-09-05T16:49:43.745 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.63564.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47694.log 2024-09-05T16:49:43.745 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63564.log.gz 2024-09-05T16:49:43.745 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.74196.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72306.log 2024-09-05T16:49:43.745 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74196.log.gz 2024-09-05T16:49:43.745 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.47694.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72432.log 2024-09-05T16:49:43.745 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47694.log.gz 2024-09-05T16:49:43.745 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.72306.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70162.log 2024-09-05T16:49:43.745 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72306.log.gz 2024-09-05T16:49:43.745 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.72432.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63354.log 2024-09-05T16:49:43.746 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72432.log.gz 2024-09-05T16:49:43.746 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.70162.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73272.log 2024-09-05T16:49:43.746 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70162.log.gz 2024-09-05T16:49:43.746 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.63354.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47768.log 2024-09-05T16:49:43.746 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63354.log.gz 2024-09-05T16:49:43.746 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.73272.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74910.log 2024-09-05T16:49:43.746 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73272.log.gz 2024-09-05T16:49:43.746 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.47768.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57300.log 2024-09-05T16:49:43.746 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47768.log.gz 2024-09-05T16:49:43.746 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.74910.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45984.log 2024-09-05T16:49:43.746 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74910.log.gz 2024-09-05T16:49:43.746 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.57300.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74616.log 2024-09-05T16:49:43.746 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57300.log.gz 2024-09-05T16:49:43.746 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41321.log 2024-09-05T16:49:43.746 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.45984.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45984.log.gz 2024-09-05T16:49:43.747 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.74616.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68229.log 2024-09-05T16:49:43.747 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74616.log.gz 2024-09-05T16:49:43.747 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.41321.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59025.log 2024-09-05T16:49:43.747 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41321.log.gz 2024-09-05T16:49:43.747 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.68229.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54515.log 2024-09-05T16:49:43.747 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68229.log.gz 2024-09-05T16:49:43.747 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.59025.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52919.log 2024-09-05T16:49:43.747 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59025.log.gz 2024-09-05T16:49:43.747 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.54515.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46068.log 2024-09-05T16:49:43.747 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54515.log.gz 2024-09-05T16:49:43.747 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.52919.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62304.log 2024-09-05T16:49:43.747 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52919.log.gz 2024-09-05T16:49:43.747 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.46068.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58058.log 2024-09-05T16:49:43.748 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46068.log.gz 2024-09-05T16:49:43.748 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.62304.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73482.log 2024-09-05T16:49:43.748 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62304.log.gz 2024-09-05T16:49:43.748 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.58058.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62766.log 2024-09-05T16:49:43.748 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58058.log.gz 2024-09-05T16:49:43.748 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.73482.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54221.log 2024-09-05T16:49:43.748 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73482.log.gz 2024-09-05T16:49:43.748 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.62766.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45816.log 2024-09-05T16:49:43.748 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62766.log.gz 2024-09-05T16:49:43.748 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.54221.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39216.log 2024-09-05T16:49:43.748 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54221.log.gz 2024-09-05T16:49:43.748 INFO:teuthology.orchestra.run.smithi087.stderr:gzip/var/log/ceph/ceph-client.admin.45816.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.53801.log 2024-09-05T16:49:43.748 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45816.log.gz 2024-09-05T16:49:43.748 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.39216.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45942.log 2024-09-05T16:49:43.749 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39216.log.gz 2024-09-05T16:49:43.749 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.53801.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42298.log 2024-09-05T16:49:43.749 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53801.log.gz 2024-09-05T16:49:43.749 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.45942.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55061.log 2024-09-05T16:49:43.749 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45942.log.gz 2024-09-05T16:49:43.749 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.42298.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42002.log 2024-09-05T16:49:43.749 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42298.log.gz 2024-09-05T16:49:43.749 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.55061.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54893.log 2024-09-05T16:49:43.749 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55061.log.gz 2024-09-05T16:49:43.749 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.42002.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69615.log 2024-09-05T16:49:43.749 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42002.log.gz 2024-09-05T16:49:43.749 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.54893.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54893.log.gz 2024-09-05T16:49:43.749 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60159.log 2024-09-05T16:49:43.749 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.69615.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66338.log 2024-09-05T16:49:43.750 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69615.log.gz 2024-09-05T16:49:43.750 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.60159.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55409.log 2024-09-05T16:49:43.750 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60159.log.gz 2024-09-05T16:49:43.750 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.66338.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65414.log 2024-09-05T16:49:43.750 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66338.log.gz 2024-09-05T16:49:43.750 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40432.log 2024-09-05T16:49:43.750 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.55409.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55409.log.gz 2024-09-05T16:49:43.750 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.65414.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68271.log 2024-09-05T16:49:43.750 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65414.log.gz 2024-09-05T16:49:43.750 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.40432.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44893.log 2024-09-05T16:49:43.750 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40432.log.gz 2024-09-05T16:49:43.750 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.68271.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62724.log 2024-09-05T16:49:43.750 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68271.log.gz 2024-09-05T16:49:43.750 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.44893.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72768.log 2024-09-05T16:49:43.751 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44893.log.gz 2024-09-05T16:49:43.751 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.62724.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61797.log 2024-09-05T16:49:43.751 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62724.log.gz 2024-09-05T16:49:43.751 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.72768.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39527.log 2024-09-05T16:49:43.751 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72768.log.gz 2024-09-05T16:49:43.751 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.61797.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69153.log 2024-09-05T16:49:43.751 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61797.log.gz 2024-09-05T16:49:43.751 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.39527.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70414.log 2024-09-05T16:49:43.751 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39527.log.gz 2024-09-05T16:49:43.751 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73356.log 2024-09-05T16:49:43.751 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.69153.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69153.log.gz 2024-09-05T16:49:43.751 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.70414.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67641.log 2024-09-05T16:49:43.751 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70414.log.gz 2024-09-05T16:49:43.752 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.73356.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57342.log 2024-09-05T16:49:43.752 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73356.log.gz 2024-09-05T16:49:43.752 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.67641.log: /var/log/ceph/ceph-client.admin.47731.log 2024-09-05T16:49:43.752 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67641.log.gz 2024-09-05T16:49:43.752 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.57342.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67263.log 2024-09-05T16:49:43.752 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57342.log.gz 2024-09-05T16:49:43.752 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.47731.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52289.log 2024-09-05T16:49:43.752 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47731.log.gz 2024-09-05T16:49:43.752 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.67263.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41395.log 2024-09-05T16:49:43.752 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67263.log.gz 2024-09-05T16:49:43.752 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.52289.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47123.log 2024-09-05T16:49:43.752 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52289.log.gz 2024-09-05T16:49:43.752 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.41395.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71970.log 2024-09-05T16:49:43.753 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41395.log.gz 2024-09-05T16:49:43.753 INFO:teuthology.orchestra.run.smithi087.stderr:gzip/var/log/ceph/ceph-client.admin.47123.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.47284.log 2024-09-05T16:49:43.753 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47123.log.gz 2024-09-05T16:49:43.753 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.71970.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45502.log 2024-09-05T16:49:43.753 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71970.log.gz 2024-09-05T16:49:43.753 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.47284.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54935.log 2024-09-05T16:49:43.753 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47284.log.gz 2024-09-05T16:49:43.753 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.45502.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42557.log 2024-09-05T16:49:43.753 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45502.log.gz 2024-09-05T16:49:43.753 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.54935.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72642.log 2024-09-05T16:49:43.753 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54935.log.gz 2024-09-05T16:49:43.753 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.42557.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40618.log 2024-09-05T16:49:43.753 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42557.log.gz 2024-09-05T16:49:43.754 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.72642.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58268.log 2024-09-05T16:49:43.754 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72642.log.gz 2024-09-05T16:49:43.754 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.40618.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72852.log 2024-09-05T16:49:43.754 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40618.log.gz 2024-09-05T16:49:43.754 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.58268.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55745.log 2024-09-05T16:49:43.754 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58268.log.gz 2024-09-05T16:49:43.754 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.72852.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51239.log 2024-09-05T16:49:43.754 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72852.log.gz 2024-09-05T16:49:43.754 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.55745.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55745.log.gz 2024-09-05T16:49:43.754 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51743.log 2024-09-05T16:49:43.754 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.51239.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66926.log 2024-09-05T16:49:43.754 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51239.log.gz 2024-09-05T16:49:43.754 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.51743.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51743.log.gz 2024-09-05T16:49:43.755 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67095.log 2024-09-05T16:49:43.755 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.66926.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65960.log 2024-09-05T16:49:43.755 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66926.log.gz 2024-09-05T16:49:43.755 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.67095.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60789.log 2024-09-05T16:49:43.755 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67095.log.gz 2024-09-05T16:49:43.755 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.65960.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39174.log 2024-09-05T16:49:43.755 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65960.log.gz 2024-09-05T16:49:43.755 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.60789.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69195.log 2024-09-05T16:49:43.755 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60789.log.gz 2024-09-05T16:49:43.755 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.39174.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68145.log 2024-09-05T16:49:43.755 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39174.log.gz 2024-09-05T16:49:43.755 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.69195.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72936.log 2024-09-05T16:49:43.755 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69195.log.gz 2024-09-05T16:49:43.756 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.68145.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52457.log 2024-09-05T16:49:43.756 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68145.log.gz 2024-09-05T16:49:43.756 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.72936.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58395.log 2024-09-05T16:49:43.756 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72936.log.gz 2024-09-05T16:49:43.756 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.52457.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50021.log 2024-09-05T16:49:43.756 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52457.log.gz 2024-09-05T16:49:43.756 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71382.log 2024-09-05T16:49:43.756 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.58395.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58395.log.gz 2024-09-05T16:49:43.756 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.50021.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72180.log 2024-09-05T16:49:43.756 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50021.log.gz 2024-09-05T16:49:43.756 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.71382.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63312.log 2024-09-05T16:49:43.756 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71382.log.gz 2024-09-05T16:49:43.757 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.72180.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60831.log 2024-09-05T16:49:43.757 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72180.log.gz 2024-09-05T16:49:43.757 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.63312.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63312.log.gz 2024-09-05T16:49:43.757 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68481.log 2024-09-05T16:49:43.757 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.60831.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65119.log 2024-09-05T16:49:43.757 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60831.log.gz 2024-09-05T16:49:43.757 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.68481.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65077.log 2024-09-05T16:49:43.757 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68481.log.gz 2024-09-05T16:49:43.757 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46698.log 2024-09-05T16:49:43.757 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.65119.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65119.log.gz 2024-09-05T16:49:43.757 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.65077.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57888.log 2024-09-05T16:49:43.757 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65077.log.gz 2024-09-05T16:49:43.757 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.46698.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39132.log 2024-09-05T16:49:43.758 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46698.log.gz 2024-09-05T16:49:43.758 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.57888.log: /var/log/ceph/ceph-client.admin.39678.log 2024-09-05T16:49:43.758 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57888.log.gz 2024-09-05T16:49:43.758 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.39132.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70877.log 2024-09-05T16:49:43.758 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39132.log.gz 2024-09-05T16:49:43.758 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.39678.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71130.log 2024-09-05T16:49:43.758 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39678.log.gz 2024-09-05T16:49:43.758 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.70877.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-05T16:49:43.758 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70877.log.gz 2024-09-05T16:49:43.758 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.71130.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39074.log 2024-09-05T16:49:43.758 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71130.log.gz 2024-09-05T16:49:43.758 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60957.log 2024-09-05T16:49:43.758 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69826.log 2024-09-05T16:49:43.758 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.39074.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39074.log.gz 2024-09-05T16:49:43.759 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.60957.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46908.log 2024-09-05T16:49:43.759 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60957.log.gz 2024-09-05T16:49:43.759 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.69826.log: gzip 0.0% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.69826.log.gz -- 2024-09-05T16:49:43.759 INFO:teuthology.orchestra.run.smithi087.stderr: /var/log/ceph/ceph-client.admin.50231.log 2024-09-05T16:49:43.759 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.46908.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52583.log 2024-09-05T16:49:43.759 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46908.log.gz 2024-09-05T16:49:43.759 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.50231.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67137.log 2024-09-05T16:49:43.759 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50231.log.gz 2024-09-05T16:49:43.759 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.52583.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52583.log.gz 2024-09-05T16:49:43.759 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65666.log 2024-09-05T16:49:43.759 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.67137.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72348.log 2024-09-05T16:49:43.759 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67137.log.gz 2024-09-05T16:49:43.760 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.65666.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70330.log 2024-09-05T16:49:43.760 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65666.log.gz 2024-09-05T16:49:43.760 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.72348.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55787.log 2024-09-05T16:49:43.760 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72348.log.gz 2024-09-05T16:49:43.760 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.70330.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61755.log 2024-09-05T16:49:43.760 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70330.log.gz 2024-09-05T16:49:43.760 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.55787.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63018.log 2024-09-05T16:49:43.760 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55787.log.gz 2024-09-05T16:49:43.760 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.61755.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51701.log 2024-09-05T16:49:43.760 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61755.log.gz 2024-09-05T16:49:43.760 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56291.log 2024-09-05T16:49:43.760 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.63018.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63018.log.gz 2024-09-05T16:49:43.760 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50987.log 2024-09-05T16:49:43.761 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.51701.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51701.log.gz 2024-09-05T16:49:43.761 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.56291.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56291.log.gz 2024-09-05T16:49:43.761 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59865.log 2024-09-05T16:49:43.761 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.50987.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40348.log 2024-09-05T16:49:43.761 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50987.log.gz 2024-09-05T16:49:43.761 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.59865.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59865.log.gz 2024-09-05T16:49:43.761 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51869.log 2024-09-05T16:49:43.761 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.40348.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75582.log 2024-09-05T16:49:43.761 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40348.log.gz 2024-09-05T16:49:43.761 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63144.log 2024-09-05T16:49:43.761 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.51869.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51869.log.gz 2024-09-05T16:49:43.761 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.75582.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61293.log 2024-09-05T16:49:43.762 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.75582.log.gz 2024-09-05T16:49:43.762 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.63144.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46404.log 2024-09-05T16:49:43.762 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63144.log.gz 2024-09-05T16:49:43.762 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67389.log 2024-09-05T16:49:43.762 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.61293.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61293.log.gz 2024-09-05T16:49:43.762 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.46404.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53633.log 2024-09-05T16:49:43.762 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46404.log.gz 2024-09-05T16:49:43.762 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.67389.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58437.log 2024-09-05T16:49:43.762 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67389.log.gz 2024-09-05T16:49:43.762 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.53633.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71676.log 2024-09-05T16:49:43.762 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53633.log.gz 2024-09-05T16:49:43.763 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.58437.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61503.log 2024-09-05T16:49:43.763 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58437.log.gz 2024-09-05T16:49:43.763 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.71676.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48101.log 2024-09-05T16:49:43.763 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71676.log.gz 2024-09-05T16:49:43.763 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73398.log 2024-09-05T16:49:43.763 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.61503.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61503.log.gz 2024-09-05T16:49:43.763 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.48101.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44980.log 2024-09-05T16:49:43.763 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48101.log.gz 2024-09-05T16:49:43.763 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.73398.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69447.log 2024-09-05T16:49:43.763 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73398.log.gz 2024-09-05T16:49:43.763 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.44980.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51491.log 2024-09-05T16:49:43.763 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44980.log.gz 2024-09-05T16:49:43.763 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.69447.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71256.log 2024-09-05T16:49:43.764 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69447.log.gz 2024-09-05T16:49:43.764 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67473.log 2024-09-05T16:49:43.764 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.51491.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51491.log.gz 2024-09-05T16:49:43.764 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40729.log 2024-09-05T16:49:43.764 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.71256.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71256.log.gz 2024-09-05T16:49:43.764 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.67473.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61083.log 2024-09-05T16:49:43.764 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67473.log.gz 2024-09-05T16:49:43.764 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.40729.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.40729.log.gz -5 2024-09-05T16:49:43.764 INFO:teuthology.orchestra.run.smithi087.stderr: --verbose -- /var/log/ceph/ceph-client.admin.64068.log 2024-09-05T16:49:43.764 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45356.log 2024-09-05T16:49:43.764 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.61083.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61083.log.gz 2024-09-05T16:49:43.764 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.64068.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73440.log 2024-09-05T16:49:43.765 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64068.log.gz 2024-09-05T16:49:43.765 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.45356.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61671.log 2024-09-05T16:49:43.765 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45356.log.gz 2024-09-05T16:49:43.765 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.73440.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47472.log 2024-09-05T16:49:43.765 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73440.log.gz 2024-09-05T16:49:43.765 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.61671.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72726.log 2024-09-05T16:49:43.765 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61671.log.gz 2024-09-05T16:49:43.765 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.47472.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71844.log 2024-09-05T16:49:43.765 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47472.log.gz 2024-09-05T16:49:43.765 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.72726.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59151.log 2024-09-05T16:49:43.765 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72726.log.gz 2024-09-05T16:49:43.765 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.71844.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54053.log 2024-09-05T16:49:43.766 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71844.log.gz 2024-09-05T16:49:43.766 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.59151.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75540.log 2024-09-05T16:49:43.766 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59151.log.gz 2024-09-05T16:49:43.766 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.54053.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57468.log 2024-09-05T16:49:43.766 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54053.log.gz 2024-09-05T16:49:43.766 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.75540.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62808.log 2024-09-05T16:49:43.766 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.75540.log.gz 2024-09-05T16:49:43.766 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.57468.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47412.log 2024-09-05T16:49:43.766 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57468.log.gz 2024-09-05T16:49:43.766 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.62808.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76002.log 2024-09-05T16:49:43.766 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62808.log.gz 2024-09-05T16:49:43.767 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.47412.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49769.log 2024-09-05T16:49:43.767 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47412.log.gz 2024-09-05T16:49:43.767 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.76002.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58479.log 2024-09-05T16:49:43.767 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.76002.log.gz 2024-09-05T16:49:43.767 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.49769.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64867.log 2024-09-05T16:49:43.767 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49769.log.gz 2024-09-05T16:49:43.767 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.58479.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51113.log 2024-09-05T16:49:43.767 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58479.log.gz 2024-09-05T16:49:43.767 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.64867.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55619.log 2024-09-05T16:49:43.767 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64867.log.gz 2024-09-05T16:49:43.767 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.51113.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41173.log 2024-09-05T16:49:43.767 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51113.log.gz 2024-09-05T16:49:43.767 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.55619.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69363.log 2024-09-05T16:49:43.768 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55619.log.gz 2024-09-05T16:49:43.768 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.41173.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58647.log 2024-09-05T16:49:43.768 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41173.log.gz 2024-09-05T16:49:43.768 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.69363.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50903.log 2024-09-05T16:49:43.768 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69363.log.gz 2024-09-05T16:49:43.768 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.58647.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66170.log 2024-09-05T16:49:43.768 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58647.log.gz 2024-09-05T16:49:43.768 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.50903.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42631.log 2024-09-05T16:49:43.768 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50903.log.gz 2024-09-05T16:49:43.768 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.66170.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65750.log 2024-09-05T16:49:43.768 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66170.log.gz 2024-09-05T16:49:43.768 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.42631.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41284.log 2024-09-05T16:49:43.768 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42631.log.gz 2024-09-05T16:49:43.769 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.65750.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48138.log 2024-09-05T16:49:43.769 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65750.log.gz 2024-09-05T16:49:43.769 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.41284.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70204.log 2024-09-05T16:49:43.769 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41284.log.gz 2024-09-05T16:49:43.769 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.48138.log: gzip -5 --verbose 0.0% -- /var/log/ceph/ceph-client.admin.72684.log -- replaced with /var/log/ceph/ceph-client.admin.48138.log.gz 2024-09-05T16:49:43.769 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-09-05T16:49:43.769 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.70204.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74238.log 2024-09-05T16:49:43.769 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70204.log.gz 2024-09-05T16:49:43.769 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.72684.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39528.log 2024-09-05T16:49:43.769 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72684.log.gz 2024-09-05T16:49:43.769 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.74238.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40877.log 2024-09-05T16:49:43.769 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74238.log.gz 2024-09-05T16:49:43.770 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64447.log 2024-09-05T16:49:43.770 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.39528.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39528.log.gz 2024-09-05T16:49:43.770 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.40877.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67893.log 2024-09-05T16:49:43.770 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40877.log.gz 2024-09-05T16:49:43.770 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.64447.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64405.log 2024-09-05T16:49:43.770 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64447.log.gz 2024-09-05T16:49:43.770 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.67893.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74952.log 2024-09-05T16:49:43.770 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67893.log.gz 2024-09-05T16:49:43.770 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.64405.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54809.log 2024-09-05T16:49:43.770 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64405.log.gz 2024-09-05T16:49:43.770 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.74952.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41641.log 2024-09-05T16:49:43.770 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74952.log.gz 2024-09-05T16:49:43.771 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.54809.log: gzip -5 --verbose 0.0% -- -- replaced with /var/log/ceph/ceph-client.admin.54809.log.gz /var/log/ceph/ceph-client.admin.67557.log 2024-09-05T16:49:43.771 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-09-05T16:49:43.771 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.41641.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33558.log 2024-09-05T16:49:43.771 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41641.log.gz 2024-09-05T16:49:43.771 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.67557.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.39358.log 2024-09-05T16:49:43.771 INFO:teuthology.orchestra.run.smithi087.stderr: -- replaced with /var/log/ceph/ceph-client.admin.67557.log.gz 2024-09-05T16:49:43.771 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.33558.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57384.log 2024-09-05T16:49:43.771 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33558.log.gz 2024-09-05T16:49:43.771 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.39358.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39358.log.gz 2024-09-05T16:49:43.771 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71466.log 2024-09-05T16:49:43.771 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.57384.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63984.log 2024-09-05T16:49:43.771 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57384.log.gz 2024-09-05T16:49:43.771 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.71466.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59403.log 2024-09-05T16:49:43.772 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71466.log.gz 2024-09-05T16:49:43.772 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.63984.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51449.log 2024-09-05T16:49:43.772 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63984.log.gz 2024-09-05T16:49:43.772 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.59403.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39359.log 2024-09-05T16:49:43.772 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59403.log.gz 2024-09-05T16:49:43.772 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.51449.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47916.log 2024-09-05T16:49:43.772 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51449.log.gz 2024-09-05T16:49:43.772 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.39359.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39359.log.gz 2024-09-05T16:49:43.772 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41891.log 2024-09-05T16:49:43.772 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.47916.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51785.log 2024-09-05T16:49:43.772 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47916.log.gz 2024-09-05T16:49:43.772 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.41891.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69699.log 2024-09-05T16:49:43.773 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41891.log.gz 2024-09-05T16:49:43.773 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.51785.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64194.log 2024-09-05T16:49:43.773 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51785.log.gz 2024-09-05T16:49:43.773 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.69699.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67179.log 2024-09-05T16:49:43.773 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69699.log.gz 2024-09-05T16:49:43.773 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.64194.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39846.log 2024-09-05T16:49:43.773 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64194.log.gz 2024-09-05T16:49:43.773 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.67179.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67179.log.gz 2024-09-05T16:49:43.773 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74784.log 2024-09-05T16:49:43.773 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.39846.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33991.log 2024-09-05T16:49:43.773 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39846.log.gz 2024-09-05T16:49:43.773 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.74784.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64909.log 2024-09-05T16:49:43.773 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74784.log.gz 2024-09-05T16:49:43.774 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.33991.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41247.log 2024-09-05T16:49:43.774 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33991.log.gz 2024-09-05T16:49:43.774 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.64909.log: gzip -5 --verbose 0.0% -- /var/log/ceph/ceph-client.admin.52415.log -- replaced with /var/log/ceph/ceph-client.admin.64909.log.gz 2024-09-05T16:49:43.774 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-09-05T16:49:43.774 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.41247.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61167.log 2024-09-05T16:49:43.774 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41247.log.gz 2024-09-05T16:49:43.774 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.52415.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74070.log 2024-09-05T16:49:43.774 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52415.log.gz 2024-09-05T16:49:43.774 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.61167.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69321.log 2024-09-05T16:49:43.774 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61167.log.gz 2024-09-05T16:49:43.774 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.74070.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63900.log 2024-09-05T16:49:43.774 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74070.log.gz 2024-09-05T16:49:43.775 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.69321.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51323.log 2024-09-05T16:49:43.775 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69321.log.gz 2024-09-05T16:49:43.775 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.63900.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53507.log 2024-09-05T16:49:43.775 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63900.log.gz 2024-09-05T16:49:43.775 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72096.log 2024-09-05T16:49:43.775 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.51323.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51323.log.gz 2024-09-05T16:49:43.775 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.53507.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67683.log 2024-09-05T16:49:43.775 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53507.log.gz 2024-09-05T16:49:43.775 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.72096.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65287.log 2024-09-05T16:49:43.775 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72096.log.gz 2024-09-05T16:49:43.775 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.67683.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54263.log 2024-09-05T16:49:43.775 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67683.log.gz 2024-09-05T16:49:43.775 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.65287.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54011.log 2024-09-05T16:49:43.775 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65287.log.gz 2024-09-05T16:49:43.776 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.54263.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40116.log 2024-09-05T16:49:43.776 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54263.log.gz 2024-09-05T16:49:43.776 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.54011.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70750.log 2024-09-05T16:49:43.776 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54011.log.gz 2024-09-05T16:49:43.776 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.40116.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53843.log 2024-09-05T16:49:43.776 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40116.log.gz 2024-09-05T16:49:43.776 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.70750.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62178.log 2024-09-05T16:49:43.776 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70750.log.gz 2024-09-05T16:49:43.776 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.53843.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59109.log 2024-09-05T16:49:43.776 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53843.log.gz 2024-09-05T16:49:43.776 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.62178.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51659.log 2024-09-05T16:49:43.776 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62178.log.gz 2024-09-05T16:49:43.776 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.59109.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59109.log.gz 2024-09-05T16:49:43.776 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69573.log 2024-09-05T16:49:43.776 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.51659.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71003.log 2024-09-05T16:49:43.777 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51659.log.gz 2024-09-05T16:49:43.777 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.69573.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33951.log 2024-09-05T16:49:43.777 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69573.log.gz 2024-09-05T16:49:43.777 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.71003.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61926.log 2024-09-05T16:49:43.777 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71003.log.gz 2024-09-05T16:49:43.777 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.33951.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33951.log.gz 2024-09-05T16:49:43.777 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48064.log 2024-09-05T16:49:43.777 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.61926.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-09-05T16:49:43.777 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61926.log.gz 2024-09-05T16:49:43.777 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.48064.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56333.log 2024-09-05T16:49:43.777 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48064.log.gz 2024-09-05T16:49:43.777 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56753.log 2024-09-05T16:49:43.777 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.56333.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52751.log 2024-09-05T16:49:43.777 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56333.log.gz 2024-09-05T16:49:43.778 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.56753.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44766.log 2024-09-05T16:49:43.778 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56753.log.gz 2024-09-05T16:49:43.778 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.52751.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46530.log 2024-09-05T16:49:43.778 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52751.log.gz 2024-09-05T16:49:43.778 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.44766.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62976.log 2024-09-05T16:49:43.778 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44766.log.gz 2024-09-05T16:49:43.778 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.46530.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63690.log 2024-09-05T16:49:43.778 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46530.log.gz 2024-09-05T16:49:43.778 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.62976.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60495.log 2024-09-05T16:49:43.778 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62976.log.gz 2024-09-05T16:49:43.778 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.63690.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41743.log 2024-09-05T16:49:43.778 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63690.log.gz 2024-09-05T16:49:43.778 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.60495.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70078.log 2024-09-05T16:49:43.779 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60495.log.gz 2024-09-05T16:49:43.779 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.41743.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69279.log 2024-09-05T16:49:43.779 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41743.log.gz 2024-09-05T16:49:43.779 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.70078.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58310.log 2024-09-05T16:49:43.779 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70078.log.gz 2024-09-05T16:49:43.779 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.69279.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57930.log 2024-09-05T16:49:43.779 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69279.log.gz 2024-09-05T16:49:43.779 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.58310.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59193.log 2024-09-05T16:49:43.779 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58310.log.gz 2024-09-05T16:49:43.779 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.57930.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40200.log 2024-09-05T16:49:43.779 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57930.log.gz 2024-09-05T16:49:43.779 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.59193.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69111.log 2024-09-05T16:49:43.779 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59193.log.gz 2024-09-05T16:49:43.780 INFO:teuthology.orchestra.run.smithi087.stderr: 96.1% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-05T16:49:43.780 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.40200.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.40200.log.gz 2024-09-05T16:49:43.780 INFO:teuthology.orchestra.run.smithi087.stderr: --verbose -- /var/log/ceph/ceph-client.admin.49517.log 2024-09-05T16:49:43.780 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.69111.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75204.log 2024-09-05T16:49:43.780 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69111.log.gz 2024-09-05T16:49:43.780 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.49517.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45228.log 2024-09-05T16:49:43.780 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49517.log.gz 2024-09-05T16:49:43.780 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.75204.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68187.log 2024-09-05T16:49:43.780 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.75204.log.gz 2024-09-05T16:49:43.780 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.45228.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50273.log 2024-09-05T16:49:43.780 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45228.log.gz 2024-09-05T16:49:43.780 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.68187.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57762.log 2024-09-05T16:49:43.780 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68187.log.gz 2024-09-05T16:49:43.781 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.50273.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63858.log 2024-09-05T16:49:43.781 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50273.log.gz 2024-09-05T16:49:43.781 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.57762.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67347.log 2024-09-05T16:49:43.781 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57762.log.gz 2024-09-05T16:49:43.781 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.63858.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64489.log 2024-09-05T16:49:43.781 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63858.log.gz 2024-09-05T16:49:43.781 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.67347.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64825.log 2024-09-05T16:49:43.781 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67347.log.gz 2024-09-05T16:49:43.781 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.64489.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68103.log 2024-09-05T16:49:43.781 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64489.log.gz 2024-09-05T16:49:43.781 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.64825.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68733.log 2024-09-05T16:49:43.781 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64825.log.gz 2024-09-05T16:49:43.781 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49349.log 2024-09-05T16:49:43.782 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.68103.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68103.log.gz 2024-09-05T16:49:43.782 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.68733.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60243.log 2024-09-05T16:49:43.782 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68733.log.gz 2024-09-05T16:49:43.782 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.49349.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49349.log.gz 2024-09-05T16:49:43.782 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66086.log 2024-09-05T16:49:43.782 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.60243.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58521.log 2024-09-05T16:49:43.782 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60243.log.gz 2024-09-05T16:49:43.782 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.66086.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66086.log.gz 2024-09-05T16:49:43.782 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58184.log 2024-09-05T16:49:43.782 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.58521.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47264.log 2024-09-05T16:49:43.782 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58521.log.gz 2024-09-05T16:49:43.782 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.58184.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.75078.log 2024-09-05T16:49:43.782 INFO:teuthology.orchestra.run.smithi087.stderr: -- replaced with /var/log/ceph/ceph-client.admin.58184.log.gz 2024-09-05T16:49:43.782 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.47264.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71298.log 2024-09-05T16:49:43.783 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47264.log.gz 2024-09-05T16:49:43.783 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.75078.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58857.log 2024-09-05T16:49:43.783 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.75078.log.gz 2024-09-05T16:49:43.783 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.71298.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68775.log 2024-09-05T16:49:43.783 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71298.log.gz 2024-09-05T16:49:43.783 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.58857.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46950.log 2024-09-05T16:49:43.783 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58857.log.gz 2024-09-05T16:49:43.783 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.68775.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60999.log 2024-09-05T16:49:43.783 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68775.log.gz 2024-09-05T16:49:43.783 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.46950.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67053.log 2024-09-05T16:49:43.783 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46950.log.gz 2024-09-05T16:49:43.783 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.60999.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46236.log 2024-09-05T16:49:43.783 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60999.log.gz 2024-09-05T16:49:43.784 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.67053.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58773.log 2024-09-05T16:49:43.784 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67053.log.gz 2024-09-05T16:49:43.784 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.46236.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41543.log 2024-09-05T16:49:43.784 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46236.log.gz 2024-09-05T16:49:43.784 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.58773.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73902.log 2024-09-05T16:49:43.784 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58773.log.gz 2024-09-05T16:49:43.784 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.41543.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57720.log 2024-09-05T16:49:43.784 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41543.log.gz 2024-09-05T16:49:43.784 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.73902.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49643.log 2024-09-05T16:49:43.784 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73902.log.gz 2024-09-05T16:49:43.784 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.57720.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60201.log 2024-09-05T16:49:43.784 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57720.log.gz 2024-09-05T16:49:43.784 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.49643.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39804.log 2024-09-05T16:49:43.784 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49643.log.gz 2024-09-05T16:49:43.785 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.60201.log: gzip -5 --verbose 0.0% -- -- replaced with /var/log/ceph/ceph-client.admin.60201.log.gz /var/log/ceph/ceph-client.admin.51029.log 2024-09-05T16:49:43.785 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-09-05T16:49:43.785 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.39804.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47657.log 2024-09-05T16:49:43.785 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39804.log.gz 2024-09-05T16:49:43.785 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.51029.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53759.log 2024-09-05T16:49:43.785 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51029.log.gz 2024-09-05T16:49:43.785 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.47657.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.47657.log.gz -5 2024-09-05T16:49:43.785 INFO:teuthology.orchestra.run.smithi087.stderr: --verbose -- /var/log/ceph/ceph-client.admin.54137.log 2024-09-05T16:49:43.785 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.53759.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.46446.log 2024-09-05T16:49:43.785 INFO:teuthology.orchestra.run.smithi087.stderr: -- replaced with /var/log/ceph/ceph-client.admin.53759.log.gz 2024-09-05T16:49:43.785 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.54137.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42224.log 2024-09-05T16:49:43.785 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54137.log.gz 2024-09-05T16:49:43.785 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.46446.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67767.log 2024-09-05T16:49:43.786 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46446.log.gz 2024-09-05T16:49:43.786 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.42224.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50651.log 2024-09-05T16:49:43.786 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42224.log.gz 2024-09-05T16:49:43.786 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.67767.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40914.log 2024-09-05T16:49:43.786 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67767.log.gz 2024-09-05T16:49:43.786 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.50651.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53297.log 2024-09-05T16:49:43.786 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50651.log.gz 2024-09-05T16:49:43.786 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.40914.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50735.log 2024-09-05T16:49:43.786 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40914.log.gz 2024-09-05T16:49:43.786 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71634.log 2024-09-05T16:49:43.786 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.53297.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53297.log.gz 2024-09-05T16:49:43.786 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.50735.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68397.log 2024-09-05T16:49:43.786 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50735.log.gz 2024-09-05T16:49:43.786 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.71634.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54095.log 2024-09-05T16:49:43.787 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71634.log.gz 2024-09-05T16:49:43.787 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.68397.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60075.log 2024-09-05T16:49:43.787 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68397.log.gz 2024-09-05T16:49:43.787 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.54095.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44768.log 2024-09-05T16:49:43.787 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54095.log.gz 2024-09-05T16:49:43.787 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.60075.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44767.log 2024-09-05T16:49:43.787 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60075.log.gz 2024-09-05T16:49:43.787 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.44768.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66506.log 2024-09-05T16:49:43.787 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44768.log.gz 2024-09-05T16:49:43.787 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.44767.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65498.log 2024-09-05T16:49:43.787 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44767.log.gz 2024-09-05T16:49:43.787 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.66506.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72810.log 2024-09-05T16:49:43.787 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66506.log.gz 2024-09-05T16:49:43.788 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.65498.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48027.log 2024-09-05T16:49:43.788 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65498.log.gz 2024-09-05T16:49:43.788 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.72810.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69910.log 2024-09-05T16:49:43.788 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72810.log.gz 2024-09-05T16:49:43.788 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.48027.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40516.log 2024-09-05T16:49:43.788 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48027.log.gz 2024-09-05T16:49:43.788 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.69910.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59319.log 2024-09-05T16:49:43.788 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69910.log.gz 2024-09-05T16:49:43.788 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.40516.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46824.log 2024-09-05T16:49:43.788 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40516.log.gz 2024-09-05T16:49:43.788 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.59319.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63102.log 2024-09-05T16:49:43.788 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59319.log.gz 2024-09-05T16:49:43.788 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.46824.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40692.log 2024-09-05T16:49:43.789 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46824.log.gz 2024-09-05T16:49:43.789 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.63102.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75750.log 2024-09-05T16:49:43.789 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63102.log.gz 2024-09-05T16:49:43.789 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.40692.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40692.log.gz 2024-09-05T16:49:43.789 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62514.log 2024-09-05T16:49:43.789 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.75750.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59235.log 2024-09-05T16:49:43.789 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.75750.log.gz 2024-09-05T16:49:43.789 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.62514.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49979.log 2024-09-05T16:49:43.789 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62514.log.gz 2024-09-05T16:49:43.789 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.59235.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-09-05T16:49:43.789 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59235.log.gz 2024-09-05T16:49:43.789 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.49979.log: gzip 0.0% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.49979.log.gz -- 2024-09-05T16:49:43.789 INFO:teuthology.orchestra.run.smithi087.stderr: /var/log/ceph/ceph-client.admin.56921.log 2024-09-05T16:49:43.789 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75456.log 2024-09-05T16:49:43.790 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.56921.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50483.log 2024-09-05T16:49:43.790 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56921.log.gz 2024-09-05T16:49:43.790 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.75456.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59487.log 2024-09-05T16:49:43.790 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.75456.log.gz 2024-09-05T16:49:43.790 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.50483.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.27835.log 2024-09-05T16:49:43.790 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50483.log.gz 2024-09-05T16:49:43.790 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.59487.log: gzip 0.0% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.59487.log.gz -- 2024-09-05T16:49:43.790 INFO:teuthology.orchestra.run.smithi087.stderr: /var/log/ceph/ceph-client.admin.66590.log 2024-09-05T16:49:43.790 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph.tmp-client.admin.27835.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65834.log 2024-09-05T16:49:43.790 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.27835.log.gz 2024-09-05T16:49:43.790 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.66590.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55241.log 2024-09-05T16:49:43.790 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66590.log.gz 2024-09-05T16:49:43.790 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.65834.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49391.log 2024-09-05T16:49:43.790 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65834.log.gz 2024-09-05T16:49:43.791 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.55241.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55241.log.gz 2024-09-05T16:49:43.791 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33855.log 2024-09-05T16:49:43.791 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.49391.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69531.log 2024-09-05T16:49:43.791 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49391.log.gz 2024-09-05T16:49:43.791 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.33855.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41469.log 2024-09-05T16:49:43.791 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33855.log.gz 2024-09-05T16:49:43.791 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.69531.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74448.log 2024-09-05T16:49:43.791 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69531.log.gz 2024-09-05T16:49:43.791 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.41469.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60705.log 2024-09-05T16:49:43.791 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41469.log.gz 2024-09-05T16:49:43.791 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.74448.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69405.log 2024-09-05T16:49:43.791 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74448.log.gz 2024-09-05T16:49:43.791 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.60705.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50819.log 2024-09-05T16:49:43.791 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60705.log.gz 2024-09-05T16:49:43.792 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.69405.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57258.log 2024-09-05T16:49:43.792 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69405.log.gz 2024-09-05T16:49:43.792 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.50819.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50819.log.gz 2024-09-05T16:49:43.792 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69741.log 2024-09-05T16:49:43.792 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.57258.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62934.log 2024-09-05T16:49:43.792 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57258.log.gz 2024-09-05T16:49:43.792 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.69741.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74406.log 2024-09-05T16:49:43.792 INFO:teuthology.orchestra.run.smithi087.stderr: 25.6% -- replaced with /var/log/ceph/ceph-client.admin.69741.log.gz 2024-09-05T16:49:43.792 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.62934.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49559.log 2024-09-05T16:49:43.792 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62934.log.gz 2024-09-05T16:49:43.792 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.74406.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53339.log 2024-09-05T16:49:43.792 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74406.log.gz 2024-09-05T16:49:43.792 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.49559.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48212.log 2024-09-05T16:49:43.793 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49559.log.gz 2024-09-05T16:49:43.793 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.53339.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63228.log 2024-09-05T16:49:43.793 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53339.log.gz 2024-09-05T16:49:43.793 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.48212.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71760.log 2024-09-05T16:49:43.793 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48212.log.gz 2024-09-05T16:49:43.793 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56837.log/var/log/ceph/ceph-client.admin.63228.log: 2024-09-05T16:49:43.793 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63228.log.gz 2024-09-05T16:49:43.793 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.71760.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52331.log 2024-09-05T16:49:43.793 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71760.log.gz 2024-09-05T16:49:43.793 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.56837.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70666.log 2024-09-05T16:49:43.793 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56837.log.gz 2024-09-05T16:49:43.793 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60747.log 2024-09-05T16:49:43.793 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.52331.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52331.log.gz 2024-09-05T16:49:43.793 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.70666.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57426.log 2024-09-05T16:49:43.794 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70666.log.gz 2024-09-05T16:49:43.794 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.60747.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71886.log 2024-09-05T16:49:43.794 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60747.log.gz 2024-09-05T16:49:43.794 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55955.log 2024-09-05T16:49:43.794 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.57426.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57426.log.gz 2024-09-05T16:49:43.794 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.71886.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33734.log 2024-09-05T16:49:43.794 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71886.log.gz 2024-09-05T16:49:43.794 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.55955.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60663.log 2024-09-05T16:49:43.794 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55955.log.gz 2024-09-05T16:49:43.794 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.33734.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41780.log 2024-09-05T16:49:43.794 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33734.log.gz 2024-09-05T16:49:43.794 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54389.log 2024-09-05T16:49:43.794 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.60663.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60663.log.gz 2024-09-05T16:49:43.794 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.41780.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70372.log 2024-09-05T16:49:43.795 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41780.log.gz 2024-09-05T16:49:43.795 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.54389.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70456.log 2024-09-05T16:49:43.795 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54389.log.gz 2024-09-05T16:49:43.795 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.70372.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49685.log 2024-09-05T16:49:43.795 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70372.log.gz 2024-09-05T16:49:43.795 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.70456.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70456.log.gz 2024-09-05T16:49:43.795 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68943.log 2024-09-05T16:49:43.795 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.49685.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62430.log 2024-09-05T16:49:43.795 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49685.log.gz 2024-09-05T16:49:43.795 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.68943.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57636.log 2024-09-05T16:49:43.795 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68943.log.gz 2024-09-05T16:49:43.795 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.62430.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71592.log 2024-09-05T16:49:43.796 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62430.log.gz 2024-09-05T16:49:43.796 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.57636.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.46782.log 2024-09-05T16:49:43.796 INFO:teuthology.orchestra.run.smithi087.stderr: -- replaced with /var/log/ceph/ceph-client.admin.57636.log.gz 2024-09-05T16:49:43.796 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.71592.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66548.log 2024-09-05T16:49:43.796 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71592.log.gz 2024-09-05T16:49:43.796 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.46782.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46488.log 2024-09-05T16:49:43.796 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46782.log.gz 2024-09-05T16:49:43.796 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.66548.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-09-05T16:49:43.796 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66548.log.gz 2024-09-05T16:49:43.796 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46614.log 2024-09-05T16:49:43.796 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.46488.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46488.log.gz 2024-09-05T16:49:43.796 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45690.log 2024-09-05T16:49:43.796 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.46614.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75666.log 2024-09-05T16:49:43.797 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46614.log.gz 2024-09-05T16:49:43.797 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.45690.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.45690.log.gz 2024-09-05T16:49:43.797 INFO:teuthology.orchestra.run.smithi087.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.40115.log 2024-09-05T16:49:43.797 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.75666.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73944.log 2024-09-05T16:49:43.797 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.75666.log.gz 2024-09-05T16:49:43.797 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.40115.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.72222.log 2024-09-05T16:49:43.797 INFO:teuthology.orchestra.run.smithi087.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40115.log.gz 2024-09-05T16:49:43.797 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.73944.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59277.log 2024-09-05T16:49:43.797 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73944.log.gz 2024-09-05T16:49:43.797 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.72222.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72222.log.gz 2024-09-05T16:49:43.797 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55661.log 2024-09-05T16:49:43.797 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.59277.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75372.log 2024-09-05T16:49:43.798 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59277.log.gz 2024-09-05T16:49:43.798 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.55661.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62388.log 2024-09-05T16:49:43.798 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55661.log.gz 2024-09-05T16:49:43.798 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.75372.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41854.log 2024-09-05T16:49:43.798 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.75372.log.gz 2024-09-05T16:49:43.798 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.62388.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54683.log 2024-09-05T16:49:43.798 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62388.log.gz 2024-09-05T16:49:43.798 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.41854.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53003.log 2024-09-05T16:49:43.798 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41854.log.gz 2024-09-05T16:49:43.798 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.54683.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41965.log 2024-09-05T16:49:43.798 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54683.log.gz 2024-09-05T16:49:43.798 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.53003.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61041.log 2024-09-05T16:49:43.798 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53003.log.gz 2024-09-05T16:49:43.799 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.41965.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.48249.log 2024-09-05T16:49:43.799 INFO:teuthology.orchestra.run.smithi087.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41965.log.gz 2024-09-05T16:49:43.799 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.61041.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54977.log 2024-09-05T16:49:43.799 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61041.log.gz 2024-09-05T16:49:43.799 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.48249.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56627.log 2024-09-05T16:49:43.799 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48249.log.gz 2024-09-05T16:49:43.799 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.54977.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75624.log 2024-09-05T16:49:43.799 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54977.log.gz 2024-09-05T16:49:43.799 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34109.log 2024-09-05T16:49:43.799 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.56627.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56627.log.gz 2024-09-05T16:49:43.799 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.75624.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71508.log 2024-09-05T16:49:43.799 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.75624.log.gz 2024-09-05T16:49:43.799 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.34109.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55157.log 2024-09-05T16:49:43.799 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34109.log.gz 2024-09-05T16:49:43.800 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42520.log 2024-09-05T16:49:43.800 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.71508.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71508.log.gz 2024-09-05T16:49:43.800 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.55157.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73146.log 2024-09-05T16:49:43.800 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55157.log.gz 2024-09-05T16:49:43.800 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.42520.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61545.log 2024-09-05T16:49:43.800 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42520.log.gz 2024-09-05T16:49:43.800 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.73146.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49475.log 2024-09-05T16:49:43.800 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73146.log.gz 2024-09-05T16:49:43.800 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.61545.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46152.log 2024-09-05T16:49:43.800 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61545.log.gz 2024-09-05T16:49:43.800 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.49475.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58226.log 2024-09-05T16:49:43.800 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49475.log.gz 2024-09-05T16:49:43.801 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.46152.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74322.log 2024-09-05T16:49:43.801 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46152.log.gz 2024-09-05T16:49:43.801 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.58226.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70540.log 2024-09-05T16:49:43.801 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58226.log.gz 2024-09-05T16:49:43.801 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.74322.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74574.log 2024-09-05T16:49:43.801 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74322.log.gz 2024-09-05T16:49:43.801 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.70540.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51911.log 2024-09-05T16:49:43.801 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70540.log.gz 2024-09-05T16:49:43.801 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.74574.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40951.log 2024-09-05T16:49:43.801 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74574.log.gz 2024-09-05T16:49:43.801 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.51911.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59655.log 2024-09-05T16:49:43.801 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51911.log.gz 2024-09-05T16:49:43.801 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.40951.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58353.log 2024-09-05T16:49:43.802 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40951.log.gz 2024-09-05T16:49:43.802 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.59655.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51995.log 2024-09-05T16:49:43.802 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59655.log.gz 2024-09-05T16:49:43.802 INFO:teuthology.orchestra.run.smithi087.stderr:gzip/var/log/ceph/ceph-client.admin.58353.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.75918.log 2024-09-05T16:49:43.802 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58353.log.gz 2024-09-05T16:49:43.802 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.51995.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62010.log 2024-09-05T16:49:43.802 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51995.log.gz 2024-09-05T16:49:43.802 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.75918.log: -- /var/log/ceph/ceph-client.admin.57678.log 2024-09-05T16:49:43.802 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.75918.log.gz 2024-09-05T16:49:43.802 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.62010.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63060.log 2024-09-05T16:49:43.802 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62010.log.gz 2024-09-05T16:49:43.802 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.57678.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.57678.log.gz --verbose 2024-09-05T16:49:43.802 INFO:teuthology.orchestra.run.smithi087.stderr: -- /var/log/ceph/ceph-client.admin.54767.log 2024-09-05T16:49:43.803 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39360.log 2024-09-05T16:49:43.803 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.63060.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63060.log.gz 2024-09-05T16:49:43.803 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.54767.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74364.log 2024-09-05T16:49:43.803 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54767.log.gz 2024-09-05T16:49:43.803 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.39360.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46740.log 2024-09-05T16:49:43.803 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39360.log.gz 2024-09-05T16:49:43.803 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.74364.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66254.log 2024-09-05T16:49:43.803 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74364.log.gz 2024-09-05T16:49:43.803 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.46740.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73230.log 2024-09-05T16:49:43.803 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46740.log.gz 2024-09-05T16:49:43.803 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.66254.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51575.log 2024-09-05T16:49:43.803 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66254.log.gz 2024-09-05T16:49:43.804 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.73230.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64573.log 2024-09-05T16:49:43.804 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73230.log.gz 2024-09-05T16:49:43.804 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.51575.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-05T16:49:43.804 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51575.log.gz 2024-09-05T16:49:43.804 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69868.log 2024-09-05T16:49:43.804 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.64573.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64573.log.gz 2024-09-05T16:49:43.804 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59613.log 2024-09-05T16:49:43.804 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.69868.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60327.log 2024-09-05T16:49:43.804 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69868.log.gz 2024-09-05T16:49:43.804 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.59613.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42409.log 2024-09-05T16:49:43.804 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59613.log.gz 2024-09-05T16:49:43.804 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.60327.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40368.log 2024-09-05T16:49:43.805 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60327.log.gz 2024-09-05T16:49:43.805 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.42409.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64321.log 2024-09-05T16:49:43.805 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42409.log.gz 2024-09-05T16:49:43.805 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.40368.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41099.log 2024-09-05T16:49:43.805 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40368.log.gz 2024-09-05T16:49:43.805 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.64321.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55451.log 2024-09-05T16:49:43.805 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64321.log.gz 2024-09-05T16:49:43.805 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.41099.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51197.log 2024-09-05T16:49:43.805 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41099.log.gz 2024-09-05T16:49:43.805 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.55451.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57594.log 2024-09-05T16:49:43.805 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55451.log.gz 2024-09-05T16:49:43.805 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.51197.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66884.log 2024-09-05T16:49:43.805 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51197.log.gz 2024-09-05T16:49:43.805 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41432.log 2024-09-05T16:49:43.806 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.57594.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57594.log.gz 2024-09-05T16:49:43.806 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.66884.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55325.log 2024-09-05T16:49:43.806 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66884.log.gz 2024-09-05T16:49:43.806 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.41432.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41432.log.gz 2024-09-05T16:49:43.806 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73860.log 2024-09-05T16:49:43.806 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.55325.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50693.log 2024-09-05T16:49:43.806 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55325.log.gz 2024-09-05T16:49:43.806 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.73860.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53423.log 2024-09-05T16:49:43.806 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73860.log.gz 2024-09-05T16:49:43.806 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.50693.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65540.log 2024-09-05T16:49:43.806 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50693.log.gz 2024-09-05T16:49:43.806 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.53423.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67515.log 2024-09-05T16:49:43.806 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53423.log.gz 2024-09-05T16:49:43.807 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.65540.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73020.log 2024-09-05T16:49:43.807 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65540.log.gz 2024-09-05T16:49:43.807 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.67515.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69952.log 2024-09-05T16:49:43.807 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67515.log.gz 2024-09-05T16:49:43.807 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.73020.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60285.log 2024-09-05T16:49:43.807 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73020.log.gz 2024-09-05T16:49:43.807 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.69952.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41062.log 2024-09-05T16:49:43.807 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69952.log.gz 2024-09-05T16:49:43.807 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.60285.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53045.log 2024-09-05T16:49:43.807 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60285.log.gz 2024-09-05T16:49:43.807 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.41062.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37546.log 2024-09-05T16:49:43.807 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41062.log.gz 2024-09-05T16:49:43.807 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.53045.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53045.log.gz 2024-09-05T16:49:43.807 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47583.log 2024-09-05T16:49:43.808 INFO:teuthology.orchestra.run.smithi087.stderr: 95.7% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-05T16:49:43.808 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.37546.log: gzip -5 --verbose 0.0% -- /var/log/ceph/ceph-client.admin.58899.log 2024-09-05T16:49:43.808 INFO:teuthology.orchestra.run.smithi087.stderr: -- replaced with /var/log/ceph/ceph-client.admin.37546.log.gz 2024-09-05T16:49:43.808 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.47583.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63816.log 2024-09-05T16:49:43.808 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47583.log.gz 2024-09-05T16:49:43.808 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.58899.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71718.log 2024-09-05T16:49:43.808 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58899.log.gz 2024-09-05T16:49:43.808 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.63816.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40284.log 2024-09-05T16:49:43.808 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63816.log.gz 2024-09-05T16:49:43.808 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61125.log 2024-09-05T16:49:43.808 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.71718.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71718.log.gz 2024-09-05T16:49:43.808 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.40284.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42446.log 2024-09-05T16:49:43.808 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40284.log.gz 2024-09-05T16:49:43.809 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.61125.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45591.log 2024-09-05T16:49:43.809 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61125.log.gz 2024-09-05T16:49:43.809 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.42446.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63396.log 2024-09-05T16:49:43.809 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42446.log.gz 2024-09-05T16:49:43.809 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.45591.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72894.log 2024-09-05T16:49:43.809 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.63396.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53171.log 2024-09-05T16:49:43.809 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63396.log.gz 2024-09-05T16:49:43.809 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.72894.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34187.log 2024-09-05T16:49:43.809 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72894.log.gz 2024-09-05T16:49:43.809 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.53171.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68565.log 2024-09-05T16:49:43.809 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53171.log.gz 2024-09-05T16:49:43.809 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.34187.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46278.log 2024-09-05T16:49:43.810 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34187.log.gz 2024-09-05T16:49:43.810 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.68565.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57132.log 2024-09-05T16:49:43.810 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68565.log.gz 2024-09-05T16:49:43.810 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.46278.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71550.log 2024-09-05T16:49:43.810 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46278.log.gz 2024-09-05T16:49:43.810 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.57132.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-09-05T16:49:43.810 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57132.log.gz 2024-09-05T16:49:43.810 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45858.log 2024-09-05T16:49:43.810 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.71550.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71550.log.gz 2024-09-05T16:49:43.810 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72516.log 2024-09-05T16:49:43.810 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.45858.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70961.log 2024-09-05T16:49:43.810 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45858.log.gz 2024-09-05T16:49:43.810 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.72516.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64278.log 2024-09-05T16:49:43.811 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72516.log.gz 2024-09-05T16:49:43.811 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.70961.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70961.log.gz 2024-09-05T16:49:43.811 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70835.log 2024-09-05T16:49:43.811 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.64278.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50567.log 2024-09-05T16:49:43.811 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64278.log.gz 2024-09-05T16:49:43.811 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.70835.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46026.log 2024-09-05T16:49:43.811 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70835.log.gz 2024-09-05T16:49:43.811 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.50567.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73062.log 2024-09-05T16:49:43.811 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50567.log.gz 2024-09-05T16:49:43.811 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.46026.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63522.log 2024-09-05T16:49:43.811 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46026.log.gz 2024-09-05T16:49:43.811 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.73062.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47879.log 2024-09-05T16:49:43.811 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73062.log.gz 2024-09-05T16:49:43.812 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.63522.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66800.log 2024-09-05T16:49:43.812 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63522.log.gz 2024-09-05T16:49:43.812 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.47879.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52499.log 2024-09-05T16:49:43.812 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47879.log.gz 2024-09-05T16:49:43.812 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41136.log 2024-09-05T16:49:43.812 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.66800.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66800.log.gz 2024-09-05T16:49:43.812 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.52499.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67935.log 2024-09-05T16:49:43.812 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52499.log.gz 2024-09-05T16:49:43.812 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.41136.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62262.log 2024-09-05T16:49:43.812 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41136.log.gz 2024-09-05T16:49:43.812 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62136.log 2024-09-05T16:49:43.812 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.67935.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67935.log.gz 2024-09-05T16:49:43.812 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.62262.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75414.log 2024-09-05T16:49:43.812 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62262.log.gz 2024-09-05T16:49:43.813 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.62136.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62136.log.gz 2024-09-05T16:49:43.813 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41817.log 2024-09-05T16:49:43.813 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.75414.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40766.log 2024-09-05T16:49:43.813 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.75414.log.gz 2024-09-05T16:49:43.813 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.41817.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47990.log 2024-09-05T16:49:43.813 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41817.log.gz 2024-09-05T16:49:43.813 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.40766.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73524.log 2024-09-05T16:49:43.813 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40766.log.gz 2024-09-05T16:49:43.813 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.47990.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59949.log 2024-09-05T16:49:43.813 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47990.log.gz 2024-09-05T16:49:43.813 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.73524.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60915.log 2024-09-05T16:49:43.813 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73524.log.gz 2024-09-05T16:49:43.813 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.59949.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42187.log 2024-09-05T16:49:43.814 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59949.log.gz 2024-09-05T16:49:43.814 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.60915.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47370.log 2024-09-05T16:49:43.814 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60915.log.gz 2024-09-05T16:49:43.814 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.42187.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69657.log 2024-09-05T16:49:43.814 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42187.log.gz 2024-09-05T16:49:43.814 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.47370.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73734.log 2024-09-05T16:49:43.814 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47370.log.gz 2024-09-05T16:49:43.814 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.69657.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40264.log 2024-09-05T16:49:43.814 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69657.log.gz 2024-09-05T16:49:43.814 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.73734.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47158.log 2024-09-05T16:49:43.814 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73734.log.gz 2024-09-05T16:49:43.814 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.40264.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68649.log 2024-09-05T16:49:43.814 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40264.log.gz 2024-09-05T16:49:43.814 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.47158.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42766.log 2024-09-05T16:49:43.815 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47158.log.gz 2024-09-05T16:49:43.815 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.68649.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45019.log 2024-09-05T16:49:43.815 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68649.log.gz 2024-09-05T16:49:43.815 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.42766.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58605.log 2024-09-05T16:49:43.815 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42766.log.gz 2024-09-05T16:49:43.815 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.45019.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72012.log 2024-09-05T16:49:43.815 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45019.log.gz 2024-09-05T16:49:43.815 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.58605.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68691.log 2024-09-05T16:49:43.815 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58605.log.gz 2024-09-05T16:49:43.815 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.72012.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59907.log 2024-09-05T16:49:43.815 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72012.log.gz 2024-09-05T16:49:43.815 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.68691.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43726.log 2024-09-05T16:49:43.815 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68691.log.gz 2024-09-05T16:49:43.816 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.59907.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66128.log 2024-09-05T16:49:43.816 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59907.log.gz 2024-09-05T16:49:43.816 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.43726.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66632.log 2024-09-05T16:49:43.816 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43726.log.gz 2024-09-05T16:49:43.816 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.66128.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68019.log 2024-09-05T16:49:43.816 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66128.log.gz 2024-09-05T16:49:43.816 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.66632.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71340.log 2024-09-05T16:49:43.816 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66632.log.gz 2024-09-05T16:49:43.816 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.68019.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59823.log 2024-09-05T16:49:43.816 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68019.log.gz 2024-09-05T16:49:43.816 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.71340.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70793.log 2024-09-05T16:49:43.816 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71340.log.gz 2024-09-05T16:49:43.817 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.59823.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69027.log 2024-09-05T16:49:43.817 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59823.log.gz 2024-09-05T16:49:43.817 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.70793.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56963.log 2024-09-05T16:49:43.817 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70793.log.gz 2024-09-05T16:49:43.817 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.69027.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54557.log 2024-09-05T16:49:43.817 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69027.log.gz 2024-09-05T16:49:43.817 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.56963.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72558.log 2024-09-05T16:49:43.817 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56963.log.gz 2024-09-05T16:49:43.817 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.54557.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62640.log 2024-09-05T16:49:43.817 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54557.log.gz 2024-09-05T16:49:43.817 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.72558.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49433.log 2024-09-05T16:49:43.817 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72558.log.gz 2024-09-05T16:49:43.817 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.62640.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56501.log 2024-09-05T16:49:43.818 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62640.log.gz 2024-09-05T16:49:43.818 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.49433.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49433.log.gz 2024-09-05T16:49:43.818 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55367.log 2024-09-05T16:49:43.818 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.56501.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56039.log 2024-09-05T16:49:43.818 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56501.log.gz 2024-09-05T16:49:43.818 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.55367.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72600.log 2024-09-05T16:49:43.818 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55367.log.gz 2024-09-05T16:49:43.818 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.56039.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71045.log 2024-09-05T16:49:43.818 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56039.log.gz 2024-09-05T16:49:43.818 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.72600.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45732.log 2024-09-05T16:49:43.818 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72600.log.gz 2024-09-05T16:49:43.818 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.71045.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58100.log 2024-09-05T16:49:43.818 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71045.log.gz 2024-09-05T16:49:43.818 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.45732.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42335.log 2024-09-05T16:49:43.819 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45732.log.gz 2024-09-05T16:49:43.819 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.58100.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64783.log 2024-09-05T16:49:43.819 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58100.log.gz 2024-09-05T16:49:43.819 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.42335.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46320.log 2024-09-05T16:49:43.819 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42335.log.gz 2024-09-05T16:49:43.819 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.64783.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40474.log 2024-09-05T16:49:43.819 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64783.log.gz 2024-09-05T16:49:43.819 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.46320.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66044.log 2024-09-05T16:49:43.819 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46320.log.gz 2024-09-05T16:49:43.819 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54179.log 2024-09-05T16:49:43.819 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.40474.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40474.log.gz 2024-09-05T16:49:43.819 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.66044.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52247.log 2024-09-05T16:49:43.819 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66044.log.gz 2024-09-05T16:49:43.820 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.54179.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62598.log 2024-09-05T16:49:43.820 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54179.log.gz 2024-09-05T16:49:43.820 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.52247.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75876.log 2024-09-05T16:49:43.820 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52247.log.gz 2024-09-05T16:49:43.820 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.62598.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51953.log 2024-09-05T16:49:43.820 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62598.log.gz 2024-09-05T16:49:43.820 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.75876.log: gzip -5 0.0% --verbose -- replaced with /var/log/ceph/ceph-client.admin.75876.log.gz -- 2024-09-05T16:49:43.820 INFO:teuthology.orchestra.run.smithi087.stderr: /var/log/ceph/ceph-client.admin.55913.log 2024-09-05T16:49:43.820 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.51953.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74028.log 2024-09-05T16:49:43.820 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51953.log.gz 2024-09-05T16:49:43.820 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.55913.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62220.log 2024-09-05T16:49:43.820 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55913.log.gz 2024-09-05T16:49:43.820 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.74028.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74994.log 2024-09-05T16:49:43.820 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74028.log.gz 2024-09-05T16:49:43.821 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.62220.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56669.log 2024-09-05T16:49:43.821 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62220.log.gz 2024-09-05T16:49:43.821 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.74994.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41506.log 2024-09-05T16:49:43.821 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74994.log.gz 2024-09-05T16:49:43.821 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.56669.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55997.log 2024-09-05T16:49:43.821 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56669.log.gz 2024-09-05T16:49:43.821 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.41506.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40655.log 2024-09-05T16:49:43.821 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41506.log.gz 2024-09-05T16:49:43.821 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.55997.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-09-05T16:49:43.821 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55997.log.gz 2024-09-05T16:49:43.821 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.40655.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67305.log 2024-09-05T16:49:43.821 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40655.log.gz 2024-09-05T16:49:43.821 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42594.log 2024-09-05T16:49:43.822 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.67305.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67599.log 2024-09-05T16:49:43.822 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67305.log.gz 2024-09-05T16:49:43.822 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.42594.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72474.log 2024-09-05T16:49:43.822 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42594.log.gz 2024-09-05T16:49:43.822 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.67599.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44894.log 2024-09-05T16:49:43.822 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67599.log.gz 2024-09-05T16:49:43.822 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.72474.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66842.log 2024-09-05T16:49:43.822 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72474.log.gz 2024-09-05T16:49:43.822 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.44894.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33895.log 2024-09-05T16:49:43.822 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44894.log.gz 2024-09-05T16:49:43.822 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.66842.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74154.log 2024-09-05T16:49:43.822 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66842.log.gz 2024-09-05T16:49:43.822 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.33895.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62850.log 2024-09-05T16:49:43.822 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33895.log.gz 2024-09-05T16:49:43.823 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.74154.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56249.log 2024-09-05T16:49:43.823 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74154.log.gz 2024-09-05T16:49:43.823 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.62850.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63942.log 2024-09-05T16:49:43.823 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62850.log.gz 2024-09-05T16:49:43.823 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.56249.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61335.log 2024-09-05T16:49:43.823 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56249.log.gz 2024-09-05T16:49:43.823 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.63942.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48286.log 2024-09-05T16:49:43.823 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63942.log.gz 2024-09-05T16:49:43.823 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.61335.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59067.log 2024-09-05T16:49:43.823 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61335.log.gz 2024-09-05T16:49:43.823 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.48286.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62472.log 2024-09-05T16:49:43.823 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48286.log.gz 2024-09-05T16:49:43.823 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.59067.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63480.log 2024-09-05T16:49:43.824 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59067.log.gz 2024-09-05T16:49:43.824 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.62472.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69069.log 2024-09-05T16:49:43.824 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62472.log.gz 2024-09-05T16:49:43.824 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.63480.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65876.log 2024-09-05T16:49:43.824 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63480.log.gz 2024-09-05T16:49:43.824 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.69069.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66716.log 2024-09-05T16:49:43.824 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69069.log.gz 2024-09-05T16:49:43.824 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.65876.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55829.log 2024-09-05T16:49:43.824 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65876.log.gz 2024-09-05T16:49:43.824 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.66716.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48360.log 2024-09-05T16:49:43.824 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66716.log.gz 2024-09-05T16:49:43.824 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.55829.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45774.log 2024-09-05T16:49:43.824 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55829.log.gz 2024-09-05T16:49:43.825 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.48360.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39258.log 2024-09-05T16:49:43.825 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48360.log.gz 2024-09-05T16:49:43.825 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.45774.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50315.log 2024-09-05T16:49:43.825 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45774.log.gz 2024-09-05T16:49:43.825 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.39258.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39258.log.gz 2024-09-05T16:49:43.825 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56165.log 2024-09-05T16:49:43.825 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.50315.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65582.log 2024-09-05T16:49:43.825 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50315.log.gz 2024-09-05T16:49:43.825 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.56165.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51827.log 2024-09-05T16:49:43.825 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56165.log.gz 2024-09-05T16:49:43.825 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.65582.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66002.log 2024-09-05T16:49:43.825 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65582.log.gz 2024-09-05T16:49:43.825 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.51827.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73188.log 2024-09-05T16:49:43.825 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51827.log.gz 2024-09-05T16:49:43.826 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.66002.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71214.log 2024-09-05T16:49:43.826 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66002.log.gz 2024-09-05T16:49:43.826 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.73188.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63648.log 2024-09-05T16:49:43.826 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73188.log.gz 2024-09-05T16:49:43.826 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.71214.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40988.log 2024-09-05T16:49:43.826 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71214.log.gz 2024-09-05T16:49:43.826 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.63648.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63648.log.gz 2024-09-05T16:49:43.826 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40040.log 2024-09-05T16:49:43.826 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.40988.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54473.log 2024-09-05T16:49:43.826 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40988.log.gz 2024-09-05T16:49:43.826 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.40040.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50525.log 2024-09-05T16:49:43.826 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40040.log.gz 2024-09-05T16:49:43.826 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.54473.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64615.log 2024-09-05T16:49:43.827 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54473.log.gz 2024-09-05T16:49:43.827 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.50525.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60411.log 0.0% 2024-09-05T16:49:43.827 INFO:teuthology.orchestra.run.smithi087.stderr: -- replaced with /var/log/ceph/ceph-client.admin.50525.log.gz 2024-09-05T16:49:43.827 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.64615.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61419.log 2024-09-05T16:49:43.827 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64615.log.gz 2024-09-05T16:49:43.827 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.60411.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72978.log 2024-09-05T16:49:43.827 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60411.log.gz 2024-09-05T16:49:43.827 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.61419.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45230.log 2024-09-05T16:49:43.827 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61419.log.gz 2024-09-05T16:49:43.827 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.72978.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52121.log 2024-09-05T16:49:43.827 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72978.log.gz 2024-09-05T16:49:43.827 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.45230.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47244.log 2024-09-05T16:49:43.827 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45230.log.gz 2024-09-05T16:49:43.827 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.52121.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56879.log 2024-09-05T16:49:43.828 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52121.log.gz 2024-09-05T16:49:43.828 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.47244.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52625.log 2024-09-05T16:49:43.828 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47244.log.gz 2024-09-05T16:49:43.828 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.56879.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47546.log 2024-09-05T16:49:43.828 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56879.log.gz 2024-09-05T16:49:43.828 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.52625.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61461.log 2024-09-05T16:49:43.828 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52625.log.gz 2024-09-05T16:49:43.828 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.47546.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51071.log 2024-09-05T16:49:43.828 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47546.log.gz 2024-09-05T16:49:43.828 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.61461.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63186.log 2024-09-05T16:49:43.828 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61461.log.gz 2024-09-05T16:49:43.829 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.51071.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61629.log 2024-09-05T16:49:43.829 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51071.log.gz 2024-09-05T16:49:43.829 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.63186.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46866.log 2024-09-05T16:49:43.829 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63186.log.gz 2024-09-05T16:49:43.829 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.61629.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48397.log 2024-09-05T16:49:43.829 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61629.log.gz 2024-09-05T16:49:43.829 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.46866.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45355.log 2024-09-05T16:49:43.829 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46866.log.gz 2024-09-05T16:49:43.829 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.48397.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51407.log 2024-09-05T16:49:43.829 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48397.log.gz 2024-09-05T16:49:43.829 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.45355.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52541.log 2024-09-05T16:49:43.829 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45355.log.gz 2024-09-05T16:49:43.830 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.51407.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56207.log 2024-09-05T16:49:43.830 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51407.log.gz 2024-09-05T16:49:43.830 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.52541.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74826.log 2024-09-05T16:49:43.830 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52541.log.gz 2024-09-05T16:49:43.830 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.56207.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55535.log 2024-09-05T16:49:43.830 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56207.log.gz 2024-09-05T16:49:43.830 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.74826.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44644.log 2024-09-05T16:49:43.830 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74826.log.gz 2024-09-05T16:49:43.830 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.55535.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57174.log 2024-09-05T16:49:43.830 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55535.log.gz 2024-09-05T16:49:43.830 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.44644.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.44644.log.gz 2024-09-05T16:49:43.830 INFO:teuthology.orchestra.run.smithi087.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.62094.log 2024-09-05T16:49:43.831 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.57174.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57510.log 2024-09-05T16:49:43.831 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57174.log.gz 2024-09-05T16:49:43.831 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.62094.log: gzip 0.0% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.62094.log.gz -- 2024-09-05T16:49:43.831 INFO:teuthology.orchestra.run.smithi087.stderr: /var/log/ceph/ceph-client.admin.52877.log 2024-09-05T16:49:43.831 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.57510.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74112.log 2024-09-05T16:49:43.831 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57510.log.gz 2024-09-05T16:49:43.831 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.52877.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.66380.log -- replaced with /var/log/ceph/ceph-client.admin.52877.log.gz 2024-09-05T16:49:43.831 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-09-05T16:49:43.831 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.74112.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53927.log 2024-09-05T16:49:43.831 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74112.log.gz 2024-09-05T16:49:43.831 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.66380.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49937.log 2024-09-05T16:49:43.832 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66380.log.gz 2024-09-05T16:49:43.832 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.53927.log: gzip 0.0% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.53927.log.gz -- 2024-09-05T16:49:43.832 INFO:teuthology.orchestra.run.smithi087.stderr: /var/log/ceph/ceph-client.admin.68607.log 2024-09-05T16:49:43.832 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.49937.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42113.log 2024-09-05T16:49:43.832 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49937.log.gz 2024-09-05T16:49:43.832 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68313.log 2024-09-05T16:49:43.832 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.68607.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68607.log.gz 2024-09-05T16:49:43.832 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.42113.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40840.log 2024-09-05T16:49:43.832 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42113.log.gz 2024-09-05T16:49:43.832 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.68313.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55104.log 2024-09-05T16:49:43.832 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68313.log.gz 2024-09-05T16:49:43.833 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.40840.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64993.log 2024-09-05T16:49:43.833 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40840.log.gz 2024-09-05T16:49:43.833 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.55104.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45900.log 2024-09-05T16:49:43.833 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55104.log.gz 2024-09-05T16:49:43.833 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.64993.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61251.log 2024-09-05T16:49:43.833 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64993.log.gz 2024-09-05T16:49:43.833 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.45900.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45900.log.gz 2024-09-05T16:49:43.833 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46362.log 2024-09-05T16:49:43.833 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.61251.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59781.log 2024-09-05T16:49:43.833 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61251.log.gz 2024-09-05T16:49:43.833 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.46362.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63438.log 2024-09-05T16:49:43.833 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46362.log.gz 2024-09-05T16:49:43.834 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.59781.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73314.log 2024-09-05T16:49:43.834 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59781.log.gz 2024-09-05T16:49:43.834 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.63438.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54347.log 2024-09-05T16:49:43.834 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63438.log.gz 2024-09-05T16:49:43.834 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68523.log 2024-09-05T16:49:43.834 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.73314.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73314.log.gz 2024-09-05T16:49:43.834 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.54347.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54305.log 2024-09-05T16:49:43.834 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54347.log.gz 2024-09-05T16:49:43.834 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.68523.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46194.log 2024-09-05T16:49:43.834 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68523.log.gz 2024-09-05T16:49:43.834 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.54305.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54305.log.gz 2024-09-05T16:49:43.834 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62892.log 2024-09-05T16:49:43.834 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.46194.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70919.log 2024-09-05T16:49:43.835 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46194.log.gz 2024-09-05T16:49:43.835 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.62892.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64741.log 2024-09-05T16:49:43.835 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62892.log.gz 2024-09-05T16:49:43.835 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.70919.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74490.log 2024-09-05T16:49:43.835 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70919.log.gz 2024-09-05T16:49:43.835 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.64741.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.64741.log.gz 2024-09-05T16:49:43.835 INFO:teuthology.orchestra.run.smithi087.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.66674.log 2024-09-05T16:49:43.835 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.74490.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55703.log 2024-09-05T16:49:43.835 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74490.log.gz 2024-09-05T16:49:43.835 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.66674.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66674.log.gz 2024-09-05T16:49:43.835 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73104.log 2024-09-05T16:49:43.835 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.55703.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42668.log 2024-09-05T16:49:43.835 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55703.log.gz 2024-09-05T16:49:43.836 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.73104.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73104.log.gz 2024-09-05T16:49:43.836 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59361.log 2024-09-05T16:49:43.836 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.42668.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65203.log 2024-09-05T16:49:43.836 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42668.log.gz 2024-09-05T16:49:43.836 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.59361.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59361.log.gz 2024-09-05T16:49:43.836 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64657.log 2024-09-05T16:49:43.836 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.65203.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49853.log 2024-09-05T16:49:43.836 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65203.log.gz 2024-09-05T16:49:43.836 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.64657.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63270.log 2024-09-05T16:49:43.836 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64657.log.gz 2024-09-05T16:49:43.836 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.49853.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71928.log 2024-09-05T16:49:43.836 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49853.log.gz 2024-09-05T16:49:43.836 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.63270.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45020.log 2024-09-05T16:49:43.837 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63270.log.gz 2024-09-05T16:49:43.837 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.71928.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66464.log 2024-09-05T16:49:43.837 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71928.log.gz 2024-09-05T16:49:43.837 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.45020.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70288.log 2024-09-05T16:49:43.837 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45020.log.gz 2024-09-05T16:49:43.837 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.66464.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75246.log 2024-09-05T16:49:43.837 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66464.log.gz 2024-09-05T16:49:43.837 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.70288.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40017.log 2024-09-05T16:49:43.837 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70288.log.gz 2024-09-05T16:49:43.837 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.75246.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42724.log 2024-09-05T16:49:43.837 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.75246.log.gz 2024-09-05T16:49:43.837 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.40017.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57804.log 2024-09-05T16:49:43.837 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40017.log.gz 2024-09-05T16:49:43.837 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50189.log 2024-09-05T16:49:43.838 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.42724.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42724.log.gz 2024-09-05T16:49:43.838 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.57804.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52667.log 2024-09-05T16:49:43.838 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57804.log.gz 2024-09-05T16:49:43.838 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.50189.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62556.log 2024-09-05T16:49:43.838 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50189.log.gz 2024-09-05T16:49:43.838 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.52667.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70708.log 2024-09-05T16:49:43.838 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52667.log.gz 2024-09-05T16:49:43.838 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.62556.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62556.log.gz -5 2024-09-05T16:49:43.838 INFO:teuthology.orchestra.run.smithi087.stderr: --verbose -- /var/log/ceph/ceph-client.admin.63774.log 2024-09-05T16:49:43.838 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.70708.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-09-05T16:49:43.838 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70708.log.gz 2024-09-05T16:49:43.838 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.63774.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39930.log 2024-09-05T16:49:43.838 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63774.log.gz 2024-09-05T16:49:43.839 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45106.log 2024-09-05T16:49:43.839 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.39930.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60117.log 2024-09-05T16:49:43.839 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39930.log.gz 2024-09-05T16:49:43.839 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.45106.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67725.log 2024-09-05T16:49:43.839 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45106.log.gz 2024-09-05T16:49:43.839 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.60117.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64699.log 2024-09-05T16:49:43.839 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60117.log.gz 2024-09-05T16:49:43.839 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.67725.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73608.log 2024-09-05T16:49:43.839 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67725.log.gz 2024-09-05T16:49:43.839 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.64699.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42372.log 2024-09-05T16:49:43.839 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64699.log.gz 2024-09-05T16:49:43.839 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.73608.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56543.log 2024-09-05T16:49:43.839 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73608.log.gz 2024-09-05T16:49:43.840 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.42372.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55019.log 2024-09-05T16:49:43.840 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42372.log.gz 2024-09-05T16:49:43.840 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.56543.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67851.log 2024-09-05T16:49:43.840 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56543.log.gz 2024-09-05T16:49:43.840 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.55019.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47032.log 2024-09-05T16:49:43.840 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55019.log.gz 2024-09-05T16:49:43.840 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.67851.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56711.log 2024-09-05T16:49:43.840 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67851.log.gz 2024-09-05T16:49:43.840 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.47032.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46656.log 2024-09-05T16:49:43.840 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47032.log.gz 2024-09-05T16:49:43.840 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.56711.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51533.log 2024-09-05T16:49:43.840 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56711.log.gz 2024-09-05T16:49:43.840 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.46656.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46656.log.gz 2024-09-05T16:49:43.841 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47030.log 2024-09-05T16:49:43.841 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.51533.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53465.log 2024-09-05T16:49:43.841 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51533.log.gz 2024-09-05T16:49:43.841 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.47030.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47030.log.gz 2024-09-05T16:49:43.841 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62052.log 2024-09-05T16:49:43.841 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.53465.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39636.log 2024-09-05T16:49:43.841 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53465.log.gz 2024-09-05T16:49:43.841 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.62052.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56585.log 2024-09-05T16:49:43.841 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62052.log.gz 2024-09-05T16:49:43.841 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.39636.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51155.log 2024-09-05T16:49:43.841 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39636.log.gz 2024-09-05T16:49:43.841 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.56585.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60033.log 2024-09-05T16:49:43.841 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56585.log.gz 2024-09-05T16:49:43.841 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.51155.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67011.log 2024-09-05T16:49:43.842 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51155.log.gz 2024-09-05T16:49:43.842 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.60033.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69784.log 2024-09-05T16:49:43.842 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60033.log.gz 2024-09-05T16:49:43.842 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.67011.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60537.log 2024-09-05T16:49:43.842 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67011.log.gz 2024-09-05T16:49:43.842 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.69784.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54599.log 2024-09-05T16:49:43.842 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69784.log.gz 2024-09-05T16:49:43.842 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40199.log 2024-09-05T16:49:43.842 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.60537.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60537.log.gz 2024-09-05T16:49:43.842 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.54599.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73986.log 2024-09-05T16:49:43.842 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54599.log.gz 2024-09-05T16:49:43.842 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.40199.log: gzip -5 --verbose 0.0% -- -- replaced with /var/log/ceph/ceph-client.admin.40199.log.gz /var/log/ceph/ceph-client.admin.52205.log 2024-09-05T16:49:43.842 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-09-05T16:49:43.842 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.73986.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39488.log 2024-09-05T16:49:43.843 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73986.log.gz 2024-09-05T16:49:43.843 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.52205.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.52205.log.gz --verbose 2024-09-05T16:49:43.843 INFO:teuthology.orchestra.run.smithi087.stderr: -- /var/log/ceph/ceph-client.admin.51617.log 2024-09-05T16:49:43.843 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.39488.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54431.log 2024-09-05T16:49:43.843 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39488.log.gz 2024-09-05T16:49:43.843 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.51617.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73650.log 2024-09-05T16:49:43.843 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51617.log.gz 2024-09-05T16:49:43.843 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.54431.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45442.log 2024-09-05T16:49:43.843 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54431.log.gz 2024-09-05T16:49:43.843 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.73650.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73650.log.gz 2024-09-05T16:49:43.843 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71424.log 2024-09-05T16:49:43.843 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50777.log 2024-09-05T16:49:43.843 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.45442.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45442.log.gz 2024-09-05T16:49:43.843 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.71424.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44854.log 2024-09-05T16:49:43.843 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71424.log.gz 2024-09-05T16:49:43.844 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.50777.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61884.log 2024-09-05T16:49:43.844 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50777.log.gz 2024-09-05T16:49:43.844 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.44854.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66968.log 2024-09-05T16:49:43.844 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44854.log.gz 2024-09-05T16:49:43.844 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.61884.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44686.log 2024-09-05T16:49:43.844 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61884.log.gz 2024-09-05T16:49:43.844 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.66968.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42483.log 2024-09-05T16:49:43.844 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66968.log.gz 2024-09-05T16:49:43.844 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.44686.log: gzip -5 --verbose 0.0% -- -- replaced with /var/log/ceph/ceph-client.admin.44686.log.gz /var/log/ceph/ceph-client.admin.41706.log 2024-09-05T16:49:43.844 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-09-05T16:49:43.844 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.42483.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53591.log 2024-09-05T16:49:43.844 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42483.log.gz 2024-09-05T16:49:43.844 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.41706.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70036.log 2024-09-05T16:49:43.845 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41706.log.gz 2024-09-05T16:49:43.845 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.53591.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39888.log 2024-09-05T16:49:43.845 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53591.log.gz 2024-09-05T16:49:43.845 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.70036.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70036.log.gz 2024-09-05T16:49:43.845 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.39888.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39888.log.gz 2024-09-05T16:49:43.845 INFO:teuthology.orchestra.run.smithi087.stderr: 95.2% -- replaced with /var/log/ceph/ceph-client.admin.45591.log.gz 2024-09-05T16:49:47.115 INFO:teuthology.orchestra.run.smithi087.stderr: 91.4% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-09-05T16:49:47.771 INFO:teuthology.orchestra.run.smithi087.stderr: 91.6% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-09-05T16:49:48.343 INFO:teuthology.orchestra.run.smithi087.stderr: 91.4% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-09-05T16:49:49.983 INFO:teuthology.orchestra.run.smithi087.stderr: 89.9% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-09-05T17:02:24.254 INFO:teuthology.orchestra.run.smithi087.stderr: 91.7% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-09-05T17:04:59.538 INFO:teuthology.orchestra.run.smithi087.stderr: 91.4% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-09-05T17:05:40.874 INFO:teuthology.orchestra.run.smithi087.stderr: 91.4% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-09-05T17:06:32.761 INFO:teuthology.orchestra.run.smithi087.stderr: 91.7% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-09-05T17:06:32.763 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-09-05T17:06:32.763 INFO:teuthology.orchestra.run.smithi087.stderr:real 16m49.359s 2024-09-05T17:06:32.764 INFO:teuthology.orchestra.run.smithi087.stderr:user 6m21.495s 2024-09-05T17:06:32.764 INFO:teuthology.orchestra.run.smithi087.stderr:sys 0m18.256s 2024-09-05T17:06:32.764 INFO:tasks.ceph:Archiving logs... 2024-09-05T17:06:32.766 DEBUG:teuthology.misc:Transferring archived files from smithi087:/var/log/ceph to /home/teuthworker/archive/skanta-2024-09-05_12:54:40-rados-wip-bharath4-testing-2024-09-02-0957-distro-default-smithi/7890779/remote/smithi087/log 2024-09-05T17:06:32.767 DEBUG:teuthology.orchestra.run.smithi087:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-05T17:07:51.467 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-05T17:07:51.504 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-05T17:07:51.504 DEBUG:teuthology.orchestra.run.smithi087:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-05T17:07:51.632 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-05T17:07:51.641 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-05T17:07:51.641 DEBUG:teuthology.orchestra.run.smithi087:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-05T17:07:51.655 INFO:teuthology.orchestra.run.smithi087.stderr:bash: line 1: ntpq: command not found 2024-09-05T17:07:51.698 INFO:teuthology.orchestra.run.smithi087.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-05T17:07:51.698 INFO:teuthology.orchestra.run.smithi087.stdout:=============================================================================== 2024-09-05T17:07:51.698 INFO:teuthology.orchestra.run.smithi087.stdout:^* hv01.front.sepia.ceph.com 3 6 377 32 +563us[ +566us] +/- 47ms 2024-09-05T17:07:51.699 INFO:teuthology.orchestra.run.smithi087.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 34 -962us[ -958us] +/- 62ms 2024-09-05T17:07:51.699 INFO:teuthology.orchestra.run.smithi087.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 35 +292us[ +296us] +/- 70ms 2024-09-05T17:07:51.699 INFO:teuthology.orchestra.run.smithi087.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-05T17:07:51.699 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-05T17:07:51.709 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-05T17:07:51.709 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-05T17:07:51.734 DEBUG:teuthology.orchestra.run.smithi087:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-05T17:07:51.860 DEBUG:teuthology.orchestra.run.smithi087:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-05T17:07:51.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-05T17:07:51.889 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-05T17:07:51.898 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-05T17:07:51.926 INFO:teuthology.task.internal:Duration was 2487.713131 seconds 2024-09-05T17:07:51.927 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-05T17:07:51.960 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-05T17:07:51.960 DEBUG:teuthology.orchestra.run.smithi087:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-05T17:07:52.026 INFO:teuthology.orchestra.run.smithi087.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-05T17:07:52.078 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-05T17:07:52.079 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi087.front.sepia.ceph.com 2024-09-05T17:07:52.079 DEBUG:teuthology.orchestra.run.smithi087:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-05T17:07:52.099 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-05T17:07:52.100 DEBUG:teuthology.orchestra.run.smithi087:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-05T17:07:52.166 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-05T17:07:52.166 DEBUG:teuthology.orchestra.run.smithi087:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-05T17:07:52.337 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-05T17:07:52.347 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-05T17:07:52.347 DEBUG:teuthology.orchestra.run.smithi087:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-05T17:07:52.404 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-05T17:07:52.414 DEBUG:teuthology.orchestra.run.smithi087:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-05T17:07:52.441 INFO:teuthology.orchestra.run.smithi087.stdout:kernel.core_pattern = core 2024-09-05T17:07:52.466 DEBUG:teuthology.orchestra.run.smithi087:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-05T17:07:52.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-05T17:07:52.522 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-05T17:07:52.531 INFO:teuthology.task.internal:Transferring archived files... 2024-09-05T17:07:52.533 DEBUG:teuthology.misc:Transferring archived files from smithi087:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/skanta-2024-09-05_12:54:40-rados-wip-bharath4-testing-2024-09-02-0957-distro-default-smithi/7890779/remote/smithi087 2024-09-05T17:07:52.533 DEBUG:teuthology.orchestra.run.smithi087:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-05T17:07:52.598 INFO:teuthology.task.internal:Removing archive directory... 2024-09-05T17:07:52.599 DEBUG:teuthology.orchestra.run.smithi087:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-05T17:07:52.648 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-05T17:07:52.658 INFO:teuthology.task.internal:Not uploading archives. 2024-09-05T17:07:52.658 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-05T17:07:52.684 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-05T17:07:52.684 DEBUG:teuthology.orchestra.run.smithi087:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-05T17:07:52.702 INFO:teuthology.orchestra.run.smithi087.stdout: 262217 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 5 17:07 /home/ubuntu/cephtest 2024-09-05T17:07:52.703 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-05T17:07:52.719 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-05T17:07:52.787 INFO:teuthology.run:Summary data: description: rados/singleton/{all/ec-lost-unfound mon_election/classic msgr-failures/few msgr/async-v2only objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-hybrid} rados supported-random-distro$/{centos_latest}} duration: 2487.7131311893463 flavor: default owner: scheduled_skanta@teuthology success: true 2024-09-05T17:07:52.788 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-05T17:07:52.866 INFO:teuthology.run:pass